Your IP : 216.73.216.85
��X� � @ sc d Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z y d d l Z
Wn e k
r� d d l
Z
Yn Xd d l Z d d l
Z
d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z m Z m Z d d l m Z m Z d d l m Z d d l m Z m Z m Z d d
l m! Z! d d l" m# Z# m$ Z$ d d l% m& Z& m' Z' d
Z( d d � Z) Gd d � d e* � Z+ d d � Z, d d � Z- d d d d d d � Z. e j/ � Z0 Gd d � d e � Z1 Gd d � d e2 � Z3 Gd d � d e2 � Z4 Gd d � d e � Z5 Gd d! � d! e � Z6 d S)"zo
A context object for caching a function's return value each time it
is called with the same input arguments.
� )�with_statementN� )�hashing)�
get_func_code�
get_func_name� filter_args)�format_signature� format_call)�open_py_source)�Logger� format_time�pformat)� numpy_pickle)�mkdirp�
rm_subdirs)� _basestring� PY3_OR_LATERz
# first line:c C sj | j t � rZ | j d � } t | d t t � d � � } d j | d d � � } n d } | | f S)zZ Extract the first line information from the function code
text if available.
�
r Nr ���)�
startswith�FIRST_LINE_TEXT�split�int�len�join)� func_code�
first_line� r �>/opt/alt/python35/lib/python3.5/site-packages/joblib/memory.py�extract_first_line8 s r c @ s e Z d Z d Z d S)�JobLibCollisionWarningzF Warn that there might be a collision between names of functions.
N)�__name__�
__module__� __qualname__�__doc__r r r r r E s r c C s/ t | � \ } } | j | � t j j | � S)ziCompute the part of part associated with a function.
See code of_cache_key_to_dir() for details
)r �append�os�pathr )�func�modules�funcnamer r r �_get_func_fullnameJ s
r+ c C sd | g } t | t � r( | j | � n | j t | � � | d k rT | j | � t j j | � S)z�Compute directory associated with a given cache key.
func can be a function or a string as returned by _get_func_fullname().
N)�
isinstancer r% r+ r& r' r )�cachedirr( �
argument_hash�partsr r r �_cache_key_to_dirT s
r0 c
C sM | d k rd } yg | d k rf d j d d � | d j � D� � } d t j j | � | f } n t j j | � } Wn t k
r� Yn X| d k r� d t t j � | � } n d } | d
k r� t d | t | � f � n t d | t | � | f � t j j | d
� } t j j
| � s:t d | � � t j | d | �S)z Load output of a computation.r � Nz, c S s& g | ] \ } } d | | f � q S)z%s=%sr )�.0�name�valuer r r �
<listcomp>k s z _load_output.<locals>.<listcomp>�
input_argsz%s(%s)z% 16s�
z[Memory]%s: Loading %s...z [Memory]%s: Loading %s from %sz
output.pklzHNon-existing cache value (may have been cleared).
File %s does not exist� mmap_mode)
r �itemsr& r' �basename�KeyErrorr �time�print�str�isfiler �load)
�
output_dir� func_name� timestamp�metadatar8 �verboseZ signature�args�t�filenamer r r � _load_outputd s0
rI c @ s^ e Z d Z d Z d d d d d d � Z d d � Z d d � Z d
d � Z d d
� Z d S)�MemorizedResulta� Object representing a cached value.
Attributes
----------
cachedir: string
path to root of joblib cache
func: function or string
function whose output is cached. The string case is intended only for
instanciation based on the output of repr() on another instance.
(namely eval(repr(memorized_instance)) works).
argument_hash: string
hash of the function arguments
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
The memmapping mode used when loading from cache numpy arrays. See
numpy.load for the meaning of the different values.
verbose: int
verbosity level (0 means no message)
timestamp, metadata: string
for internal use only
Nr c C s t j | � t | t � r( | | _ n t | � | _ | | _ | | _ | | _ t | | j | � | _
| d k r� | | _ nU i | _ yA t t
j j | j
d � d � � } t j | � | _ Wd QRXWn Yn X| j j d d � | _ | | _ | | _ d S)Nz
metadata.json�rb�duration)r �__init__r, r r( r+ r. r- r8 r0 � _output_dirrD �openr& r' r �jsonr@ �getrL rE rC ) �selfr- r( r. r8 rE rC rD �fr r r rM � s*
zMemorizedResult.__init__c C s= t | j t | j � d | j d | j d | j d | j �S)z$Read value from cache and return it.rC rD r8 rE )rI rN r+ r( rC rD r8 rE )rR r r r rQ � s zMemorizedResult.getc C s t j | j d d �d S)zClear value from cache�
ignore_errorsTN)�shutil�rmtreerN )rR r r r �clear� s zMemorizedResult.clearc C s1 d j d | j j d | j d | j d | j � S)NzS{class_name}(cachedir="{cachedir}", func="{func}", argument_hash="{argument_hash}")�
class_namer- r( r. )�format� __class__r! r- r( r. )rR r r r �__repr__� s
zMemorizedResult.__repr__c C s+ | j | j | j | j f d | j i f S)Nr8 )rZ r- r( r. r8 )rR r r r �
__reduce__� s zMemorizedResult.__reduce__) r! r" r# r$ rM rQ rW r[ r\ r r r r rJ � s rJ c @ sd e Z d Z d Z d Z d d � Z d d � Z d d � Z d
d � Z d d
� Z d d � Z
d S)�NotMemorizedResultzxClass representing an arbitrary value.
This class is a replacement for MemorizedResult when there is no cache.
r4 �validc C s | | _ d | _ d S)NT)r4 r^ )rR r4 r r r rM � s zNotMemorizedResult.__init__c C s | j r | j St d � � d S)NzNo value stored.)r^ r4 r; )rR r r r rQ � s zNotMemorizedResult.getc C s d | _ d | _ d S)NF)r^ r4 )rR r r r rW � s zNotMemorizedResult.clearc C s@ | j r. d j d | j j d t | j � � S| j j d Sd S)Nz{class_name}({value})rX r4 z with no value)r^ rY rZ r! r
r4 )rR r r r r[ � s
zNotMemorizedResult.__repr__c C s d | j d | j i S)Nr^ r4 )r^ r4 )rR r r r � __getstate__ s z NotMemorizedResult.__getstate__c C s | d | _ | d | _ d S)Nr^ r4 )r^ r4 )rR �stater r r � __setstate__ s
z NotMemorizedResult.__setstate__N)r4 r^ ) r! r" r# r$ � __slots__rM rQ rW r[ r_ ra r r r r r] � s
r] c @ sa e Z d Z d Z d d � Z d d � Z d d � Z d d � Z d
d � Z d d
d � Z d S)�NotMemorizedFunca No-op object decorating a function.
This class replaces MemorizedFunc when there is no cache. It provides an
identical API but does not write anything on disk.
Attributes
----------
func: callable
Original undecorated function.
c C s
| | _ d S)N)r( )rR r( r r r rM s zNotMemorizedFunc.__init__c O s | j | | � S)N)r( )rR rF �kwargsr r r �__call__ s zNotMemorizedFunc.__call__c O s t | j | | � � S)N)r] r( )rR rF rd r r r �call_and_shelve s z NotMemorizedFunc.call_and_shelvec C s | j | j f f S)N)rZ r( )rR r r r r\ s zNotMemorizedFunc.__reduce__c C s d | j j | j f S)Nz %s(func=%s))rZ r! r( )rR r r r r[ # s zNotMemorizedFunc.__repr__Tc C s d S)Nr )rR �warnr r r rW ) s zNotMemorizedFunc.clearN)
r! r" r# r$ rM re rf r\ r[ rW r r r r rc s
rc c @ s e Z d Z d Z d d d d d d d � Z d d � Z d d
� Z d d � Z d
d � Z d d � Z d d � Z
d d � Z d d � Z e Z
d d d � Z d d � Z d d � Z d d d � Z d d! d" � Z d# d$ � Z d% d& � Z d' d( d) � Z d* d+ � Z d, d- � Z d S).�
MemorizedFunca� Callable object decorating a function for caching its return value
each time it is called.
All values are cached on the filesystem, in a deep directory
structure. Methods are provided to inspect the cache or clean it.
Attributes
----------
func: callable
The original, undecorated, function.
cachedir: string
Path to the base cache directory of the memory context.
ignore: list or None
List of variable names to ignore when choosing whether to
recompute.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the different
values.
compress: boolean, or integer
Whether to zip the stored data on disk. If an integer is
given, it should be between 1 and 9, and sets the amount
of compression. Note that compressed arrays cannot be
read by memmapping.
verbose: int, optional
The verbosity flag, controls messages that are issued as
the function is evaluated.
NFr c
C s5 t j | � | | _ | | _ | d k r1 g } | | _ | | _ | | _ | | _ | r} | j d k r} t j d d d �| d k r� t
j
� } | | _ t | j � y t
j | | � Wn Yn Xt j | � rt j � j | � } | j d d d � } t j d d | � } n | j } d
| | _ d S) a{
Parameters
----------
func: callable
The function to decorate
cachedir: string
The path of the base directory to use as a data store
ignore: list or None
List of variable names to ignore.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the
arguments.
compress : boolean, or integer
Whether to zip the stored data on disk. If an integer is
given, it should be between 1 and 9, and sets the amount
of compression. Note that compressed arrays cannot be
read by memmapping.
verbose: int, optional
Verbosity flag, controls the debug messages that are issued
as functions are evaluated. The higher, the more verbose
timestamp: float, optional
The reference time from which times in tracing messages
are reported.
Nz&Compressed results cannot be memmapped�
stacklevel� r z
r z.r1 zMemoized version of %s)r rM r8 r( �ignore�_verboser- �compress�warningsrg r< rC r � functools�update_wrapper�inspectZ
isfunction�pydocZTextDocZdocument�replace�re�subr$ ) rR r( r- rk r8 rm rE rC �docr r r rM W s4
zMemorizedFunc.__init__c C s | j | | � \ } } d } | j d d � o? t j j | � s� | j d k r� t | j � \ } } | j d | | | f � | j | | � \ } } | j
d k r�t | t | j � d | j
d | j
d | j �} n y� t j � } t | t | j � d | j
d | d | j
d | j �} | j d k r�t j � | }
t | j � \ } } d
| t |
� f } t t d d t | � � d
| � Wnc t k
r�| j d | | t j � f � t j | d d �| j | | � \ } } d } Yn X| | | f S)a� Call wrapped function and cache result, or read cache if available.
This function returns the wrapped function output and some metadata.
Returns
-------
output: value or tuple
what is returned by wrapped function
argument_hash: string
hash of function arguments
metadata: dict
some metadata about wrapped function call (see _persist_input())
Nri � r7 z3Computing func %s, argument hash %s in directory %srC r8 rE rD z%s cache loaded - %sr �P �_z<Exception while loading results for (args=%s, kwargs=%s)
%srT T)�_get_output_dir�_check_previous_func_coder& r' �existsrl r r( rg �callr8 rI r+ rC r< r r= �maxr � Exception� traceback�
format_excrU rV ) rR rF rd rA r. rD ry r3 �outZt0rG �msgr r r � _cached_call� s@ )
zMemorizedFunc._cached_callc
O sM | j | | � \ } } } t | j | j | d | d | j d d | j �S)a\ Call wrapped function, cache result and return a reference.
This method returns a reference to the cached result instead of the
result itself. The reference object is small and pickeable, allowing
to send or store it easily. Call .get() on reference object to get
result.
Returns
-------
cached_result: MemorizedResult or NotMemorizedResult
reference to the value returned by the wrapped function. The
class "NotMemorizedResult" is used when there is no cache
activated (e.g. cachedir=None in Memory).
rD rE r rC )r� rJ r- r( rl rC )rR rF rd ry r. rD r r r rf � s z MemorizedFunc.call_and_shelvec O s | j | | � d S)Nr )r� )rR rF rd r r r re � s zMemorizedFunc.__call__c C s1 | j | j | j | j | j | j | j f f S)z� We don't store the timestamp when pickling, to avoid the hash
depending from it.
In addition, when unpickling, we run the __init__
)rZ r( r- rk r8 rm rl )rR r r r r\ � s zMemorizedFunc.__reduce__c O s# t j d t � t | j | | � S)NzMMemorizedFunc.format_signature will be removed in a future version of joblib.)rn rg �DeprecationWarningr r( )rR rF rd r r r r � s z MemorizedFunc.format_signaturec O s# t j d t � t | j | | � S)NzHMemorizedFunc.format_call will be removed in a future version of joblib.)rn rg r� r r( )rR rF rd r r r r � s zMemorizedFunc.format_callc O s1 t j t | j | j | | � d | j d k �S)NZ coerce_mmap)r �hashr r( rk r8 )rR rF rd r r r �_get_argument_hash� s z MemorizedFunc._get_argument_hashc O s= | j | | � } t j j | j | j � | � } | | f S)z} Return the directory in which are persisted the result
of the function called with the given arguments.
)r� r& r' r �
_get_func_dirr( )rR rF rd r. rA r r r rz s z MemorizedFunc._get_output_dirTc C s, t | j | j d � } | r( t | � | S)zT Get the directory corresponding to the cache for the
function.
N)r0 r- r( r )rR �mkdir�func_dirr r r r� s
zMemorizedFunc._get_func_dirc C s: t t | j d d � � } t | j � t | j � | f S)z'Hash a function to key the online cache�__code__N)r� �getattrr( �id)rR Z func_code_hr r r �
_hash_func s zMemorizedFunc._hash_funcc C s� d t | | f } t j | d d d �� } | j | � Wd QRXd } t ru t | j d � oo | j j d k } n$ t | j d � o� | j j d k } | r� | j � } y | t
| j <Wn t k
r� Yn Xd S)
z= Write the function code and the filename to a file.
z%s %i
%s�w�encodingzUTF-8NFr! z<lambda>rB ) r �iorO �writer �hasattrr( r! rB r� �_FUNCTION_HASHES� TypeError)rR rH r r r� Zis_named_callable� func_hashr r r �_write_func_code s
z MemorizedFunc._write_func_coderj c % C s� y6 | j t k r5 | j � } | t | j k r5 d SWn t k
rJ Yn Xt | j � \ } } } | j � } t j j | d � } y; t j
| d d �� } t | j � � \ } }
Wd QRXWn) t
k
r� | j | | | � d SYn X| | k r� d St | j d d d d �\ } } |
| k o3d k n sD| d
k r�| d k sfd | | | f }
n | }
t j t d |
� d
| �|
| k r�| d k r�d } t j j | � rPt | j d d �\ } } t | j d � � } t | � �- } | j � |
d |
d | d � } Wd QRXd j | � } | j � | j � k } n | j d � } | r�t j t d | | |
| | | f � d
| �| j d k r�t | j d d �\ } } | j d | | f � | j d d � d S)z�
stacklevel is the depth a which this function is called, to
issue useful warnings to the user.
Tz func_code.pyr� zUTF-8NFZ resolv_aliasZwin_charactersr z<lambda>z
%s (%s:%i)z/Cannot detect name collisions for function '%s'ri r r1 z <doctest zHPossible name collisions between functions '%s' (%s:%i) and '%s' (%s:%i)r7 z'Function %s (stored in %s) has changed.rg r r ) r( r� r� r� r r� r& r' r r� rO r �read�IOErrorr� r rn rg r r| r r r
� readlines�rstripr rl rW )rR ri r� r Z source_filer r� �func_code_fileZinfileZ
old_func_codeZold_first_linery rB Zfunc_descriptionZpossible_collisionZ num_linesrS Zon_disk_func_coder r r r{ 6 sh
#
( $ z'MemorizedFunc._check_previous_func_codec C s� | j d d � } | j d k r8 | r8 | j d | � t j j | � r] t j | d d �t | � t | j
� \ } } } t j j | d � } | j | | | � d S) z% Empty the function's cache.
r� Fr zClearing cache %srT Tz func_code.pyN)
r� rl rg r& r' r| rU rV r r r( r r� )rR rg r� r ry r r� r r r rW � s
zMemorizedFunc.clearc O s� t j � } | j | | � \ } } | j d k rL t t | j | | � � | j | | � } | j | | � t j � | } | j | | | | � } | j d k r� t | j � \ } } d | t | � f }
t t
d d t |
� � d |
� | | f S)zq Force the execution of the function with the given arguments and
persist the output values.
r z%s - %srx ry ) r< rz rl r= r r( �_persist_output�_persist_inputr r r~ r ) rR rF rd �
start_timerA ry �outputrL rD r3 r� r r r r} � s %zMemorizedFunc.callc C sr yY t | � t j j | d � } t j | | d | j �| j d k rX t d | � Wn t k
rm Yn Xd S)z: Persist the given output tuple in the directory.
z
output.pklrm r7 zPersisting in %sN)
r r& r' r r �dumprm rl r= �OSError)rR r� �dirrH r r r r� � s
z MemorizedFunc._persist_outputg �?c C s� t j � } t | j | j | | � } t d d � | j � D� � } d | d | i } yF t | � t t j j
| d � d � � }
t j | |
� Wd QRXWn Yn Xt j � | } | | k r� t
j d | d d
�| S) a% Save a small summary of the call using json format in the
output directory.
output_dir: string
directory where to write metadata.
duration: float
time taken by hashing input arguments, calling the wrapped
function and persisting its output.
args, kwargs: list and dict
input arguments for wrapped function
this_duration_limit: float
Max execution time for this function before issuing a warning.
c s s' | ] \ } } | t | � f Vq d S)N)�repr)r2 �k�vr r r � <genexpr>� s z/MemorizedFunc._persist_input.<locals>.<genexpr>rL r6 z
metadata.jsonr� Na~ Persisting input arguments took %.2fs to run.
If this happens often in your code, it can cause performance problems
(results will be correct in all cases).
The reason for this is probably some large input arguments for a wrapped
function (e.g. large strings).
THIS IS A JOBLIB ISSUE. If you can, kindly provide the joblib's team with an
example so that they can fix the problem.ri � )r< r r( rk �dictr9 r rO r& r' r rP r� rn rg ) rR rA rL rF rd Zthis_duration_limitr� Z
argument_dictZ
input_reprrD rS Z
this_durationr r r r� � s
!
z MemorizedFunc._persist_inputc C sA t j d t � t | t | j � d | j d | j d | j �S)ze Read the results of a previous calculation from the directory
it was cached in.
z�MemorizedFunc.load_output is deprecated and will be removed in a future version
of joblib. A MemorizedResult provides similar featuresrC r8 rE ) rn rg r� rI r+ r( rC r8 rl )rR rA r r r � load_output� s
zMemorizedFunc.load_outputc C s# d | j j | j t | j � f S)Nz%s(func=%s, cachedir=%s))rZ r! r( r� r- )rR r r r r[ s zMemorizedFunc.__repr__)r! r" r# r$ rM r� rf re r\ r r r� rz Zget_output_dirr� r� r� r{ rW r} r� r� r� r[ r r r r rh 1 s, !: ; X
5 rh c @ sv e Z d Z d Z d d d d d � Z d d d d d d � Z d d
d � Z d d
� Z d d � Z d d � Z d S)�Memoryz� A context object for caching a function's return value each time it
is called with the same input arguments.
All values are cached on the filesystem, in a deep directory
structure.
see :ref:`memory_reference`
NFr c C s� t j | � | | _ | | _ t j � | _ | | _ | r\ | d k r\ t j d d d �| d k rt d | _ n% t
j j | d � | _ t
| j � d S)a�
Parameters
----------
cachedir: string or None
The path of the base directory to use as a data store
or None. If None is given, no caching is done and
the Memory object is completely transparent.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the
arguments.
compress: boolean, or integer
Whether to zip the stored data on disk. If an integer is
given, it should be between 1 and 9, and sets the amount
of compression. Note that compressed arrays cannot be
read by memmapping.
verbose: int, optional
Verbosity flag, controls the debug messages that are issued
as functions are evaluated.
Nz&Compressed results cannot be memmappedri rj Zjoblib)r rM rl r8 r< rC rm rn rg r- r& r' r r )rR r- r8 rm rE r r r rM s
zMemory.__init__c C s� | d k r. t j | j d | d | d | �S| j d k rG t | � S| d k r\ | j } | d k rq | j } t | t � r� | j } t | d | j d | d | d | j
d | d | j �S) a{ Decorates the given function func to only compute its return
value for input arguments not cached on disk.
Parameters
----------
func: callable, optional
The function to be decorated
ignore: list of strings
A list of arguments name to ignore in the hashing
verbose: integer, optional
The verbosity mode of the function. By default that
of the memory object is used.
mmap_mode: {None, 'r+', 'r', 'w+', 'c'}, optional
The memmapping mode used when loading from cache
numpy arrays. See numpy.load for the meaning of the
arguments. By default that of the memory object is used.
Returns
-------
decorated_func: MemorizedFunc object
The returned object is a MemorizedFunc object, that is
callable (behaves like a function), but offers extra
methods for cache lookup and management. See the
documentation for :class:`joblib.memory.MemorizedFunc`.
Nrk rE r8 Fr- rm rC ) ro �partial�cacher- rc rl r8 r, rh r( rm rC )rR r( rk rE r8 r r r r� @ s"
z Memory.cacheTc C s3 | r | j d � | j d k r/ t | j � d S)z- Erase the complete cache directory.
z Flushing completely the cacheN)rg r- r )rR rg r r r rW o s
z Memory.clearc O s2 | j d k r | | | � S| j | � | | � S)a Eval function func with arguments `*args` and `**kwargs`,
in the context of the memory.
This method works similarly to the builtin `apply`, except
that the function is called only if the cache is not
up to date.
N)r- r� )rR r( rF rd r r r �evalw s
z Memory.evalc C s d | j j t | j � f S)Nz%s(cachedir=%s))rZ r! r� r- )rR r r r r[ � s zMemory.__repr__c C sJ | j d k r" | j d d � n d } | j | | j | j | j f f S)z� We don't store the timestamp when pickling, to avoid the hash
depending from it.
In addition, when unpickling, we run the __init__
N� i����)r- rZ r8 rm rl )rR r- r r r r\ � s ( zMemory.__reduce__)
r! r" r# r$ rM r� rW r� r[ r\ r r r r r� s $ . r� )7r$ Z
__future__r r& rU r< rr rt �sysZcPickle�pickle� ImportErrorro r� rn rq rP �weakrefr� r1 r Z func_inspectr r r r r Z_memory_helpersr
Zloggerr r r
r Zdiskr r Z_compatr r r r � UserWarningr r+ r0 rI �WeakKeyDictionaryr� rJ �objectr] rc rh r� r r r r �<module> sP
% Q*&� �