Your IP : 216.73.216.61
�u�W�a � @ s� d Z d d l m Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l
Z
d d l m Z y e
Wn e k
r� d Z
Yn Xd d l m Z y$ d d l m Z d d l m Z Wn> e k
r)d d l m Z d d l m Z d d l Z Yn Xd d l m Z d d l m Z d d
l m Z d d l m Z m Z d d
l m Z y d d l Z d d l! m" Z" Wn e k
r�d Z Yn Xd d l# m$ Z$ d d l# m% Z% d d l& m' Z' d Z( e j) e j* Be j+ BZ, e j) e j* BZ- d d � Z. d d � Z/ d d � Z0 d d � Z1 d d � Z2 Gd d � d e3 � Z4 Gd d � d e � Z5 Gd! d" � d" e3 � Z6 Gd# d$ � d$ e � Z7 d% d& � Z8 Gd' d( � d( e7 � Z9 d S))a� Custom implementation of multiprocessing.Pool with custom pickler.
This module provides efficient ways of working with data stored in
shared memory with numpy.memmap arrays without inducing any memory
copy between the parent and child processes.
This module should not be imported if multiprocessing is not
available as it implements subclasses of multiprocessing Pool
that uses a custom alternative to SimpleQueue.
� )�mmapN)�sleep)� whichmodule)�loads)�dumps)�Pickler)�HIGHEST_PROTOCOL)�BytesIO� )�mp�assert_spawning)�Pool)�
as_strided)�load)�dump)�hashz/dev/shmc C sC t | d d � } | d k r" d St | t � r5 | St | � Sd S)z@Recursively look up the original np.memmap instance base if any.�baseN)�getattr�
isinstancer �_get_backing_memmap)�a�b� r �</opt/alt/python35/lib/python3.5/site-packages/joblib/pool.pyr O s r c C s t | � d k S)z?Return True if a is backed by some mmap buffer directly or not.N)r )r r r r �has_shareable_memory` s r c C s� | d k r d } | d k rI t j | d | d | d | d | d | �St j | d | d | d | d | d | �} t | d | d | �Sd S)
z2Reconstruct an array view on a memory mapped file.zw+zr+N�dtype�shape�mode�offset�order�strides)�np�memmapr ) �filenamer r r r r r �total_buffer_lenr r r r �_strided_from_memmape s
r% c C s� t j | � \ } } t j | � d } | | } | | j 7} | j d rU d } n d } | j d su | j d r� d } d } n | j } | | | j } t | j | j | j | | | j
| | f f S)a Pickling reduction for memmap backed arrays.
a is expected to be an instance of np.ndarray (or np.memmap)
m is expected to be an instance of np.memmap on the top of the ``base``
attribute ancestry of a. ``m.base`` should be the real python mmap object.
r � F_CONTIGUOUS�F�C� C_CONTIGUOUSN) r! � byte_boundsr �flagsr �itemsizer% r# r r r ) r �m�a_start�a_end�m_startr r r r$ r r r �_reduce_memmap_backedx s
!r1 c C sK t | � } | d k r% t | | � St t t j | � d t �f f Sd S)zCPickle the descriptors of a memmap instance to reopen on same file.N�protocol)r r1 r r r! �asarrayr )r r- r r r �
reduce_memmap� s
r4 c @ s7 e Z d Z d Z d d d d d � Z d d � Z d S) �ArrayMemmapReducera� Reducer callable to dump large arrays to memmap files.
Parameters
----------
max_nbytes: int
Threshold to trigger memmaping of large arrays to files created
a folder.
temp_folder: str
Path of a folder where files for backing memmaped arrays are created.
mmap_mode: 'r', 'r+' or 'c'
Mode for the created memmap datastructure. See the documentation of
numpy.memmap for more details. Note: 'w+' is coerced to 'r+'
automatically to avoid zeroing the data on unpickling.
verbose: int, optional, 0 by default
If verbose > 0, memmap creations are logged.
If verbose > 1, both memmap creations, reuse and array pickling are
logged.
prewarm: bool, optional, False by default.
Force a read on newly memmaped array to make sure that OS pre-cache it
memory. This can be useful to avoid concurrent disk access when the
same data array is passed to different worker processes.
r NTc C sS | | _ | | _ | | _ t | � | _ | | _ | d k rO t j d t � d S)NzPcontext_id is deprecated and ignored in joblib 0.9.4 and will be removed in 0.11) � _max_nbytes� _temp_folder�
_mmap_mode�int�verbose�_prewarm�warnings�warn�DeprecationWarning)�self�
max_nbytes� temp_folder� mmap_moder: �
context_id�prewarmr r r �__init__� s zArrayMemmapReducer.__init__c C s t | � } | d k r% t | | � S| j j r�| j d k r�| j | j k r�y' t j | j � t j | j t
� Wn= t k
r� } z | j t j
k r� | � WYd d } ~ Xn Xd t j � t t j � � t | � f } t j j | j | � } t j j | � s�| j d k r;t d | j | j | f � x' t | | � D] } t j | t � qKW| j r�t | d | j �j � n, | j d k r�t d | j | j | f � t | | j f f S| j d k r�t d | j | j f � t t | d t! �f f Sd S) Nz %d-%d-%s.pklr z-Memmaping (shape=%r, dtype=%s) to new file %srB r
z-Memmaping (shape=%s, dtype=%s) to old file %sz$Pickling array (shape=%r, dtype=%s).r2 )"r r1 r � hasobjectr6 �nbytes�os�makedirsr7 �chmod�FOLDER_PERMISSIONS�OSError�errno�EEXIST�getpid�id� threading�current_threadr �path�join�existsr: �printr r �FILE_PERMISSIONSr; r r8 �maxr r r )r? r r- �e�basenamer# �dumped_filenamer r r �__call__� s>
( zArrayMemmapReducer.__call__)�__name__�
__module__� __qualname__�__doc__rE r\ r r r r r5 � s r5 c @ s4 e Z d Z d Z d e d d � Z d d � Z d S)�CustomizablePickleraw Pickler that accepts custom reducers.
HIGHEST_PROTOCOL is selected by default as this pickler is used
to pickle ephemeral datastructures for interprocess communication
hence no backward compatibility is required.
`reducers` is expected to be a dictionary with key/values
being `(type, callable)` pairs where `callable` is a function that
give an instance of `type` will return a tuple `(constructor,
tuple_of_objects)` to rebuild an instance out of the pickled
`tuple_of_objects` as would return a `__reduce__` method. See the
standard library documentation on pickling for more details.
Nc C s� t j | | d | �| d k r( i } t t d � rL t j j � | _ n t j j � | _ x* | j � D] \ } } | j | | � qk Wd S)Nr2 �dispatch) r rE �hasattrrb �copy�copyreg�dispatch_table�items�register)r? �writer�reducersr2 �type� reduce_funcr r r rE s z CustomizablePickler.__init__c sB t t d � r1 � f d d � } | | j | <n
� | j | <d S)z@Attach a reducer function to a given type in the dispatch table.rb c s# � | � } | j | d | � d S)N�obj)� save_reduce)r? rm �reduced)rl r r �
dispatcher3 s z0CustomizablePickler.register.<locals>.dispatcherN)rc r rb rf )r? rk rl rp r )rl r rh . s z CustomizablePickler.register)r] r^ r_ r` r rE rh r r r r ra s ra c @ sU e Z d Z d Z d d d � Z d d � Z d d � Z d d
� Z d d � Z d S)
�CustomizablePicklingQueuea� Locked Pipe implementation that uses a customizable pickler.
This class is an alternative to the multiprocessing implementation
of SimpleQueue in order to make it possible to pass custom
pickling reducers, for instance to avoid memory copy when passing
memory mapped datastructures.
`reducers` is expected to be a dict with key / values being
`(type, callable)` pairs where `callable` is a function that, given an
instance of `type`, will return a tuple `(constructor, tuple_of_objects)`
to rebuild an instance out of the pickled `tuple_of_objects` as would
return a `__reduce__` method.
See the standard library documentation on pickling for more details.
Nc C sn | | _ | j d d � \ | _ | _ | j � | _ t j d k rQ d | _ n | j � | _ | j � d S)N�duplexF�win32)
� _reducers�Pipe�_reader�_writer�Lock�_rlock�sys�platform�_wlock�
_make_methods)r? �contextrj r r r rE L s z"CustomizablePicklingQueue.__init__c C s, t | � | j | j | j | j | j f S)N)r rv rw ry r| rt )r? r r r � __getstate__V s
z&CustomizablePicklingQueue.__getstate__c C s2 | \ | _ | _ | _ | _ | _ | j � d S)N)rv rw ry r| rt r} )r? �stater r r � __setstate__[ s $z&CustomizablePicklingQueue.__setstate__c C s | j j � S)N)rv �poll)r? r r r �empty` s z CustomizablePicklingQueue.emptyc s� � j j � _ � � j j � j j � � � � � f d d � } | � _ � j rt � f d d � � � � _ n � j j
� _ � � j d k r� � � _ n: � j j � j j � � � � � f d d � } | � _ d S)Nc
s! � � z � � SWd � � Xd S)Nr r )�racquire�recv�rreleaser r �getg s z4CustomizablePicklingQueue._make_methods.<locals>.getc s< t � } t | � j � j | � � j j | j � � d S)N)r ra rt r rw �
send_bytes�getvalue)rm �buffer)r? r r �sendq s z5CustomizablePicklingQueue._make_methods.<locals>.sendc
s$ � � z � | � SWd � � Xd S)Nr )rm )r� �
wlock_acquire�
wlock_releaser r �put s z4CustomizablePicklingQueue._make_methods.<locals>.put)
rv r� �_recvry �acquire�releaser� rt �_sendrw r� r| r� )r? r� r� r )r� r� r� r? r� r� r� r r} c s z'CustomizablePicklingQueue._make_methods) r] r^ r_ r` rE r r� r� r} r r r r rq ; s
rq c s= e Z d Z d Z d d d � f d d � Z d d � Z � S)� PicklingPoola� Pool implementation with customizable pickling reducers.
This is useful to control how data is shipped between processes
and makes it possible to use shared memory without useless
copies induces by the default pickling methods of the original
objects passed as arguments to dispatch.
`forward_reducers` and `backward_reducers` are expected to be
dictionaries with key/values being `(type, callable)` pairs where
`callable` is a function that, given an instance of `type`, will return a
tuple `(constructor, tuple_of_objects)` to rebuild an instance out of the
pickled `tuple_of_objects` as would return a `__reduce__` method.
See the standard library documentation about pickling for more details.
Nc sr | d k r t � } | d k r* t � } | | _ | | _ t d | � } | j | � t t | � j | � d S)N� processes)�dict�_forward_reducers�_backward_reducers�update�superr� rE )r? r� �forward_reducers�backward_reducers�kwargs�poolargs)� __class__r r rE � s
zPicklingPool.__init__c C s^ t | d t � } t | | j � | _ t | | j � | _ | j j | _ | j j | _
d S)N�_ctx) r r rq r� �_inqueuer� � _outqueuer� �
_quick_putr� �
_quick_get)r? r~ r r r �
_setup_queues� s zPicklingPool._setup_queues)r] r^ r_ r` rE r� r r )r� r r� � s r� c C sM y# t j j | � r" t j | � Wn# t k
rH t j d | � Yn Xd S)zAUtility function to cleanup a temporary folder if still existing.z$Failed to clean temporary folder: %sN)rH rS rU �shutil�rmtree� WindowsErrorr< r= )� folder_pathr r r �
delete_folder� s
r� c sU e Z d Z d Z d d d d d d d d d � f d d � Z � f d d
� Z � S) �
MemmapingPoolau
Process pool that shares large arrays to avoid memory copy.
This drop-in replacement for `multiprocessing.pool.Pool` makes
it possible to work efficiently with shared memory in a numpy
context.
Existing instances of numpy.memmap are preserved: the child
suprocesses will have access to the same shared memory in the
original mode except for the 'w+' mode that is automatically
transformed as 'r+' to avoid zeroing the original data upon
instantiation.
Furthermore large arrays from the parent process are automatically
dumped to a temporary folder on the filesystem such as child
processes to access their content via memmaping (file system
backed shared memory).
Note: it is important to call the terminate method to collect
the temporary folder used by the pool.
Parameters
----------
processes: int, optional
Number of worker processes running concurrently in the pool.
initializer: callable, optional
Callable executed on worker process creation.
initargs: tuple, optional
Arguments passed to the initializer callable.
temp_folder: str, optional
Folder to be used by the pool for memmaping large arrays
for sharing memory with worker processes. If None, this will try in
order:
- a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,
- /dev/shm if the folder exists and is writable: this is a RAMdisk
filesystem available by default on modern Linux distributions,
- the default system temporary folder that can be overridden
with TMP, TMPDIR or TEMP environment variables, typically /tmp
under Unix operating systems.
max_nbytes int or None, optional, 1e6 by default
Threshold on the size of arrays passed to the workers that
triggers automated memory mapping in temp_folder.
Use None to disable memmaping of large arrays.
mmap_mode: {'r+', 'r', 'w+', 'c'}
Memmapping mode for numpy arrays passed to workers.
See 'max_nbytes' parameter documentation for more details.
forward_reducers: dictionary, optional
Reducers used to pickle objects passed from master to worker
processes: see below.
backward_reducers: dictionary, optional
Reducers used to pickle return values from workers back to the
master process.
verbose: int, optional
Make it possible to monitor how the communication of numpy arrays
with the subprocess is handled (pickling or memmaping)
prewarm: bool or str, optional, "auto" by default.
If True, force a read on newly memmaped array to make sure that OS pre-
cache it in memory. This can be useful to avoid concurrent disk access
when the same data array is passed to different worker processes.
If "auto" (by default), prewarm is set to True, unless the Linux shared
memory partition /dev/shm is available and used as temp_folder.
`forward_reducers` and `backward_reducers` are expected to be
dictionaries with key/values being `(type, callable)` pairs where
`callable` is a function that give an instance of `type` will return
a tuple `(constructor, tuple_of_objects)` to rebuild an instance out
of the pickled `tuple_of_objects` as would return a `__reduce__`
method. See the standard library documentation on pickling for more
details.
Ng ��.A�rr Fc
sP | d k r t � } | d k r* t � } | d k rF t j d t � d } d t j � t | � f } | d k r� t j j d d � } | d k rt j j
t � ryD t } t j j | | � � t j j
� � s� t j
� � d } Wn t k
rd } Yn X| d k r t j � } t j j t j j | � � } t j j | | � � � | _ t t d � � � � f d d � }
t j |
� t d k r| d k r�| } t | � | | d
| �} | | t j <t | t j <t d � | | � } | | t j <t | t j <t d | d | d
| � } | j |
� t t | � j | � d S)NzPcontext_id is deprecated and ignored in joblib 0.9.4 and will be removed in 0.11Fzjoblib_memmaping_pool_%d_%d�JOBLIB_TEMP_FOLDERTr� c s& t � d d g �j } | � � d S)N�fromlistr� )�
__import__r� )r� )� pool_folder�pool_module_namer r �_cleanup/ s z(MemmapingPool.__init__.<locals>._cleanup�autorD r� r� r� )!r� r< r= r> rH rO rP �environr� rS rU �SYSTEM_SHARED_MEM_FSrT rI �IOError�tempfile�
gettempdir�abspath�
expanduserr7 r r� �atexitrh r! r5 �ndarrayr4 r" r� r� r� rE )r? r� rA r@ rB r� r� r: rC rD r� �use_shared_mem�pool_folder_namer� �forward_reduce_ndarray�backward_reduce_ndarrayr� )r� )r� r� r rE s`
zMemmapingPool.__init__c s� d } x t | � D]q } y t t | � j � PWq t k
r� } z0 t d � | d | k rq t j d | � WYd d } ~ Xq Xq Wt | j � d S)N�
g�������?r
zAFailed to terminate worker processes in multiprocessing pool: %r)
�ranger� r� � terminater� r r< r= r� r7 )r? � n_retries�irY )r� r r r� Z s
zMemmapingPool.terminate)r] r^ r_ r` rE r� r r )r� r r� � s
F Wr� ):r` r rM rH �statrz rQ r� r� r� r< �timer r� � NameError�pickler �cPickler r � ImportErrorre r r �ior �_multiprocessing_helpersr r �multiprocessing.poolr
�numpyr! �numpy.lib.stride_tricksr � numpy_pickler r �hashingr r� �S_IRUSR�S_IWUSR�S_IXUSRrK rW r r r% r1 r4 �objectr5 ra rq r� r� r� r r r r �<module> sd
$ _2N'