Your IP : 216.73.216.213
���W> � @ s� d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m
Z
m Z d d l m Z d d l m
Z
d d
l m Z d d l m Z d d l m Z e d k r&d d
l m Z d d l m Z d d l m Z d d l m Z d a d d � Z d d � Z d d � Z d d � Z e e e � Z d d � Z d d � Z e
e e d d � � � � Z e
e e d d � � � � Z e
e e d! d" � � � � Z! e
e e d# d$ � � � � Z" e
e e d% d& � � � � Z# e
e e d' d( � � � � Z$ e
e e d) d* � � � � Z% e
e e d+ d, � � � � Z& d- d. � Z' e
e e d/ d0 � � � � Z( d S)1� N)�
with_setup)� assert_equal)�
assert_raises)� assert_false)� assert_true)�
with_numpy�np)�setup_autokill)�teardown_autokill)�with_multiprocessing)� with_dev_shm)�mp)�
MemmapingPool)�has_shareable_memory)�ArrayMemmapReducer)�
reduce_memmapc C s t t d d �d S)N�timeouti, )r �__name__� r r �F/opt/alt/python35/lib/python3.5/site-packages/joblib/test/test_pool.py� setup_module s r c C s t t � d S)N)r
r r r r r �teardown_module s r c C s t j d d � a d S)N�prefixZjoblib_test_pool_)�tempfile�mkdtemp� TEMP_FOLDERr r r r �setup_temp_folder# s r c C s# t d k r t j t � d a d S)N)r �shutil�rmtreer r r r �teardown_temp_folder( s
r c C s* | \ } } } t j j | | | � d S)z�Dummy helper function to be executed in subprocesses
Check that the provided array has the expected values in the provided
range.
N)r �testing�assert_array_equal)�args�data�position�expectedr r r � check_array2 s r& c C sM | \ } } } t | | | � | | d 9<t j j | | d | � d S)z�Dummy helper function to be executed in subprocesses
Check that the input array has the right values in the provided range
and perform an inplace modification to double the values in the range by
two.
� N)r r r r! )r" r# r$ r% r r r �inplace_double= s r( c s� t j j } t j j t d � } t j | d t j d d d d �} d t j | j
d d | j �| d d � <| j � t j | d t j d d d d
d d d d �} t j d � j
| j
� | d d � <| d d � d d � d d � f } t j | � } | j } t d t d � � � f d d � } d d � } | | � } t t | � � t t | t j � � | | | � | | � }
t t |
� � | |
| � | | � } t t | t j � � t t | � � | | | � | | � } t t | t j � � t t | � � | | | � | d
}
t t |
� � | |
� } t t | � � t t | t j � � | | | d
� t j |
� } t t | � � | | � } t t | t j � � t t | � � | | | � d S) z9Check that it is possible to reduce a memmap backed arrayz test.mmap�dtype�shapei� �modezw+g �?r N� � � zr+�order�F�offset�< � r' �cc s � | � \ } } | | � S)Nr )�x�consr" )�reducerr r �reconstruct_arrayn s z;test_memmap_based_array_reducing.<locals>.reconstruct_arrayc S s t | � \ } } | | � S)N)r )r5 r6 r" r r r �reconstruct_memmapr s z<test_memmap_based_array_reducing.<locals>.reconstruct_memmapg �)r, r- r. ���r: )r r r! �os�path�joinr �memmap�float64�aranger* r) �flush�reshape�asarray�Tr r r �
isinstancer �ndarray)r! �filename�buffer�a�br4 �dr8 r9 �a_reconstructed�b_reconstructed�c_reconstructed�d_reconstructed�a3Za3_reconstructedZb3Zb3_reconstructedr )r7 r � test_memmap_based_array_reducingL sR $-
%%
rQ c
C s� t j j } t j j t d � } t j | d t j d d d d �} t j d � j
| j � | d d � <| d d
� } | d d � d d
� f } | d d � d d � d d � d f } | d d d
� } d d � } | | � } t t
| � � t t | t j � � | | | � | | � } t t
| � � | | | � | | � }
t t
|
� � | |
| � | | � } t t
| � � | | | � | | � } t t
| � � | | | � d S)Nz test.mmapr) r* �d � r, r+ zw+r �
r- r3 r. c S s" t | � \ } } | | � } | S)N)r )r5 r6 r" �resr r r r9 � s zEtest_high_dimension_memmap_array_reducing.<locals>.reconstruct_memmap)rR rS rS r, i� i�W i� )r r r! r; r< r= r r> r? r@ rB r* r r rE )
r! rG rI rJ r4 rK �er9 rL rM rN rO Ze_reconstructedr r r �)test_high_dimension_memmap_array_reducing� s4 % (
rW c s2 t j j } t j j t d � } t j | � t d d d d | �} z�t j j t d � } t j | d t j
d d d d �� � j d
� | j t
� f d d � t � j d � D� � | � d t j � j � � t j | d t j
d d d d �� | j t
� f d d � t � j d � D� � t t j | � g � | � d t j � j � � | � d t j � j � � t j | d t j
d d d d d d �� t t | j t � f d d � t � j d � D� � t t t f | j t
� f d d � t � j d � D� � Wd | j � ~ Xd S) z@Check that subprocess can access and update shared memory memmap�poolrT �
max_nbytesr' � temp_folderz test.mmapr) r* r, r- r+ zw+g �?c s? g | ]5 } t � j d � D] } � | | f d f � q q S)r3 g �?)�ranger* )�.0�i�j)rI r r �
<listcomp>� s z)test_pool_with_memmap.<locals>.<listcomp>r r4 c s? g | ]5 } t � j d � D] } � | | f d f � q q S)r3 g @)r[ r* )r\ r] r^ )rJ r r r_ � s �rr1 r. c s g | ] } � | d f � q S)g @r )r\ r] )r4 r r r_ � s c s g | ] } � | d f � q S)g @r )r\ r] )r4 r r r_ � s N)r, r- )r- r, )rT � )r r r! r; r< r= r �makedirsr r> �float32�fill�mapr( r[ r* �onesr �listdirr �AssertionErrorr& � RuntimeError�
ValueError� terminate)r! �pool_temp_folder�prG r )rI rJ r4 r �test_pool_with_memmap� s2
$
$ ! '+
rn c s] t j j } t j j t d � } t j | � t d d d d | �} zt j j t d � } t j | d t j
d d d d �� � j d
� t j � � � t
t � t j � � t t � � � | j t � � f d d � t � j d � D� � | � d t j � j � � | � d t j � j � � t t j | � g � Wd | j � ~ Xd S)z?Check that subprocess can access and update shared memory arrayrX rT rY r' rZ z test.mmapr) r* r, r- r+ zw+g �?c s? g | ]5 } t � j d � D] } � | | f d f � q q S)r3 g �?)r[ r* )r\ r] r^ )rI �a_viewr r r_ s z4test_pool_with_memmap_array_view.<locals>.<listcomp>r N)r, r- )r r r! r; r< r= r rb r r> rc rd rC r rE r r re r( r[ r* rf r rg rk )r! rl rm rG r )rI ro r � test_pool_with_memmap_array_view s$
$
rp c s� t t j t � g � t d d d d t �} zvt t j t � g � t t j j | j � � t j
d d t j �� t � j d � | j
t � f d d � t � j d
� D� � t t j t � g � t j
d d t j �� t � j d � | j
t � f d
d � t � j d
� D� � t t j j | j � � t j | j � } t t | � d � t j d g d d d �} | j
t | g � } t | d
� Wd | j � t t j j | j � � ~ Xd S)z0Check that large arrays are not copied in memoryr, rY �( rZ r- r) ra c s g | ] } � | d f � q S)g �?r )r\ r] )�smallr r r_ = s z8test_memmaping_pool_for_large_arrays.<locals>.<listcomp>r rR i c s g | ] } � | d f � q S)g �?r )r\ r] )�larger r r_ E s r3 �abc�objectN)r r; rg r r r r< �exists� _temp_folderr rf rc �nbytesre r& r[ r* r? r �isdir�len�arrayr rk )rm Zdumped_filenames�objects�resultsr )rs rr r �$test_memmaping_pool_for_large_arrays* s* 00
r~ c s� t d d d d t �} z� t t j t � g � t j d d t j �� t � j d � | j t
� f d d � t � j d
� D� � t t j t � g � Wd | j
� ~ Xd S) z1Check that large arrays memmaping can be disabledr, rY NrZ rR r) i c s g | ] } � | d f � q S)g �?r )r\ r] )rs r r r_ i s zAtest_memmaping_pool_for_large_arrays_disabled.<locals>.<listcomp>r )r r r r; rg r rf r? rx re r& r[ r* rk )rm r )rs r �-test_memmaping_pool_for_large_arrays_disabledZ s 0
r c C s3 t d d d �} z� | j } d } t | j | � � t t j j | � � t j d d t j �} t
| j d � | j t
| g d � t
t t j | � � d � t j d d t j �d } t
| j d � | j t
| g d � t
t t j | � � d � Wd
| j � ~ Xt t j j | � � d
S) z4Check that MemmapingPool uses /dev/shm when possibler, rY rT z /dev/shm/joblib_memmaping_pool_rR r) i r3 r' N)r rw r �
startswithr; r< rv r rf r? r rx re �idrz rg rk r )rm rl Z
folder_prefixrI rJ r r r �test_memmaping_on_dev_shmt s"
r� c
C s� t j j } t d d d d t �} zN | j t j d d �} | j � } t t | � � | | t j d � � Wd | j
� ~ Xd S) z:Check that large arrays are not copied in memory in returnr, rY rT rZ r" �� N)r� ) r r r! r r � apply_asyncrf �getr r rk )r! rm rU rs r r r �.test_memmaping_pool_for_large_arrays_in_return� s
r� c C s t t | � � | | S)z4Multiplication function to be executed by subprocess)r r )rI Zn_timesr r r �_worker_multiply� s r� c C s� t j j } t d d d d t �} z� t j t j d � j d
� d d �d
d
� d
d � d
d
� f } | j t d | d f �j
� } t t | � � | | d | � Wd
| j
� ~ Xd
S)z�Check that memmaps with a bad buffer are returned as regular arrays
Unary operations and ufuncs on memmap instances return a new memmap
instance with an in-memory buffer (probably a numpy bug).
r, rY rT rZ ip �� r' r/ r0 Nr3 r" )r� r' r, )r r r! r r rC r@ rB r� r� r� r r rk )r! rm rI rJ r r r �6test_workaround_against_bad_memmap_with_copied_buffers� s (!
r� ))r; r r �
nose.toolsr r r r r Zjoblib.test.commonr r r r
r r Z joblib._multiprocessing_helpersr
Z joblib.poolr r r r r r r r r Zwith_temp_folderr& r( rQ rW rn rp r~ r r� r� r� r� r r r r �<module> sl Q,4%.&