Your IP : 216.73.216.213
��X�X � @ s� d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l
m Z m Z m
Z
m Z y d d l m Z Wn" e k
r� d d l m Z Yn Xe r� e j Z e j Z e Z n e j Z e j Z y d d l Z Wn e k
r.d Z Yn Xy d d l Z Wn e k
rYd Z Yn Xd Z d Z d Z d
Z d Z d Z! d
d d d d f Z" e j# e j$ g Z% e d k r�e% j& e j' � e( d d � e e e e e! f D� � Z) d d Z* d d � Z+ d d � Z, d d � Z- e j. d d d � � Z/ d
d f d d � Z0 d Z1 d Z2 d Z3 d Z4 d! Z5 Gd" d# � d# e j6 � Z7 Gd$ d% � d% e7 � Z8 d d& Z9 d' d( d) � Z: d S)*zFUtilities for fast persistence of big data, with optional compression.� N)�closing� )� PY3_OR_LATER�PY26�PY27� _basestring)�RLocks ZFs xs �s BZs �7zXZs ] �zlib�bz2�lzma�xz�gzipc c s | ] } t | � Vq d S)N)�len)�.0�prefix� r �J/opt/alt/python35/lib/python3.5/site-packages/joblib/numpy_pickle_utils.py� <genexpr>: s r i � c C s� | j d � | j t � } | j d � | j t � r< d S| j t � rO d S| j t � rb d S| j t � ru d S| j t � r� d S| j t � r� d Sd S) z�Return the compressor matching fileobj.
Parameters
----------
fileobj: file object
Returns
-------
str in {'zlib', 'gzip', 'bz2', 'lzma', 'xz', 'compat', 'not-compressed'}
r r r
r
r r �compatznot-compressed)
�seek�read�_MAX_PREFIX_LEN�
startswith� _ZLIB_PREFIX� _GZIP_PREFIX� _BZ2_PREFIX� _LZMA_PREFIX�
_XZ_PREFIX�
_ZFILE_PREFIX)�fileobj� first_bytesr r r �_detect_compressorD s
r" c C s9 t s t r" t | t j � r" | St j | d t �Sd S)z0Return a buffered version of a read file object.� buffer_sizeN)r r �
isinstancer
�BZ2File�io�BufferedReader�_IO_BUFFER_SIZE)�fobjr r r �_buffered_read_filed s r* c C s? t s t r( t | t j � r( t | � St j | d t �Sd S)z1Return a buffered version of a write file object.r# N) r r r$ r
r% r r& �BufferedWriterr( )r) r r r �_buffered_write_fileo s
r, c c s� t | � } t | t t � � r- | j j } | d k r[ t j d | t d d �| Vn�| t k s| t | t t � � r� | d k r� t j d t
� d d �t | t j � r� | d k r� t j d t
� d d �t | t t � � r� | Vn� | d k rt
t | d � � Vn� | d
k r=t
t | d � � Vn� | d k r�t rit
t j | d � � Vq�t
t j | j d � � Vnl | d k s�| d
k r�t d k r�t
t j | d � � Vq�t d j t j d t j d � � � n | Vd S)a- Utility function opening the right fileobject from a filename.
The magic number is used to choose between the type of file object to open:
* regular file object (default)
* zlib file object
* gzip file object
* bz2 file object
* lzma file object (for xz and lzma compressor)
Parameters
----------
fileobj: file object
compressor: str in {'zlib', 'gzip', 'bz2', 'lzma', 'xz', 'compat',
'not-compressed'}
filename: str
filename path corresponding to the fileobj parameter.
mmap_mode: str
memory map mode that should be used to open the pickle file. This
parameter is useful to verify that the user is not trying to one with
compression. Default: None.
Returns
-------
a file like object
r zjThe file '%s' has been generated with a joblib version less than 0.10. Please regenerate this pickle file.�
stacklevelr Nz�File "%(filename)s" is compressed using "%(compressor)s" which is not compatible with mmap_mode "%(mmap_mode)s" flag passed. mmap_mode option will be ignored.zuIn memory persistence is not compatible with mmap_mode "%(mmap_mode)s" flag passed. mmap_mode option will be ignored.r �rbr
r
r r zHLzma decompression is not available for this version of python ({0}.{1})r r )r" r$ �tuple�_COMPRESSOR_CLASSES� __class__�__name__�warnings�warn�DeprecationWarning� _COMPRESSORS�localsr& �BytesIOr* �BinaryZlibFile�BinaryGzipFiler r
r% �namer �LZMAFile�NotImplementedError�format�sys� version_info)r �filename� mmap_mode�
compressorr r r �_read_fileobject| sB
rD � c C s� | d } | d } | d k r9 t t | d d | �� S| d k ra t t j | d d | �� St d k r� | d k r� t t j | d d t j d
| �� St d k r� | d k r� t t j | d d
| d t j �� St t | d d | �� Sd S)
z6Return the right compressor file object in write mode.r r r
�wb�
compresslevelr
Nr �check�presetr r> ) r, r: r
r% r r< �
CHECK_NONE� FORMAT_ALONEr9 )rA �compress�compressmethodrG r r r �_write_fileobject� s$
rN i c @ s' e Z d Z d Z e j Z d d d d � Z d d � Z e d d � � Z
d
d � Z d d
� Z d d � Z
d d � Z d d � Z d d � Z d d � Z d d � Z d d � Z d d d � Z d d d � Z d0 d" d# � Z d$ d% � Z d& d' � Z d( d) � Z d* d+ d, � Z d- d. � Z d/ S)1r9 a� A file object providing transparent zlib (de)compression.
A BinaryZlibFile can act as a wrapper for an existing file object, or refer
directly to a named file on disk.
Note that BinaryZlibFile provides only a *binary* file interface: data read
is returned as bytes, and data to be written should be given as bytes.
This object is an adaptation of the BZ2File object and is compatible with
versions of python >= 2.6.
If filename is a str or bytes object, it gives the name
of the file to be opened. Otherwise, it should be a file object,
which will be used to read or write the compressed data.
mode can be 'rb' for reading (default) or 'wb' for (over)writing
If mode is 'wb', compresslevel can be a number between 1
and 9 specifying the level of compression: 1 produces the least
compression, and 9 (default) produces the most compression.
r. � c C s t � | _ d | _ d | _ t | _ d | _ d | _ t | t � sf d | k o` d k n r{ t
d j | � � � | d k r� t } t
j | j � | _ d | _ d | _ nO | d k r� t } t
j | t
j | j t
j d � | _ n t
d | f � � t | t � r<t | | � | _ d
| _ | | _ n? t | d � sZt | d � ro| | _ | | _ n t d
� � d S)NFr r rO zFcompresslevel must be between an integer between 1 and 9, you gave {0}r. � rF zInvalid mode: %rTr �writez1filename must be a str or bytes object, or a file���) r �_lock�_fp�_closefp� _MODE_CLOSED�_mode�_pos�_sizer$ �int�
ValueErrorr> �
_MODE_READr �
decompressobj�wbits�
_decompressor�_buffer�_buffer_offset� _MODE_WRITE� compressobj�DEFLATED�
DEF_MEM_LEVEL� _compressorr �open�hasattr� TypeError)�selfrA �moderG � mode_coder r r �__init__ s< - zBinaryZlibFile.__init__c C s� | j �� | j t k r d SzV | j t t f k rA d | _ n1 | j t k rr | j j | j j
� � d | _ Wd z | j r� | j j � Wd d | _ d | _ t | _ d | _
d | _ XXWd QRXd S)z�Flush and close the file.
May be called more than once without error. Once the file is
closed, any other operation on it will raise a ValueError.
NFrP r )rS rW rV r\ �_MODE_READ_EOFr_ rb rT rQ rf �flushrU �closer` ra )rj r r r rp 9 s"
zBinaryZlibFile.closec C s
| j t k S)z True if this file is closed.)rW rV )rj r r r �closedS s zBinaryZlibFile.closedc C s | j � | j j � S)z3Return the file descriptor for the underlying file.)�_check_not_closedrT �fileno)rj r r r rs X s
zBinaryZlibFile.filenoc C s | j � o | j j � S)z)Return whether the file supports seeking.)�readablerT �seekable)rj r r r ru ] s zBinaryZlibFile.seekablec C s | j � | j t t f k S)z/Return whether the file was opened for reading.)rr rW r\ rn )rj r r r rt a s
zBinaryZlibFile.readablec C s | j � | j t k S)z/Return whether the file was opened for writing.)rr rW rb )rj r r r �writablef s
zBinaryZlibFile.writablec C s] | j rY t | j d d � } d } | d k rC | d j | � 7} | d 7} t | � � d S)Nr; z I/O operation on closed filez {0}�.)rq �getattrrT r> r[ )rj �fname�msgr r r rr m s
z BinaryZlibFile._check_not_closedc C s2 | j t t f k r. | j � t j d � � d S)NzFile not open for reading)rW r\ rn rr r& �UnsupportedOperation)rj r r r �_check_can_readv s
z BinaryZlibFile._check_can_readc C s, | j t k r( | j � t j d � � d S)NzFile not open for writing)rW rb rr r& r{ )rj r r r �_check_can_write{ s
z BinaryZlibFile._check_can_writec C sP | j t t f k r. | j � t j d � � | j j � sL t j d � � d S)Nz3Seeking is only supported on files open for readingz3The underlying file object does not support seeking)rW r\ rn rr r& r{ rT ru )rj r r r �_check_can_seek� s
z BinaryZlibFile._check_can_seekc C s� | j t k r d Sx� | j t | j � k r� y. | j j pL | j j t � } | s[ t
� Wn+ t
k
r� t | _ | j | _ d SYn X| j j
| � | _ d | _ q Wd S)NFr T)rW rn ra r r` r_ � unused_datarT r � _BUFFER_SIZE�EOFErrorrX rY �
decompress)rj �rawblockr r r � _fill_buffer� s
zBinaryZlibFile._fill_bufferTc C s� | j | j d � | _ d | _ g } xG | j � rq | rM | j | j � | j t | j � 7_ d | _ q+ W| r� d j | � Sd S)Nr rP )r` ra r� �appendrX r �join)rj � return_data�blocksr r r � _read_all� s
zBinaryZlibFile._read_allc C s@ | j | } | t | j � k rd | j | j | � } | | _ | j t | � 7_ | r` | Sd S| j | j d � | _ d | _ g } x� | d k r(| j � r(| t | j � k r� | j d | � } | | _ n | j } d | _ | r | j | � | j t | � 7_ | t | � 8} q� W| r<d j | � Sd S)Nr rP )ra r r` rX r� r� r� )rj �n_bytesr� �end�datar� r r r � _read_block� s*
zBinaryZlibFile._read_blockr c C sR | j �B | j � | d k r$ d S| d k r: | j � S| j | � SWd QRXd S)z�Read up to size uncompressed bytes from the file.
If size is negative or omitted, read until EOF is reached.
Returns b'' if the file is already at EOF.
r rP N)rS r| r� r� )rj �sizer r r r � s
zBinaryZlibFile.readc
C s( | j � t j j | | � SWd QRXd S)z_Read up to len(b) bytes into b.
Returns the number of bytes read (0 for EOF).
N)rS r& �BufferedIOBase�readinto)rj �br r r r� � s
zBinaryZlibFile.readintoc C s� | j �r | j � t r6 t | t � r6 | j � } | j j | � } | j j | � | j
t | � 7_
t | � SWd QRXd S)z�Write a byte string to the file.
Returns the number of uncompressed bytes written, which is
always len(data). Note that due to buffering, the file on disk
may not reflect the data written until close() is called.
N) rS r} r r$ �
memoryview�tobytesrf rL rT rQ rX r )rj r� �
compressedr r r rQ � s
zBinaryZlibFile.writec C sP | j j d d � t | _ d | _ t j | j � | _ d | _ d | _
d S)Nr rP ) rT r r\ rW rX r r] r^ r_ r` ra )rj r r r �_rewind� s zBinaryZlibFile._rewindr c C s� | j �� | j � | d k r# nj | d k r? | j | } nN | d k rz | j d k rj | j d d � | j | } n t d | f � � | | j k r� | j � n
| | j 8} | j | d d �| j SWd QRXd S)a� Change the file position.
The new position is specified by offset, relative to the
position indicated by whence. Values for whence are:
0: start of stream (default); offset must not be negative
1: current stream position
2: end of stream; offset must not be positive
Returns the new file position.
Note that seeking is emulated, so depending on the parameters,
this operation may be extremely slow.
r r r r� Fz Invalid value for whence: %sN)rS r~ rX rY r� r[ r� r� )rj �offset�whencer r r r s
zBinaryZlibFile.seekc C s& | j � | j � | j SWd QRXd S)z!Return the current file position.N)rS rr rX )rj r r r �tell* s
zBinaryZlibFile.tellNrR ) r2 �
__module__� __qualname__�__doc__r � MAX_WBITSr^ rm rp �propertyrq rs ru rt rv rr r| r} r~ r� r� r� r r� rQ r� r r� r r r r r9 � s, )
*r9 c @ s e Z d Z d Z d Z d S)r: a A file object providing transparent gzip (de)compression.
If filename is a str or bytes object, it gives the name
of the file to be opened. Otherwise, it should be a file object,
which will be used to read or write the compressed data.
mode can be 'rb' for reading (default) or 'wb' for (over)writing
If mode is 'wb', compresslevel can be a number between 1
and 9 specifying the level of compression: 1 produces the least
compression, and 9 (default) produces the most compression.
� N)r2 r� r� r� r^ r r r r r: 1 s r: � zran out of datac C s� t � } xh yL | j | t | � � } | | 7} t | � d k sV t | � | k rW PWq t j k
ro Yq Xq Wt | � | k r� d } t | | | t | � f � � n | Sd S)a Read from file-like object until size bytes are read.
Raises ValueError if not EOF is encountered before size bytes are read.
Non-blocking objects only supported if they derive from io objects.
Required as e.g. ZipExtFile in python 2.6 can return less data than
requested.
This function was taken from numpy/lib/format.py in version 1.10.2.
Parameters
----------
fp: file-like object
size: int
error_template: str
Returns
-------
a bytes object
The data read in bytes.
r z)EOF: reading %s, expected %d bytes got %dN)�bytesr r r& �BlockingIOErrorr[ )�fpr� �error_templater� �rrz r r r � _read_bytesH s
$ "r� );r� �pickler? r& r r
r
r3 �
contextlibr �_compatr r r r � threadingr � ImportError�dummy_threading�
_Unpickler� Unpickler�_Pickler�Pickler�range�xrange�numpy�npr r r r r r r r6 �GzipFiler% r0 r� r< �maxr r( r" r* r, �contextmanagerrD rN rV r\ rn rb r� r� r9 r: � BUFFER_SIZEr� r r r r �<module> st "
Y� ;