Your IP : 216.73.216.213
3
� f`( �
@ s d dl Z d dlZd dlZd dlZd dlmZ d dlmZm Z y d dl
Z
W n e k
r` dZ
Y nX G dd� d�Z G dd� de ej
�ZG dd � d e ej
�ZG d
d � d e ej
�ZG d d
� d
e �ZG dd� deej
�ZG dd� de�ZG dd� de ej
�ZG dd� de ej
�ZG dd� de ej
�ZG dd� de ej
�ZG dd� de�ZG d d � d e ej
�ZG d d � d e ej
�ZG d d!� d!e ej
�ZG d"d#� d#e ej
�Z G d$d%� d%eej
�Z G d&d'� d'e ej
�Z G d(d)� d)e�Z ej e
d*�G d+d,� d,ej
��Z!G d-d.� d.ej
�Z"e#d/k�r
ej$� dS )0� N)�support)�BaseHTTPRequestHandler�
HTTPServerc @ s< e Zd ZdZdZg Zg Zdd� Zdd� Zdd� Z d d
� Z
d S ) �
BaseRobotTest� Ztest_robotparserc C s, t j| j�j� }tjj� | _| jj|� d S )N) �io�StringIO�
robots_txt� readlines�urllib� robotparser�RobotFileParser�parser�parse)�self�lines� r �:/opt/alt/python36/lib64/python3.6/test/test_robotparser.py�setUp s zBaseRobotTest.setUpc C s$ t |t�r|\}}||fS | j|fS )N)�
isinstance�tuple�agent)r �urlr r r r �get_agent_and_url s
z BaseRobotTest.get_agent_and_urlc C sP xJ| j D ]@}| j|�\}}| j||d�� | j| jj||�� W d Q R X qW d S )N)r r )�goodr �subTest�
assertTruer � can_fetch)r r r r r r �test_good_urls s z BaseRobotTest.test_good_urlsc C sP xJ| j D ]@}| j|�\}}| j||d�� | j| jj||�� W d Q R X qW d S )N)r r )�badr r � assertFalser r )r r r r r r �
test_bad_urls$ s zBaseRobotTest.test_bad_urlsN) �__name__�
__module__� __qualname__r r r r r r r r! r r r r r
s r c @ s" e Zd ZdZddgZdddgZdS )�UserAgentWildcardTestz�User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/z
/test.htmlz/cyberworld/map/index.htmlz/tmp/xxxz /foo.htmlN)r" r# r$ r r r r r r r r% + s r% c @ s e Zd ZdZdddgZdgZdS )� CrawlDelayAndCustomAgentTestz�# robots.txt for http://www.example.com/
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
r& z
/test.html� cybermapper�/cyberworld/map/index.htmlN)r( r) )r" r# r$ r r r r r r r r' 6 s
r' c @ s e Zd ZdZg ZdddgZdS )�RejectAllRobotsTestz(# go away
User-agent: *
Disallow: /
z/cyberworld/map/index.htmlr& z/tmp/N)r" r# r$ r r r r r r r r* G s r* c @ s e Zd Zdd� ZdS )�BaseRequestRateTestc C s� x�| j | j D ]�}| j|�\}}| j||d��t | jrL| j| jj|�| j� | jr�| j| jj|�t j
j � | j| jj|�j | jj � | j| jj|�j
| jj
� W d Q R X qW d S )N)r r )r r r r � crawl_delay� assertEqualr � request_rateZassertIsInstancer r � RequestRateZrequestsZseconds)r r r r r r �test_request_rateS s
z%BaseRequestRateTest.test_request_rateN)r" r# r$ r0 r r r r r+ Q s r+ c @ s> e Zd ZdZdZejjdd�ZdZ dgZ
ddd d
d d d
gZ dS )� CrawlDelayAndRequestRateTestz�User-agent: figtree
Crawl-delay: 3
Request-rate: 9/30
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
�figtree� � � � /foo.htmlz/tmpz /tmp.htmlz /tmp/a.htmlz /a%3cd.htmlz /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.htmlN)r2 r6 ) r" r# r$ r r r r r/ r. r, r r r r r r r1 j s
r1 c @ s e Zd ZdZdZdZdS )�DifferentAgentTestz FigTree Robot libwww-perl/5.04N)r" r# r$ r r. r, r r r r r7 | s r7 c @ s* e Zd ZdZdgZddddddgZd Zd
S ) �InvalidRequestRateTestz�User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
Crawl-delay: 3
Request-rate: 9/banana
z/tmpz/tmp/z /tmp/a.htmlz /a%3cd.htmlz /a%3Cd.htmlz /a/b.htmlz/%7Ejoe/index.htmlr5 N)r" r# r$ r r r r, r r r r r8 � s
r8 c @ s e Zd ZdZdgZg ZdS )�InvalidCrawlDelayTestz2User-Agent: *
Disallow: /.
Crawl-delay: pears
z /foo.htmlN)r" r# r$ r r r r r r r r9 � s r9 c @ s e Zd ZdZdZdgZdgZdS )� AnotherInvalidRequestRateTestzeUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Request-rate: whale/banana
� Googlebotz/folder1/myfile.htmlz/folder1/anotherfile.htmlN)r" r# r$ r r r r r r r r r: � s r: c @ s e Zd ZdZdZdgZdS )�UserAgentOrderingTestzMUser-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
r; z/something.jpgN)r" r# r$ r r r r r r r r< � s
r< c @ s e Zd ZdZdS )�UserAgentGoogleMobileTestzGooglebot-MobileN)r" r# r$ r r r r r r= � s r= c @ s e Zd ZdZdZdgZdgZdS )�GoogleURLOrderingTestzJUser-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
Z googlebotz/folder1/myfile.htmlz/folder1/anotherfile.htmlN)r" r# r$ r r r r r r r r r> � s r> c @ s e Zd ZdZdgZdgZdS )�DisallowQueryStringTestz2User-agent: *
Disallow: /some/path?name=value
z
/some/pathz/some/path?name=valueN)r" r# r$ r r r r r r r r? � s r? c @ s e Zd ZdZdgZdgZdS )� UseFirstUserAgentWildcardTestzNUser-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/pathz
/some/pathN)r" r# r$ r r r r r r r r@ � s r@ c @ s e Zd ZdZdgZdgZdS )�EmptyQueryStringTestz>User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?N)r" r# r$ r r r r r r r rA � s rA c @ s0 e Zd ZdZejjdd�ZdZddgZ dgZ
dS ) �DefaultEntryTestzOUser-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
r5 � � r& z
/test.htmlz/cyberworld/map/index.htmlN) r" r# r$ r r r r/ r. r, r r r r r r rB � s
rB c @ s e Zd ZdZdZdd� ZdS )�StringFormattingTestz�User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow: /some/path
zzUser-agent: cybermapper
Disallow: /some/path
User-agent: *
Crawl-delay: 1
Request-rate: 3/15
Disallow: /cyberworld/map/
c C s | j t| j�| j� d S )N)r- �strr �expected_output)r r r r �test_string_formatting s z+StringFormattingTest.test_string_formattingN)r" r# r$ r rG rH r r r r rE � s
rE c @ s e Zd Zdd� Zdd� ZdS )� RobotHandlerc C s | j dd� d S )Ni� zForbidden access)Z
send_error)r r r r �do_GET s zRobotHandler.do_GETc G s d S )Nr )r �format�argsr r r � log_message s zRobotHandler.log_messageN)r" r# r$ rJ rM r r r r rI s rI z threading required for this testc @ s* e Zd Zdd� Zdd� Zejdd� �ZdS )� PasswordProtectedSiteTestCasec C sB t tjdft�| _tjd| jjddid�| _d| j_ | jj
� d S )Nr zHTTPServer servingZ
poll_intervalg{�G�z�?)�name�target�kwargsT) r r �HOSTrI �server� threadingZThreadZ
serve_forever�tZdaemon�start)r r r r r # s z#PasswordProtectedSiteTestCase.setUpc C s" | j j� | jj� | j j� d S )N)rS ZshutdownrU �joinZ server_close)r r r r �tearDown0 s
z&PasswordProtectedSiteTestCase.tearDownc C s\ | j j}dtj d t|d � }|d }tjj� }|j|� |j � | j
|j d|�� d S )Nzhttp://�:rD z /robots.txt�*) rS Zserver_addressr rR rF r r r
Zset_url�readr r )r Zaddrr Z
robots_urlr r r r �testPasswordProtectedSite5 s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSiteN)r" r# r$ r rX r Z reap_threadsr\ r r r r rN s
rN c @ sF e Zd ZdZdje�Zedd� �Zdd� Zdd� Z d d
� Z
d d � Z d
S )�NetworkTestCasezhttp://www.pythontest.net/z{}elsewhere/robots.txtc C s@ t jd� t j| j�� tjj| j�| _| jj � W d Q R X d S )NZnetwork)
r ZrequiresZtransient_internet�base_urlr r r
r r r[ )�clsr r r �
setUpClassE s
zNetworkTestCase.setUpClassc C s$ dj | j|tjj|�d s dnd�S )Nz{}{}{}rD r& r )rK r^ �os�path�splitext)r rb r r r r L s zNetworkTestCase.urlc C sV | j | jj� | j | jj� | j| jj� d� | j | jjd�� | j | jjd�� d S )Nr rZ )r r � disallow_all� allow_allZ
assertGreater�mtimer, r. )r r r r �
test_basicQ s
zNetworkTestCase.test_basicc C s� | j | jjd| jd��� | j| jjd| j�� | j| jjd| jd��� | j| jjd| jd��� | j| jjd| jd��� | j | jjd| j�� d S )NrZ Z elsewhereZNutchZbrianZwebstats)r r r r r r^ )r r r r �test_can_fetchX s z NetworkTestCase.test_can_fetchc C sf t jj| jd��}|j� | j|j� | j|j� | j |j
� d� | j |j d�� | j |j
d�� d S )Nz i-robot.txtr rZ )r r r
r r[ r re r rd r- rf Z assertIsNoner, r. )r r r r r �
test_read_404` s z NetworkTestCase.test_read_404N) r" r# r$ r^ rK r � classmethodr` r rg rh ri r r r r r] @ s
r] �__main__)%r ra ZunittestZurllib.robotparserr Ztestr Z http.serverr r rT � ImportErrorr ZTestCaser% r' r* r+ r1 r7 r8 r9 r: r< r= r> r? r@ rA rB rE rI Z
skipUnlessrN r] r" �mainr r r r �<module> sB
)