Your IP : 216.73.216.213
�
g fi � @ s) d d l Z d d l Z d d l Z d d l m Z m Z d d l m Z d d l m
Z
d d l m Z m
Z
y d d l Z Wn e k
r� d Z Yn XGd d � d e j � Z e j � Z d d d
� Z d Z d d
g Z d d d g Z e d e e e � d Z d d
dO g Z d g Z e d e e e � d Z g Z d d d g Z e d e e e � d Z g Z d d d d d d d g Z e d e e e d! � e d" e e e d# � d$ Z d g Z d d d d d% d& g Z e d' e e e � d( Z d g Z g Z e d) e e e � d* Z d+ g Z d, g Z e d- e e e d. d/ �d0 Z g Z d1 g Z e d2 e e e d. d/ �g Z d1 g Z e d3 e e e d. d4 �d5 Z g Z d1 g Z e d6 e e e d. d/ �d1 g Z g Z e d7 e e e d. d4 �d* Z d+ g Z d, g Z e d8 e e e d. d9 �d: Z d; g Z d< g Z e d= e e e � d> Z d? g Z d; g Z e d@ e e e � dA Z dB g Z dC g Z e dD e e e � GdE dF � dF e � Z e j e dG � GdH dI � dI e j � � Z GdJ dK � dK e j � Z dL dM � Z e dN k r%e j � n d S)P� N)�URLError� HTTPError)�urlopen)�support)�BaseHTTPRequestHandler�
HTTPServerc @ sC e Z d Z d d d d d d d � Z d d � Z d d � Z d S)�
RobotTestCaseNc C sz t | t � s d St j j | � | r? d | | f | _ n d | | f | _ | | _ | | _ | | _ | | _ d S)NzRobotTest(%d, good, %s)zRobotTest(%d, bad, %s))
�
isinstance�int�unittest�TestCase�__init__�str�parser�url�good�agent)�self�indexr r r r � r �:/opt/alt/python34/lib64/python3.4/test/test_robotparser.pyr
s zRobotTestCase.__init__c C s~ t | j t � r$ | j \ } } n | j } | j } | j r^ | j | j j | | � � n | j | j j | | � � d S)N) r r �tupler r �
assertTruer � can_fetch� assertFalse)r r r r r r �runTest s zRobotTestCase.runTestc C s | j S)N)r )r r r r �__str__( s zRobotTestCase.__str__)�__name__�
__module__� __qualname__r
r r r r r r r s r Ztest_robotparserc C s� t j | � j � } t j j � } | j | � x- | D]% } t j t | | | d | � � q8 Wx- | D]% } t j t | | | d | � � qh Wd S)N� r )
�io�StringIO� readlines�urllib� robotparser�RobotFileParser�parse�tests�addTestr )r Z
robots_txtZ good_urlsZbad_urlsr �linesr r r r r � RobotTest- s
#
r+ z�
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html
�/z
/test.html�/cyberworld/map/index.htmlz/tmp/xxxz /foo.htmlr z�
# robots.txt for http://www.example.com/
User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
# Cybermapper knows where to go.
User-agent: cybermapper
Disallow:
� cybermapper� z%
# go away
User-agent: *
Disallow: /
z/tmp/� zm
User-agent: figtree
Disallow: /tmp
Disallow: /a%3cd.html
Disallow: /a%2fb.html
Disallow: /%7ejoe/index.html
z/tmpz /tmp.htmlz /tmp/a.htmlz /a%3cd.htmlz /a%3Cd.htmlz /a%2fb.htmlz/~joe/index.html� Zfigtree� z FigTree Robot libwww-perl/5.04zf
User-agent: *
Disallow: /tmp/
Disallow: /a%3Cd.html
Disallow: /a/b.html
Disallow: /%7ejoe/index.html
z /a/b.htmlz/%7Ejoe/index.html� z
User-Agent: *
Disallow: /.
� zG
User-agent: Googlebot
Allow: /folder1/myfile.html
Disallow: /folder1/
z/folder1/myfile.htmlz/folder1/anotherfile.html� r Z GooglebotzJ
User-agent: Googlebot
Disallow: /
User-agent: Googlebot-Mobile
Allow: /
z/something.jpg� �
zGooglebot-MobilezJ
User-agent: Googlebot-Mobile
Allow: /
User-agent: Googlebot
Disallow: /
� � �
Z googlebotz/
User-agent: *
Disallow: /some/path?name=value
z
/some/pathz/some/path?name=value� zK
User-agent: *
Disallow: /some/path
User-agent: *
Disallow: /another/path
z
/another/path� z;
User-agent: *
Allow: /some/path?
Disallow: /another/path?
z /some/path?z/another/path?� c @ s( e Z d Z d d � Z d d � Z d S)� RobotHandlerc C s | j d d � d S)Ni� zForbidden access)Z
send_error)r r r r �do_GET s zRobotHandler.do_GETc G s d S)Nr )r �format�argsr r r � log_message s zRobotHandler.log_messageN)r r r r? rB r r r r r> s r> z threading required for this testc @ sL e Z d Z d d � Z d d � Z d d � Z d d � Z d d
� Z d S) � PasswordProtectedSiteTestCasec C sf t t j d f t � | _ t j d d d | j j d i d d 6� | _ d | j _ | j j
� d S) Nr �namezHTTPServer serving�target�kwargsg{�G�z�?Z
poll_intervalT) r r �HOSTr> �server� threadingZThreadZ
serve_forever�tZdaemon�start)r r r r �setUp s z#PasswordProtectedSiteTestCase.setUpc C s+ | j j � | j j � | j j � d S)N)rH ZshutdownrJ �joinZ server_close)r r r r �tearDown s
z&PasswordProtectedSiteTestCase.tearDownc C s | j � d S)N)�testPasswordProtectedSite)r r r r r s z%PasswordProtectedSiteTestCase.runTestc C sx | j j } d t j d t | d � } | d } t j j � } | j | � | j � | j
| j d | � � d S)Nzhttp://�:r z /robots.txt�*) rH Zserver_addressr rG r r$ r% r& Zset_url�readr r )r Zaddrr Z
robots_urlr r r r rO ! s
z7PasswordProtectedSiteTestCase.testPasswordProtectedSitec C s d | j j S)Nz%s)� __class__r )r r r r r * s z%PasswordProtectedSiteTestCase.__str__N)r r r rL rN r rO r r r r r rC s
rC c @ s+ e Z d Z e j d � d d � � Z d S)�NetworkTestCasez7does not handle the gzip encoding delivered by pydotorgc C s\ t j d � t j d � �: t j j d � } | j � | j | j d d � � Wd QXd S)NZnetworkzwww.python.orgz http://www.python.org/robots.txtrQ ) r ZrequiresZtransient_internetr$ r% r& rR r r )r r r r r �
testPythonOrg/ s
z NetworkTestCase.testPythonOrgN)r r r r �skiprU r r r r rT - s rT c C s0 t j t � } | j t � | j t � � | S)N)r Z makeSuiterT r) r( rC )�loaderZsuite�patternr r r �
load_tests9 s
rY �__main__)r. r- ) r! r Zurllib.robotparserr$ Z urllib.errorr r Zurllib.requestr Ztestr Z http.serverr r rI � ImportErrorr r Z TestSuiter( r+ �docr Zbadr> Z
skipUnlessrC rT rY r �mainr r r r �<module> s�
#