robotparser do not allow some url which are allowed

[FedeGerva]

Bug report

Bug description:

I was using robotparser to check some websites and I realized that the there was a strange behavior. I gave it some urls that I knew could be crawled and it return me False. After some debugging I realize that the url was ok but I think there is a problem in the function that is reading the url.

Line 62 of https://github.com/python/cpython/blob/main/Lib/urllib/robotparser.py

f = urllib.request.urlopen(self.url)

returned my an error code 404, but actually the url/robots.txt file was present and I could see it online.

What I did was to add these two lines of code, the error disappear and the function told me the url could be crawled, as I expected. :

header= {'User-Agent': '*'}  
req = urllib.request.Request(url=self.url, headers=header) 
f = urllib.request.urlopen(req)

Could you give me a feedback on that? Do you think it is correct and could be fixed in the main version?

CPython versions tested on:

3.8, 3.9

Operating systems tested on:

Windows

[sobolevn]

Can you please provide a URL to reproduce this locally?

[FedeGerva]

Yes, sure. here my code.

If I try it using the original function it returns False. If I add the header (see above) it return True

  url='https://www.researchgate.net/profile/Federica-Gervasoni'
  
  parse_url=urlparse(url)
  base_url=parse_url.scheme + '://' + parse_url.netloc

  #Check robots
  rp = urllib.robotparser.RobotFileParser()
  rp.set_url(base_url + '/robots.txt')
  rp.read()
  print(rp.modified())

  if rp.can_fetch("*", url):
    print(f"{url}")
  else:
    print(f"Crowling cannot be performed {url}")

[serhiy-storchaka]

The robotparser module is old, it is based on the pre-standard document http://www.robotstxt.org/orig.html. It does not support all features of RFC 9309, which was only published in 2022. In particularly, it does not support the $ special character.

So, this is a request for a new feature. Unfortunately, the full support of RFC 9309 requires changing the internal structure of the module.