Trying to fetch a dust map raises a JSONDecodeError

[ahlamaq]

I am trying to download one of the dust maps and encounter the following error, regardless of whether I try to download the map via pip [ eg. dustmaps.bayestar.fetch() ] or manually via setup.py [ eg. python setup.py fetch --map-name=bayestar ]:

File "/anaconda3/lib/python3.11/site-packages/dustmaps/bayestar.py", line 628, in fetch
    fetch_utils.dataverse_download_doi(
  File "/anaconda3/lib/python3.11/site-packages/dustmaps/fetch_utils.py", line 400, in dataverse_download_doi
    metadata = dataverse_search_doi(doi)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda3/lib/python3.11/site-packages/dustmaps/fetch_utils.py", line 367, in dataverse_search_doi
    return json.loads(r.text)
           ^^^^^^^^^^^^^^^^^^
  File "/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This error is actually raised regardless of which dust map I try to download. Any help in resolving this would be much appreciated, since I cannot currently access any of the dust maps from this package.

[gregreen]

I am unable to reproduce this error. Can you check if it still occurs for you? I suspect that it had something to do with the Harvard Dataverse returning an invalid JSON response.

[gregreen]

Just to expand on this, it's possible that this is related to Issue IQSS/dataverse.harvard.edu#40, which was a problem with the Harvard Dataverse.

[genghisken]

Hi - I see this DOI retrieval issue has recurred for us today (my code initiates the fetch of the dustmaps if they are not already present). This appears to be a recurring problem with the Dataverse server at Harvard. Can anyone confirm that this is a server issue?

Exception Type: JSONDecodeError
Expecting value: line 1 column 1 (char 0)

[nesquivelr]

Hi, I'm experiencing the same issue. How can I fix this?

I looked into why the metadata can't be dumped into JSON, and it seems the server is responding with a robots.txt file instead of the expected metadata. This only happens when I'm running this code inside a container on Google Cloud Platform (GCP).

Here's the server response that the code is trying to parse as JSON:

User-agent: *
Disallow: /
User-agent: Googlebot
User-agent: soscan (+https://dataone.org/)
Allow: /$
Allow: /dataset.xhtml
Allow: /sitemap*
Allow: /api/datasets/:persistentId/thumbnail
Disallow: /dataverse/*?q
Disallow: /dataverse/*/search
Disallow: /file.xhtml
Disallow: /dataset.xhtml?*&version=&q=
Disallow: /
Disallow: /api
Disallow: /api/*
Crawl-delay: 100
Sitemap: https://dataverse.harvard.edu/sitemap_index.xml
# Created initially using: http://www.mcanerin.com/EN/search-engine/robots-txt.asp
# Verified using: http://tool.motoricerca.info/robots-checker.phtml

[gregreen]

This looks like an issue with the Harvard Dataverse. One way to get around this problem is to see if you can download the files manually and put them in the directories that dustmaps wants them to be in. Then, you can run fetch() again, and it will detect the presence of the file and skip the download (it may do additional processing on the file, which is why you should still run fetch()).

[hombit]

The mechanism behind this seems to have changed. dataverse.harvard.edu is now behind an AWS WAF that answers non-browser clients with a JS challenge rather than a robots.txt:

$ curl -sD- -o /dev/null "https://dataverse.harvard.edu/api/datasets/:persistentId?persistentId=doi:10.7910/DVN/2EJ9TX"
HTTP/2 202
x-amzn-waf-action: challenge
content-length: 0

202 is a success status, so raise_for_status() in dataverse_search_doi() doesn't raise, and json.loads() then gets an empty body: hence the JSONDecodeError.

This isn't cloud-specific anymore: I see it both from a home connection and from CI, on dustmaps 1.0.14 / requests 2.34.2 / Python 3.12. /api/info/version, /api/access/datafile/<id> and the site root all return the same challenge, so every Dataverse-hosted map is affected. Setting a browser User-Agent doesn't help: the WAF wants a JS-solved aws-waf-token cookie, which requests can't produce.

[gregreen]

The Harvard Dataverse appears to have effectively broken unauthenticated API access, probably in response to an increase in scraping bots. I think the only long-term solution to this problem is to migrate data to a host that is not actively clamping down on API access. Zenodo is probably ideal.

[hombit]

Related Harvard Dataverse issue:
https://github.com/IQSS/dataverse/issues/12656