Skip to content

feat(research): add Serply Scholar source - #191

Merged
chauncygu merged 1 commit into
SAIL-Research-Lab:mainfrom
googio:feat/serply-scholar-source
Oct 30, 2026
Merged

chauncygu merged 1 commit into
SAIL-Research-Lab:mainfrom
googio:feat/serply-scholar-source

Conversation

@googio

@googio googio commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds serply_scholar, an optional /research source that gets Google Scholar results through the Serply API. It sits beside the scholarly-based google_scholar source and gives the same Scholar citation counts without scraping, so a query takes about 1-2s and does not hit CAPTCHAs.

It follows the Brave and Tavily sources: tier optional, SERPLY_API_KEY (or /config serply_api_key=...), and it raises SourceSkipped when no key is set, so nothing changes for users without one.

Changes

  • cheetahclaws/research/sources/serply_scholar.py: the new source, registered for the academic domain.
    • Scholar's "Cited by N" goes to engagement_raw and engagement_label.
    • The year comes from the byline and goes to published.
    • The first three author names and the PDF and "cited by" links go to extra.
    • --range/--since/--until map to Scholar's own year filter (as_ylo/as_yhi), so the date filter runs on the server instead of after the fetch.
    • Results whose link is not an http(s) URL are dropped.
  • cheetahclaws/research/sources/__init__.py: one import line.
  • cheetahclaws/research/ranker.py: calibration of 100, the same as semantic_scholar and the default google_scholar falls back to.
  • docs/guides/research.md: a row in the optional-sources table and a row in the time-range mapping table.
  • tests/test_research.py: a skip-without-key test like the Brave and Tavily ones, plus a parse test with a mocked get. The parse test covers the citation count, year, author truncation, the year filter params, the key header and dropping a row with no link.

How to test

export SERPLY_API_KEY=...     # or /config serply_api_key=...
/research --sources serply_scholar --range 2y retrieval augmented generation

Validation

  • python -m pytest tests/test_research.py -q: 90 passed.
  • Full suite: 2915 passed, 25 skipped (Python 3.13, Linux).
  • Live call through SOURCES["serply_scholar"].search(...) with a real key and parse_range("2y"): 5 results, all dated 2024-2026, each with a citation count, authors and a publisher URL.

Disclosure: I work on Serply. Happy to rename the source or adjust anything to match project conventions.

Optional academic source that reads Google Scholar through the Serply
API: Scholar citation counts without scraping, year filter mapped to
as_ylo/as_yhi. Skips without SERPLY_API_KEY.
@chauncygu
chauncygu merged commit db61acb into SAIL-Research-Lab:main Oct 30, 2026
6 checks passed
chauncygu added a commit that referenced this pull request Oct 30, 2026
v3.5.89.1: harden Serply Scholar parsing after merging #191/#193
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants