Repository navigation
Add web-search: configurable web search, extraction, crawl and research - #23
piromagnus wants to merge 2 commits into
Conversation
|
The source repository is now licensed AGPL-3.0-or-later (Copyright 2026 Pierre Marrec, see the |
Thanks for the proposal. I will keep the names. donsetch-pi-durable is only a bridge. It does not add features. It exposes the local donsetch binary's own MCP tools to Pi Pocket. The tool names, descriptions, and schemas are generated from the binary at each release. web_search, web_crawl, web_fetch, and web_screenshot are donsetch's names, not names I chose. A rename would make the addon diverge from upstream and add a custom surface that nobody maintains. |
Adds
entries/web-search.jsonfor my extension piromagnus/pi-pocket-web-search at commitce4314b306b52554793a862444243c3ed8a4d6b9. This supersedes TannerMidd/pi-pocket#126, which the maintainer asked to become a standalone extension; its core part (host.fetchPublic) already landed in 0.20.0.What it does
Owner-managed web access for agents: tiered provider search, public-page extraction, crawl and Tavily research, with an owner-only settings sheet (menu/launcher entry), per-provider BYOK keys, and an observed request ledger.
minPocket: 0.20.0,pocketApi: 2, and it needs the experimentalhost.fetchPublic; it refuses to load without it. All direct fetches go throughhost.fetchPublic(public-address checks, redirect checks, size/time limits).Why each tool
web_search— search with quick/standard/deep tiers; fallback stays inside the tier. Providers: Tavily, Exa, Brave, Serper, OpenAI (synthesis), keyless DonSeTch and DuckDuckGo. Read-only.fetch_content— readable text of a public URL; extraction order direct → Tavily → Exa → DonSeTch stealth fetch. Read-only.web_crawl— site crawl (Tavily, or keyless same-domain DonSeTch). Read-only.web_research— starts a billable Tavily research job; not replay-safe, no auto resubmit. Not marked read-only.web_research_status— polls a saved job. Read-only.Routes (
GET/POST state,POST test,POST search) are owner-only; the browser module is owner-only (client: "owner").Hosts and data (G12/G13)
api.tavily.com,api.exa.ai,api.search.brave.com,google.serper.dev,api.openai.com; keyless queries tohtml.duckduckgo.com; and the public URLs the user/agent supplies. Full per-host detail is in the entry's description and the source README.0700/0600, masked in responses). It does not read~/.pi,auth.json, provider credentials or the environment.host.text) before Pi reads it.DonSeTch binary (G14)
When installed, it runs the local
donsetchbinary as a keyless provider (search, stealth fetch, same-domain crawl). The child gets onlyPATH/HOME/locale/proxy/DONSETCH_*variables — no provider keys; the binary's own key store (donsetch keys) is untouched and still works. A missing binary degrades to a recorded failed attempt. No code is downloaded or updated by the extension.Name conflict with the
donsetchentry — and a proposalweb_searchandweb_crawlare claimed by thedonsetchentry (first entry keeps the name), so CI will fail on those two claims until this is settled. Rather than renaming and shipping two overlapping web extensions, I think the cleaner end state is one:This extension runs the same DonSeTch binary as a provider inside its tier chains and extraction fallback, plus BYOK providers (Tavily, Exa, Brave, Serper, OpenAI), tier routing, the ledger and research. The
donsetchentry'sweb_search/web_crawl/web_fetchare then a strict subset of it. @peyloride — would you be open to reducing your entry to its unique niche (web_screenshot, and the login/JS-render fetch, e.g. asdonsetch_fetch), freeingweb_search/web_crawlfor this one? I'll happily take the coordination in whichever direction you and the maintainer prefer, including renaming my tools if you would rather keep yours.Validation
tsc, Prettier, and 25 tests pass; 2 integration tests install the extension into an unmodified Pi Pocket 0.20.0 (owner-only routes, disable cleanup, public-fetch guards, gallery manifest validation).scripts/validate.ts --pocket <unmodified 0.20.0> --local <repo>passes the files, hashes and load test, anddeclaresmatches what the module installs. The only remaining failures are the two name claims above.