Skip to content

Add web-search: configurable web search, extraction, crawl and research - #23

Open
piromagnus wants to merge 2 commits into
TannerMidd:mainfrom
piromagnus:web-search-entry
Open

piromagnus wants to merge 2 commits into
TannerMidd:mainfrom
piromagnus:web-search-entry

Conversation

@piromagnus

Copy link
Copy Markdown
Contributor

Adds entries/web-search.json for my extension piromagnus/pi-pocket-web-search at commit ce4314b306b52554793a862444243c3ed8a4d6b9. This supersedes TannerMidd/pi-pocket#126, which the maintainer asked to become a standalone extension; its core part (host.fetchPublic) already landed in 0.20.0.

What it does

Owner-managed web access for agents: tiered provider search, public-page extraction, crawl and Tavily research, with an owner-only settings sheet (menu/launcher entry), per-provider BYOK keys, and an observed request ledger. minPocket: 0.20.0, pocketApi: 2, and it needs the experimental host.fetchPublic; it refuses to load without it. All direct fetches go through host.fetchPublic (public-address checks, redirect checks, size/time limits).

Why each tool

  • web_search — search with quick/standard/deep tiers; fallback stays inside the tier. Providers: Tavily, Exa, Brave, Serper, OpenAI (synthesis), keyless DonSeTch and DuckDuckGo. Read-only.
  • fetch_content — readable text of a public URL; extraction order direct → Tavily → Exa → DonSeTch stealth fetch. Read-only.
  • web_crawl — site crawl (Tavily, or keyless same-domain DonSeTch). Read-only.
  • web_research — starts a billable Tavily research job; not replay-safe, no auto resubmit. Not marked read-only.
  • web_research_status — polls a saved job. Read-only.

Routes (GET/POST state, POST test, POST search) are owner-only; the browser module is owner-only (client: "owner").

Hosts and data (G12/G13)

  • Sends queries and API keys to api.tavily.com, api.exa.ai, api.search.brave.com, google.serper.dev, api.openai.com; keyless queries to html.duckduckgo.com; and the public URLs the user/agent supplies. Full per-host detail is in the entry's description and the source README.
  • Keys are entered in the settings and stored only in this module's own data (0700/0600, masked in responses). It does not read ~/.pi, auth.json, provider credentials or the environment.
  • All text from outside is defanged and fenced (host.text) before Pi reads it.

DonSeTch binary (G14)

When installed, it runs the local donsetch binary as a keyless provider (search, stealth fetch, same-domain crawl). The child gets only PATH/HOME/locale/proxy/DONSETCH_* variables — no provider keys; the binary's own key store (donsetch keys) is untouched and still works. A missing binary degrades to a recorded failed attempt. No code is downloaded or updated by the extension.

Name conflict with the donsetch entry — and a proposal

web_search and web_crawl are claimed by the donsetch entry (first entry keeps the name), so CI will fail on those two claims until this is settled. Rather than renaming and shipping two overlapping web extensions, I think the cleaner end state is one:

This extension runs the same DonSeTch binary as a provider inside its tier chains and extraction fallback, plus BYOK providers (Tavily, Exa, Brave, Serper, OpenAI), tier routing, the ledger and research. The donsetch entry's web_search/web_crawl/web_fetch are then a strict subset of it. @peyloride — would you be open to reducing your entry to its unique niche (web_screenshot, and the login/JS-render fetch, e.g. as donsetch_fetch), freeing web_search/web_crawl for this one? I'll happily take the coordination in whichever direction you and the maintainer prefer, including renaming my tools if you would rather keep yours.

Validation

  • tsc, Prettier, and 25 tests pass; 2 integration tests install the extension into an unmodified Pi Pocket 0.20.0 (owner-only routes, disable cleanup, public-fetch guards, gallery manifest validation).
  • scripts/validate.ts --pocket <unmodified 0.20.0> --local <repo> passes the files, hashes and load test, and declares matches what the module installs. The only remaining failures are the two name claims above.
  • Tests use fake provider responses; no paid provider calls were made.

@piromagnus

Copy link
Copy Markdown
Contributor Author

The source repository is now licensed AGPL-3.0-or-later (Copyright 2026 Pierre Marrec, see the LICENSE file), so I pinned the entry to the commit that carries it: c374a68ea6dca91b3c1221e5e5393ca3cb021b92. No installed file changed; hashes and declares are identical. scripts/validate.ts --pocket <unmodified 0.20.0> --local <repo> still passes the files and load test, with only the two known name claims (web_search, web_crawl) outstanding.

@peyloride

Copy link
Copy Markdown
Contributor

Name conflict with the donsetch entry — and a proposal

web_search and web_crawl are claimed by the donsetch entry (first entry keeps the name), so CI will fail on those two claims until this is settled. Rather than renaming and shipping two overlapping web extensions, I think the cleaner end state is one:

This extension runs the same DonSeTch binary as a provider inside its tier chains and extraction fallback, plus BYOK providers (Tavily, Exa, Brave, Serper, OpenAI), tier routing, the ledger and research. The donsetch entry's web_search/web_crawl/web_fetch are then a strict subset of it. @peyloride — would you be open to reducing your entry to its unique niche (web_screenshot, and the login/JS-render fetch, e.g. as donsetch_fetch), freeing web_search/web_crawl for this one? I'll happily take the coordination in whichever direction you and the maintainer prefer, including renaming my tools if you would rather keep yours.

Thanks for the proposal. I will keep the names.

donsetch-pi-durable is only a bridge. It does not add features. It exposes the local donsetch binary's own MCP tools to Pi Pocket. The tool names, descriptions, and schemas are generated from the binary at each release. web_search, web_crawl, web_fetch, and web_screenshot are donsetch's names, not names I chose. A rename would make the addon diverge from upstream and add a custom surface that nobody maintains.

@TannerMidd TannerMidd closed this Oct 25, 2026
@TannerMidd TannerMidd reopened this Oct 25, 2026
@TannerMidd TannerMidd closed this Oct 25, 2026
@TannerMidd TannerMidd reopened this Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants