Repository navigation
Package web search as a standalone Pi Pocket extension - #126
piromagnus wants to merge 3 commits into
Conversation
Add the shared provider manager, validated settings, atomic request ledger, owner-only API and configuration sheet. Keep the built-in tool module in a separate commit under contribution policy C17. Local review: trail check reported keyword matches against subscription-usage notes. Those refer to unrelated account credential and cache code; this patch has no subscription lookup or account-report cache. Web provider tests explicitly reject subscription tokens.
Register search, extraction, crawl, research creation and research status tools. Use only the host service at runtime; research submission is not replay-safe. Local review: trail check matches against unrelated subscription usage notes do not apply. This module does not read subscription credentials or maintain account-report caches.
A module that fetches an address from outside (a page Pi names, a search result's link, a crawl) had only fetch, which reaches this machine, the local network, and a cloud's metadata address. PR #126 had to put web search in core to keep that check there (C4). `host.fetchPublic(url, init)` resolves names where it connects and refuses one with any address that is not public, so DNS rebinding gets nowhere; IP literals are checked first. Each redirect is checked again; a 303 becomes a GET without its body's headers, and another origin gets no credentials. It stops at 10 MiB decoded, 30 s, and 5 redirects. Refusals carry a code: POCKET_NOT_PUBLIC, POCKET_TOO_LARGE, POCKET_REDIRECTS. Experimental in 0.20.0. Its first user is docs/examples/read-page.ts, a read_page tool that fences the page's text with host.text (C15). Docs: host reference and guide, limits, API versions, security model, and the web access case in core-or-extension. Tested: test/public-fetch.test.ts (address ranges, literal forms, names, redirects, limits, timeout), the examples test (read_page refuses localhost), npm run check, lint, and npm test (font-ui is flaky once, passes alone). A read-only review found no SSRF bypass; its four correctness findings are fixed.
Thanks for this, @piromagnus: it's a thorough piece of work, and the tiered routing and request accounting are nice. The plan is for this to live as a standalone extension rather than in core, so owners who want web search can install it and others don't carry it. Under core-or-extension, search providers, their keys and settings are an outside-service integration (C6, C8, C12). The one piece that does belong in core is the check that stops a fetched URL from reaching this machine or its network (C4), so that is the part that went into core. The upcoming 0.20.0 release adds
Everything else in this PR already maps onto the public extension API:
The only thing you lose is the link inside the Providers sheet; a menu row or launcher action covers it. If you'd like to share it, you can submit it to the gallery (pi-pocket-extensions, see publishing and gallery review). A few things the review will look for:
This PR stays open for you to decide whether to close it in favour of the extension. If anything in the extension API turns out not to be enough, open an issue titled "Extension point: …". |
Use public 0.20 extension APIs, extension-owned keys/data and host.fetchPublic. Retain tier routing and observed request accounting; fence external tool output. TypeScript, lint, formatting and 22 tests passed for this exact prepared package. Browser validation remains pending. Scoped trail check reports keyword matches in negative security assertions and mocked tests, not semantic regressions: production has no unrestricted fetch or external credential fallback, no core Providers-sheet integration, and no cross-tier escalation. These boundaries are covered by tests; no branch overlaps were reported. Record that review in a new team learning.
piromagnus
changed the title
|
Closing in favour of the standalone extension, as reviewed: the gallery entry is TannerMidd/pi-pocket-extensions#23, with the code at piromagnus/pi-pocket-web-search (pinned commit ce4314b). The core part — |
Provides an installable
web-searchextension for Pi Pocket 0.20.0+, using the experimentalhost.fetchPublicAPI. It exposes owner-only settings through public extension routes and a menu/launcher sheet, with its own stored keys and data underhost.dataDir.Search supports configurable quick/standard/deep provider ordering, manual overrides and fallback confined to the selected tier. Provider options cover Exa search/content and Tavily search, extraction, crawl and research. The local ledger tracks observed requests, outcomes and reported credits separately.
External tool output is defanged and fenced. The README documents installation, every contacted hostname and transmitted data, key isolation, experimental API requirements and accounting limits. Research creation remains non-replay-safe.
Validation: TypeScript, ESLint, formatting and 22 tests pass, including installation, owner/non-owner routes, disable cleanup, gallery manifest validation and public-network guards against unmodified Pi Pocket 0.20. Tests use fake provider responses. Chromium could not launch in this environment, so browser validation remains pending; no paid provider calls were made.