Repository navigation
fix(discovery): fix budget starvation, name-only merging, and 34 other issues in discovery - #170
Merged
Merged
Conversation
The sweep filtered dependency caches and skip-listed trees only after walk() had already listed and charged them, so the skip lists saved no budget. walk() now takes a descend predicate that is consulted before a directory joins the frontier, and the sweep passes its scope test.
A directory holding a valid CACHEDIR.TAG, or listing more than Budget.max_children entries (default 5000, --max-children), is still read but its subdirectories are not enqueued. Each case is recorded as a cache_dir or fan_out boundary, the latter with the true child count.
walk() used one global FIFO, so a single huge subtree drained the shared entry ceiling before its siblings got past the first levels. It now keeps one queue per top-level child of the root and serves them round-robin, breadth-first within each. Visit order stays deterministic.
State roots were only listed one level, so plugin-cache skills depended on the home sweep reaching depth 6+ before the budget ran out. The sweep now walks the existing STATE_ROOTS first, to the depth cap, under a share of the ceiling (Budget.state_root_share, default 25%) and with the same marker logic. Budget.reserved_walk_entries keeps entries back from the generic sweep for later stages.
Roots were walked independently, so /home re-listed /home/<user> and a home re-listed its own code roots and state roots, charging them twice. walk() now takes a shared set of listed directories; the sweep passes one set to every walk. Directories are marked when listed, so subtrees a capped walk never reached stay available to the next one.
The sweep could spend the whole entry ceiling, after which extraction's listing of the surface directories it had just found was refused, so no skills or commands were read. Listings of marker directories are now retained by the gate and re-served free, and reserved_walk_entries defaults to 10% of max_entries so later stages always have budget left.
CONFIG_FILE_TEMPLATES held mostly Linux paths, so Claude Desktop, VS Code, and Cline configs on macOS and Windows were reached only by chance. Add their documented per-user paths, ~/.claude/settings*.json, and the macOS Claude Code managed files. Zed documents ~/.config/zed on macOS, so only its Windows %APPDATA% path is new. Windows absolute (ProgramData) paths are not expressible in the logical posix path layer and are skipped.
opencode declares servers with a list `command`, `environment` for env, and type local/remote, so every record was rejected. Accept an argv list (argv[0] is the command), treat `environment` as `env` (names only), map local/remote to stdio/http, and emit servers with enabled:false carrying a `disabled` flag on the asset.
A well-formed .mcpb/manifest.json produced no declaration because its `server` key (singular) is not one of the config server blocks; only the unparseable case yielded an mcp_bundle. Read name, version and server.mcp_config (command, args, env names) into an mcp_bundle, and carry the declared version onto the asset.
A file reached by both the config-template source and the sweep was extracted twice, giving merged_2 flags and duplicated evidence for every server in it. Dedupe config surfaces by real path before extraction, keeping the first candidate and recording the other sources in its detail.
Block scalars, folded descriptions, nested maps and flow lists made the strict subset parser reject the whole record. Split the block into top-level entries without reading values, parse only allowlisted keys (flow and block lists, block scalars, quoted strings, CRLF), and skip other unreadable keys with a coverage note. The line cap rises from 60 to 200 beside the 64 KiB byte cap; bodies are still never returned.
Hook assets were named by a per-file counter, so adding one hook renamed the rest and changed their asset ids. Derive identity from settings path, event, matcher and a hash of the scrubbed command (hook:<path>:<digest>), keep a readable name, and carry hook_event and hook_matcher onto the asset as optional trailing fields. The command itself is not serialized.
Every CLAUDE.md, every same-named skill and every MCP server name merged into one asset, and the judge picked an arbitrary declaration by name. File-shaped kinds now carry their path in the identity, MCP servers their declaring config file, CI agents workflow+job+step. The judge looks up the declaration by asset identity. Bump schema to 1.1.
Instructions, skills, commands and similar files were given launch risk derived from a command that does not exist: transport "stdio" and an inherited environment that reaches every credential. Only MCP servers, bundles, hooks and CI agents now get launch risk; other declarations keep an empty Risk.
The degraded "unreadable launch shape" probe fired for every hook and for any command the pinning parser had no verdict on, such as absolute-path binaries or `python -m mod`. Add a launch_shape classifier (package runners incl. pnpm dlx/bun x/uv tool run, containers, local programs) and probe only MCP launches that are truly unparseable.
Sanction worked only for catalogued tools, so MCP servers, skills and package-launched servers could never be approved or forbidden. Policy gains approved_indicators/forbidden_indicators (skill_sha256, file_sha256, mcp_endpoint, package, name), state_for takes the asset, and a forbidden match raises a high forbidden_asset finding. Skills carry skill_sha256.
The catalog id a workflow step resolves to stayed in the declaration, so CI agents could never be judged or sanctioned. Copy it onto the observation. A declaration only names the tool: it never merges by catalog id, takes no local session telemetry, and CI agents raise no unattended_execution finding.
Entries such as brew:claude under proofs.provenance were never indexed, so a brew package record named claude did not identify claude-code. Index them with the package fingerprints at load, under the same ambiguity rule; the shipped catalog loads without conflicts.
…ad of failing them A directory under plugins/ or skills/ without the manifest the host loads (cache/, data/, marketplaces/, shared reference dirs) is not a member of the surface. It was raised per entry and recorded as a degraded extractor probe, which made bookkeeping look like coverage gaps.
…layout Claude Code's native installer keeps each release under ~/.local/share/claude/versions/<version> with a launcher link in ~/.local/bin. There is no package record, so the provenance rung never concluded and an installed, running claude was not inventoried. A catalog entry may now declare fingerprints.install_layouts: a directory its updater manages, globbed in the last segment only, and how to read the version from the path. A binary (its symlink followed through the gate), a process or a network peer whose executable lies there is concluded at the provenance rung; the version is kept only when it has the entry's version_shape. The layout directory is the install root and the file owner the asset owner, so the launcher, each release and the process running one merge into one asset whose id does not depend on whether it runs. The installed version wins over a process still running an older release. Layouts are validated at load (prefix, glob placement, version_from) and a layout claimed by two entries is ambiguous. The enumerator is unchanged: launcher links are already listed by the bin-root source.
…ence fires Socket tables carry peer addresses, but model providers were matched by hostname suffix only, so no live connection ever produced network evidence. Because the review queue threshold is reachable in practice only through that signal, the queue was always empty as well. On Linux, IPv6 peers were additionally left as raw /proc hex. When an established peer is a bare address, the provider hostnames are resolved once through the host's resolver (a fixed list, a 3 s deadline, through the gate) and peers are matched against the result. A peer matched this way is weighted lower than a peer the socket named, because a CDN address can be shared with other tenants. Failure to resolve is recorded as an unavailable resolver; --no-resolve turns lookups off. /proc/net/tcp6 addresses are decoded, IPv4-mapped peers fold to IPv4, and bracketed IPv6 peers from lsof are accepted. An unknown process talking to a provider now reaches the review queue, and a catalogued install carries the connection as network evidence.
The gate already stats every file it reads, but the mtime never left the pipeline. Assets now carry `modified` (ISO-8601 UTC): for an identified binary or process its executable, for a declaration the declaring file, and for a skill its SKILL.md rather than the directory, whose own mtime moves only when files are added or removed. A merged asset reports the latest time of its observations. It round-trips through from_dict and does not take part in delta change detection.
Eleven catalog entries declare risk_factors (executes_shell, local_inference, browses_autonomously), but nothing read them, so an identified agent that runs shell commands carried no such fact. The judge now appends an entry's factors to the risk of every asset identified as that entry, after the factors it derives itself and without duplicates. Uncatalogued assets are unchanged. A snapshot taken before this reports these factors as a one-time risk_delta on the next diff.
…process A socket listening on a local model port became a candidate named "tcp:<port>", which no rung can identify and whose score is below the review threshold, so the fact never reached the snapshot. The candidate now names the owning process's executable when the pid is known. It is then identified like the process (package ownership is consulted for socket candidates with an executable path, and installer layouts apply), merges into the runtime's asset, and adds "listening on local model port <n>" network evidence. A socket candidate with a live pid also marks the asset running. Connections keep their own provider evidence.
coverage.out_of_scope still listed instruction_files and agent_hooks, both of which the collector inventories, so a reader could not trust the field (the README said as much). It now names only scheduling_mechanisms, and the README caveat is removed.
… snapshot back from_dict rebuilt findings and review items without their evidence, so a snapshot read back for --diff had lost why each finding was raised. Both now decode evidence with the same helper assets use, and a test asserts that from_dict(to_json(snapshot)) equals the snapshot, findings and review queue included.
… nothing produces Channel.SIGNATURE, EXEC_JOURNAL and TELEMETRY, Rung.BEHAVIOUR and Observation.signature_id (with its "signature" merge key) were never produced by any stage, so they could only mislead a reader of the schema or of CONCLUSIVE_RUNGS. No snapshot can contain them. Kept: Channel.RUNTIME, the channel a running process is observed through (the resolver tests already model it that way); it is the independent second channel for an install that is also running. Kind.CLOUD_AGENT, because the README's asset schema lists cloud_agent as a reserved kind, in the same way location reserves values not yet produced.
- identifier/verify.py check_version: a stub refusing a removed API - ladder.binary_format and its magic table: only a test called it - Budget.max_strings_bytes, resolver independent_channels, merge group_only, contracts Boundary: never referenced - findings: the `"declared" in flags` clause (no asset carries that flag; declared MCP servers pass through their confidence band) - risk.unattended: `headless and bypass or bypass` is just `bypass` - openworld.signals_for: the credential_kinds parameter no caller passes Kept: Candidate.priority. It is part of the M2 candidate contract, every source sets it and config-surface dedupe keeps the strongest. Removing it would touch every source for no behaviour change.
…o fixed-address hosts A scan now sends nothing over the network unless --resolve is given (replaces --no-resolve). Without it, address-only peers stay unmatched and the resolver is recorded as unavailable in coverage. Only hosts served from a small fixed set of addresses are resolved: a published block or a CDN anycast pair, so the address a client used is the one a later lookup returns. Google APIs and CloudFront-served hosts rotate pools and share addresses with unrelated services (Google Fonts and Maps resolve into the Gemini/Vertex sets), and per-tenant Azure and regional Bedrock hosts have no single name to resolve.
An identified process carried only the identifier's filesystem proof, so an install that was also running was banded on one channel, the same as an idle one. The process table is an independent observation: it now adds a runtime evidence entry, and a running install is banded medium.
A launcher reached from two enumerator sources became two candidates with the same proof, and the merged asset listed that entry twice.
An install's version is the release it runs next. A process can still be executing an older or newer release, and that was visible only as an evidence path. Assets now carry running_versions, sorted numerically. The delta still compares version only, so processes starting and stopping do not show up as changes.
Common/adr_common/host_identity.py is now the single source for the account, host name and OS a package reports. The Sensor is published to PyPI and supports Python 3.9 while Discovery needs 3.11, so neither can depend on a shared distribution; each carries a checked-in copy generated by Common/sync.py, which also has a --check mode for CI. The Sensor's copy keeps its import path (adr_sensor.host_identity) and behaviour; Discovery's lives in world/, the only package allowed to touch the host.
getpass.getuser() reads LOGNAME, USER, LNAME and USERNAME first, so any caller could set the account a snapshot names. The snapshot now takes username and hostname from the shared host identity, which asks the OS (the effective uid on POSIX) and is the same value the Sensor reports. The host name is unchanged except that a failing lookup yields unknown_hostname instead of an exception.
Discovery's suite and lint never ran in CI. Add a Discovery job on its minimum Python (3.11) and a job that fails when a generated copy of Common/ source has drifted from its source.
…ink as evidence Reading each real file once (501e509) kept whichever path the walk met first, so a CLAUDE.md -> AGENTS.md link either named the asset or vanished, depending on order. The kept candidate now takes the real path, so the asset ID does not depend on walk order, and an instruction file records every link that reaches it: the link is what shows a second agent loads the same file.
barisozbas
approved these changes
Oct 12, 2026
atovkes
added a commit
that referenced
this pull request
Oct 15, 2026
…pture (#176) Brings main's 49 commits (Rohan's #170 and #176, plus #161, #162, #164) on top of stage A, so main is an ancestor of desktop-dev again. Conflict resolutions: - Discovery lives in Sensor/adr_discovery since split_stage 02. Main's new modules and tests go there; its policy example moves to Sensor/examples and its GenAI mapping test to Sensor/tests/sensor. - Walk budget: main's stage-ceiling reserve (reserved_walk_entries, 10% by default) replaces desktop-dev's walk-level reserve. Desktop-dev's coverage records stay in main's fair-share walk: scope_excluded, symlink_not_followed, the walk-stop boundary, and swept only when the root was listed. - Hooks: main's digest identity, with desktop-dev's shlex-aware scrubbing (applied before hashing) and readable names. - Schema 1.2 carries both sides' additions; identifier/verify.py stays deleted. - Common/sync.py writes the Discovery copy under Sensor/. Main's Discovery CI job becomes a lint step in Sensor tests, which already run Discovery's tests. - Desktop's import contracts list the new adr_discovery.archive module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Related issue: N/A
What changed?
This PR fixes the 36 bugs and gaps below.
"Before" is
mainat930e5cb; "after" is this branch.Traversal and budget
.cache,node_modules,.git/objects,.venv, …) were filtered after being listed and queued, so they still used up the budgetbudget_exhaustedin one content-addressed cache; 0 skills foundCACHEDIR.TAG(cache_dir) or more than--max-childrenentries (fan_out) are listed, not walked~/.claude,~/.codex, …) were listed one level and never walked/homeand/home/<user>)/homere-walked: 1,264,231 entrieslist_dirrefused after the sweepHost identity
usernamecame fromgetpass.getuser(), which readsUSER/LOGNAMEfirstUSER=john.doegavejohn.doeCommon/adr_common/host_identity.py;Common/sync.pywrites generated copies into each package (Sensor keeps its import path), checked in CIMCP configuration and surfaces
~/.claude/settings*.json, managed settings)commandis an argv list, env isenvironment,typeislocal/remotestdio/http,disabledhonoured.mcpbbundle manifest produced nothingmcp_bundleassetCLAUDE.md -> AGENTS.md) took whichever path the walk met firstskills/orplugins/without its manifest (cache/,data/,marketplaces/, shared reference dirs) was reported as a failed readFrontmatter
frontmatternote, not a failureBinary identification
~/.local/share/claude/versions/<v>behind~/.local/bin/claude, with no package recordcli_agent, although installed and runninginstall_layouts. The launcher is followed through the gate and matched at the provenance rung. Version from the path; running processes merge into the installproofs.provenanceentries (e.g.brew:claude) were never indexedrunning_versions, e.g.version2.1.287 withrunning_versions[2.1.287, 2.1.296]; the delta still comparesversiononlyAsset identity
"<kind>:<name>")CLAUDE.mdasset for 431 files; 1 asset for an MCP server declared in 210 files; skills = distinct names (2,360); 18 of 41 plugin-cache skills own an assetSCHEMA_VERSION1.1"PreToolUse hook 11")hook:<path>:<digest>; unchanged when siblings changeRisk, findings and policy
CLAUDE.mdcarriedtransport=stdio,credential_kinds=["inherited"]mcp_server,mcp_bundle,hook,ci_agentcommandsanction: unknownapproved_indicators/forbidden_indicatorsmatchskill_sha256(rawSKILL.mdbytes),file_sha256,mcp_endpoint,package,name. Forbidden beats approved; a forbidden asset raises a highforbidden_assetfindingcatalog_idwas not copied onto the observationunattended_executionapplyNetwork and review queue
--resolve, peers are matched by address against provider hosts served from fixed addresses--resolve/proc/net/tcp6andlsofwere left unparsed--resolveis passed; otherwise the resolver is recorded as unavailableData collected but never emitted
hook_event,hook_matcher; the command is never serializedcontent_hash,modified,skill_sha256,package_idrisk_factorshad no consumerrisk.factors(e.g. Claude Code:executes_shell)out_of_scopelisted categories that are collectedinstruction_files,agent_hooks,scheduling_mechanismsscheduling_mechanismsfrom_dictdropped finding and review-queue evidenceDead code
Channel.SIGNATURE,EXEC_JOURNAL,TELEMETRY;Rung.BEHAVIOUR;Observation.signature_idand its merge keyChannel.RUNTIME(now produced);Kind.CLOUD_AGENT(reserved in the README)check_versionstub,Budget.max_strings_bytes,binary_format,independent_channels,group_only,Boundary, unreachable branches, an unused parameterCandidate.priority: set by every source and read by config dedupeCI
Why?
See "bug" column of each of the tables above.
How did you test it?
.mcpbmanifests, CI agents and the review queue. Every commit passespytestandruffon its own; the Sensor suite andCommon/sync.py --checkpass.USER/LOGNAMEspoofed;skill_sha256, and approving another;Potential risks