From 9d392737a0ad9a0ac1dbbbea68b87c69950c0c86 Mon Sep 17 00:00:00 2001 From: Lihua <1017343802@qq.com> Date: Sun, 1 Nov 2026 23:09:51 -0800 Subject: [PATCH] feat(examples): add a source-verified repository-context pilot Signed-off-by: Lihua <1017343802@qq.com> --- examples/repository-context-pilot/README.md | 141 ++++++++++++++++++ examples/repository-context-pilot/evidence.ts | 128 ++++++++++++++++ .../repository-context-pilot/pilot.test.ts | 125 ++++++++++++++++ .../repository-context-pilot/providers.ts | 114 ++++++++++++++ .../repository-context-pilot/queries.json | 23 +++ examples/repository-context-pilot/run.ts | 122 +++++++++++++++ loopx/capabilities/issue_fix/README.md | 6 + loopx/capabilities/issue_fix/README.zh-CN.md | 5 + 8 files changed, 664 insertions(+) create mode 100644 examples/repository-context-pilot/README.md create mode 100644 examples/repository-context-pilot/evidence.ts create mode 100644 examples/repository-context-pilot/pilot.test.ts create mode 100644 examples/repository-context-pilot/providers.ts create mode 100644 examples/repository-context-pilot/queries.json create mode 100644 examples/repository-context-pilot/run.ts diff --git a/examples/repository-context-pilot/README.md b/examples/repository-context-pilot/README.md new file mode 100644 index 0000000000..f4543a03f5 --- /dev/null +++ b/examples/repository-context-pilot/README.md @@ -0,0 +1,141 @@ +# Repository-context pilot + +First bounded experiment for [#4926](https://github.com/loopx-project/loopx/issues/4926). +It asks whether an optional repository-intelligence provider can supply useful, +checkable context to the existing issue-fix workflow. It does not register a new +capability, change default context or claim improved repair outcomes. + +## Run + +Use a source checkout prepared with `uv sync --extra test`. Node must support +TypeScript stripping; use the repository's supported Node version. Git, tar and +ripgrep must be on PATH. + +```sh +mkdir -p .local +node --experimental-strip-types examples/repository-context-pilot/run.ts \ + --repository . \ + --revision 789694fa42972db17acd47aa2604b1907060028d \ + --out .local/repository-context-first-run \ + --prepare-rune \ + --loopx-python "$PWD/.venv/bin/python" +``` + +`--prepare-rune` explicitly clones the MIT-licensed Rune upstream at +[`4042a088`](https://github.com/thecolourfoundation/rune/tree/4042a088f1672d32110f38a5ec10c59717ea5412) +and installs its locked dependencies with scripts disabled in disposable +storage. Preparation time is included. Shared package caches are not cleared. +Alternatively, `--rune-source` uses an already prepared checkout of that exact +revision with no tracked edits; its acquisition and installation cost is then **unknown/excluded**. +Omitting both options runs only direct search and records Rune as disabled. + +The source is exported from the exact commit. Dirty edits, untracked files, +existing provider indexes and symlinks are excluded; the live repository and +LoopX runtime are never given to Rune. Only the exported copy receives an index. +Source hashes are captured before provider startup; a provider cannot validate +its own replacement text as evidence for the pinned revision. +The MCP client calls `rune_agent` with `deep:false`; no model is configured or +called by this runner. The optional provider's dependencies remain outside +LoopX's installation. Temporary copies and the provider process are cleaned up. + +Each run requires a new output directory and retains failed rows. Keep generated +reports owner-local; review them before sharing. Raw MCP responses, source +snippets, absolute source locations and server logs are not retained. The report +contains source-relative citations, file hashes, coverage, elapsed times and +provider/native-readback dispositions. `*.context.json` files use the existing +`issue_fix_repository_context_input_v0` contract; a normal read-only +`issue-fix workflow-plan` consumes them when `--loopx-python` is supplied. + +Verified citation currency does not establish relevance or authority. Sources +stay advisory and cover only change-scope discovery; reproduction, validation +and any protected operation still require their own evidence and authorization. +The prototype is a CLI/artifact experiment, not an installed frontend provider. + +## Frozen comparison + +`queries.json` is declared before running either provider. Two historical public +issues have reference files derived from their independently reviewed fixes: +[#7817](https://github.com/loopx-project/loopx/pull/7817) and +[#7814](https://github.com/loopx-project/loopx/pull/7814). A third query names an +absent symbol. The pinned source predates those repairs. + +Rune receives the issue question; direct search receives the declared literal +terms from the same report. These are different retrieval interfaces, not +matched model treatments. Direct results are file-sorted; Rune keeps its own +ranking. Both retain at most eight distinct files after source verification. +Reference-file precision/recall measures this declared set, not every potentially +useful file. Extra files can be relevant; they are reported as unexpected rather +than certified false positives. Unsupported-language misses and wrong answers +on the absent-symbol query remain in the report. + +Costs include source export, optional provider preparation, MCP startup/indexing, +all query/verification attempts, native CLI readback and cleanup. Native-plan +time is separate from retrieval time. One process search and one MCP tool call +are not equivalent units of work. No model, repair, handoff, held-out trial or +long-horizon benefit is qualified by this small comparison. + +## Design research and placement + +| Source | Design considered | Decision for this pilot | +| --- | --- | --- | +| [Rune](https://github.com/thecolourfoundation/rune), MIT | Source-cited graph facts through MCP; JavaScript/TypeScript strongest, thinner Python coverage; optional deep calls use a model | Use only the explicit no-model MCP path, pin the source and independently reread citations | +| [Repomix](https://github.com/yamadashy/repomix), MIT | Repository packing, bounded output and security checks through CLI/MCP | A packed repository is a useful alternative; this pilot tests retrieval rather than adding a second repository packer | +| [Existing LoopX provider comparison](../context-provider-comparison.py) | Content-based relevance and explicit unavailable rows through Decision Context | Preserve that distinction; issue-fix already owns repository-context input, so no new Kernel or provider-selection owner | + +No upstream implementation was copied. Rune and Repomix are independently +maintained optional tools, not new mandatory dependencies. Production adoption +would need provider lifecycle, applicable dependency/security qualification, +the existing frontend journey and real task-outcome evidence. Reuse #4926 for +that decision after this experiment; do not treat a successful MCP read as +acceptance of a full integration. + +## Validation + +### Observed source qualification + +On macOS with Node 26.9.0, the declared commit exported 7,174 files +(182,435,840 archive bytes); Rune indexed 6,240 files. The instrumented run kept +the declared queries, eight-file limit and 4,096 estimated-token Rune budget. + +| Query | Direct search | Rune | Retrieval and verification time, direct / Rune | +| --- | --- | --- | --- | +| Windows worktree | 0/1 reference files; 46 additional matches omitted by the declared cap | 0/1; eight insights dropped by Rune's budget before citations reached the client | 0.100 s / 21.201 s | +| Provider key shapes | 2/2 reference files | 0/2; eight insights dropped by Rune's budget | 0.220 s / 7.496 s | +| Absent symbol | Empty result | Five source-current files unrelated to the requested symbol | 0.081 s / 2.266 s | + +Provider preparation took 23.740 s, startup/indexing 11.302 s, and shared source +export/hash capture 2.358 s. The complete run took 73.151 s, including all six +read-only native CLI observations and cleanup. Native context remained +`ungrounded` in every case; no reproduction or validation evidence was invented. +`failed:false` in this report means the observation pipeline completed, not that +retrieval quality passed. The Windows issue is a historical source query, not +native Windows execution qualification. + +Two earlier attempts are retained separately: the first failed before retrieval +because the archive exceeded a 128 MiB in-memory subprocess buffer. Export now +uses a temporary archive file without raising that buffer limit. The second +completed the same queries but lacked provider-budget telemetry; the third added +that observation without changing retrieval settings or reference files. Results +are not pooled, and no failed attempt was replaced or omitted. Preparation was +measured again; the differing setup times are not a stable latency estimate. + +This does not support default integration of Rune on this workload. The direct +control is also incomplete: file sorting plus the declared cap misses the +Windows owner, and this one-shot search is not a full agent investigation. +Budget-aware citation delivery, unknown-symbol abstention and task-level utility +need evidence before adoption. Do not increase the budget after the fact and +relabel this observation as a pass. + +A separate `npm audit` of the same pinned Rune dependency tree reported nine +findings (one critical, three high, five moderate). Dependencies are disposable +and this runner uses local stdio; production applicability and remediation have +not been qualified. This is another limit on adoption, not a clean security bill. + +```sh +node --experimental-strip-types --test examples/repository-context-pilot/pilot.test.ts +``` + +These tests cover source isolation, citation tampering/escapes, default-off and +unavailable behavior, independent coverage accounting and the real LoopX CLI. +They do not mock a successful Rune run or enter the active benchmark fleet. +Run the explicit experiment above to qualify the real provider separately. diff --git a/examples/repository-context-pilot/evidence.ts b/examples/repository-context-pilot/evidence.ts new file mode 100644 index 0000000000..f7fa4309c8 --- /dev/null +++ b/examples/repository-context-pilot/evidence.ts @@ -0,0 +1,128 @@ +/** Local experiment formats; LoopX's existing issue-fix contract is unchanged. */ +import {createHash} from "node:crypto"; +import {execFileSync} from "node:child_process"; +import {lstatSync, readFileSync, readdirSync, realpathSync, rmSync} from "node:fs"; +import {dirname, isAbsolute, join, relative, sep} from "node:path"; + +export interface Query { + id: string; + issue: string; + objective: string; + terms: string[]; + relevant_files: string[]; +} + +export interface Citation {file: string; line: number; snippet: string} +export interface Evidence {file: string; line: number; sha256: string} + +export const digest = (text: string | Buffer): string => + createHash("sha256").update(text).digest("hex"); + +export function object(value: unknown): Record { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("invalid_object"); + return value as Record; +} + +export function loadQueries(value: unknown): Query[] { + if (!Array.isArray(value) || value.length < 1 || value.length > 12) throw new Error("invalid_queries"); + const ids = new Set(); + return value.map(raw => { + const row = object(raw); + if (typeof row.id !== "string" || !/^[a-z0-9-]{1,64}$/u.test(row.id) || ids.has(row.id)) + throw new Error("invalid_query_id"); + ids.add(row.id); + if (typeof row.issue !== "string" || !/^https:\/\/github\.com\/[^/]+\/[^/]+\/issues\/\d+$/u.test(row.issue) || + typeof row.objective !== "string" || !row.objective.trim() || row.objective.length > 500) + throw new Error("invalid_query"); + for (const key of ["terms", "relevant_files"]) { + if (!Array.isArray(row[key]) || row[key].length > 8 || row[key].some(x => typeof x !== "string" || !x || x.length > 260)) + throw new Error("invalid_query_list"); + } + if (!(row.terms as string[]).length) throw new Error("missing_search_terms"); + return row as unknown as Query; + }); +} + +export function exportSnapshot(repository: string, revision: string, destination: string): { + archive_bytes: number; source_hashes: Map; +} { + if (!/^[a-f0-9]{40}$/u.test(revision)) throw new Error("immutable_revision_required"); + const resolved = execFileSync("git", ["-C", repository, "rev-parse", "--verify", `${revision}^{commit}`], + {encoding: "utf8", timeout: 10_000}).trim(); + if (resolved !== revision) throw new Error("revision_mismatch"); + const archive = join(dirname(destination), "source.tar"); + let archiveBytes: number; + try { + execFileSync("git", ["-C", repository, "archive", "--format=tar", `--output=${archive}`, revision], + {timeout: 30_000, stdio: "pipe"}); + archiveBytes = lstatSync(archive).size; + execFileSync("tar", ["-xf", archive, "-C", destination], {timeout: 30_000, stdio: "pipe"}); + } finally { rmSync(archive, {force: true}); } + // Never give the provider an existing index, a live worktree or an escape link. + const sourceHashes = new Map(); + function prune(directory: string): void { + for (const entry of readdirSync(directory, {withFileTypes: true})) { + const path = join(directory, entry.name); + if (entry.isSymbolicLink() || entry.name === ".rune") rmSync(path, {recursive: true, force: true}); + else if (entry.isDirectory()) prune(path); + else if (entry.isFile()) sourceHashes.set(relative(destination, path).split(sep).join("/"), digest(readFileSync(path))); + } + } + prune(destination); + return {archive_bytes: archiveBytes, source_hashes: sourceHashes}; +} + +export function verifyCitations(root: string, raw: unknown[], limit: number, sourceHashes?: Map): { + evidence: Evidence[]; rejected: number; omitted: number; +} { + const evidence: Evidence[] = [], seen = new Set(); + let rejected = 0, omitted = 0; + for (const candidate of raw) { + try { + const row = object(candidate); + if (typeof row.file !== "string" || !row.file || row.file.includes("\\") || + row.file.split("/").some(part => !part || part === "." || part === "..") || isAbsolute(row.file) || + !Number.isSafeInteger(row.line) || Number(row.line) < 1 || typeof row.snippet !== "string" || + !row.snippet.trim() || row.snippet.length > 2048) throw new Error("invalid_citation"); + const path = realpathSync(join(root, row.file)); + const contained = relative(realpathSync(root), path); + if (!contained || contained.startsWith(`..${sep}`) || contained === ".." || isAbsolute(contained) || + !lstatSync(path).isFile() || lstatSync(path).size > 2 * 1024 * 1024) throw new Error("outside_source"); + const bytes = readFileSync(path); + if (sourceHashes && sourceHashes.get(row.file) !== digest(bytes)) throw new Error("source_changed"); + const text = bytes.toString("utf8").replace(/\r\n/gu, "\n"); + const line = Number(row.line), snippet = row.snippet.trim().replace(/\r\n/gu, "\n"); + const window = text.split("\n").slice(line - 1, line - 1 + snippet.split("\n").length).join("\n"); + if (!window.includes(snippet)) throw new Error("stale_citation"); + if (seen.has(row.file)) continue; + seen.add(row.file); + if (evidence.length === limit) { omitted += 1; continue; } + evidence.push({file: row.file, line, sha256: digest(bytes)}); + } catch { rejected += 1; } + } + return {evidence, rejected, omitted}; +} + +export function assess(evidence: Evidence[], expected: string[]) { + const relevant = new Set(expected), returned = new Set(evidence.map(x => x.file)); + const found = [...returned].filter(file => relevant.has(file)); + return { + relevant_file_count: relevant.size, relevant_files_found: found, + missing_relevant_files: [...relevant].filter(file => !returned.has(file)), + unexpected_files: [...returned].filter(file => !relevant.has(file)), + recall: relevant.size ? found.length / relevant.size : null, + precision: returned.size ? found.length / returned.size : null, + unanswerable_abstained: relevant.size ? null : returned.size === 0, + }; +} + +export function contextInput(revision: string, evidence: Evidence[]) { + return { + schema_version: "issue_fix_repository_context_input_v0", repository_revision: revision, + sources: evidence.map(row => ({ + source_id: `context-${digest(row.file).slice(0, 16)}`, source_kind: "knowledge_bundle", + reference: row.file, trust: "advisory", freshness: "current", supports: ["change_scope"], + summary: `Source-checked citation at line ${row.line}; file SHA-256 ${row.sha256}. Verify relevance before acting.`, + })), + }; +} diff --git a/examples/repository-context-pilot/pilot.test.ts b/examples/repository-context-pilot/pilot.test.ts new file mode 100644 index 0000000000..b90e96cbc3 --- /dev/null +++ b/examples/repository-context-pilot/pilot.test.ts @@ -0,0 +1,125 @@ +import assert from "node:assert/strict"; +import {execFileSync, spawnSync} from "node:child_process"; +import {existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync, writeFileSync} from "node:fs"; +import {tmpdir} from "node:os"; +import {dirname, join} from "node:path"; +import {fileURLToPath} from "node:url"; +import {test} from "node:test"; +import {assess, contextInput, exportSnapshot, loadQueries, verifyCitations} from "./evidence.ts"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ROOT = join(HERE, "../.."); +const PYTHON = join(ROOT, process.platform === "win32" ? ".venv/Scripts/python.exe" : ".venv/bin/python"); + +function fixture(t: {after: (cleanup: () => void) => void}) { + const temporary = mkdtempSync(join(tmpdir(), "context-pilot-test-")); + t.after(() => rmSync(temporary, {recursive: true, force: true})); + const repository = join(temporary, "repo"); mkdirSync(repository); + const git = (...args: string[]) => execFileSync("git", ["-C", repository, ...args], {encoding: "utf8", stdio: "pipe"}).trim(); + git("init", "--quiet"); git("config", "user.name", "Synthetic Fixture"); git("config", "user.email", "fixture@example.invalid"); + writeFileSync(join(repository, "source.ts"), "export const visible = 1;\n"); + git("add", "source.ts"); git("commit", "--quiet", "-m", "synthetic source"); + return {temporary, repository, revision: git("rev-parse", "HEAD"), git}; +} + +test("an immutable export excludes dirty/untracked work and provider-owned indexes", t => { + const f = fixture(t), snapshot = join(f.temporary, "snapshot"); mkdirSync(snapshot); + mkdirSync(join(f.repository, ".rune")); writeFileSync(join(f.repository, ".rune", "graph.json"), "{}"); + f.git("add", ".rune/graph.json"); f.git("commit", "--quiet", "-m", "synthetic old index"); + const revision = f.git("rev-parse", "HEAD"); + writeFileSync(join(f.repository, "source.ts"), "changed working copy\n"); + writeFileSync(join(f.repository, "private-untracked.txt"), "never export this\n"); + exportSnapshot(f.repository, revision, snapshot); + assert.equal(readFileSync(join(snapshot, "source.ts"), "utf8"), "export const visible = 1;\n"); + assert.equal(existsSync(join(snapshot, "private-untracked.txt")), false); + assert.equal(existsSync(join(snapshot, ".rune")), false); + assert.equal(readFileSync(join(f.repository, "source.ts"), "utf8"), "changed working copy\n"); +}); + +test("citations require exact current source at the claimed line and cannot escape", t => { + const f = fixture(t); + writeFileSync(join(f.temporary, "outside.ts"), "external data\n"); + const good = {file: "source.ts", line: 1, snippet: "export const visible = 1;"}; + const checked = verifyCitations(f.repository, [good, + {...good, line: 2}, {...good, snippet: "export const visible = 2;"}, + {file: "../outside.ts", line: 1, snippet: "external data"}, {...good, line: 1.5}], 8); + assert.deepEqual(checked.evidence.map(x => [x.file, x.line]), [["source.ts", 1]]); + assert.equal(checked.rejected, 4); + const context = contextInput(f.revision, checked.evidence); + assert.equal(context.sources[0].trust, "advisory"); + assert.deepEqual(context.sources[0].supports, ["change_scope"]); + assert.equal(JSON.stringify(context).includes("export const"), false); +}); + +test("a symlink cannot turn an outside file into repository evidence", {skip: process.platform === "win32"}, t => { + const f = fixture(t); + writeFileSync(join(f.temporary, "outside.ts"), "outside\n"); + symlinkSync(join(f.temporary, "outside.ts"), join(f.repository, "alias.ts")); + assert.equal(verifyCitations(f.repository, [{file: "alias.ts", line: 1, snippet: "outside"}], 8).rejected, 1); +}); + +test("coverage counts unique reference files and preserves an incorrect non-answer", () => { + const evidence = [{file: "other.ts", line: 1, sha256: "a"}]; + assert.deepEqual(assess(evidence, ["owner.ts"]).missing_relevant_files, ["owner.ts"]); + assert.equal(assess(evidence, []).unanswerable_abstained, false); + assert.equal(assess([], []).unanswerable_abstained, true); +}); + +test("a provider cannot make changed source current by citing its own replacement text", t => { + const f = fixture(t), snapshot = join(f.temporary, "snapshot"); mkdirSync(snapshot); + const source = exportSnapshot(f.repository, f.revision, snapshot); + writeFileSync(join(snapshot, "source.ts"), "export const visible = 999;\n"); + const checked = verifyCitations(snapshot, [{file: "source.ts", line: 1, snippet: "export const visible = 999;"}], 8, source.source_hashes); + assert.equal(checked.evidence.length, 0); + assert.equal(checked.rejected, 1); +}); + +test("duplicate or empty query identities cannot silently overwrite experiment rows", () => { + const query = {id: "one", issue: "https://github.com/example/repo/issues/1", objective: "visible", terms: ["visible"], relevant_files: []}; + assert.throws(() => loadQueries([query, query]), /invalid_query_id/u); + assert.throws(() => loadQueries([{...query, terms: []}]), /missing_search_terms/u); +}); + +function runPilot(f: ReturnType, name: string, extra: string[]) { + const queryFile = join(f.temporary, "queries.json"), output = join(f.temporary, name); + writeFileSync(queryFile, JSON.stringify([{id: "visible", issue: "https://github.com/loopx-project/loopx/issues/4926", + objective: "find visible", terms: ["visible"], relevant_files: ["source.ts"]}])); + const result = spawnSync(process.execPath, ["--experimental-strip-types", join(HERE, "run.ts"), + "--repository", f.repository, "--revision", f.revision, "--queries", queryFile, "--out", output, ...extra], + {cwd: ROOT, encoding: "utf8", timeout: 40_000}); + return {result, report: JSON.parse(readFileSync(join(output, "report.json"), "utf8")), output}; +} + +test("the default direct-search path consumes context through the real CLI without Rune", {skip: !existsSync(PYTHON)}, t => { + const f = fixture(t), {result, report} = runPilot(f, "direct", ["--loopx-python", PYTHON]); + assert.equal(result.status, 0, result.stderr); + const direct = report.rows.find((row: {provider: string}) => row.provider === "direct-search"); + assert.deepEqual(direct.relevant_files_found, ["source.ts"]); + assert.equal(direct.native_plan, "accepted_read_only"); + assert.equal(direct.native_context_revision, f.revision); + assert.equal(report.rows.find((row: {provider: string}) => row.provider === "rune").status, "disabled"); +}); + +test("unavailable optional providers retain the successful direct-search observation", t => { + const f = fixture(t), {result, report} = runPilot(f, "missing-provider", ["--rune-source", join(f.temporary, "missing")]); + assert.equal(result.status, 1); + assert.equal(report.failed, true); + assert.deepEqual(report.rows[0].relevant_files_found, ["source.ts"]); + assert.equal(report.rows[1].status, "unavailable"); +}); + +test("failed native readback remains distinct from successful retrieval", t => { + const f = fixture(t), {result, report} = runPilot(f, "missing-native", ["--loopx-python", join(f.temporary, "missing-python")]); + assert.equal(result.status, 1); + assert.equal(report.rows[0].status, "completed"); + assert.deepEqual(report.rows[0].relevant_files_found, ["source.ts"]); + assert.equal(report.rows[0].native_plan, "unavailable"); +}); + +test("a later attempt cannot overwrite the first observation", t => { + const f = fixture(t), first = runPilot(f, "retained", []); + const before = readFileSync(join(first.output, "report.json"), "utf8"); + const second = runPilot(f, "retained", []); + assert.equal(second.result.status, 1); + assert.equal(readFileSync(join(first.output, "report.json"), "utf8"), before); +}); diff --git a/examples/repository-context-pilot/providers.ts b/examples/repository-context-pilot/providers.ts new file mode 100644 index 0000000000..fd4506f6a5 --- /dev/null +++ b/examples/repository-context-pilot/providers.ts @@ -0,0 +1,114 @@ +/** Explicit optional Rune MCP provider and the direct-search control. No model calls. */ +import {execFileSync, spawnSync} from "node:child_process"; +import {createRequire} from "node:module"; +import {join} from "node:path"; +import {pathToFileURL} from "node:url"; +import {object, type Citation, type Query} from "./evidence.ts"; + +export const RUNE_REVISION = "4042a088f1672d32110f38a5ec10c59717ea5412"; +const TIMEOUT = 60_000; + +export function directSearch(root: string, query: Query): Citation[] { + const args = ["--no-config", "--json", "--line-number", "--fixed-strings", "--max-count", "1", + "--hidden", "--no-ignore", "--glob", "!.rune/**"]; + for (const term of query.terms) args.push("-e", term); + args.push("--", "."); + const result = spawnSync("rg", args, {cwd: root, encoding: "utf8", timeout: TIMEOUT, maxBuffer: 8 * 1024 * 1024}); + if (result.error || result.status === null || result.status > 1) throw new Error("direct_search_unavailable"); + return result.stdout.split("\n").filter(Boolean).flatMap(line => { + const event = object(JSON.parse(line)); + if (event.type !== "match") return []; + const data = object(event.data), file = object(data.path).text, snippet = object(data.lines).text; + return typeof file === "string" && typeof snippet === "string" + ? [{file: file.replace(/^\.\//u, ""), line: Number(data.line_number), snippet: snippet.trim()}] : []; + }).sort((left, right) => left.file.localeCompare(right.file)); +} + +export function prepareRune(destination: string): void { + // Trusted fixed upstream and immutable revision, never a suite-supplied command. + execFileSync("git", ["clone", "--quiet", "https://github.com/thecolourfoundation/rune.git", destination], + {timeout: TIMEOUT, stdio: "pipe"}); + execFileSync("git", ["-C", destination, "checkout", "--quiet", "--detach", RUNE_REVISION], + {timeout: TIMEOUT, stdio: "pipe"}); + execFileSync(process.platform === "win32" ? "npm.cmd" : "npm", ["ci", "--ignore-scripts"], + {cwd: destination, timeout: TIMEOUT, stdio: "pipe"}); +} + +interface Client { + connect(transport: unknown): Promise; + listTools(): Promise; + callTool(request: unknown, schema?: unknown, options?: {timeout: number}): Promise; + close(): Promise; +} + +function toolData(value: unknown): Record { + const result = object(value); + if (result.isError || !Array.isArray(result.content)) throw new Error("provider_tool_failed"); + const text = result.content.find(item => object(item).type === "text"); + if (!text || typeof object(text).text !== "string") throw new Error("provider_payload_invalid"); + return object(JSON.parse(object(text).text as string)); +} + +export class RuneSession { + client: Client; + startup_ms = 0; + file_count: number | null = null; + constructor(client: Client) { this.client = client; } + + static async start(source: string, root: string): Promise { + const revision = execFileSync("git", ["-C", source, "rev-parse", "HEAD"], {encoding: "utf8", timeout: TIMEOUT}).trim(); + if (revision !== RUNE_REVISION) throw new Error("provider_revision_mismatch"); + if (execFileSync("git", ["-C", source, "status", "--porcelain", "--untracked-files=no"], + {encoding: "utf8", timeout: TIMEOUT}).trim()) throw new Error("provider_source_modified"); + const require = createRequire(join(source, "package.json")); + const {Client} = await import(pathToFileURL(require.resolve("@modelcontextprotocol/sdk/client/index.js")).href); + const {StdioClientTransport} = await import(pathToFileURL(require.resolve("@modelcontextprotocol/sdk/client/stdio.js")).href); + const transport = new StdioClientTransport({ + command: process.execPath, args: [join(source, "bin/rune.js"), "serve", root], + cwd: root, stderr: "pipe", maxBufferSize: 2 * 1024 * 1024, + }); + transport.stderr?.resume(); // Do not retain provider logs or absolute paths. + const client = new Client({name: "loopx-repository-context-pilot", version: "0.1.0"}, {capabilities: {}}) as Client; + const session = new RuneSession(client), start = performance.now(); + let timer: ReturnType | undefined; + try { + await Promise.race([ + client.connect(transport), + new Promise((_, reject) => { timer = setTimeout(() => reject(new Error("provider_start_timeout")), TIMEOUT); }), + ]); + const listed = object(await client.listTools()); + if (!Array.isArray(listed.tools) || !listed.tools.some(tool => object(tool).name === "rune_agent")) + throw new Error("provider_tool_missing"); + const overview = toolData(await client.callTool({name: "rune_get_overview", arguments: {}}, undefined, {timeout: TIMEOUT})); + const meta = object(overview.meta); + session.file_count = typeof meta.fileCount === "number" ? meta.fileCount : null; + session.startup_ms = performance.now() - start; + return session; + } catch (error) { await client.close(); throw error; } + finally { if (timer) clearTimeout(timer); } + } + + async search(query: Query): Promise<{candidates: unknown[]; observation: Record}> { + const report = toolData(await this.client.callTool({ + name: "rune_agent", arguments: {objective: query.objective, maxTokens: 4096, deep: false, force: false}, + }, undefined, {timeout: TIMEOUT})); + const synthesis = object(report.synthesis); + if (!Array.isArray(synthesis.insights)) throw new Error("provider_payload_invalid"); + const candidates = synthesis.insights.flatMap(raw => { + const insight = object(raw); + return Array.isArray(insight.evidence) ? insight.evidence : []; + }); + const budget = object(report.budget), sections = object(budget.sections); + const kept = object(sections.synthesisInsights), triage = object(report.triage); + return {candidates, observation: { + relevant_fact_count: report.relevantFactCount, + returned_insight_count: synthesis.insights.length, + dropped_insight_count: kept.dropped, + reported_token_estimate: budget.usedTokens, + max_token_estimate: budget.maxTokens, + triage_mode: triage.mode, + }}; + } + + close(): Promise { return this.client.close(); } +} diff --git a/examples/repository-context-pilot/queries.json b/examples/repository-context-pilot/queries.json new file mode 100644 index 0000000000..5e69e1df2f --- /dev/null +++ b/examples/repository-context-pilot/queries.json @@ -0,0 +1,23 @@ +[ + { + "id": "windows-worktree", + "issue": "https://github.com/loopx-project/loopx/issues/7805", + "objective": "Why does --write-worktree reject existing in-scope paths on native Windows?", + "terms": ["write-worktree", "realpath"], + "relevant_files": ["loopx/control_plane/work_items/task_lease_workspace.ts"] + }, + { + "id": "provider-key-shapes", + "issue": "https://github.com/loopx-project/loopx/issues/7799", + "objective": "Why are unlabeled sk-proj and sk-ant keys accepted by the public boundary scan?", + "terms": ["sk-proj", "sk-ant", "PROVIDER_TOKEN_SHAPE_PATTERN_SOURCE"], + "relevant_files": ["loopx/public_safe_text.py", "loopx/contract.py"] + }, + { + "id": "absent-symbol", + "issue": "https://github.com/loopx-project/loopx/issues/4926", + "objective": "Locate loopx_quantum_shuffle_v999 for this issue", + "terms": ["loopx_quantum_shuffle_v999"], + "relevant_files": [] + } +] diff --git a/examples/repository-context-pilot/run.ts b/examples/repository-context-pilot/run.ts new file mode 100644 index 0000000000..15029a65c4 --- /dev/null +++ b/examples/repository-context-pilot/run.ts @@ -0,0 +1,122 @@ +/** Reproducible, explicit public-source experiment for issue #4926. */ +import {execFileSync} from "node:child_process"; +import {mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync} from "node:fs"; +import {tmpdir} from "node:os"; +import {dirname, join, resolve} from "node:path"; +import {fileURLToPath, pathToFileURL} from "node:url"; +import {parseArgs} from "node:util"; +import {assess, contextInput, digest, exportSnapshot, loadQueries, object, verifyCitations} from "./evidence.ts"; +import {directSearch, prepareRune, RUNE_REVISION, RuneSession} from "./providers.ts"; + +const HERE = dirname(fileURLToPath(import.meta.url)); + +export async function main(argv: string[]): Promise { + const {values} = parseArgs({args: argv, options: { + repository: {type: "string"}, revision: {type: "string"}, out: {type: "string"}, + queries: {type: "string", default: join(HERE, "queries.json")}, + "rune-source": {type: "string"}, "prepare-rune": {type: "boolean", default: false}, + "loopx-python": {type: "string"}, + }}); + if (!values.repository || !values.revision || !values.out || (values["prepare-rune"] && values["rune-source"])) + throw new Error("usage: --repository REPO --revision FULL_SHA --out NEW_DIRECTORY [--prepare-rune | --rune-source SOURCE] [--loopx-python PYTHON]"); + const queryBytes = readFileSync(values.queries!), queries = loadQueries(JSON.parse(queryBytes.toString("utf8"))); + const output = resolve(values.out); + mkdirSync(output); // Never overwrite a previous observation or selectively rerun failed rows. + const temporary = mkdtempSync(join(tmpdir(), "loopx-repository-context-")); + const snapshot = join(temporary, "source"); mkdirSync(snapshot); + const report: Record = { + source_revision: values.revision, query_sha256: digest(queryBytes), node_version: process.version, + platform: process.platform, rune_revision: RUNE_REVISION, + scope: "file discovery and citation currency; no repair-outcome or model-benefit qualification", + order: "direct search then Rune per query; one shared immutable export; no cache flushing", + result_file_limit: 8, model_calls: 0, live_goal_state_read: false, + provider_preparation_included: values["prepare-rune"], provider_preparation_ms: null, + provider_preparation_commands_planned: values["prepare-rune"] ? 3 : 0, + snapshot_commands_planned: 3, snapshot_ms: null, rune_startup_ms: null, rune_file_count: null, rows: [], + }; + const rows: Record[] = [], started = performance.now(); + let rune: RuneSession | undefined, providerUnavailable = !values["prepare-rune"] && !values["rune-source"]; + let failed = false; + try { + const snapshotStart = performance.now(); + const source = exportSnapshot(resolve(values.repository), values.revision, snapshot); + report.snapshot_ms = performance.now() - snapshotStart; + report.snapshot_archive_bytes = source.archive_bytes; + report.snapshot_source_file_count = source.source_hashes.size; + let runeSource = values["rune-source"]; + if (values["prepare-rune"]) { + runeSource = join(temporary, "rune"); const start = performance.now(); + try { prepareRune(runeSource); } + catch { providerUnavailable = true; failed = true; report.provider_preparation_error = "provider_setup_unavailable"; } + report.provider_preparation_ms = performance.now() - start; + } + if (runeSource && !providerUnavailable) { + const start = performance.now(); + try { + rune = await RuneSession.start(resolve(runeSource), snapshot); + report.rune_startup_ms = rune.startup_ms; report.rune_file_count = rune.file_count; + } catch { providerUnavailable = true; failed = true; report.rune_startup_error = "provider_unavailable"; } + report.rune_startup_attempt_ms = performance.now() - start; + } + for (const query of queries) { + for (const provider of ["direct-search", "rune"]) { + const start = performance.now(); + let row: Record; + try { + if (provider === "rune" && !rune) { + row = {query_id: query.id, provider, status: providerUnavailable && (values["prepare-rune"] || values["rune-source"]) ? "unavailable" : "disabled"}; + } else { + const result = provider === "rune" ? await rune!.search(query) : {candidates: directSearch(snapshot, query), observation: {}}; + const raw = result.candidates; + const checked = verifyCitations(snapshot, raw, 8, source.source_hashes); + const context = contextInput(values.revision, checked.evidence); + const contextName = `${query.id}.${provider}.context.json`; + writeFileSync(join(output, contextName), JSON.stringify(context, null, 2) + "\n"); + row = {query_id: query.id, provider, status: "completed", elapsed_ms: performance.now() - start, + ...checked, ...assess(checked.evidence, query.relevant_files), context_file: contextName, + provider_observation: result.observation, + search_requests: 1, context_bytes: Buffer.byteLength(JSON.stringify(context)), native_plan: "not_requested"}; + if (values["loopx-python"]) { + const nativeStart = performance.now(); + try { + const bytes = execFileSync(values["loopx-python"], ["-m", "loopx.cli", "--format", "json", "issue-fix", "workflow-plan", + "--url", query.issue, "--repository-context-json", join(output, contextName)], + {cwd: snapshot, encoding: "utf8", timeout: 30_000, maxBuffer: 2 * 1024 * 1024}); + const plan = object(JSON.parse(bytes)); + const contextReadback = object(plan.repository_context); + if (plan.ok !== true || plan.external_writes_performed !== false || plan.todo_write_performed !== false || + contextReadback.repository_revision !== values.revision) throw new Error("native_plan_invalid"); + row.native_plan = "accepted_read_only"; + row.native_context_revision = contextReadback.repository_revision; + row.native_context_status = contextReadback.context_status; + } catch { failed = true; row.native_plan = "unavailable"; } + row.native_plan_ms = performance.now() - nativeStart; + } + } + } catch { + failed = true; row = {query_id: query.id, provider, status: "unavailable", elapsed_ms: performance.now() - start, + reason_code: "provider_or_native_readback_failed"}; + } + rows.push(row); + report.rows = rows; + writeFileSync(join(output, "report.json"), JSON.stringify(report, null, 2) + "\n"); + } + } + } catch { + failed = true; report.run_error = "source_snapshot_unavailable"; + } finally { + try { if (rune) await rune.close(); } + catch { failed = true; report.teardown_error = "provider_close_failed"; } + rmSync(temporary, {recursive: true, force: true}); + report.total_ms = performance.now() - started; + report.failed = failed; + report.rows = rows; + writeFileSync(join(output, "report.json"), JSON.stringify(report, null, 2) + "\n"); + } + console.log(JSON.stringify({report: join(output, "report.json"), failed})); + if (failed) process.exitCode = 1; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + main(process.argv.slice(2)).catch(() => { console.error("repository_context_pilot_failed; inspect the retained report or command arguments"); process.exitCode = 1; }); +} diff --git a/loopx/capabilities/issue_fix/README.md b/loopx/capabilities/issue_fix/README.md index c00b713d3e..bf1f593af4 100644 --- a/loopx/capabilities/issue_fix/README.md +++ b/loopx/capabilities/issue_fix/README.md @@ -869,6 +869,12 @@ until verified. The public pilot applies that evidence order without introducing a repository-specific control path. +The optional [repository-context pilot](../../../examples/repository-context-pilot/README.md) +compares Rune's no-model MCP retrieval with direct search on pinned public issue +source. It exports source-checked advisory evidence through this same input and +reads it back through `workflow-plan`. This is a reproducible experiment; it +does not enable Rune, change default context or qualify a production provider. + They also accept either `--repository-memory-json ` or a configured context provider. The provider path is deliberately layered: the reusable LoopX context-provider diff --git a/loopx/capabilities/issue_fix/README.zh-CN.md b/loopx/capabilities/issue_fix/README.zh-CN.md index cd24a6e32b..1d5d59449b 100644 --- a/loopx/capabilities/issue_fix/README.zh-CN.md +++ b/loopx/capabilities/issue_fix/README.zh-CN.md @@ -769,6 +769,11 @@ repo-relative source ref。当前 checkout 证据保持最高权威；memory/exp [OpenViking pilot handoff](docs/openviking-pilot-handoff.md) 展示真实 pilot 如何应用该证据 顺序，同时不引入仓库特判控制路径。 +可选的[仓库上下文实验](../../../examples/repository-context-pilot/README.md)在固定版本的 +公开 issue 源码上，对照 Rune 的无模型 MCP 检索与直接搜索。经过源码核验的 advisory +证据复用同一输入，并通过 `workflow-plan` 读回。这是可复现实验，不启用 Rune、 +不改变默认上下文，也不代表生产 provider 已验收。 + 两者既接受 `--repository-memory-json `，也可以调用已配置的 context provider。这里采用明确分层：LoopX 通用 context-provider module 负责 OpenViking CLI/版本/服务 preflight、有上限的显式 `search -> read`、超时与结果 cap、fail-open，