Repository navigation
feat: adopt reviewed Sol security consumer policy - #3998
Merged
Merged
Conversation
…l-policy-20261019
…l-policy-20261019
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Collaborator
Author
|
Parent-inspected proof for consumer head Fresh installed ClawScan source:
SkillSpector: 7 completed requests, Sol/medium, explicit wire default. AIG: 4, Sol/medium, explicit wire default. Judge: 7, Sol/high, explicit CLI default, wire field omitted by Codex 0.160, observed default response tier. All 18 completed responses identify Sol/default. This distinction is intentional, not an assertion that Codex sent an explicit API field. Original built-in-provider acceptance also passed both fixtures; full observed judge transport telemetry comes from a separate transparent loopback custom-provider rerun. No artifact instructions were executed. This is bounded synthetic/macOS-host acceptance, not statistical quality evaluation or local Linux-image certification. Exact acceptance receipt{
"consumerCommit": "6818f639b4b3bbee54f472b125db9467047240c1",
"upstreamCommit": "5be7f8e97fa474f0aa9eb443f38d9e3d5f51226b",
"binarySha256": "1c8ae7f28ece2921ffb86342d7e489f81864cad6579d8fa1ca0d89357ecad036",
"host": "macOS arm64",
"sourceClean": true,
"fixtures": [
{
"fixture": "benign",
"sha256": "d81d8fd54a49ecca0e64b1dcb0b8640855ca2b76f6b5e34661e3035e8e1a696a",
"seconds": 41.384,
"verdict": "benign",
"confidence": "high",
"consumerCheck": "clean",
"aigFindings": 0,
"scanners": {
"aig": "completed",
"clawscan-static": "completed",
"skillspector": "completed"
},
"configuredJudgeModel": "gpt-6.1-sol"
},
{
"fixture": "malicious",
"sha256": "3bc2ea37dd5014a70d35dd29e1cff72c4f42e64389c459468167d024c63d71c5",
"seconds": 86.051,
"verdict": "malicious",
"confidence": "high",
"consumerCheck": "blocked",
"aigFindings": 2,
"scanners": {
"aig": "completed",
"clawscan-static": "completed",
"skillspector": "completed"
},
"configuredJudgeModel": "gpt-6.1-sol"
}
],
"lanes": [
{
"fixture": "benign",
"lane": "skillspector",
"requests": 3,
"completed": 3,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "medium",
"configuredTier": "default",
"wireTier": "default",
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
},
{
"fixture": "benign",
"lane": "aig",
"requests": 2,
"completed": 2,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "medium",
"configuredTier": "default",
"wireTier": "default",
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
},
{
"fixture": "benign",
"lane": "judge",
"requests": 3,
"completed": 3,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "high",
"configuredTier": "default",
"wireTier": null,
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
},
{
"fixture": "malicious",
"lane": "skillspector",
"requests": 4,
"completed": 4,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "medium",
"configuredTier": "default",
"wireTier": "default",
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
},
{
"fixture": "malicious",
"lane": "aig",
"requests": 2,
"completed": 2,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "medium",
"configuredTier": "default",
"wireTier": "default",
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
},
{
"fixture": "malicious",
"lane": "judge",
"requests": 4,
"completed": 4,
"requestedModel": "gpt-6.1-sol",
"requestedEffort": "high",
"configuredTier": "default",
"wireTier": null,
"observedModels": [
"gpt-6.1-sol"
],
"observedTiers": [
"default"
]
}
],
"safety": {
"syntheticTextOnly": true,
"artifactCodeExecuted": false,
"credentialsPersisted": false,
"credentialAbsenceCheckedFiles": 78
},
"limitations": [
"Two tiny fixtures are acceptance smoke, not an accuracy benchmark.",
"Original built-in OpenAI Codex runs passed both fixtures; complete observed judge telemetry is a separate full rerun using a loopback-only transparent custom provider and HTTPS transport. Request bytes are forwarded without model, reasoning, or service tier edits.",
"Codex 0.160 intentionally omits explicit default service_tier on the wire. Original CLI flags request default; provider responses report default. This is not an explicit judge API tier field.",
"No production deployment or public push; macOS host dependencies, not a local Linux image run."
]
}Redacted request/response transport metadata{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1347, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 1344}, "output_tokens": 16, "output_tokens_details": {"reasoning_tokens": 0}, "total_tokens": 1363}}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 962, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 0}, "output_tokens": 16, "output_tokens_details": {"reasoning_tokens": 0}, "total_tokens": 978}}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "benign", "pid": 35699, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1718, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 1715}, "output_tokens": 86, "output_tokens_details": {"reasoning_tokens": 68}, "total_tokens": 1804}}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "http", "status": 200}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 8931, "input_tokens_details": {"cache_write_tokens": 136, "cached_tokens": 8792}, "output_tokens": 260, "output_tokens_details": {"reasoning_tokens": 104}, "total_tokens": 9191}}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "http", "status": 200}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "benign", "pid": 35821, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 9541, "input_tokens_details": {"cache_write_tokens": 610, "cached_tokens": 8928}, "output_tokens": 141, "output_tokens_details": {"reasoning_tokens": 48}, "total_tokens": 9682}}
{"fixture":"benign","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"benign","lane":"judge","kind":"http","status":200}
{"fixture":"benign","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":23767,"input_tokens_details":{"cache_write_tokens":23764,"cached_tokens":0},"output_tokens":147,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":23914}}
{"fixture":"benign","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"benign","lane":"judge","kind":"http","status":200}
{"fixture":"benign","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":24194,"input_tokens_details":{"cache_write_tokens":427,"cached_tokens":23764},"output_tokens":127,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":24321}}
{"fixture":"benign","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"benign","lane":"judge","kind":"http","status":200}
{"fixture":"benign","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":28831,"input_tokens_details":{"cache_write_tokens":4637,"cached_tokens":24191},"output_tokens":507,"output_tokens_details":{"reasoning_tokens":79},"total_tokens":29338}}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1033, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 1030}, "output_tokens": 100, "output_tokens_details": {"reasoning_tokens": 82}, "total_tokens": 1133}}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1417, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 1414}, "output_tokens": 457, "output_tokens_details": {"reasoning_tokens": 115}, "total_tokens": 1874}}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1789, "input_tokens_details": {"cache_write_tokens": 0, "cached_tokens": 1786}, "output_tokens": 350, "output_tokens_details": {"reasoning_tokens": 182}, "total_tokens": 2139}}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": 128000}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "http", "status": 200}
{"lane": "skillspector", "fixture": "malicious", "pid": 39524, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 1421, "input_tokens_details": {"cache_write_tokens": 1418, "cached_tokens": 0}, "output_tokens": 909, "output_tokens_details": {"reasoning_tokens": 0}, "total_tokens": 2330}}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "http", "status": 200}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 9065, "input_tokens_details": {"cache_write_tokens": 270, "cached_tokens": 8792}, "output_tokens": 258, "output_tokens_details": {"reasoning_tokens": 84}, "total_tokens": 9323}}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "request", "path": "/v1/responses", "model": "gpt-6.1-sol", "reasoning": {"effort": "medium"}, "serviceTier": "default", "maxOutputTokens": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "http", "status": 200}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "in_progress", "serviceTier": "default", "usage": null}
{"lane": "aig", "fixture": "malicious", "pid": 42135, "kind": "response", "model": "gpt-6.1-sol", "status": "completed", "serviceTier": "default", "usage": {"input_tokens": 9835, "input_tokens_details": {"cache_write_tokens": 770, "cached_tokens": 9062}, "output_tokens": 662, "output_tokens_details": {"reasoning_tokens": 81}, "total_tokens": 10497}}
{"fixture":"malicious","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"malicious","lane":"judge","kind":"http","status":200}
{"fixture":"malicious","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":25136,"input_tokens_details":{"cache_write_tokens":25133,"cached_tokens":0},"output_tokens":75,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":25211}}
{"fixture":"malicious","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"malicious","lane":"judge","kind":"http","status":200}
{"fixture":"malicious","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":25307,"input_tokens_details":{"cache_write_tokens":171,"cached_tokens":25133},"output_tokens":134,"output_tokens_details":{"reasoning_tokens":0},"total_tokens":25441}}
{"fixture":"malicious","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"malicious","lane":"judge","kind":"http","status":200}
{"fixture":"malicious","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":32657,"input_tokens_details":{"cache_write_tokens":7350,"cached_tokens":25304},"output_tokens":153,"output_tokens_details":{"reasoning_tokens":100},"total_tokens":32810}}
{"fixture":"malicious","lane":"judge","kind":"request","path":"/v1/responses","model":"gpt-6.1-sol","reasoning":{"effort":"high","context":"all_turns"},"serviceTier":null}
{"fixture":"malicious","lane":"judge","kind":"http","status":200}
{"fixture":"malicious","lane":"judge","kind":"response","model":"gpt-6.1-sol","status":"completed","serviceTier":"default","usage":{"input_tokens":33017,"input_tokens_details":{"cache_write_tokens":360,"cached_tokens":32654},"output_tokens":749,"output_tokens_details":{"reasoning_tokens":111},"total_tokens":33766}} |
clawsweeper
Bot
added
clawsweeper:intake:feature
Contributor
|
Codex review: needs maintainer review before merge. What this changesThis PR adopts Sol scanner and judge defaults with matching source-pinned Responses adapters and accurate configured-model provenance. Example: A newly uploaded skill receives its security scan.
Review scores
ProductKind: Feature · Worth it: Yes · Fix scope: Complete Merge readiness✅ Ready for maintainer review Keep open: this maintainer-authored PR delivers authorized scanner-policy work absent from main, with no blocking findings identified. Priority: P2 Before mergeNone. FindingsNone. Agent review detailsHow this fits togetherSecurity workers receive claimed skill or plugin artifacts, run scanners and a final judge, and submit validated evidence and verdicts to the existing publication and moderation owners. flowchart LR
A[Claimed artifact] --> B[Security worker]
B --> C[Matched scanner adapters]
C --> D[Scanner evidence]
D --> E[ClawScan judge]
E --> F[Validated verdict and provenance]
F --> G[Publication and moderation]
Technical reviewBest possible solution: Adopt the matched upstream implementation while retaining independent operator controls, honest provenance, and existing moderation boundaries. Do we have a high-confidence way to reproduce the issue? This is policy adoption rather than a bug report; source comparison establishes the old defaults, and exact-head provider traces exercise the proposed behavior. Is this the best way to solve the issue? Best-fix verdict: best for the approved scope, because matched adapters remain upstream and both workers share their environment policy; changing environment values alone would leave incompatible transports, while copying adapters into ClawHub would create another owner. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against ec1016ae8b2b. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment Reviewed October 19, 2026, 11:29 PM ET / October 20, 2026, 03:29 UTC. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
5be7f8e97fa474f0aa9eb443f38d9e3d5f51226band install its exact matching source-pinned Responses adapters, instead of using an older npm profile with incompatible dependencies.Verification
gpt-6.1-solanddefault. The two scanner APIs explicitly sendservice_tier: default. Codex 0.160 explicitly configures Standard in CLI argv but intentionally omits that field on the wire; judge responses reportdefault.ci:static,ci:types-build,ci:unit(7,913 passed, three skipped),ci:packages, 120 targeted security tests and seven installer tests passed.Boundaries
Two synthetic text-only fixtures prove wiring and basic acceptance, not scanner-quality benchmarks. Complete judge telemetry uses a transparent loopback custom-provider transport without modifying model/effort/tier bytes; separate built-in-provider runs also passed. Host acceptance is macOS arm64; upstream native Linux amd64/arm64 runtime-image proof is linked from ClawScan #68. No production queue processing, deployment, Endor activation, or public ClawScan package release is part of this PR.