Skip to content

fix(ci): prepare Luna high scanner settings - #3800

Draft
vincentkoc wants to merge 1 commit into
mainfrom
fix/clawhub-gpt6-luna-high
Draft

vincentkoc wants to merge 1 commit into
mainfrom
fix/clawhub-gpt6-luna-high

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

What Problem This Solves

The requested scanner policy is GPT-6 Luna with high reasoning for the ClawScan judge, SkillSpector, and A.I.G. Current released scanner dependencies do not yet implement that policy end to end.

User Impact

Draft — activation is incomplete and must not merge until the dependency and live-proof checklist below is complete. This draft sets explicit model and effort values in both worker workflows. Coverage, moderation decisions, credential isolation, and exact-artifact result reuse stay unchanged. Skill Card generation is outside this change.

The independently safe preparation has landed in #3802 at 9a614dc195afbe1b6834065c6521018b8100afde: optional controls now pass through both restricted worker environments, and the mobile metadata layout assertion uses a controlled local fixture. Deployed workflow defaults and scanner pins were preserved by that merge.

Remaining Dependency Order

  1. A.I.G upstream: merge fix(skill-scan): forward optional reasoning effort to the model API Tencent/AI-Infra-Guard#676 and publish an actual aig-skill-scan PyPI version supporting REASONING_EFFORT. The existing PR is the upstream route; do not publish a duplicate. At the latest check on September 24, 2026, PR 676 was open at de2c5db39855e7e619c8e2d6098ad30b599f7084, and the latest PyPI release was 0.2.2, which ignores this setting.
  2. ClawScan maintainers: update the A.I.G runtime pin to that supported release, and publish the runtime/package containing the Luna/high profiles and environment forwarding from fix(profiles): use Luna judges and forward scanner reasoning clawscan#62, merged as 490bd167ea431697ac42f566adb06281db53c720. The latest published ClawScan is still 0.2.0, whose immutable source uses the older judge profile; source landing alone does not update the package or runtime image.
  3. ClawHub team: adopt those real releases in both .github/workflows/prepublication-publish-checks.yml and .github/workflows/security-scan-codex.yml; regenerate scripts/security/aig-worker-requirements.txt with hashes; update the matching version assertions in scripts/security/prepublication-worker-workflow.test.ts and scripts/security/security-scan-worker-workflow.test.ts. Both workflows currently retain ClawScan 0.1.8 and A.I.G 0.2.2. Keep the existing SkillSpector commit unless a verified dependency requirement changes.
  4. ClawHub team: run a small credentialed Luna/high proof through both worker paths, including the direct bundled-skill SkillSpector path. Record sanitized endpoint, model, effort, parsed result, and fail-closed behavior. Verify the active ClawScan judge and runtime actually consume the released controls; a serialized payload alone does not establish provider authorization. No production scan batch is needed.
  5. ClawHub team: complete relevant scanner/workflow tests, required CI, and a clean review of the final activation head before merging.

The existing dependency notification is #3800 (comment). No package/image release or production scan dispatch has been performed by this PR.

Why This Change Was Made

The worker environment boundary is already repaired on main. This draft now changes only the two workflow settings, their corresponding assertions, and rollout documentation. The judge remains owned by ClawScan's embedded profiles; there is no launcher shim, copied scanner implementation, or invented dependency version.

Application production TypeScript: unchanged. Workflow configuration: net +6 lines. Tests: net +6 lines. Changelog/specs: +6 lines.

Evidence And Limits

Current draft head: 3c2b00c596ae47519ed38e7ed009d7732772d509, rebased onto the landed preparation. The four workflow/test files are byte-identical to the previously reviewed 6c2c9757d5b532eb5f55ccd28ff016e791d1aff3; all worker and browser-fixture files match the landed preparation. The only conflict resolutions preserve the landed documentation and describe the requested policy as pending. git diff --check passed.

  • Historical activation proof: https://github.com/openclaw/clawhub/actions/runs/35950585240 — 67 focused scanner/workflow tests, static checks, and 7,195 unit tests passed (3 skipped); independent Sol/high autoreview was scoped-clean. This predates the rebase and does not replace final activation CI/review after dependency updates.
  • Landed preparation: final-head CI passed; exact-head review found no actionable issue or remaining before-merge move.
  • The previously failing public mobile-metadata fixture is repaired in PR 3802. Local fixture proof passed all four profile-context tests and 18 remaining public smoke tests with retries disabled. Runner-mode proof confirmed the test skips outside the local-auth runner and executes all assertions against the seeded app when enabled.
  • Pinned SkillSpector 69dcdfb was tested against a loopback HTTP server with minimum-supported langchain-openai 1.1.10 / openai 2.54.0 and currently resolved 1.6.5 / 3.19.2: serialized POST /v1/chat/completions, model=gpt-6-luna, reasoning_effort=high, response_format=json_schema; parsed output passed. No function tools, temperature, top_p, top_logprobs, or logprobs were sent. No provider call was made.
  • Codex model and effort forwarding were inspected directly in exec/src/lib.rs and core/src/client.rs. An existing-account Luna/high Codex canary succeeded; it does not validate the ClawHub scanner API credentials.

The original local A.I.G patch had separate historical tests; those results are not transferred to upstream PR 676, whose implementation differs. Use PR 676's final merged/released code for activation acceptance.

AI-assisted implementation; scoped changes and dependency contracts were inspected and validated as described above.

@clawsweeper

clawsweeper Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@vercel

vercel Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
clawhub Ready Ready Preview Sep 24, 2026 8:07am UTC

Request Review

@blacksmith-sh

This comment has been minimized.

@clawsweeper

clawsweeper Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs changes before merge.

What this changes

This PR configures both security worker workflows to request GPT-6 Luna with high reasoning and updates their assertions and rollout documentation.

Example: A new skill is scanned through the prepublication worker.

  • Before: The workflow supplies DEFAULT_MODEL=gpt-5.6.
  • After: The workflow supplies gpt-6-luna and high reasoning, but pinned ClawScan 0.1.8 still explicitly runs its judge with gpt-5.5.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The direction is useful and authorized, but the activation settings disagree with the installed scanner implementation and supplied proof does not demonstrate production-path activation.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality 🦪 silver shellfish (2/6) 1 actionable review finding remain.

Product

Kind: Feature · Worth it: Yes · Fix scope: Partial
User problem: Scanner reviews do not yet consistently use the maintainer-requested Luna/high policy.
Reason: The maintainer-authored PR authorizes the direction and keeps the change scoped, but the current configuration cannot deliver the complete policy.

Merge readiness

⛔ Needs changes before merge - 2 items remain

Keep this maintainer-owned draft open: current main preserves the existing scanner policy, and the previously reported incompatible-pin blocker remains unresolved.

Priority: P2
Reviewed head: 3c2b00c596ae47519ed38e7ed009d7732772d509

Before merge

  • Pin compatible scanners before changing worker defaults (P1) - An ordinary skill scan still installs ClawScan 0.1.8 and invokes its clawhub profile, whose judge explicitly passes --model gpt-5.5. DEFAULT_MODEL=gpt-6-luna cannot override that command, so these settings change supporting scanner configuration while leaving the authoritative judge on the older model. The retained A.I.G 0.2.2 pin also predates the upstream reasoning-support change. Update compatible immutable scanner pins, the A.I.G hash lock, and both workflows' version assertions together, or retain the existing defaults. The same defect affects security-scan-codex.yml and remains unresolved on the previously reviewed head.
  • Complete next step - Resolve GitHub's reported merge conflict while preserving current main's staging and Test worker routing.

Findings

  • [P1] Pin compatible scanners before changing worker defaults — .github/workflows/prepublication-publish-checks.yml:123-126

Tests

  • Missing end-to-end proof: Supply a credentialed run through both worker entry points, including bundled-skill SkillSpector, showing Luna/high consumption, parsed verdicts, and fail-closed behavior; configuration assertions do not exercise that scenario.
Agent review details

How this fits together

GitHub Actions installs pinned scanners and runs workers against materialized skill or package artifacts; ClawScan combines scanner evidence into verdicts used by publication and moderation.

flowchart TD
  A[Skill or package artifact] --> B[GitHub Actions workflow]
  B --> C[Restricted worker environment]
  C --> D[ClawScan profile]
  D --> E[SkillSpector and AIG evidence]
  E --> F[Codex judge]
  F --> G[Publication and moderation verdict]
Loading

Technical review

Best possible solution:

Activate Luna/high through compatible immutable scanner releases and matching workflow pins while preserving credential isolation, scanner gating, and result reuse.

Do we have a high-confidence way to reproduce the issue?

The incomplete activation is reproducible from source: both workers select the clawhub profile, whose pinned ClawScan 0.1.8 judge explicitly requests gpt-5.5; no target code was executed.

Is this the best way to solve the issue?

The workflow is the right activation layer once compatible dependencies are pinned; environment-only activation is too narrow, and copying or overriding the upstream judge locally would create a competing owner.

Full review comments:

  • [P1] Pin compatible scanners before changing worker defaults — .github/workflows/prepublication-publish-checks.yml:123-126
    An ordinary skill scan still installs ClawScan 0.1.8 and invokes its clawhub profile, whose judge explicitly passes --model gpt-5.5. DEFAULT_MODEL=gpt-6-luna cannot override that command, so these settings change supporting scanner configuration while leaving the authoritative judge on the older model. The retained A.I.G 0.2.2 pin also predates the upstream reasoning-support change. Update compatible immutable scanner pins, the A.I.G hash lock, and both workflows' version assertions together, or retain the existing defaults. The same defect affects security-scan-codex.yml and remains unresolved on the previously reviewed head.
    Confidence: 0.99

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 3635eb2fcc3b.

Provenance checked

Testing

Proof path: unit tests only.

Security

None.

Evidence

What I checked:

Review metrics

Metric Value Why it matters
Scoped change size Workflow configuration +8/-2; tests +8/-2; documentation +6/-0 across 6 files. The small diff stays within scanner configuration and rollout intent; it adds no application runtime implementation.

Labels

Label changes:

  • add merge-risk: 🚨 auth-provider: The configured model changes while the pinned judge still selects GPT-5.5, producing inconsistent model routing that configuration assertions do not detect.
  • remove merge-risk: 🚨 automation: Current PR review merge-risk labels are merge-risk: 🚨 auth-provider.

Label justifications:

  • P2: This is a planned scanner-policy improvement in a draft, with no demonstrated current-main outage.
  • merge-risk: 🚨 auth-provider: The configured model changes while the pinned judge still selects GPT-5.5, producing inconsistent model routing that configuration assertions do not detect.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (23 earlier review cycles; latest 8 shown)
  • reviewed 2026-10-07T12:51:40.351Z sha 3c2b00c :: blocked before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-07T20:37:37.129Z sha 3c2b00c :: blocked before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-08T04:43:18.918Z sha 3c2b00c :: blocked before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-08T16:41:36.288Z sha 3c2b00c :: blocked before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-08T22:36:23.855Z sha 3c2b00c :: needs changes before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-09T05:39:43.091Z sha 3c2b00c :: needs changes before merge. :: [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-09T15:36:29.916Z sha 3c2b00c :: needs changes before merge. :: [P1] [P1] Pin compatible scanners before changing worker defaults
  • reviewed 2026-10-10T18:41:17.575Z sha 3c2b00c :: needs changes before merge. :: [P1] [P1] Pin compatible scanners before changing worker defaults

Reviewed October 12, 2026, 11:09 AM ET / 15:09 UTC (Revision 24).

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Sep 24, 2026
@vincentkoc

Copy link
Copy Markdown
Member Author

@Patrick-Erichsen we’re using Tencent/AI-Infra-Guard#676 for ClawHub’s Luna/high activation. Once it is merged and published, please share the aig-skill-scan version and release link here so we can update the hash lock and ClawScan runtime pin. Activation stays draft pending that release and scanner verification; the optional worker plumbing will land separately with current scan defaults preserved.

@vincentkoc
vincentkoc force-pushed the fix/clawhub-gpt6-luna-high branch 2 times, most recently from 6c2c975 to 3c2b00c Compare September 24, 2026 08:04
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. label Sep 30, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. and removed merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. labels Oct 9, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale due to inactivity.
Please update it or it will be closed.

@github-actions github-actions Bot added the stale label Oct 20, 2026

This branch was successfully deployed

1 active deployment
Preview – clawhub — 3c2b00c5 Deployed Sep 24, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. stale status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant