[jasonnvidia]

The issue

When a regeneration run fails, the tracking issue tells us which skill broke but never why.

The generator asks an AI model to assign five metadata fields to each skill. For a skill already in metadata.json, a failed enrichment is harmless — the previous values are carried forward. A brand new skill has nothing to fall back on, so a failed enrichment fails the whole run and blocks metadata updates for every skill.

That happened five times in a row on the three Run:ai skills synced in #782, reported in #783. The model is non-deterministic, so a different skill failed on different runs: runai-get-cluster-health-summary three times, runai-get-utilization-overview twice. The sixth run succeeded on its own with no code change, and all three skills have been in metadata.json since.

Throughout, we could not tell which field failed enrichment or why. The reason was being collected in skill_warnings, but main() returns at the if errors: gate before reaching the block that prints it and writes /tmp/skill-warnings.json, so the detail was gathered and then discarded.

The recovered case (a published skill kept from the baseline) does not set errors, so it always reported correctly. That asymmetry is why this stayed hidden for so long — the only path that loses the reason is a brand new skill, which is exactly the path that hard-fails.

The fix

Write the warnings file before the error gate, and print the reasons alongside the summary line.

The workflow already reads /tmp/skill-warnings.json and renders it as a "Skills that could not be regenerated" section in the issue body, so no workflow change is needed — the generator simply never wrote that file on the one path that fails.

This is reporting only. Enrichment, entry building, validation, and output serialization are untouched, so nothing that decides the contents of metadata.json changed.

Example output

Before — the actual log from run 40729153708, as posted to #783:

VALIDATION FAILED:
  - no entry could be built for these skills, and no usable entry exists in the previous metadata.json to fall back on: skills/runai-get-cluster-health-summary. Re-run, or add the skill name to metadata-exclusions.yaml to drop it deliberately.
Skill change classification:
  added: 3
    - skills/runai-audit-assigned-quota
    - skills/runai-get-cluster-health-summary
    - skills/runai-get-utilization-overview

Actionable content: re-run, or exclude the skill. Nothing about the cause.

After — same failure, reproduced through the test harness:

VALIDATION FAILED:
  - no entry could be built for these skills, and no usable entry exists in the previous metadata.json to fall back on: skills/runai-get-cluster-health-summary. Re-run, or add the skill name to metadata-exclusions.yaml to drop it deliberately.

Per-skill reasons:
  - skills/runai-get-cluster-health-summary: AI returned UNRESOLVED for `discovery.activity_tags` on skill `runai-get-cluster-health-summary`; metadata cannot be auto-generated. Please set this field manually or update the skill.
Skill change classification:
  added: 3
    ...

And /tmp/skill-warnings.json is now produced on this path, which the workflow surfaces at the top of the issue:

{
  "stale_skills": [
    "skills/runai-get-cluster-health-summary: AI returned UNRESOLVED for `discovery.activity_tags` on skill `runai-get-cluster-health-summary`; metadata cannot be auto-generated. Please set this field manually or update the skill."
  ]
}

Risks

Low, and deliberately narrow in scope. Old vs new compared across all six mode/path combinations:

scenario	exit old/new	warnings file old/new	log
healthy, --check --no-ai (PR CI)	0 / 0	no / no	identical
--report-only	0 / 0	no / no	identical
new skill, --no-ai	0 / 0	yes / yes	identical
new skill, AI declines	1 / 1	no / yes	differs (intended)
published skill recovered	1 / 1	yes / yes	identical
healthy with AI	0 / 0	no / no	identical

Exit codes identical everywhere. The only changed scenario is the target, and its diff is purely additive — three lines added, nothing removed or reworded.

Two residual risks worth knowing:

1. The failure path now has a filesystem side effect where it previously had none. If /tmp were unwritable, a run that used to exit 1 with a clean validation message would instead raise. Not reachable on ubuntu-latest, and the success path already wrote to the same location, so CI exposure is unchanged — but the trigger condition is new. Left unguarded to match existing style; happy to wrap it if reviewers prefer.

2. Failures with an unrelated root cause may now lead with a skills section. If a run fails on something else (schema violation, unknown subdomain) and some other skill separately produced a warning, the issue body shows the "could not be regenerated" block above the collapsed log. Everything in it is accurate, but it sits above the actual blocker.

Not verified end to end: the github-script step cannot run locally. The generator is confirmed to emit the file in the exact shape that step already consumed on the partial-success path, so the parsing contract is unchanged — only the trigger condition widened.

Testing

Added case 6 to test_generate_skill_metadata.py (4 assertions). The three behavioral ones fail against the unmodified generator and pass with the fix; the fourth guards against regressing the summary line. This closes a real gap: case 3 asserted only that the warning is collected, and case 5 asserted only the exit code — nothing asserted the reason is emitted, which is how this survived.

Verified against current main (430 skills):

• test_generate_skill_metadata.py — 19/19 pass
• generate-skill-metadata.py --check --no-ai — OK: generated outputs are byte-stable

Follow-up, not included here

This makes the next failure diagnosable; it does not prevent failures. The actual fix is to retry when the model returns UNRESOLVED or an empty field, the way the client already retries 403/429/5xx and malformed JSON. Measured against the real client with a stubbed HTTP layer:

model response	HTTP attempts (budget is 4)
UNRESOLVED	1
empty field	1
malformed JSON	4

The field validation sits after the retry loop's break, so it never retries. Note the scope is wider than it first appears: there are four non-retried terminal checks after the loop (non-dict, unexpected keys, missing/empty/non-string, and UNRESOLVED), and until this PR lands we cannot tell which one actually hit the Run:ai skills. Deferred deliberately so the retry change can be driven by a real logged reason rather than a guess.

[jasonnvidia]

@jim-nvidia
@mosheabr please review

[siqinvidia]

Reviewed f3d3de2. The change writes collected per-skill warnings before the validation error gate and includes their reasons in the failure log, preserving the existing structured warnings contract and validation decisions. Independently ran test_generate_skill_metadata.py: 19/19 assertions pass. generate-skill-metadata.py --check --no-ai confirms byte-stable output for all 430 skills. All applicable GitHub CI checks pass.