fix(ci): nightly installer job plants a harness before install.sh (red 5 nights) - #27
Merged
Merged
Conversation
… install.sh The `installer` job in nightly-install.yml (A4, b5339db, 2026-09-14) has failed every night since it was added (runs 34949338879 .. 35431368317). It isolates HOME to `${{ runner.temp }}/home` so `studyloop install agents` writes into scratch, but an empty HOME has no harness: on the runner no codex/opencode/pi/grok binary is on PATH and no ~/.kiro, ~/.claude, ~/.pi or ~/.grok exists, so `detect_available_agent_tools()` returns [] and `install.sh` exits 1 at "Installing agent definitions" — the script's documented behaviour when no supported AI tool is present. The two tools the job was written to prove had already installed fine by then; the verify step never ran. Reproduced locally with an empty HOME and a PATH holding only uv and the system bins: exit 1, same message. Planting ~/.kiro (or ~/.claude, ~/.pi, ~/.grok — the markers the detector reads without a binary) makes the same command exit 0 and write the links. This test pins the fixture: the run step plants a marker before the script, and a verify step reads what `install agents` wrote into the isolated HOME. Red on the current workflow for that reason (1 failed, 13 passed).
…d HOME GREEN for ee43863. The `installer` job isolates HOME so `install agents` writes into scratch; the fixture now creates ~/.kiro, ~/.claude, ~/.pi and ~/.grok in that HOME first — the four markers detect_available_agent_tools() reads without a binary — so `install.sh` reaches and exercises all four link sets instead of refusing with "No supported AI tools detected". A new step verifies what `install agents` wrote: two artefacts each for kiro, claude and grok, one for pi, every path taken from installers.py (_TOOL_LINKS / _configure_*). Without a read of the isolated HOME the isolation proved nothing. Proved locally the way the runner runs it: full ./scripts/install.sh --non-interactive --no-smoke with HOME, UV_TOOL_DIR and UV_TOOL_BIN_DIR in scratch and a PATH holding only uv and the system bins — exit 0, "Installation complete!", both entry points answer, all seven artefacts present. Contract module 14/14. The nightly run itself can only be observed at 03:30 UTC or via workflow_dispatch on main after merge.
…tus posted, left open Both issues carry the proposals' content with every cited test id, symbol and path verified against main a03fc9b before posting. #21 stays open because its definition of done names OpenCode in the core tuple and the harness receipt records OpenCode as preview on two named blockers.
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Strengthen the workflow contract tests to enforce complete harness coverage and verification-step ordering.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (1)
What changed in this PR
Fixes the nightly installer job by seeding harness markers in its isolated HOME and verifying installation outputs.
Changes:
- Plants Kiro, Claude, Pi, and Grok harness markers.
- Verifies seven installed artifacts.
- Adds workflow contract coverage and updates task status.
| File | Summary |
|---|---|
packages/studyloop/tests/test_ci_workflow_contract.py |
Tests the workflow contract. Moderate (3 votes): assert all expected markers. Moderate (1 vote): ensure verification follows installation. |
openspec/changes/plan-integration-followons/tasks.md |
Records completed follow-up work. |
.github/workflows/nightly-install.yml |
Seeds harness directories and verifies installation outputs. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+273
to
+281
| planted = [ | ||
| marker | ||
| for marker in ("$HOME/.kiro", "$HOME/.claude", "$HOME/.pi", "$HOME/.grok") | ||
| if marker in run | ||
| ] | ||
| assert planted, ( | ||
| "the installer job runs install.sh in an empty HOME; plant at least one " | ||
| "directory-detected harness marker first or `install agents` exits 1" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Nightly Install Check's
installerjob (A4, added inb5339db9on 2026-09-14) has failed every night since it was added (runs 34949338879 … 35431368317; the last green nightly, 34826479107, predates the job). Cause: the job isolatesHOMEto an empty scratch directory sostudyloop install agentswrites there — but an empty HOME has no harness, the runner has nocodex/opencode/pi/grokonPATH, sodetect_available_agent_tools()returns[]andinstall.shexits 1 at "Installing agent definitions" with "No supported AI tools detected" — the script's documented behaviour. The two tools the job exists to prove had already installed fine; its verify step never ran.ee43863eRED —test_nightly_installer_job_plants_a_harness_before_running_install_shintest_ci_workflow_contract.py: the run step must plant a directory-detected harness marker in the isolated HOME before the script, and a verify step must read whatinstall agentswrote. Red on the old workflow for that reason (1 failed / 13 passed).57b38838GREEN — the step now creates~/.kiro ~/.claude ~/.pi ~/.grok(the four markers the detector reads without a binary), so all four link sets are exercised; a new "Verify installed agent definitions" step checks seven artefacts, each a path frominstallers.py(_TOOL_LINKS/_configure_*).4f8e3e0f—tasks.md: item 7 step 4 done (Plan integration follow-on (D-D): derive the plan bias from prerequisite edges, milestone order and energy demand #25, Plan integration follow-on (D-E): age-aware nudge and a learner-issued retire/snooze for an overdue item #26 filed; Promote Pi and OpenCode from preview to core harness tier — with live release evidence, then docs #21 status posted, left open).Verification
uvand the system bins → exit 1 with the runner's exact message. Planting the markers → exit 0../scripts/install.sh --non-interactive --no-smokerun the way the fixed job runs it (HOME,UV_TOOL_DIR,UV_TOOL_BIN_DIRall in scratch, uv-only PATH): exit 0, "Installation complete!", both entry points answer, all seven artefacts present.test_ci_workflow_contract.py14/14; all pre-commit hooks (incl. pyright) on both commits.workflow_dispatchonmainafter merge, or 03:30 UTC.Not touched:
install.sh's refusal when no harness exists (correct; the README states a harness is a precondition).