Skip to content

[test] verify eval builtins across module policies (BOX-15) - #41

Merged
vt128 merged 3 commits into
masterfrom
codex/goidiomatic-eval-policy-box15
Nov 8, 2026
Merged

vt128 merged 3 commits into
masterfrom
codex/goidiomatic-eval-policy-box15

Conversation

@vt128

@vt128 vt128 commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Starlet's go_idiomatic collection now provides eval and try_eval. Add integration tests for their availability in Safe/Network/Full and explicit Empty-set opt-in, data ownership and frozen results, excluded host capabilities, shared step limits and fatal cancellation. This changes only ctor_test.go; production code, module allowlists and dependency files match current master. Addresses BOX-15.

The currently published Starlet pin exposes neither helper, so the new feature test explicitly skips there; partial registration fails. A dependency upgrade runs every new case without changing the existing exact module-set golden. This PR does not adopt a new dependency version.

Validation:

  • Fresh four-leg build/race-count2/atomic-coverage/vet comparison on Go 1.27.3 and Go 1.19.13: identical PR source with original Starlet v0.2.4/Starlight v0.2.2 pins versus temporary replacements to Starlet 91918836 and Starlight cd5f885d. All four legs pass. Each original-pin leg records the two expected feature-test skips; each upgraded leg executes every feature case. Both compiler profile pairs retain identical statement/hit blocks and 749/754 covered statements. The interpreter, Go floor and all non-core selected dependency versions stay unchanged.
  • An initial refresh exposed an overstrict cancellation assertion: the outer VM can stop before the evaluator runs and return its typed cancellation error instead of the context sentinel. Its original failure is retained. The corrected test accepts only the context cause or the pinned VM's exact typed cancellation message, and proves no following host callback runs. Separate direct-builtin and pre-cancelled-run tests require the context cause and zero script side effects. The revised source has its own four fresh passing legs; the old failed execution was not rerun.
  • The original normal pre-commit hook passes. Earlier b12f37e9/3c5e2ab8 evidence includes the full 84-leg ecosystem comparison and sixteen real CLI scenarios; those records retain their original scope and were not relabelled as a rerun of this refreshed head.

The producer Starlet #187 has been accepted and merged. Its disclosed performance costs remain documented; this test-only companion adds no production implementation change. Other held runtime fixes remain separate. No tag, release or auto-merge.

@codacy-production

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 31 complexity · 0 duplication

Metric Results
Complexity 31
Duplication 0

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@codacy-production

codacy-production Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 36 complexity · 0 duplication

Metric Results
Complexity 36
Duplication 0

View in Codacy

🟢 Coverage ∅ diff coverage · +0.00% coverage variation

Metric Results
Coverage variation ✅ +0.00% coverage variation (-1.00%)
Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (9d1cf16) 1177 1168 99.24%
Head commit (943d0b8) 1177 (+0) 1168 (+0) 99.24% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#41) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (9d1cf16) to head (943d0b8).

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #41   +/-   ##
=======================================
  Coverage   98.97%   98.97%           
=======================================
  Files          12       12           
  Lines         879      879           
=======================================
  Hits          870      870           
  Misses          5        5           
  Partials        4        4           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vt128
vt128 marked this pull request as ready for review November 8, 2026 17:51
@vt128
vt128 merged commit 36a5f61 into master Nov 8, 2026
12 checks passed
@vt128
vt128 deleted the codex/goidiomatic-eval-policy-box15 branch November 8, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant