Skip to content

[test] classify optional Starlark evaluator (BOX-14) - #40

Closed
vt128 wants to merge 1 commit into
masterfrom
codex/starlark-eval-policy-box14
Closed

vt128 wants to merge 1 commit into
masterfrom
codex/starlark-eval-policy-box14

Conversation

@vt128

@vt128 vt128 commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Superseded by #41, which implements the approved go_idiomatic builtin design. This earlier separate-module candidate is closed without merging; its original description and validation history remain below.


When Starlet registers its data-only starlark evaluator, Starbox's fail-closed module-classification test rejects the new builtin even though Safe and Network correctly keep it unavailable.

Classify this one reviewed optional module as Full-only when it is present. Preserve the exact existing Safe/Network lists and continue rejecting every other unclassified future module. Add integration tests proving that Safe/Network return ModuleWithheldError, while Full and an explicit AddNamedModules("starlark") opt-in can evaluate native data, report a friendly try-error on attempted mutation, and leave the caller's list writable and independent.

This test-only change addresses BOX-14. Current published Starlet pins remain supported: the module-set golden still runs, while only the new evaluator-specific test skips when the module is absent. Production files, dependencies, and policy configuration are unchanged.

Validation:

  • Reproduced the classification failure on the unmodified consumer against the new-module source; the two-file test change passes against that same source snapshot. Original red evidence is retained.
  • Fresh paired validation of this final consumer commit passes with Starlet actual master bf94a756 and the new-module candidate 179266c2, on host Go 1.27.3 and Docker Go 1.19.13. Full build, race tests repeated twice, and vet pass in all four legs.
  • Consumer coverage remains 749/754 covered statements, with identical hit blocks inside each compiler. Baseline absent-module skips are explicit; the new-module legs execute all policy cases.
  • The original pre-commit hook ran vet and the full current-pin suite successfully.

The producer is Starlet #186. Its directed framework cohort totals 16 fresh legs with exact coverage, and 16 real CLI invocations; independent terminal review passes. The two original CLI budget-exit assertion failures are retained: the existing source contract requires exit 5, and a separate audit corrected the expectation without rerunning.

Exact-head 12 required / 12 reported checks, both Codecov contexts, and checks-watch pass; post-watch head/base remain unchanged. Keep this PR in draft alongside the producer pending full-ecosystem and final performance acceptance. This PR does not change production policy, and no tag, release, or auto-merge is requested.

@codacy-production

codacy-production Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 10 complexity · 0 duplication

Metric Results
Complexity 10
Duplication 0

View in Codacy

🟢 Coverage ∅ diff coverage · +0.00% coverage variation

Metric Results
Coverage variation ✅ +0.00% coverage variation (-1.00%)
Diff coverage ✅ ∅ diff coverage

View coverage diff in Codacy

Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (77c6bad) 1177 1168 99.24%
Head commit (6ef62ac) 1177 (+0) 1168 (+0) 99.24% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#40) 0 0 ∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@codecov

codecov Bot commented Nov 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.97%. Comparing base (77c6bad) to head (6ef62ac).

Additional details and impacted files
@@           Coverage Diff           @@
##           master      #40   +/-   ##
=======================================
  Coverage   98.97%   98.97%           
=======================================
  Files          12       12           
  Lines         879      879           
=======================================
  Hits          870      870           
  Misses          5        5           
  Partials        4        4           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant