test: execute the compiled tool-grammar path and bound chat-test builds

[leehack]

Summary

Split-out of #420, which carries a further 5,300 lines of CI policy pending a
maintainer decision. This PR takes the executable files — the ones that run
rather than describe — so they can land independently; nothing here depends on
that machinery.

#420 retains .github/high-risk-policy.json, the workflow, the validator, the
prose, and three test/unit/tooling/ files (trusted_high_risk_contract_test,
test_matrix_test, run_llama_cpp_chat_tests_script_test). The last one also
updates workflow files this PR does not touch, so rather than import it, this
PR adds its own self-contained coverage for the shell change it ships.

No lib/ changes: the new test runs against production code unchanged from
main.

What changed

test/integration/core/grammar/generated_tool_schema_grammar_test.dart (new, +286)

Issue #419 required the generated tool grammar to survive llama.cpp's own
grammar compiler; existing coverage stopped at the generated GBNF string. Five
tests now execute that requirement:

• the grammar generated for a ToolChoice.required tool drives a real
  generation through LlamaEngine.create(tools:, toolChoice:) and yields the
  expected tool call;
• a prompt asking for a schema-invalid call still produces the valid call,
  i.e. the constraint holds against the model, not just the prompt;
• the grammar compiled via llama_sampler_init_grammar accepts a valid
  envelope token-by-token and drives an invalid one to -inf;
• a valid-but-incomplete prefix is rejected at EOG;
• mutating the generated root rule to reference an undefined rule makes the
  native parser reject it, surfacing as LlamaInferenceException.

Gating follows the existing convention for model-backed integration tests
(grammar_regression_test.dart, engine_integration_test.dart,
chat_session_integration_test.dart): @TestOn('vm') plus a timeout, no
local-only tag. It needs the same tiny stories15M.gguf fixture and the same
build-hook native asset those siblings already need, and the CI VM job
(ci.yml, dart test -p vm -j 1 --exclude-tags local-only) already provisions
both, so it adds no new CI prerequisite. It is covered by the existing
root-vm row in tool/testing/test_matrix.dart; no new matrix row is needed.

tool/testing/run_llama_cpp_chat_tests.sh (+7/-1)

Replaces a bare cmake --build --parallel (one job per core) with
--parallel "${build_jobs}", from LLAMA_CPP_CHAT_TEST_BUILD_JOBS, default
2. Non-positive-integer values are rejected with exit 64 before the llama.cpp
clone and configure, rather than being handed to cmake.

test/e2e/template/llama_cpp_chat_tests_e2e_test.dart (+1)

Adds includeParentEnvironment: true on the Process.run that passes
environment:. This is Dart's default, so it is a no-op at runtime; it is
there so the reader of a call that overrides environment: can see that the
child still inherits the parent PATH/HOME the build script needs.

test/integration/core/template/llama_cpp_template_detection_integration_test.dart (+5)

Asserts the fixture count equals the expectation-map size. The pre-existing
missing isEmpty check only caught templates present on disk but unmapped; the
reverse — a registered fixture disappearing and silently shrinking the parity
matrix — passed quietly. Now it fails.

Porting notes

All four files applied to ea2fe4e05 with no conflicts. The blobs are
byte-identical at 161b8df6a and at #420's current head 4457f71d9, so
nothing was ported from a superseded revision. git apply --check:

Checking patch test/e2e/template/llama_cpp_chat_tests_e2e_test.dart...
Checking patch test/integration/core/template/llama_cpp_template_detection_integration_test.dart...
Checking patch tool/testing/run_llama_cpp_chat_tests.sh...

No changelog entry: AGENTS.md scopes changelog bullets to user-facing changes,
and this PR changes only tests and a local test-runner script. Runtime and
public API behavior are unchanged.

Validation

Flutter 3.47.1 / Dart from the pinned CI SDK. Every command below was run on
this branch at 861b8c86b.

dart run tool/prepare_workspace.dart → exit 0, then git status --porcelain:

(no output)

dart analyze:

Analyzing wt-grammar-cov...
No issues found!

dart format --output=none --set-exit-if-changed .:

Formatted 573 files (0 changed) in 1.44 seconds.

New grammar test — dart test -p vm -j 1 --exclude-tags local-only test/integration/core/grammar/generated_tool_schema_grammar_test.dart:

00:05 +0: native compiler accepts generated tool schema grammar
00:05 +1: compiled grammar rejects a schema-invalid requested tool call
00:06 +2: compiled grammar deterministically filters invalid candidates
00:06 +3: compiled grammar rejects an incomplete valid prefix at EOG
00:06 +4: native compiler rejects a broken generated grammar dependency
00:06 +5: All tests passed!

Template detection test with the pinned b10549 fixtures
(LLAMA_CPP_TEMPLATES_DIR=... REQUIRE_LLAMA_CPP_TEMPLATES=1 dart test -p vm -j 1 --exclude-tags local-only test/integration/core/template/llama_cpp_template_detection_integration_test.dart)
— 68 .jinja fixtures (the extract also holds a README.md, which the test filters out), 74 tests, including the new count assertion:

00:00 +68: llama.cpp template detection parity maps every vendored llama.cpp template
00:01 +74: All tests passed!

Full CI VM lane — dart test -p vm -j 1 --exclude-tags local-only (same env):

01:24 +1732 ~3: All tests passed!
exit=0

Shell script — bash -n and shellcheck both clean, and the new guard:

--- guard: 0 ---
LLAMA_CPP_CHAT_TEST_BUILD_JOBS must be a positive integer.
exit=64
--- guard: abc ---
LLAMA_CPP_CHAT_TEST_BUILD_JOBS must be a positive integer.
exit=64
--- guard: -4 ---
LLAMA_CPP_CHAT_TEST_BUILD_JOBS must be a positive integer.
exit=64

Not run

test/e2e/template/llama_cpp_chat_tests_e2e_test.dart — tagged
['local-only', 'e2e'] and excluded from CI. Executing it means cloning
llama.cpp and cmake-building test-chat-* plus test-gbnf-validator (and, for
the second case, the full tools/server build). I did not run that build, so I
have no evidence for the --parallel 2 build path or the
includeParentEnvironment change beyond the static checks and guard-rejection
runs above. The valid-value path of the script is unexercised here.

dart test -p chrome --exclude-tags local-only — not run; the new test is
@TestOn('vm') and no Chrome-visible file changed.

[Copilot]

🟢 Approval recommended

Changes are confined to tests and a local-only test runner script, and the added assertions/guards are consistent with existing repo patterns and CI expectations.

Pull request overview

Adds integration coverage ensuring the generated “tool schema grammar” is not only well-formed as a GBNF string, but also compiles and enforces constraints through the native llama.cpp grammar compiler and production LlamaEngine.create(...) tool-calling path. Also tightens a local-only llama.cpp chat-test build helper and strengthens template fixture parity assertions.

Changes:

• Add a new VM integration test that compiles and exercises the generated tool-schema grammar end-to-end (native compiler + real generation + token-by-token accept/reject checks).
• Make the llama.cpp chat-test build parallelism explicitly configurable and validate the setting early.
• Strengthen template fixture parity by asserting the vendored fixture count matches the expectation map, preventing silent shrinkage.

File summaries

File	Description
tool/testing/run_llama_cpp_chat_tests.sh	Adds validated LLAMA_CPP_CHAT_TEST_BUILD_JOBS and uses it for cmake --build --parallel to avoid unbounded core usage.
test/integration/core/template/llama_cpp_template_detection_integration_test.dart	Adds a fixture-count assertion so missing registered templates fail loudly.
test/integration/core/grammar/generated_tool_schema_grammar_test.dart	New integration test executing compiled tool-schema grammar paths (production generation + native compiled acceptance/rejection).
test/e2e/template/llama_cpp_chat_tests_e2e_test.dart	Makes parent environment inheritance explicit when overriding environment: in Process.run.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

A new unit test executes bash without a Windows skip despite CI running dart test on windows-latest, and the new FFI tokenization helper should guard against non-positive required token counts before allocating native memory.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/integration/core/grammar/generated_tool_schema_grammar_test.dart:236

• _tokenize assumes -llama_tokenize(...) returns a positive required token count, and then passes it directly to calloc<llama_token>(required). If required <= 0 (error/unexpected return), this can crash or throw before the test produces a useful failure. Add an expect(required, greaterThan(0)) (or equivalent guard) before allocating and tokenizing, similar to the production _tokenize... helpers in lib/src/backends/llama_cpp/llama_cpp_service.dart.

List<int> _tokenize(Pointer<llama_vocab> vocab, String value) {
  final text = value.toNativeUtf8();
  try {
    final required = -llama_tokenize(
      vocab,
      text.cast<Char>(),
      text.length,
      nullptr,
      0,
      false,
      true,
    );
    final tokens = calloc<llama_token>(required);

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

The new compiled-grammar test has a native sampler lifetime issue where llama_sampler_free is not guaranteed if tokenization/prefix assertions throw, which can leak native resources and destabilize subsequent tests.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

The changes are test- and tooling-focused, consistent with existing patterns in the repo, and include direct coverage for the new script behavior and compiled-grammar requirements.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟢 Approval recommended

Changes are isolated to tests and a local runner script, and the added coverage aligns with the stated regression-prevention goals without impacting production APIs.

Review details

• Files reviewed: 5/5 changed files
• Comments generated: 0 new
• Review effort level: Lite