Repository navigation
test: coverage gates (90 % lines, 80 % branches) and property-based tests, with three bugs they found - #21
Merged
Merged
Conversation
Slice 07's merge rule says an incoming gloss becomes a form only when it differs from the word's own gloss. Both runtimes added it whenever it wasn't already a form, so a form the learner had removed came back on the next re-add, and restoring a backup without ids a second time reported its words as merged (and changed them) instead of identical. A property test (restoring the same backup twice changes nothing) found it. The shared merge fixture gains the case, so the server and the extension are both held to it, and the backup tests gain a regression test for the restore. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
requireExt wrapped each library in "(function (module, exports) {"
before evaluating it, which shifted every code offset by the wrapper's
length. When the same file was also run unwrapped (importScripts in the
background's vm context, or a page's jsdom window) in another test
process, Node's coverage merge could not line the two up and reported
lines as uncovered that tests do run: lib/local-mode.js showed 40 % of
its lines covered while its own tests cover 89 %.
vm.compileFunction takes the file as the function body, so the offsets
match the file whichever way it is loaded. The libraries behave exactly
as before; only the coverage numbers become right (extension branch
coverage 79.1 % -> 84.0 %, lines 92.2 % -> 93.7 %, with no test added).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
privacy.js, ui/theme.js and content/connect.js were the three extension scripts no test loaded. The new tests check that the policy renders as elements (HTML in the file stays text), follows Kotiko's language and falls back to English, and says so when nothing can be read; that the theme is applied from the cache before storage answers, refreshed from storage, follows changes, and survives blocked site data and a failing storage read; and that the connect page hands the code to the background, reports done, expired or failed, takes the code out of the address, and does nothing on any other page. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
fast-check (a pinned dev dependency; the extension still ships no npm code) runs properties over generated input, 100 cases each with a new seed on every run, so CI keeps exploring: the matcher on any page and vocabulary (bounds, order, no overlaps, exactly the vocabulary's words, case-blind, leftmost-longest phrases, nothing across a block edge), server address normalising (never throws, idempotent, no user info, http only where it is local), the bulk list reader (any bytes and text, RFC 4180 round trip, UTF-8 and UTF-16 decoding, line lists), the backup (any file is safe to read, export then read is a fixed point, restoring twice changes nothing, no secret reaches a backup), casing, the word-list response check, and the privacy policy's Markdown. A failure prints the seed and path; FC_SEED and FC_PATH replay it. Assertion mode: npm test loads test/helpers/assert-mode.mjs, which sets globalThis.__KOTIKO_ASSERT__; the test loaders pass it on to every jsdom window and vm context. With it set, the matcher checks every scan's matches, the backup checks every restore plan, and the word-list check checks every filtered list, throwing when an invariant breaks. The shipped extension never sets the flag, and npm run perf doesn't load the file, so the budgets measure the code as it ships. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
prepare_input/1 replaced line breaks with ~r/[\t\n\r\v\f]/ in byte mode, where \v is any vertical space, the byte 0x85 (NEL) included. That byte is inside the UTF-8 of Å (C3 85), Ņ (C5 85) and other letters, so the text became invalid and the next step raised: POST /api/v1/words with "Åland" answered 500, and the bot replied with an internal error. The pattern is now Unicode-aware and names the vertical tab itself (\x0B), as \v means in the extension's JavaScript. A StreamData property on prepare_input/1 found it; the shared input fixture gains the case, so both runtimes check it, and the server has its own regression test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
GET /api/v1/words?lang[]=ru (and base, status, limit) and GET /api/v1/export?include[]=pending reach the router as a list or a map, which String.split/3 and Integer.parse/1 can't take, so the server answered 500. A non-string value is now refused with invalid_request, naming the field. Found by the new StreamData properties on the v1 router: any word body for POST /words, /words/batch and PATCH /words/:id, any profile and job control body, and any query string for /words, /export and the legacy list are answered below 500, with slice 25's error shape, and a batch answers every word by its index. test/support/gen.ex holds the shared generators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
mix test --cover now fails below 90 % of lines (test_coverage in mix.exs); the helpers in test/support are test code and don't count. Coverage goes from 87.8 % to 91.4 % with tests of what had none: the Telegram voice path end to end (download, transcription, lookup), the bot's commands, the janitor, pronunciation refresh and Wiktionary pass processes running a step when their timer fires and surviving one that raises, the lookup cache, word edge cases and the boot summary. Transcriber reads optional Req options (empty in production) so a test can stub it like Telegram and the model. StreamData properties, run by plain mix test: word validation and patches, input preparation and answer extraction (WordSpec), any .env parses to typed settings or lists its problems without echoing a secret (Config), language tags, text cleaning, UUIDv7, log redaction, and backup import and export round trips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
CI's extension job runs npm run coverage instead of npm test: the same tests, failing below 90 % of lines or 80 % of branches in extension/ (Node 22's --test-coverage-lines and --test-coverage-branches). The server job runs mix test --cover on the newest Elixir, failing below 90 % of lines (mix.exs), and plain mix test on 1.15.8, where line counts differ a little. The release build ran the extension's tests only. It now runs the same coverage gates for both halves, so the property tests (the dynamic analysis) run on every tree that is released. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
CONTRIBUTING.md's Tests section says which gates CI enforces and how to run them, that the server has no branch number (Elixir's cover counts lines), where the property tests live and how to replay a failure from its seed, and what the extension's assertion mode checks. The docs site's developer setup lists the coverage commands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
…into silver/gold-tests
# Conflicts: # .github/workflows/ci.yml
…r answers With this branch merged, these are Met, each with its evidence: - gold test_statement_coverage90 (server 91.6 % lines, extension 93.9 %, both gated) and test_branch_coverage80 (extension 84.5 %, gated; no FLOSS branch tool exists for Elixir); - dynamic_analysis and dynamic_analysis_enable_assertions (passing and gold, one shared answer): fast-check and StreamData properties in CI and the release workflow, and the extension's test-only assertion mode; - passing dynamic_analysis_fixed: every defect the properties found was fixed with a regression test. test_most and silver test_statement_coverage80 carry the new numbers, and the passing summary is recounted (62 Met, 1 Unmet, 4 N/A). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
…r gitleaks CI's gitleaks scan (full history) flagged "whisper-key-123" in voice_test.exs as a generic API key. It's a test value, but it broke the CONTRIBUTING rule that fake keys contain 0123456789, which is what tells the scanner they're fake. The test now uses whisper-key-0123456789, and .gitleaksignore records the one historical finding by its fingerprint with the reason, instead of rewriting signed commits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
The secrets job checks out with fetch-depth 0, which fetches every branch, and ran gitleaks with --log-opts=--all. So a finding on one pull request's branch failed every other pull request (#22 failed on a test key in #21's branch). It now scans HEAD: the whole history of main plus the pull request's own commits, which is everything that would land. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Ayla Croft <dev@scriptkittyos.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gold's test criteria, enforced. The property tests found three real bugs; all are fixed with regression tests.
mix test --cover)mix.exs)npm run coverage)fast-check4.10.2, a dev dependency): the matcher, URL checks, bulk-add parsing (CSV round trips, UTF-8/16), backup read/restore (export→read changes nothing, restoring twice changes nothing, no secret in a backup, a restore never changes where requests go), casing, the word-list check and the privacy page Markdown. Each run uses a new seed;FC_SEED/FC_PATHreplay one..envparsing never echoing a secret, language tags, UUIDv7, redaction, backup round trips.__KOTIKO_ASSERT__. The shipped extension never sets it.requireExtwrapped each library in a function, which shifted code offsets, so Node merged coverage wrongly (lib/local-mode.jsshowed 40 % instead of 89 %).vm.compileFunctionfixes it; this alone took branches from 79.1 to 84.0 %.npm run coverage. The server runsmix test --warnings-as-errorson 1.15.8 and adds--coveron the newest Elixir (line counts differ slightly between compilers). The release workflow now runs both before building.test_statement_coverage90,test_branch_coverage80,dynamic_analysis,dynamic_analysis_enable_assertions, and passingdynamic_analysis_fixedbecome Met.Bugs the properties found (shipped behaviour changes)
spec/fixtures/merge.jsongains the case.POST /api/v1/wordsanswer 500.\vwithout the Unicode flag matched byte 0x85, which is inside those letters' UTF-8. The fix is~r/[\t\n\r\x0B\f]/u, with a regression test inspec/fixtures/input.json.?lang[]=ru) answered 500. It now answers 400invalid_request, naming the field.Checked (after merging main, under #16's strict rules)
npm run lint: 0 errors.npm run coverage: 1942 pass (93.88 % lines, 84.46 % branches).mix format,compile --warnings-as-errorsandmix credo --strict("found no issues") pass.mix test --warnings-as-errors --cover: 23 properties, 1086 tests, 0 failures, 91.63 %.mix sobelow: exit 0.bestpractices-links --check(209 valid), DCO and forbidden-files checks pass.npm run perfall ok;npx playwright test99 passed, 4 skipped;npm testlooped 6 times with no failures.🤖 Generated with Claude Code