Repository navigation
Conversation
…r file The core suite spent more worker time in the Postgres harness than in test bodies. Two costs made that time. Each test truncated all 25 tables, about 12 ms, even when no table had rows. The reset now truncates only the tables that have rows. It moves from the setup file of packages/junior to @sentry/junior-testing, so every package gets the same reset. Each test file drops and clones a database, and Postgres does a checkpoint for each drop. CI now keeps the Postgres data directory in memory, so a checkpoint does not wait for the disk. Each file still gets a new database. The check that the harness only runs on a local server had three copies. It is now in createPostgresHarnessConfig. Local runs of the whole core suite, four workers, on a new Postgres: 198 s before, 128 to 130 s with both changes. Two runs had no timeout. Refs #2001 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The integration evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37977349844. The evals of this commit replay them in strict mode. Eval-Suite: integration
Sentry Evalsjunior-behavioral
Base commit junior-guardian
Base commit junior-integration
Base commit junior-router
Base commit |
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/37977349924. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
pnpm installs a copy of each workspace package in node_modules. It updates a copy only after the build script of that package, and @sentry/junior-testing has no build, because tests read its TypeScript source. So the copy keeps old code after a pull or a checkout, and pnpm install does not update it. Tests and typecheck then run against an old harness. Each Vitest config that uses the package now resolves it from source, as the configs already do for junior-plugin-api and junior-memory. The two tsconfig files that typecheck tests map it too. Refs #2001 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/38007002624. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
The eval had no recording, so each run made its model requests live. The last CI run failed on the judge score of a live answer, and CI dropped the recording. A local live run passed, and these are its recordings. Refs #2001 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The integration evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/38015297836. The evals of this commit replay them in strict mode. Eval-Suite: integration
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/38015297818. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
dcramer
marked this pull request as ready for review
October 10, 2026 15:40
The reset truncated only the tables that had rows. That check does not see rows of a transaction that is still open, so a test that leaves one open could pass its rows to the next test. A TRUNCATE of every table waits for that transaction and has no such gap. Refs #2001 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dcramer
changed the title
The behavioral evals recorded these model responses in https://github.com/getsentry/junior/actions/runs/38065716902. The evals of this commit replay them in strict mode. Eval-Suite: behavioral
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Postgres test harness is now in one place, and CI runs it on a Postgres that keeps its data in memory. Isolation does not change: each test file gets a new database, and each test starts with every table truncated.
packages/juniorto@sentry/junior-testing/postgres, with no change to what it does. Every package that uses the harness gets the same reset.createPostgresHarnessConfig.@sentry/junior-testinginnode_modulesand does not update it after a pull, so a checkout could run an old harness. The Vitest configs and the two tsconfig files that typecheck tests now resolve it from source, as they already do forjunior-plugin-apiandjunior-memory. With the copy broken on purpose, typecheck and the tests of all five packages that use it pass.Vitest duration of the core jobs in CI, one sample each. CI times vary a lot between runs, so read these as a direction:
An earlier version of this pull request truncated only the tables that had rows. It is removed: that check does not see rows of a transaction that is still open.
Local runs do not get faster with this change.
compose.yamlkeeps one Postgres on disk for tests and for the development database.One commit adds the recordings of the Google Calendar free/busy eval, which
maindid not have. The other recording commits are from the eval bot.Refs #2001
🤖 Generated with Claude Code