Skip to content

feat: implement .jitCode() support for Assertion actions#2170

Merged
rafal-hawrylak merged 1 commit into
mainfrom
jit_in_assertions
May 26, 2026
Merged

feat: implement .jitCode() support for Assertion actions#2170
rafal-hawrylak merged 1 commit into
mainfrom
jit_in_assertions

Conversation

@rafal-hawrylak

@rafal-hawrylak rafal-hawrylak commented May 21, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator
  • Add jit_code field to the Assertion message in protos/core.proto
  • Introduce JitAssertionResult type alias and jitCode method to Assertion builder (core/actions/assertion.ts)
  • Add strictly typed jitContextable parameter to Session.sqlxAction in core/session.ts
  • Update assertion compile() to handle jitCode execution bypassing standard string evaluation
  • Parse and apply jitContextable safely for assertions in core/session.ts
  • Add unit test coverage in core/main_test.ts to verify compilation output

@rafal-hawrylak rafal-hawrylak requested a review from a team as a code owner May 21, 2026 12:46
@rafal-hawrylak rafal-hawrylak requested review from Ceridan, andrzej-grudzien, apilaskowski and ikholopov-omni and removed request for a team May 21, 2026 12:46
ikholopov-omni
ikholopov-omni requested changes May 25, 2026
View reviewed changes
Comment thread core/session.ts Outdated
Comment thread core/session.ts Outdated
ikholopov-omni
ikholopov-omni approved these changes May 25, 2026
View reviewed changes
Comment thread core/actions/assertion.ts
@rafal-hawrylak
feat: implement .jitCode() support for Assertion actions
219873a 
- Add `jit_code` field to the `Assertion` message in `protos/core.proto`
- Fix missing protobuf imports and configuration in `protos/BUILD`
- Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`)
- Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts`
- Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation
- Parse and apply `jitContextable` safely for assertions in `core/session.ts`
- Add unit test coverage in `core/main_test.ts` to verify compilation output
@rafal-hawrylak rafal-hawrylak merged commit 8488238 into main May 26, 2026
10 checks passed
@rafal-hawrylak rafal-hawrylak deleted the jit_in_assertions branch May 26, 2026 07:04
spatel11 pushed a commit to spatel11/dataform that referenced this pull request Jul 8, 2026
@rafal-hawrylak @spatel11
feat: implement .jitCode() support for Assertion actions (dataform-co…
c8d5d6a 
…#2170)

- Add `jit_code` field to the `Assertion` message in `protos/core.proto`
- Fix missing protobuf imports and configuration in `protos/BUILD`
- Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`)
- Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts`
- Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation
- Parse and apply `jitContextable` safely for assertions in `core/session.ts`
- Add unit test coverage in `core/main_test.ts` to verify compilation output
apilaskowski pushed a commit that referenced this pull request Jul 13, 2026
@spatel11 @dependabot
@rafal-hawrylak @SuchodolskiEdvin @kolina @ikholopov-omni @aicayzer @AmmanuelT
ISSUE-2103 Handle trailing and trailing statement comments (#2108)
135dc27 
* ISSUE-2103 Handle trailing and trailing statement comments

* Bump vm2 from 3.10.2 to 3.11.3 (#2168)

* Bump vm2 from 3.10.2 to 3.11.3

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.10.2 to 3.11.3.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.10.2...v3.11.3)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix vm2 sandbox execution errors and CallSite path stripping

The upgrade of vm2 to 3.11.3 tightened sandbox security, stripping file path information from V8 CallSite objects during Error.prepareStackTrace and breaking the resolution of symlinked modules. This caused critical test failures ("Unable to find valid caller file") during Dataform compilation.

This commit resolves these issues by:
* Resolving `projectDir` to its absolute realpath before sandbox compilation to circumvent symlink boundaries (especially important for bazel runfiles).
* Injecting `global.__dataform_current_file` into the NodeVM compiler configuration with a try-finally block to track cross-boundary macro scopes.
* Adding a fallback in `core/utils.ts:getCallerFile` to read the injected global file path when the V8 stack trace path is stripped or undefined.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>

* Bump DF_VERSION from 3.0.56 to 3.0.57 (#2173)

* Support onSchemaChange for incremental tables (#2101)

* Support onSchemaChange for incremental tables

This change introduces support for the onSchemaChange option in incremental tables for the BigQuery adapter. It adds the incrementalSchemaChangeBody() strategy to handle schema changes.

End-to-end tests have been created to verify the new functionality.

* fix: Private functions moved after public functions

* - Split onSchemaChange queries to sub-tasks
- Split sql queries to sun-functions
- Fixed function names

* fix: Address feedback for PR #2101 and fix presubmit failures

- Updated e2e tests - Added golden files - Consolidated merge and insert functions

* feat: implement .jitCode() support for Assertion actions (#2170)

- Add `jit_code` field to the `Assertion` message in `protos/core.proto`
- Fix missing protobuf imports and configuration in `protos/BUILD`
- Introduce `JitAssertionResult` type alias and `jitCode` method to `Assertion` builder (`core/actions/assertion.ts`)
- Add strictly typed `jitContextable` parameter to `Session.sqlxAction` in `core/session.ts`
- Update assertion `compile()` to handle `jitCode` execution bypassing standard string evaluation
- Parse and apply `jitContextable` safely for assertions in `core/session.ts`
- Add unit test coverage in `core/main_test.ts` to verify compilation output

* fix: Resolve presubmit failure introduced by PR #2101

- extend onSchemaChange e2e test timeout (#2175)

* Bump DF_VERSION from 3.0.57 to 3.0.58 (#2176)

* Guard workflow_settings.yaml and dataform.json require with existence check (#2178)

* Refactor CompileChildProcess to use BaseWorker (#2179)

* Rename worker timeout error to "Compilation timed out" (#2186)

* Extract task creation into executionSql helpers (#2180)

* Add JiT compilation support for assertion actions (#2193)

Adds JIT_COMPILATION_TARGET_TYPE_ASSERTION alongside the existing
TABLE / OPERATION / INCREMENTAL_TABLE target types, plus a matching
JitAssertionResult { string query } proto and a jitCompileAssertion()
dispatcher in core/jit_compiler.ts. Assertions reuse SqlActionJitContext
since AssertionContext implements IActionContext, and the result is a
single SELECT query string (matching the existing
JitAssertionResult = string alias in core/actions/assertion.ts).

* Fix caller-file error (#2177) and enforce Core/CLI version match (#2191)

vm2 3.11.3 strips file paths from V8 CallSite objects inside the sandbox,
so `getCallerFile()` in @dataform/core can no longer resolve the current
file from the stack. Track it via a host-side stack exposed through
`__df_enter`/`__df_exit`/`__df_current` sandbox helpers and read it through
a getter on `global.__dataform_current_file`.

For @dataform/core <= 3.0.56 (no global fallback in getCallerFile), patch
the bundle text at load time to consult the global. Gated on the installed
Core version so newer bundles are never touched.

Add a CLI/Core compatibility check: reject installed Core whose major
differs from the CLI's, or whose minor is below the CLI's minor. Resolve
the installed Core's version by running
`require("@dataform/core").version` inside the index-generator vm so any
install layout (package.json, workflow_settings.yaml stateless install,
JiT) reports the version actually loaded. The error tells the user the
exact `dataformCoreVersion:` line to set in `workflow_settings.yaml`.
Skipped when the CLI version can't be determined (unbundled local dev).

Mirror the same wrapper pattern in `testing/run_core.ts`.

Add `compile rejects @dataform/core with incompatible version` test in
`cli/index_compile_test.ts` that drives the workflow_settings.yaml
stateless install path with dataformCoreVersion: "2.9.0" (latest 2.x on
the registry) to exercise the real `npm i` install flow.

Add `compile succeeds with @dataform/core <= 3.0.56 via caller-file shim`
test that installs @dataform/core@3.0.50 and asserts the compiled action's
`fileName` matches the source path, end-to-end proving the bundle-text
patch and host-side file stack restore getCallerFile under vm2 3.11.3.

* Raise default compile timeout to 60s and improve timeout error message (#2192)

Default compilation timeout was 30s. Bump to 60s so larger projects
don't hit it on a single compile pass.

Extends the "Compilation timed out" error to point at --timeout with
concrete examples (--timeout=2m, --timeout=1h) so users can self-mitigate
without reading docs.

* Bump DF_VERSION from 3.0.58 to 3.0.59 (#2194)

* Bump vm2 from 3.11.3 to 3.11.4 (#2188)

Bumps [vm2](https://github.com/patriksimek/vm2) from 3.11.3 to 3.11.4.
- [Release notes](https://github.com/patriksimek/vm2/releases)
- [Changelog](https://github.com/patriksimek/vm2/blob/main/CHANGELOG.md)
- [Commits](patriksimek/vm2@v3.11.3...v3.11.4)

---
updated-dependencies:
- dependency-name: vm2
  dependency-version: 3.11.4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Extract BigQueryClientProvider for injectable BQ clients (#2181)

* Extract BigQueryClientProvider for injectable BQ clients

* Address review: collapse BigQueryClientProvider to a function type

* Bump protobufjs from 7.5.5 to 7.5.8 (#2172)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.5 to 7.5.8.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.5.8/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.5...protobufjs-v7.5.8)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.5.8
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add go_package option to db_adapter.proto and jit.proto (#2197)

* Migrate to bazel 7.3.2 (#2187)

* Migrate to bazel 7.3.2

* Use fixed ref for github action

* Update gcloud to 572.0.0 (#2207)

KMS has an org-policy enabled, requiring the usage of client certificate aware endpoints when performing the decryption of credentials.

Updating gcloud, so that it uses an updated compatible endpoint.

* Add metadata field to AssertionConfig (#2208)

Mirrors TableConfig.metadata / ViewConfig.metadata so callers can
attach key/value data that lands on
Assertion.action_descriptor.metadata.extra_properties.

* Bump DF_VERSION from 3.0.59 to 3.0.60 (#2209)

* Ignore trailing positional arguments in the global flag parser (#2199)

The flag parser in common/flags/index.ts runs in a static initialiser at
module load and parses process.argv. Once it had seen a --flag, any later
token that was neither another flag nor a flag value caused it to throw
"Arg neither flag name nor flag value", crashing the entire CLI before yargs
ran. yargs accepts positional arguments after flags (for example
`dataform run --some-flag value my_project`), so this lightweight parser was
rejecting valid argument orderings.

Ignore any token that is neither a flag nor a flag value instead of throwing,
exactly as already happened for positional arguments before the first flag;
yargs remains responsible for parsing positionals. Extract the parsing loop
into an exported parseArgv function so it can be unit-tested in isolation, and
add common/flags/index_test.ts covering the regression and related orderings.

Fixes #2198.

* Bump protobufjs-cli from 1.0.0 to 1.3.3 (#2202)

* Bump protobufjs-cli from 1.0.0 to 1.3.3

Bumps [protobufjs-cli](https://github.com/protobufjs/protobuf.js) from 1.0.0 to 1.3.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/master/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-cli-v1.0.0...protobufjs-cli-v1.3.3)

---
updated-dependencies:
- dependency-name: protobufjs-cli
  dependency-version: 1.3.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix(tools): patch generated proto typings to use ES6 import for 'long'

This avoids syntax errors in older rollup-plugin-dts which does not support
'import Long = require("long")' syntax. We patch the generated ts.d.ts file
to use 'import Long from "long"' instead.

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nick Nalivaika <ikolina@google.com>

* Bump js-yaml from 4.1.1 to 4.2.0 (#2205)

Bumps [js-yaml](https://github.com/nodeca/js-yaml) from 4.1.1 to 4.2.0.
- [Changelog](https://github.com/nodeca/js-yaml/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nodeca/js-yaml/commits)

---
updated-dependencies:
- dependency-name: js-yaml
  dependency-version: 4.2.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat: add getContents() for reading external markdown files into table descriptions pr fix (#2164)

* initial attempt at md reader

* adding relevant tests and better path resolving for md

* cleaner get contents path handling

* escaping newlines for md

* adding documentation

* add check if file in rootdir for get contents and update documentation on getContents to include limits

* cleaner and filesystem agnostic check if getContents target is in the root  dir

* making sure root dir has seperator on check if getcontents file is in dir

* removing node module usage

* removing redundant normalize and root variable

* adding seperator when checking if file is in rootDir

* Bump protobufjs from 7.5.8 to 7.6.3 (#2204)

Bumps [protobufjs](https://github.com/protobufjs/protobuf.js) from 7.5.8 to 7.6.3.
- [Release notes](https://github.com/protobufjs/protobuf.js/releases)
- [Changelog](https://github.com/protobufjs/protobuf.js/blob/protobufjs-v7.6.3/CHANGELOG.md)
- [Commits](protobufjs/protobuf.js@protobufjs-v7.5.8...protobufjs-v7.6.3)

---
updated-dependencies:
- dependency-name: protobufjs
  dependency-version: 7.6.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(gcloud): support client certificate environment in gcloud actions and tests (#2213)

- Enable `use_default_shell_env` in `gcloud_secret` rule to pass host environment variables (like `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE`) to the gcloud tool.
- Pass `CLOUDSDK_CONTEXT_AWARE_USE_CLIENT_CERTIFICATE` to bazel test in `scripts/publish`.

* Prevent .sqlx files in actions.yaml configs with a clear error (#2189)

Putting a .sqlx file under the ``filename`` field of an entry in
``definitions/actions.yaml`` used to fall through to ``nativeRequire``,
which then failed with a cryptic error far from the source of the
mistake. ``.sqlx`` files are loaded directly from the ``definitions/``
directory by the sqlx compiler, not referenced through ``actions.yaml``.

Add an explicit validation step in ``loadActionConfigs`` that runs before
each action is constructed. If the action's ``filename`` ends in
``.sqlx`` (case-insensitive), emit a ``compileError`` naming the action
type, the offending filename, and pointing at the fix. The action is
then skipped so we don't double-up with a downstream ``nativeRequire``
error.

The check covers every action type that accepts a ``filename``: ``table``,
``view``, ``incrementalTable``, ``assertion``, ``operation``,
``declaration``, ``notebook`` and ``dataPreparation``. Data preparations
are included because a ``.dp.sqlx`` file is also auto-compiled from
``definitions/`` - referencing one from ``actions.yaml`` doesn't resolve
its path, ignores its ``config {}`` block and double-registers the
action, so it's the same mistake. ``.dp.yaml`` data preparations are
unaffected.

Closes #1785

* Bump tmp from 0.2.4 to 0.2.7 (#2183)

Bumps [tmp](https://github.com/raszi/node-tmp) from 0.2.4 to 0.2.7.
- [Changelog](https://github.com/raszi/node-tmp/blob/master/CHANGELOG.md)
- [Commits](raszi/node-tmp@v0.2.4...v0.2.7)

---
updated-dependencies:
- dependency-name: tmp
  dependency-version: 0.2.6
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump @tootallnate/once from 2.0.0 to 2.0.1 (#2171)

Bumps [@tootallnate/once](https://github.com/TooTallNate/once) from 2.0.0 to 2.0.1.
- [Release notes](https://github.com/TooTallNate/once/releases)
- [Changelog](https://github.com/TooTallNate/once/blob/v2.0.1/CHANGELOG.md)
- [Commits](TooTallNate/once@2.0.0...v2.0.1)

---
updated-dependencies:
- dependency-name: "@tootallnate/once"
  dependency-version: 2.0.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump fast-uri from 3.0.6 to 3.1.2 (#2161)

Bumps [fast-uri](https://github.com/fastify/fast-uri) from 3.0.6 to 3.1.2.
- [Release notes](https://github.com/fastify/fast-uri/releases)
- [Commits](fastify/fast-uri@v3.0.6...v3.1.2)

---
updated-dependencies:
- dependency-name: fast-uri
  dependency-version: 3.1.2
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove extra utils_test.ts from api build

* allow pbts command to run on mac

* refactor concatenateQueries to be a bit more readable

* update on_schema_change tests to reflect the new format

* update format for api spec

* replace drop table in statement comment test because it was causing a test failure

* add new tests for comments within a query

* update test count in bigquery spec

* multi line comment tests

* update new golden tests to updated format

* fix bigquery spec test to add ending semi colons

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Stephen <spatel11@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Rafal Hawrylak <hawrylak@google.com>
Co-authored-by: rafal-hawrylak <33635872+rafal-hawrylak@users.noreply.github.com>
Co-authored-by: Edvin <56413069+SuchodolskiEdvin@users.noreply.github.com>
Co-authored-by: Nick Nalivayka <ikolina@google.com>
Co-authored-by: ikholopov-omni <73560138+ikholopov-omni@users.noreply.github.com>
Co-authored-by: August Cayzer <august@cayzer.me>
Co-authored-by: Amman <87317538+AmmanuelT@users.noreply.github.com>
Co-authored-by: August Cayzer <a@cyzr.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ikholopov-omni ikholopov-omni ikholopov-omni approved these changes
@Ceridan Ceridan Awaiting requested review from Ceridan Ceridan is a code owner automatically assigned from dataform-co/dataform-reviewers
@apilaskowski apilaskowski Awaiting requested review from apilaskowski
@andrzej-grudzien andrzej-grudzien Awaiting requested review from andrzej-grudzien

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rafal-hawrylak @ikholopov-omni