Let TriggerDagRunOperator own its execution via a new accessor - #69135
Open
1fanwang wants to merge 7 commits into
Open
Let TriggerDagRunOperator own its execution via a new accessor#691351fanwang wants to merge 7 commits into
1fanwang wants to merge 7 commits into
Conversation
1fanwang
force-pushed
the
triggerdagrun-owns-execution
branch
2 times, most recently
from
June 29, 2026 17:25
35cc421 to
ea1d794
Compare
1fanwang
marked this pull request as draft
June 29, 2026 17:28
1fanwang
force-pushed
the
triggerdagrun-owns-execution
branch
from
June 30, 2026 00:04
ea1d794 to
32d5c54
Compare
1fanwang
marked this pull request as ready for review
June 30, 2026 00:04
Contributor
|
Hey, thanks for this big refacto. airflow 2.11 is EOL , so maybe you could remove eveyrthing related to 2.11 ? |
Airflow already exposes the dag-run poll half to task code as ti.get_dagrun_state(); the trigger half had no first-class accessor and was only reachable through the DagRunTriggerException side channel. Adding the symmetric ti.trigger_dag_run() routes a trigger through the same execution-API endpoint and scoped token the task runner already uses, so an operator can own its trigger-and-wait execution directly.
1fanwang
force-pushed
the
triggerdagrun-owns-execution
branch
2 times, most recently
from
June 30, 2026 23:57
94e9525 to
c320bd5
Compare
…ble reconnect On Airflow 3, TriggerDagRunOperator.execute() raised DagRunTriggerException and the task runner did the trigger and the wait loop, so the synchronous wait-and-reconnect contract was duplicated between the operator and the runner and could drift. With the new ti.trigger_dag_run() accessor the operator does the submit and poll itself and reuses ResumableJobMixin directly, keeping that contract in one place. The opt-in durable flag persists the triggered run id before polling so a worker crash mid-wait reconnects to the in-flight run on retry instead of triggering a duplicate. Deferrable still needs the triggerer handoff, so it keeps the exception path; Airflow < 3.3 and Airflow 2 are unchanged.
On Airflow 3.3+ the operator owns its synchronous trigger and wait through the execution-API accessors, so the task instance no longer reaches the task runner's _handle_trigger_dag_run for that path. Two tests still asserted the old task-runner call sequence and failed: assert the resolved task state in the wait test (keyed by message type, so it holds on both paths and is timezone-independent), and pin the original-error guard to the < 3.3 task-runner path it actually covers. The 3.3+ behaviour is verified in the provider test_trigger_dagrun.py.
ResumableJobMixin only exists on Airflow 3.3+, so the operator carried an inline import-or-stub to keep it in its bases on older Airflow. That kind of version-conditional compatibility logic is what common.compat exists to hold, so the standard operator's regular code can stay focused on the 3.3+ accessor path it now owns and the back-compat retires cleanly once the provider's floor reaches 3.3. The stub moves next to the existing BaseAsyncOperator stub in common.compat.standard.operators; behaviour is unchanged on every supported Airflow version.
1fanwang
force-pushed
the
triggerdagrun-owns-execution
branch
from
July 1, 2026 00:44
c320bd5 to
4b8a2fd
Compare
The execution-API accessors this path relies on (ti.trigger_dag_run / ti.get_dagrun_state) are introduced here, so they first ship in 3.4.0. Gating on 3.3 made the provider call methods that do not exist on released Airflow 3.3, which failed the OpenLineage compat run with an AttributeError. Signed-off-by: 1fanwang <1fannnw@gmail.com>
The operator-owned execution path is gated on Airflow 3.4, but a task-SDK test still patched the 3.3 symbol on the operator module, so it raised AttributeError instead of exercising the pre-3.4 task-runner branch it is meant to cover. Signed-off-by: 1fanwang <1fannnw@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opened for the
[DISCUSS] Let TriggerDagRunOperator own its execution logicthread, so the accessor direction can be weighed as an actual diff rather than in the abstract — see Scope for what it does and doesn't cover.Why
On Airflow 3,
TriggerDagRunOperator.execute()raisesDagRunTriggerExceptionand the task runner does the trigger and the synchronous wait. So the trigger-and-wait contract (and any crash recovery) lives in two places that can drift. The poll half is already a first-class Task SDK accessor (ti.get_dagrun_state()); only the trigger half was missing, so it had to go through the exception side channel.What
ti.trigger_dag_run(), the counterpart toti.get_dagrun_state()— same execution-API endpoint and scoped token the runner already uses, so no new authz surface.ResumableJobMixin, so the durability contract lives in one place.durableflag (defaultFalse): the triggered run id is persisted before polling, so a worker crash mid-wait reconnects to the in-flight run on retry instead of triggering a duplicate.ResumableJobMixinshim (real on 3.3+, a stub below) lives incommon.compat.standard, so the operator body stays on the clean 3.3+ path and the older-version back-compat retires as a contained delete once the provider floor reaches 3.3.DagRunTriggerExceptionpath.Scope
This is the 3.3+ end-state, not a drop-in for the full supported range: the operator owns execution on 3.3+, while 3.0–3.2 and 2.11 stay on today's path, so durability applies on 3.3+ only. Whether and when standard can require 3.3+ (and drop the fallbacks) is the provider
min_versionquestion under discussion on the thread. The lower-duplication interim that keeps a single runner path is #68952 / #68955.Testing done
Standard provider suite: 41 passed, 41 skipped (the skips are Airflow-2-only, skipped on 3.4). The existing synchronous-path tests are re-homed to assert the accessor calls instead of the raised exception; deferrable / Airflow-2 tests are unchanged. New coverage: the accessor, the operator-owned path (trigger, skip/reset, sync wait success/failure, fire-and-forget), and the four durable states (persist-before-poll, reconnect, short-circuit an already-succeeded prior run, resubmit after a failed one). Task SDK accessor/runner tests pass.
Live E2E — Airflow 3.4 standalone (LocalExecutor + Postgres). Two DAGs: a parent
TriggerDagRunOperator(wait_for_completion=True, durable=True, poke_interval=3)triggers a child whose only task sleeps 45s.1. Clean path (no crash) — the operator owns execution via the accessors; the runner's
DagRunTriggerExceptionpath is never taken.2. Worker
SIGKILLmid-wait (durable reconnect) — the parent's worker iskill -9ed while it polls; the scheduler retries the task.SIGKILLthe parent workerRaw state progression — crash run
Refs
The accessor-based option discussed on the
[DISCUSS]thread, alongside the hand-rolled-in-the-runner durability in #68936 and the share-the-core refactor in #68952 / #68955.Was generative AI tooling used to co-author this PR?