Derive a mixed-language marker for Dags with task.stub - #71213
Draft
jason810496 wants to merge 1 commit into
Draft
Derive a mixed-language marker for Dags with task.stub#71213jason810496 wants to merge 1 commit into
jason810496 wants to merge 1 commit into
Conversation
boring-cyborg
Bot
added
area:DAG-processing
area:dev-tools
backport-to-v3-3-test
jason810496
force-pushed
the
feature/lang-sdk/mixed-language-dag-flag
branch
from
August 6, 2026 05:48
2dff735 to
68c208e
Compare
A Lang-SDK artifact can either define a Dag natively or only supply the implementations behind a Python Dag's @task.stub tasks. A Dag processor has to treat those oppositely, persisting the first and discarding the second since the Python Dag already owns that dag_id, and nothing in the serialized Dag told them apart. The flag is derived, never authored: a Python Dag reports it from the presence of stub tasks, and neither the constructor nor attribute assignment can set it, so a Dag cannot claim to be mixed-language without actually holding one.
jason810496
force-pushed
the
feature/lang-sdk/mixed-language-dag-flag
branch
from
August 6, 2026 06:06
68c208e to
8e0c2a4
Compare
jason810496
added
area:coordinator
jason810496
marked this pull request as ready for review
August 6, 2026 06:15
jason810496
requested review from
amoghrajesh,
ashb,
bolkedebruin,
bugraoz93,
gopidesupavan,
jscheffl,
kaxil and
potiuk
as code owners
August 6, 2026 06:15
jason810496
changed the title
uranusjr
reviewed
Aug 6, 2026
| Always derived from the Dag's tasks, never author-set, so that a Lang-SDK Dag processor can | ||
| tell a Dag that merely backs stub tasks from one authored natively in that language. | ||
| """ | ||
| return any(task.operator_name == STUB_OPERATOR_NAME for task in self.task_dict.values()) |
Member
There was a problem hiding this comment.
Operator name should not be used for logical conditions. Add a flag instead.
jason810496
marked this pull request as draft
August 7, 2026 10:03
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Lang-SDK artifact can play two roles that a Dag processor must treat oppositely, and nothing in the serialized Dag tells them apart:
@task.stubFor a native Dag the processor parses the artifact and the result is persisted. For a mixed-language Dag it must persist nothing: the Python Dag carrying the
@task.stubtasks already owns thatdag_id, so persisting the Lang-SDK side too would give conflicts.How
The new
is_mixed_language_dagflag at Dag level solves the problem.is_mixed_language_dagflag at Dag level argument.is_mixed_language_dagfield is a compute attribute that if there's anyStubOperatorin the Dag, the property will be true.DAGgains nothing, a Python Dag can never set theis_mixed_language_dagflag.The further mixed language Dag processing flow will be like:
JavaDagImporterdon't emit those Dag withis_mixed_language_dag=Trueby default.PythonDagImporter-> for all Dags whichis_mixed_language_dag=True(the flag will be on if there's StubOperator in the Dag) -> validate the corresponding Java Dag structure (viaTI.queue-> Coordinator ->JavaDagImporter(discover_mixed_language_dags=True)-> Java Dag structure) -> raise import error if there's validation errorWas generative AI tooling used to co-author this PR?