PHOENIX-7907 :- Cutover lifecycle with PENDING_PARTIAL_PASS state, UCF-wait, and link removal at cutover commit - #2592
Open
lokiore wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Introduces a restartable transform cutover lifecycle that delays partial repair until after cache refresh, removes dual-write links during cutover, and persists the wait deadline.
Changes:
- Adds pending/running partial-pass states with retry handling.
- Removes base-table and view dual-write links during cutover.
- Extends
SYSTEM.TRANSFORMand adds lifecycle tests.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
MetaDataUtilTest.java |
Tests system-table timestamp invariant. |
TransformMonitorTaskWaitTest.java |
Tests wait-bound arithmetic. |
CutoverLifecycleIT.java |
Adds cutover lifecycle integration coverage. |
Transform.java |
Removes links during cutover. |
TransformMonitorTask.java |
Implements partial-pass state machine. |
TransformClient.java |
Reads and writes the deadline. |
SystemTransformRecord.java |
Models the deadline and states. |
PTable.java |
Adds lifecycle statuses. |
QueryConstants.java |
Extends transform-table DDL. |
ConnectionQueryServicesImpl.java |
Adds upgrade migration. |
PhoenixDatabaseMetaData.java |
Defines the new column name. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+204
to
+206
| updateTransformRecordClearingJobId(conn, systemTransformRecord, | ||
| PTable.TransformStatus.PARTIAL_PASS_RUNNING, waitUntilTs); | ||
| conn.commit(); |
Comment on lines
+400
to
+403
| String logicalTableName = SchemaUtil.getTableName(systemTransformRecord.getSchemaName(), | ||
| systemTransformRecord.getLogicalTableName()); | ||
| PTable logicalTable = conn.getTable(systemTransformRecord.getTenantId(), logicalTableName); | ||
| updateCacheFrequency = logicalTable.getUpdateCacheFrequency(); |
Comment on lines
+565
to
+569
| assertTrue( | ||
| "Expected to observe the running partial pass in PARTIAL_PASS_RUNNING " | ||
| + "or a legitimately-completed transform", | ||
| observedRunning | ||
| || PTable.TransformStatus.COMPLETED.name().equals(completed.getTransformStatus())); |
Comment on lines
+851
to
+852
| @Test | ||
| public void testSystemTransformHasPendingPartialPassColumn() throws Exception { |
lokiore
marked this pull request as ready for review
August 19, 2026 18:55
lokiore
force-pushed
the
PHOENIX-7907/cutover-lifecycle
branch
2 times, most recently
from
August 26, 2026 17:04
1bb6ffb to
97e4321
Compare
…F-wait, and link removal at cutover commit Replace the inline partial pass that previously ran during transform cutover with an explicit, restartable cutover state machine in TransformMonitorTask, and tear down the dual-write links at the cutover commit so a single client cache-invalidation cycle propagates both the physical-table pointer swap and the dual-write shutoff. Lifecycle (all states committed to SYSTEM.TRANSFORM, monitored by the self-healing TransformMonitorTask): * PENDING_CUTOVER -> PENDING_PARTIAL_PASS: capture the cutover instant (CUTOVER_TS) BEFORE doCutover, then after doCutover swaps the physical-table pointer, persist a wait deadline instead of running the partial pass immediately. The deadline is the logical table's update-cache-frequency scaled by a safety margin (1.10), floored at 30 minutes and capped at 24 hours, so clients still holding a cached pointer to the old physical table refresh before the partial pass runs and late writes are not stranded as unverified rows. The raw frequency is clamped to the 24-hour ceiling BEFORE scaling because a table configured to never refresh its cache resolves its update-cache-frequency to Long.MAX_VALUE; scaling that and adding it to the current time would saturate into a negative (past) deadline that would defeat the wait entirely. The deadline is stored in a new nullable BIGINT column PENDING_PARTIAL_PASS_UNTIL_TS on SYSTEM.TRANSFORM. * PENDING_PARTIAL_PASS -> PARTIAL_PASS_RUNNING: once the wait window elapses, commit the transition (clearing the inherited full-pass job id so the monitoring branch cannot mistake the already-successful full pass for the partial pass and complete early), then kick the partial-pass TransformTool run. * PARTIAL_PASS_RUNNING -> COMPLETED / FAILED: monitor the partial-pass job. A job that cannot be confirmed successful (no job id registered because the initial kick failed before its STARTED transition, an unsuccessful job, or a job id that no longer resolves) is routed through a retry budget; once retries are exhausted the record reaches terminal FAILED rather than stranding a pointer-swapped table with unverified rows. Partial-pass repair floor (correctness): the partial pass re-verifies rows on the new physical table from a lower-bound timestamp. Deriving that floor from the post-wait lastStateTs stranded every write made to the old physical pointer during the [cutover, cutover + waitWindow] window -- exactly the writes the wait exists to let stale-cached clients drain -- because that floor sits past the window. The floor is instead derived from CUTOVER_TS (captured before doCutover, the most conservative instant) minus one so it is inclusive of writes stamped exactly at cutover; repairScanFloor centralizes this and falls back to lastStateTs only for pre-existing records that predate the CUTOVER_TS column. CUTOVER_TS is persisted durably (in its own commit) BEFORE doCutover swaps the pointer, and preserved across every downstream transition by the record copy-constructor. doCutover commits the pointer swap durably, so persisting CUTOVER_TS only in the later PENDING_PARTIAL_PASS commit would leave a crash window in between: a crash there would lose the instant and, on re-entry, recapture a later one that pushes the repair floor past the real cutover and silently drops the post-cutover-window writes. The PENDING_CUTOVER handling therefore commits CUTOVER_TS first and, on re-entry, reuses the already-persisted instant (resolveCutoverTs) rather than recapturing; a crash before that first commit is harmless because the pointer has not yet swapped. Dual-write link teardown at cutover commit (Transform.doCutover): * The base-table TRANSFORMING_NEW_TABLE link is deleted uncommitted and batched into the same commit as the base-table pointer swap. * Each child view's link is deleted inside the existing MUTATE_BATCH_SIZE view loop, paired with that view's pointer swap. A base table can have millions of views, so folding per-view link teardown into the bounded batch loop keeps every commit bounded while still pairing each link removal with its swap in one cache-invalidation cycle. doGetTable attaches the transforming-new-table per row from link presence, so a surviving view link would keep dual-write alive for view-routed writes after cutover. Schema: two nullable BIGINT columns are added to SYSTEM.TRANSFORM -- PENDING_PARTIAL_PASS_UNTIL_TS (the wait deadline) and CUTOVER_TS (the cutover instant used as the partial-pass repair floor); they are the first columns ever added to SYSTEM.TRANSFORM. A column added to a system table on upgrade only takes effect if SYSTEM.CATALOG's own header timestamp advances to the new min system-table timestamp, because the client upgrade gate reports SYSTEM.CATALOG's header timestamp: a min not backed by a genuine SYSTEM.CATALOG column-add at that timestamp leaves the catalog below the gate after an in-place upgrade and loops clients on UpgradeRequiredException. MIN_SYSTEM_TABLE_TIMESTAMP_5_4_0 is therefore bumped by one and a no-op marker column, UPGRADE_TS_ANCHOR_5_4_0, is added to SYSTEM.CATALOG at the new min so the header genuinely advances; the existing 5.4.0 SYSTEM.CATALOG column-add cascade is re-offset by one so every column keeps its original absolute timestamp (INDEX_CONSISTENCY stays at min-1, the anchor lands at min). The two transform columns are then added unconditionally via idempotent addColumnsIfNotExists; the previous strict-less-than gate is dropped because it was unreachable on a cluster already snapshotted at the pre-bump timestamp and would have left the columns unadded (ColumnNotFoundException on every transform read thereafter). A fresh install gets the anchor from the SYSTEM.CATALOG CREATE DDL and both transform columns from the SYSTEM.TRANSFORM CREATE DDL. SystemTransformRecord / TransformClient read and write both new columns with explicit BIGINT null handling. retry-count accounting: TransformTool's STARTED transition unconditionally increments (and auto-commits) the retry count. The first partial-pass kick pre-decrements to net zero (the initial pass is not a retry and must not consume budget); a genuine retry skips the decrement so the count strictly increases and the retries-exhausted -> FAILED transition stays reachable. Testing: CutoverLifecycleIT drives real and seeded cutovers with an injected clock (no real 30-minute sleep) and a job-lookup seam, covering the happy path (mutable / immutable / secondary-index / child-view tables), the wait-deadline honoring, the inherited-job-id clearing, the strand regressions (null job id, not-found job, retries exhausted) each asserting a terminal state, a never-cached table (update-cache-frequency NEVER) yielding a bounded future wait deadline rather than an overflowed past one, and a repair-floor regression (testPartialPassRepairFloorCoversPostCutoverWaitWindow) asserting CUTOVER_TS is captured at cutover, preserved across the transition to PARTIAL_PASS_RUNNING, and strictly precedes the post-wait lastStateTs -- the exact interval that a lastStateTs-derived floor would strand, and a re-entry regression (testCutoverReentryReusesPersistedCutoverTs) that resets a pointer-swapped record back to PENDING_CUTOVER with its CUTOVER_TS preserved, advances the clock far past it, re-runs the monitor, and asserts the persisted instant is reused unchanged rather than recaptured at the later clock. TransformMonitorTaskWaitTest is a fast unit test that pins the clamp-before-scale wait arithmetic (boundedPartialPassWaitMs) across the whole input domain (Long.MAX_VALUE / zero / negative / mid-range / at-and-above ceiling) so the deadline is always bounded and positive, the repair floor (repairScanFloor) resolving to cutoverTs-1, to the lastStateTs-1 fallback, and to 0 when neither is set, and the cutover-instant resolution (resolveCutoverTs) reusing a persisted instant on re-entry and capturing the current time on a first run. MetaDataUtilTest asserts the min system-table timestamp equals the highest timestamp SYSTEM.CATALOG's header reaches and pins its absolute offset, guarding against a future timestamp bump not backed by a SYSTEM.CATALOG column-add. Heavy user-table cutover ITs run on CI; the seeded strand and repair-floor regressions run locally. Generated-by: Claude Code (Opus 4.8) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lokiore
force-pushed
the
PHOENIX-7907/cutover-lifecycle
branch
from
August 26, 2026 18:02
97e4321 to
fefdf82
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This replaces the inline partial pass that previously ran during transform cutover with an explicit, restartable cutover state machine in
TransformMonitorTask, and tears down the dual-write links at the cutover commit so a single client cache-invalidation cycle propagates both the physical-table pointer swap and the dual-write shutoff.Lifecycle (each state is committed to
SYSTEM.TRANSFORMand monitored by the self-healingTransformMonitorTask):PENDING_CUTOVER -> PENDING_PARTIAL_PASS— afterdoCutoverswaps the physical-table pointer, persist a wait deadline instead of running the partial pass immediately. The deadline is the logical table's update-cache-frequency scaled by a safety margin (1.10), floored at 30 minutes and capped at 24 hours, stored in a new nullableBIGINTcolumnPENDING_PARTIAL_PASS_UNTIL_TS. The raw frequency is clamped to the 24-hour ceiling before scaling: a table configured to never refresh its cache resolves its update-cache-frequency toLong.MAX_VALUE, and scaling that then adding it to the current time would saturate into a negative (past) deadline that defeats the wait. This lets clients still holding a cached pointer to the old physical table refresh before the partial pass runs, so late writes routed to the old table are not stranded as unverified rows.PENDING_PARTIAL_PASS -> PARTIAL_PASS_RUNNING— once the wait window elapses, commit the transition (clearing the inherited full-pass job id so the monitoring branch cannot mistake the already-successful full pass for the partial pass and complete early), then kick the partial-passTransformToolrun.PARTIAL_PASS_RUNNING -> COMPLETED / FAILED— monitor the partial-pass job. A pass that cannot be confirmed successful (no job id registered because the initial kick failed before itsSTARTEDtransition, an unsuccessful job, or a job id that no longer resolves) is routed through a retry budget; once retries are exhausted the record reaches terminalFAILEDrather than stranding a pointer-swapped table with unverified rows forever.Dual-write link teardown at cutover commit (
Transform.doCutover):TRANSFORMING_NEW_TABLElink is deleted uncommitted and batched into the same commit as the base-table pointer swap.MUTATE_BATCH_SIZEview loop, paired with that view's pointer swap. A base table can have millions of views, so folding per-view link teardown into the bounded batch loop keeps every commit bounded while still pairing each link removal with its swap in one cache-invalidation cycle.doGetTableattaches the transforming-new-table per row from link presence, so a surviving view link would keep dual-write alive for view-routed writes after cutover.Schema:
PENDING_PARTIAL_PASS_UNTIL_TSis the first column ever added toSYSTEM.TRANSFORM. On upgrade,ConnectionQueryServicesImplsnapshots the table and runsaddColumnsIfNotExistsguarded onMIN_SYSTEM_TABLE_TIMESTAMP_5_4_0— the same (unreleased) 5.4.0 system-table timestamp thatSYSTEM.CATALOG's header already reaches via itsINDEX_CONSISTENCYcolumn add — so no new min system-table timestamp is introduced. The client upgrade gate compares againstSYSTEM.CATALOG's own reported timestamp, so introducing a new min not matched by aSYSTEM.CATALOGcolumn-add at that timestamp would leave the catalog below the gate after an in-place upgrade and loop clients onUpgradeRequiredException; riding the existing 5.4.0 timestamp avoids that. A fresh install gets the column directly from theCREATE TABLEDDL.SystemTransformRecord/TransformClientread and write the new column with explicitBIGINTnull handling.Why are the changes needed?
Running the partial pass inline at cutover repaired unverified rows immediately, but clients could still hold a cached pointer to the old physical table for up to their update-cache-frequency window. Writes routed to the old table during that window landed after the partial pass had already run, so they were never repaired and remained as unverified rows. Deferring the partial pass until after the cache-refresh window closes, and shutting off dual-write in the same cache cycle as the pointer swap, closes that gap. Making cutover an explicit, committed state machine also lets a partial pass that fails reach a terminal state (retry-budgeted, then
FAILED) instead of being lost when the monitor process restarts.Does this PR introduce any user-facing change?
No. This targets the
PHOENIX-7904-featurebranch (Online Schema Change gap-fix initiative), which is pre-launch. Two newTransformStatusvalues (PENDING_PARTIAL_PASS,PARTIAL_PASS_RUNNING) and one nullableSYSTEM.TRANSFORMcolumn are added; both are internal to the transform lifecycle.How was this patch tested?
New
CutoverLifecycleIT(@Category(ParallelStatsDisabledTest.class)) drives real and seeded cutovers with an injected clock (so no real 30-minute wait) and a job-lookup seam:PENDING_PARTIAL_PASS -> PARTIAL_PASS_RUNNINGtransition clears the inherited full-pass job id.PARTIAL_PASS_RUNNINGrecord with a null job id, with a not-found job, and with retries exhausted all reach terminalFAILED.NULL.UPDATE_CACHE_FREQUENCY=NEVER, which resolves toLong.MAX_VALUE) yields a bounded, future wait deadline (> cutover time,<= cutover + 24h) rather than an overflowed past one.Two fast unit tests (no cluster) guard the arithmetic-only invariants:
TransformMonitorTaskWaitTestexercises the extractedboundedPartialPassWaitMsclamp across the whole input domain (Long.MAX_VALUE, zero, negative, mid-range, at/above the 24h ceiling), asserting the wait is always within[30min, 24h]and strictly positive sonow + waitcannot overflow.MetaDataUtilTest.testMinSystemTableTimestampIsSystemCatalogReachableassertsMIN_SYSTEM_TABLE_TIMESTAMP == MIN_SYSTEM_TABLE_TIMESTAMP_5_4_0, tripping if a future change bumps the min system-table timestamp without a correspondingSYSTEM.CATALOGcolumn-add at that timestamp (which would strand in-place-upgraded clients onUpgradeRequiredException).Heavy user-table cutover ITs run on CI; the seeded strand regressions run locally.
mvn spotless:checkand main+test compile are clean onphoenix-core-client,phoenix-core-server, andphoenix-core.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)