feat: enforce scheduled content retention by partition drop - #1481
Merged
Merged
Conversation
Deploying control-layer with
|
| Latest commit: |
8acb6ef
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://c70f09d1.control-layer.pages.dev |
| Branch Preview URL: | https://peter-retention-lifecycle.control-layer.pages.dev |
pjb157
force-pushed
the
peter/privacy-aware-request-logging
branch
from
August 13, 2026 15:10
1214843 to
77c7cd3
Compare
pjb157
force-pushed
the
peter/retention-lifecycle
branch
2 times, most recently
from
August 14, 2026 08:52
2a954a9 to
68db145
Compare
pjb157
changed the base branch from
peter/privacy-aware-request-logging
to
main
August 14, 2026 08:52
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
August 14, 2026 09:17
68db145 to
70f57d0
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Adds policy-driven, disabled-by-default content retention to the fusillade daemon and Postgres storage backend, exposing a configurable sweep worker that expires files, ages out terminal batches, and deletes/redacts eligible batchless requests while emitting aggregate metrics.
Changes:
- Introduces retention policy/cutoff/outcome types in
fusillade-coreand re-exports them throughfusilladeandfusillade-arsenal. - Adds a retention sweep worker to the daemon with startup validation and bounded per-tick chunking.
- Implements Postgres retention sweep logic plus a migration for retention-related access-path indexes and corresponding operator documentation/config validation.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| fusillade/src/manager/mod.rs | Re-exports retention sweep types from fusillade-core. |
| fusillade/src/lib.rs | Publicly re-exports retention sweep types from the manager module. |
| fusillade/src/daemon/transitions.rs | Updates test storage stub to satisfy new DaemonStorage retention API. |
| fusillade/src/daemon/mod.rs | Adds retention startup validation, shutdown-aware helper, and retention sweep background task with metrics. |
| fusillade/src/daemon/config.rs | Adds retention policy + sweep interval fields to daemon config and round-trip tests. |
| fusillade/README.md | Documents automated content retention behavior and operational rollout steps. |
| fusillade-core/src/manager.rs | Defines RetentionSweepPolicy, immutable cutoffs, sweep outcome, and extends DaemonStorage. |
| fusillade-core/src/lib.rs | Re-exports retention sweep types from manager. |
| fusillade-core/src/daemon_record/transitions.rs | Updates test storage stub to satisfy new retention API. |
| fusillade-arsenal/src/postgres.rs | Implements Postgres retention sweeping plus lock-ordering changes and extensive regression tests. |
| fusillade-arsenal/src/lib.rs | Re-exports retention sweep types from fusillade-core. |
| fusillade-arsenal/migrations/20260813000000_add_retention_sweep_indexes.up.sql | Adds (non-concurrent) creation of candidate retention indexes with precreate guidance in comments. |
| fusillade-arsenal/migrations/20260813000000_add_retention_sweep_indexes.down.sql | Drops the new retention sweep indexes. |
| dwctl/src/config.rs | Wires retention config into dwctl, adds validation, and tests for config invariants. |
| .github/fixtures/fusillade-migration-sha384.txt | Updates migration checksum fixture to include the new migration. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
8442
to
8444
| let lease_acquired: bool = sqlx::query_scalar( | ||
| "SELECT pg_try_advisory_xact_lock(hashtextextended('fusillade.retention.sweep', 0))", | ||
| ) |
Comment on lines
+8544
to
+8549
| SELECT 1 FROM requests r | ||
| WHERE r.batch_id = batches.id | ||
| AND r.state = 'canceled' | ||
| AND r.claimed_at IS NOT NULL | ||
| AND r.canceled_at > NOW() - make_interval(secs => $3) | ||
| ) |
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
August 14, 2026 09:30
70f57d0 to
38678f8
Compare
Contributor
Author
|
Re-review follow-up is available in
Verification:
|
pjb157
changed the title
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
August 16, 2026 10:43
8f4aa4d to
3a432e9
Compare
pjb157
changed the title
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
August 18, 2026 09:22
26687f4 to
57720e7
Compare
pjb157
changed the title
pjb157
force-pushed
the
peter/retention-lifecycle
branch
3 times, most recently
from
August 28, 2026 16:38
accd28f to
16c5d1b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 45 out of 69 changed files in this pull request and generated 9 comments.
Files not reviewed (20)
- .sqlx/query-0e3c39515a74144bf61120be2f385ada3738bf7dc103f6820ea190a99436d8b6.json: Generated file
- .sqlx/query-0ea266b1cfd0bbeec6bed1109c38940ef373e1474f7f286220bd6b905fbb35b9.json: Generated file
- .sqlx/query-2b18b64a09470cee2ebb3f1c06a4095288aefe10a55d6cce07eaed4626b6c601.json: Generated file
- .sqlx/query-3c783fe881372a9177855d281a5fa906895f53ad1d555ddadc994258e4b05577.json: Generated file
- .sqlx/query-3c9a4ebeec6e5889f366c376a46f2e34b2a48a64d55b820574f9b143112acf76.json: Generated file
- .sqlx/query-4e8d32f7217b3715aa3069edc1ecf17ba880c37ba2b02e914484a37daf771ed9.json: Generated file
- .sqlx/query-4fb673d433e357250c903514f2941b122a7b4893fff7d7738a760a09df49b941.json: Generated file
- .sqlx/query-510a40c1fae33eda5611cd2478a78fe9edf262712a929354833f0ef990c75968.json: Generated file
- .sqlx/query-62e80a95cf89295b67c04690d00466d5705dca7d7a4c93fafa16a6468448e3d3.json: Generated file
- .sqlx/query-6f4084ac640045c445935f0e8e68821b4b56c7dfe6494cfb0eb8a5bcf33db634.json: Generated file
- .sqlx/query-6f7622b6deb65c0bb31d90322f8eecae0fdc9fa5c09c5bbacf193af8c97dfe13.json: Generated file
- .sqlx/query-7b2adf40a1b00739c30a4cf9b44ee1c7f93cde2614e92de7ecc2e15d008b4173.json: Generated file
- .sqlx/query-7c05cbf270bf88931106f291956e61d7a6411da0ead5203755cdb28c9c95244a.json: Generated file
- .sqlx/query-8697bbc253414f56c65f12813550249a5e67d9f283bad5985e3b16ae4271abc3.json: Generated file
- .sqlx/query-86badacda60db381aa221fa2c0e96317887283f776a49e49734116622abacb2a.json: Generated file
- .sqlx/query-967c8d2bb06462fabea767d1ada53939e56495e4ac95cdff7224875604c05067.json: Generated file
- .sqlx/query-aa821d673e92d20f27f9e35f8826cb6f65d28a713edc622c73f8dd1a081d11f5.json: Generated file
- .sqlx/query-b2288aa8eb278dd09aae8a22e9b3cbba6dbb79143509fba10e3c422f282fd4f9.json: Generated file
- .sqlx/query-f2f6500e57dc2783a6ae16f86b7d9cd6aa7653dd8dc01ff904d9fc20ccf21bb5.json: Generated file
- .sqlx/query-fb744db440734acb1bb6b86586d94b6c86de2cdc6b3093cedbbc7769450266ba.json: Generated file
Comment on lines
+686
to
+687
| Ok(archive_daemon_enabled | ||
| && (daemon.retained_response_retirement_enabled || daemon.batch_archive_retirement_enabled || state.unfinished_retirements > 0)) |
Comment on lines
+1889
to
+1893
| /// Allow file-content expiry and weekly template partition retirement. | ||
| /// Disabled by default and additionally requires an explicit retention | ||
| /// period. | ||
| #[serde(default)] | ||
| pub template_retirement_enabled: bool, |
Comment on lines
+311
to
+322
| # Optional retained-response lifecycle. It has no implicit retention | ||
| # period; movement and retirement default off. Current scheduled support | ||
| # covers terminal batchless response tiers only. Each configured tier | ||
| # requires an explicit positive duration and late-writer fence. Scheduled | ||
| # file and file-backed batch retention is rejected at startup. | ||
| # retention: | ||
| # batchless_seconds_by_service_tier: | ||
| # flex: <seconds> | ||
| # max_late_writer_seconds: <seconds> | ||
| # batchless_archive_sweep_enabled: false | ||
| # batchless_archive_backfill_enabled: false | ||
| # retained_response_retirement_enabled: false # Reserved; true is rejected until retirement ships |
Comment on lines
+4
to
+6
| DO $$ | ||
| BEGIN | ||
| IF EXISTS (SELECT 1 FROM request_templates_g2 LIMIT 1) THEN |
Comment on lines
+2829
to
+2833
| Ok(Ok(Err(error))) => { | ||
| tracing::error!( | ||
| error = %error, | ||
| "Fusillade daemon failed while stopping after leadership loss" | ||
| ); |
| -e POSTGRES_HOST_AUTH_METHOD=trust \ | ||
| -p 5432:5432 \ | ||
| postgres:17 \ | ||
| postgres:latest \ |
Comment on lines
+292
to
+298
| Current scheduled lifecycle support is limited to terminal batchless response | ||
| graphs in the `priority`, `flex`, and `background` service tiers. Every | ||
| configured tier needs an explicit positive duration, and batchless policy also | ||
| requires an explicit positive late-writer fence. Scheduled file retention and | ||
| terminal file-backed batch retention are rejected until their complete payload | ||
| lifecycle is supported. Explicit deletion continues through the ordinary | ||
| orphan purge and is independent of scheduled retention. |
Comment on lines
+192
to
+205
| parent_table <> 'batch_requests_archive' | ||
| OR ( | ||
| partition_schema IS NOT NULL | ||
| AND partition_schema_oid IS NOT NULL | ||
| AND parent_oid IS NOT NULL | ||
| AND lower_bound IS NOT NULL | ||
| AND upper_bound IS NOT NULL | ||
| AND upper_bound = lower_bound + 7 | ||
| AND lower_bound = date_trunc('week', lower_bound)::date | ||
| AND partition_table = | ||
| 'batch_requests_archive_y' || to_char(lower_bound, 'IYYY') | ||
| || 'w' || to_char(lower_bound, 'IW') | ||
| ) | ||
| ) |
Comment on lines
+3259
to
+3262
| if self.storage.supports_retained_response_route_cleanup() | ||
| && self.config.purge_interval_ms > 0 | ||
| && self.config.purge_batch_size > 0 | ||
| { |
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
September 1, 2026 06:35
16c5d1b to
964f3df
Compare
The mover deliberately excluded graphs whose deletion day had already passed (a partition never drops on its own day), leaving them to an 'explicitly gated legacy path' that no longer existed after the row-deleting sweeper was removed. Such content therefore stayed live, and undeletable, forever. Found in the cl-1548 preview, where 27M terminal rows older than the (minimal) retention period never moved. The backfill worker now owns that gated path: it discovers due graphs oldest-first with no per-tier lower bound, and move_graph lands anything already due on the day after observation, the earliest day that can still be dropped. Retention is only ever extended, by at most the time a graph spent overdue. The ordinary sweep is unchanged and its fairness guarantees keep their tests.
The oldest batchless traffic (April-May 2026) shares its input template with a file. The mover treated any file-owned template as an incomplete graph and, worse, one failed graph aborted the whole pass, so the cl-1548 backfill logged 'graph is incomplete' every second and never moved a row. Templates are now classified per graph: a dedicated template nothing else references is owned (moved and deleted with the graph); a file's template or a shared dedicated one is external input content governed by its own lifecycle, snapshotted into the retained record so it stays self-describing and left live. An incomplete graph is counted in RetainedResponseArchiveOutcome::incomplete_skipped and left live while the pass continues, so a malformed graph can never stall the worker again. Verified: fusillade-arsenal (all suites), fusillade and fusillade-core green; lint clean. The dwctl suite's failures locally are connection exhaustion on a 100-connection dev Postgres (they pass in isolation).
In a split topology the API pods share the daemon pods' environment but
run with batch_daemon.enabled = never. Config validation rejected every
retention flag on such an instance ('automated retention requires the
batch daemon to run'), so the cl-1548 preview's API pods crash-looped on
startup while the old pod kept serving, silently blocking every rollout.
An instance that never runs the daemon owns no archive maintenance, so
the ownership-gated validations no longer apply to it; the hard error
becomes a warning so a single-instance deployment cannot configure
retention that nothing enforces without noticing.
Verified: dwctl config tests, lint clean.
…nce session
The startup preflight that builds the retained-response maintenance
session rejected any instance with retirement flags but the daemon
disabled ('requires an enabled archive daemon'). Split deployments hit
exactly that: the API pods share the daemon pods' environment, so the
cl-1548 preview's API pod crash-looped and its stuck rollout exhausted
the namespace quota, blocking the daemon pods too.
An instance whose daemon never runs owns no maintenance: it starts
without a session and warns if durable work is pending, leaving recovery
to the daemon pods. Corrupted lifecycle identity still fails every
instance. Verified: preflight and config tests, lint clean.
The claim, batch-results and request-detail paths joined the two-generation template views with a plain equi-join, which the planner turned into a hash of the entire legacy template table on every claim (a 40M-cost plan that never finished against the production-sized preview, so cutover batches were never claimed). Correlated LATERAL lookups with LIMIT 1 keep the join as a primary-key probe on both arms. dwctl still counted a file's templates from the legacy table, so batches created after the cutover were validated against zero templates and lost their model label.
hydrate_previous_response deserialised the compact object the store persists straight into the full wire-shape struct, so extending any stored single-step response failed with 500 (missing `tools`). Backfill the same defaults the retrieval endpoint applies. The new test also pins that extension keeps working after the first turn's graph has moved to the retained store. Also make the leader-election retry test's polling loops real-time bounded: a fixed iteration count only bounds paused virtual time, and the unlock round-trip on a loaded CI runner outlived it (three consecutive CI failures on unrelated pushes).
Review follow-ups for the template cutover: - request_templates_g2 is keyed (created_on, id); the routed by-id probe was a range scan of the whole week's primary key on PostgreSQL 17 (production runs 17.11, no skip scan). Add an id-leading partitioned index. - Nothing called ensure_request_template_partitions, so the first upload of each UTC week paid CREATE TABLE + ATTACH inside its own upload transaction. The weekly archive-partition loop now extends the template runway too, with its own gauge. - The mover hashed both payload byte strings server-side to compare them; both are already in memory, so compare directly and save two round trips per graph. - Document the created_at-based index expire_file_content actually needs.
…request The server-side tool loop that wrote response_steps was removed from the edge in August 2026; the table stopped growing on 2026-08-12, only ever linked five batches, and no retained snapshot contains a step. Carrying it cost the retention lifecycle its most complex code paths: a recursive step-chain closure in the mover, step routes and fences, a sweeper guard that wedged whole archive weeks behind step-linked batches, and a head-step indirection on every response read. - Drop the table (new migration) and strip step routes, the step object kind, and the head_step_id/step_sequence columns from the unreleased retention migration. - A retained graph is now exactly a request plus its template; the group id is the request id. Discovery, locking, movement, erasure, and retirement lose their step branches. - dwctl treats resp_<uuid> as the request id directly and no longer wires a step store; the unused multi-step config block goes with it. - Tests about chains and multi-member graphs are removed; tests that used a step only as a fixture now use the request directly. Bumps the storage crates' major version: ResponseStepStore and the step types are gone from the public API.
It passes locally on every run but times out on the CI runner even with a 60s real-time budget while the tokio clock is paused. It has failed three consecutive CI runs on unrelated pushes; ignore it until the paused-clock database interaction is reworked on main.
Four production batches were blocked from moving (by the old response-steps guard) until after their own archive week had been retired and dropped. The sweep then retried them into the missing partition on every tick, logging an error each time, and they could never leave the live table. When the frozen week's bucket is no longer active, stamp the batch into the current week instead: a retired week never receives rows, and a later retirement date is the safe direction.
Correctness: - Two dwctl readers (tier-2 ingest validation, analytics recompute) and the request-detail/list paths still read the legacy template table; after the cutover they saw nothing. All read the generation-transparent view. - template_retirement_enabled never opened a maintenance session, so the documented rollout step crash-looped. Wired in and validated with its retention period, mirroring the batch-archive flag. - A failed leadership renew ping returned the leader connection to the pool with its advisory lock possibly still held, stranding leadership until the connection was reaped. Route it through release_leader_connection. - Erasure in the live+retained overlap deleted only the live copy, so the retained snapshot resurfaced once the live row was gone. Both go. - previous_response_id hydration performed no ownership check; another tenant's turn could be spliced into the caller's prompt. The store now resolves the row and requires the caller's user id before returning any content (pre-existing on main, now reachable through the retained store). - After the template FK drop, a pending request whose template was purged occupied a claim slot forever. The orphan purge fails such rows. - The batch archive mover takes a share lock on the target week's registry row so a retirement fence can never land between its check and its insert. - Archived-request reads dropped templates for batches whose input file was deleted (file_id compared with '=' against NULL). Operability: - The list total count runs on the read pool with a 100ms budget per arm and falls back to the planner estimate, as main did; it no longer pins a primary connection for a full scan. - Maintenance loops log the content-free error class; erasure maps a mid-retirement conflict to 409 instead of an opaque 500; leadership-loss stop failures go through background_error!; startup preflight errors keep their (payload-free) cause; the drop migration bounds its lock wait. Tests added for each behavioural fix; suites green.
pjb157
force-pushed
the
peter/retention-lifecycle
branch
from
September 4, 2026 08:41
516c06e to
e21d94f
Compare
This was referenced Sep 3, 2026
pjb157
added a commit
that referenced
this pull request
Sep 4, 2026
🤖 I have created a release *beep* *boop* --- ## [11.2.0](v11.1.1...v11.2.0) (2026-09-04) ### Features * **analytics:** record engine cached tokens and content-free request params ([#1553](#1553)) ([411fabc](411fabc)) * **clickhouse:** shared warehouse connection and a generic best-effort insert sink ([#1554](#1554)) ([6faf6c0](6faf6c0)) * enforce scheduled content retention by partition drop ([#1481](#1481)) ([fc5aeae](fc5aeae)) ### Bug Fixes * **deps:** bump browserslist to 4.28.8 (Dependabot [#175](https://github.com/doublewordai/control-layer/issues/175)/[#176](https://github.com/doublewordai/control-layer/issues/176)) ([#1552](#1552)) ([9f4d912](9f4d912)) * **deps:** finish OpenTelemetry 0.32 migration via outlet 0.10 (CVE-2… ([#1550](#1550)) ([7c6462f](7c6462f)) * search customer-visible model fields ([#1560](#1560)) ([486383d](486383d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
fergusfinn
pushed a commit
that referenced
this pull request
Sep 5, 2026
🤖 I have created a release *beep* *boop* --- ## [1.0.0](onwards-v0.38.1...onwards-v1.0.0) (2026-09-05) ### ⚠ BREAKING CHANGES * remove tool injection, tool_sources, and the onwards Responses adapter ([#1518](#1518)) ### Features * enforce scheduled content retention by partition drop ([#1481](#1481)) ([fc5aeae](fc5aeae)) * midstream retries ([#1453](#1453)) ([48be316](48be316)) * **onwards:** count gateway failures by reason and upstream status ([#1519](#1519)) ([fce9cbd](fce9cbd)) * remove tool injection, tool_sources, and the onwards Responses adapter ([#1518](#1518)) ([4276d91](4276d91)) ### Bug Fixes * **dwctl:** tolerate backend-omitted fields in Responses translation, stop retry loops on translate failure ([#1544](#1544)) ([04c07c9](04c07c9)) * fold SSE streams incrementally and drop async-openai from dwctl ([#1503](#1503)) ([2995d73](2995d73)) * **onwards:** drain active responses during shutdown ([#1586](#1586)) ([416ef7f](416ef7f)) * **onwards:** retry slow empty SSE streams ([#1510](#1510)) ([4df7a30](4df7a30)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
sejori
pushed a commit
that referenced
this pull request
Sep 15, 2026
🤖 I have created a release *beep* *boop* --- ## [6.0.0](fusillade-core-v5.0.0...fusillade-core-v6.0.0) (2026-09-15) ### ⚠ BREAKING CHANGES * **dwctl:** move stream reassembly out of the batch daemon ([#1535](#1535)) ### Features * enforce scheduled content retention by partition drop ([#1481](#1481)) ([fc5aeae](fc5aeae)) * **fusillade:** add configurable batch leak intervals ([#1712](#1712)) ([c4bfb3d](c4bfb3d)) * **fusillade:** move overdue batchless graphs concurrently in the backfill ([#1567](#1567)) ([b0ed45c](b0ed45c)) * **fusillade:** own batch finalization in a daemon loop, decouple no… ([#1462](#1462)) ([83066bb](83066bb)) ### Bug Fixes * prune retained partitions in the trailing-demand query by retent… ([#1587](#1587)) ([909ec68](909ec68)) ### Code Refactoring * **dwctl:** move stream reassembly out of the batch daemon ([#1535](#1535)) ([e9c1795](e9c1795)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
sejori
added a commit
that referenced
this pull request
Sep 15, 2026
🤖 I have created a release *beep* *boop* --- ## [4.1.0](fusillade-arsenal-v4.0.0...fusillade-arsenal-v4.1.0) (2026-09-15) ### Features * **dwctl:** pooled and direct connection pools per database ([#1565](#1565)) ([73a2080](73a2080)) * enforce scheduled content retention by partition drop ([#1481](#1481)) ([fc5aeae](fc5aeae)) * **fusillade:** add configurable batch leak intervals ([#1712](#1712)) ([c4bfb3d](c4bfb3d)) * **fusillade:** move overdue batchless graphs concurrently in the backfill ([#1567](#1567)) ([b0ed45c](b0ed45c)) * **fusillade:** own batch finalization in a daemon loop, decouple no… ([#1462](#1462)) ([83066bb](83066bb)) * Improve page load for batches page ([#1744](#1744)) ([e720a9e](e720a9e)) ### Bug Fixes * add index for finalizer sweep ([#1728](#1728)) ([a983682](a983682)) * bound retained history reads for single-model response lists ([#1778](#1778)) ([28a4746](28a4746)) * **dwctl:** count pending requests outside the admission lock; index the batched demand branch ([#1531](#1531)) ([73e8f80](73e8f80)) * Formalize indices missing from migration and drop unused ones ([#1568](#1568)) ([0866443](0866443)) * **fusillade:** bound retained response page scans ([#1699](#1699)) ([f7c64cf](f7c64cf)) * **fusillade:** drop the stranded-request purge step that times out ([#1576](#1576)) ([b9cae7a](b9cae7a)) * **fusillade:** give the count-estimate EXPLAIN its own budget ([#1661](#1661)) ([325ab85](325ab85)) * **fusillade:** keep request polling stable during retention ([#1742](#1742)) ([0ccdf8c](0ccdf8c)) * **fusillade:** page batchless archive discovery and hot-loop busy movers ([#1715](#1715)) ([f4d8737](f4d8737)) * **fusillade:** probe the template per picked row in the claim-time stranded check ([#1580](#1580)) ([1433908](1433908)) * keep the trailing-demand retained arms under the statement timeout ([#1723](#1723)) ([c4390d3](c4390d3)) * prune retained partitions in the trailing-demand query by retent… ([#1587](#1587)) ([909ec68](909ec68)) * support scoped owners for pooled component connections ([#1721](#1721)) ([1bd4127](1bd4127)) ### Performance Improvements * **fusillade:** index the batchless branch of the demand query ([#1516](#1516)) ([6833cb0](6833cb0)) * **fusillade:** make the unscoped Responses page usable ([#1740](#1740)) ([ddfc06a](ddfc06a)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Seb <seb@doubleword.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the data retention policy for every class of customer content the API stores. Scheduled deletion is partition drop only — content lives in time-keyed partitions and expires by irreversibly dropping whole relations through one shared, journaled, crash-safe engine. There is no recurring row-by-row content deletion anywhere after this PR, and every destructive control ships disabled.
How each data class is deleted
retained_response_objectspartitions, keyed by deletion datebatch_requests_archivepartitionsbatches.retention_expired_atin the same transactionrequest_templates_g2partitions; new writes cut over behind a flag, the legacy heap freezes in place untouchedretention_expired_at, row survives) once every referencing batch's content is gone; batches are stamped at partition drop. Account-lifetime retention, exactly as the policy statesOne retirement engine, three families
Every partition drop runs through a single state machine (
partition_retirement.rs): journal the exact partition identity (schema + OID, parent OID, child name + OID, bounds — with a CHECK constraint that makes tampered identities unrepresentable), fence the bucketretiringin the same commit so reads fail closed,DETACH PARTITION CONCURRENTLY,FINALIZErecovery after crashes, drop only the journaled relation, and complete atomically. Families differ only in a declarative spec: names, bounds width, eligibility SQL, and an optional metadata stamp. Lock/statement timeouts are retryable no-ops on a durable journal; renamed, re-bounded, or replaced relations are refused; an unfinished journal remains recoverable even after its flag or period is withdrawn.Destructive DDL runs only on an explicitly installed single-session maintenance pool (max 1 connection), attested at startup against the primary, with server-side lock and statement timeouts.
Already-due content (found in the cl-1548 preview)
The mover deliberately never lands content on a day that is already droppable, and its candidate query excluded graphs past their retention period — leaving them to a "gated legacy path" that no longer existed once the row-deleting sweeper was removed. Such content would have stayed live, and undeletable, forever. The backfill worker now owns that path: it discovers already-due graphs oldest-first (no per-tier lower bound) and moves them onto the day after observation, the earliest day that can still be dropped. Retention is only ever extended, by at most the time a graph spent overdue; the ordinary sweep and its fairness guarantees are unchanged.
Reads and writers
Point, list, and count reads are byte-identical before and after content moves; everything in a
retiring/retiredbucket answers not-found before any physical DDL. Template reads are generation-transparent through one view (the claim path resolves generation-2 ids through a route oracle that prunes to a single weekly partition). Late writers to moved or dropped response identities are blocked by durable content-free fences; claim, pending, and mutation paths remain live-only.Safety defaults and flags (all off)
batchless_archive_sweep_enabled,batchless_archive_backfill_enabled,retained_response_retirement_enabled,batch_archive_retirement_enabled+batch_archive_retention_days,template_generation_writes_enabled,template_retirement_enabled+template_retention_days. Retention periods have no defaults — enabling any retirement without an explicit positive period fails startup validation, as does enabling retirement without the maintenance endpoint on a dedicated database, or template retirement without the write cutover.Observability and evidence
Aggregate-only metrics with fixed labels for every phase (movement counts/bytes, partition runway and readiness, per-family retirement and retry counters, route/fence cleanup counters, file-content expiry). The retirement journal and bucket tombstones are permanent, dated, content-free records of every deletion — audit evidence by construction. No request identifiers, owners, models, or payloads appear in any log, metric, or error (enforced by the repo's no-payload-logging guard). A read-only preflight script verifies index readiness and exact partition attachment before any enablement.
Testing
Rollout (each step independently reversible until noted)
🤖 Generated with Claude Code