Skip to content

Add safe audit-event partition maintenance - #90

Merged
ivanvyd merged 12 commits into
mainfrom
feature/audit-partition
Sep 1, 2026
Merged

Add safe audit-event partition maintenance#90
ivanvyd merged 12 commits into
mainfrom
feature/audit-partition

Conversation

@ivanvyd

@ivanvyd ivanvyd commented Aug 30, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Adds a production-oriented, migration-role-only PostgreSQL maintenance tool that safely converts audit_events to monthly range partitions, validates the migrated copy, supports explicit finalize/rollback lifecycles, and applies seven-year retention without exposing privileged DDL through the shipped LakeWright packages.

What changed

  • Adds the non-packable LakeWright.DatabaseMaintenance tool and internal maintenance core.
  • Migrates existing rows transactionally while retaining audit_events_unpartitioned_backup until validation/finalization.
  • Preserves explicit safe ACLs, RLS policies, and exact enabled/FORCE RLS state, including non-superuser owners with hidden rows.
  • Maintains global audit Id uniqueness through a protected registry and SECURITY DEFINER insert trigger.
  • Pre-creates current plus two future UTC calendar-month partitions.
  • Drops only managed partitions whose complete range precedes the seven-year retention cutoff.
  • Adds versioned lifecycle state and fail-closed topology checks for relations, helper functions, registry indexes, trigger placement/state, and SECURITY DEFINER semantics.
  • Bounds in-transaction migration by lock/statement timeouts, rows, bytes, and historical partition count.
  • Documents deployment, recovery, compliance, retention, and architecture in ADR 0020 and the deployment guide.

Safety properties verified

  • The original destructive implementation was replaced; existing audit rows are copied and compared in both directions before success.
  • UTC month arithmetic remains correct when the PostgreSQL session crosses a DST boundary.
  • A reproduced FORCE RLS loss scenario now preserves all physical rows through migrate, validate, rollback, cleanup/remigration, and finalize.
  • ACCESS EXCLUSIVE is acquired before RLS flags are read, closing the concurrent ALTER TABLE race.
  • Rollback restores application access to the canonical table and revokes access to retained migration artifacts.
  • Sparse history is rejected before an unbounded number of partitions can be created (120-month default, configurable up to 1,200).
  • Corrupted lifecycle state, missing indexes, a missing registry primary key, a disabled identity trigger, and a downgraded SECURITY INVOKER function are refused before maintenance DDL.

Current verification

  • dotnet restore LakeWright.slnx --locked-mode
  • dotnet restore scripts/load/Lakewright.LoadHarness/Lakewright.LoadHarness.csproj --locked-mode
  • dotnet build LakeWright.slnx -c Release --no-restore — 0 warnings, 0 errors
  • dotnet build scripts/load/Lakewright.LoadHarness/Lakewright.LoadHarness.csproj -c Release --no-restore — 0 warnings, 0 errors
  • dotnet format LakeWright.slnx --no-restore --verify-no-changes
  • Focused real-PostgreSQL audit partition suite — 21/21 passed
  • Independent correctness, performance, security, requirements, and structural reviews completed; all blockers closed

The complete whole-application regression, Docker/sample smoke, live non-mutating Databricks bundle checks, package consumer smoke, and release evidence will run on merged main before publishing.

ivanvyd added 11 commits August 30, 2026 16:35
@ivanvyd
Partition audit_events by month (ADR 0016)
da9f485 
The audit table grows without bound; the append-only guarantee keeps its
rows honest but does nothing about their size. A retention sweep on a
single non-partitioned table becomes a DELETE the size of the table, and
the auditor is the person who notices. Postgres native range partitioning
by OccurredAt keeps each month's rows in a child table; a retention sweep
is now a metadata operation.

What changed:

- DatabasePartitioning.EnsurePartitionedAuditAsync drops the EF-created
  audit_events table and recreates it as a partitioned parent. The DDL is
  the only place that knows the shape.
- DatabasePartitioning.EnsureCurrentAndNextMonthAsync pre-creates the
  current and next month, so a row whose OccurredAt lands just after a
  boundary has a partition to land in.
- LakeWrightDbContext's primary key on AuditEvent becomes (Id, OccurredAt).
  Postgres requires every unique constraint on a partitioned table to
  include the partition key, and the Id + month combination is the only
  identity that survives across months.
- The (OrganizationId, OccurredAt) index moves from the parent to each
  child partition. Reads still go through 'SELECT FROM audit_events' —
  Postgres native partitioning makes parent and children look like one
  table to SQL clients.
- PostgresFixture calls EnsurePartitionedAuditAsync after
  EnsureCreatedAsync so every test database is partitioned; the harness
  smoke and the test suite both exercise the path.
- AuditPartitionTests assert the partitioned-table property, the
  current/next-month partitions exist, and a round-trip insert lands in
  the right place.
- ADR 0016 records the decision and the breaking change to the primary
  key.

The migration is a one-time cost: a production deploy that runs against
a database with data has to copy rows into the partitioned parent before
the swap. The shipped manager does not do this copy; tests run against
fresh databases, and the only place an existing audit table exists today
is the harness's ephemeral container. A production migration is a
separate change.
@ivanvyd
Add post-ship review of #85, #86, #88
f9e87d6 
The recent work (cost proxy, load harness, Postgres sizing, harness auth)
was reviewed with the post-ship-review lenses: correctness, security/
tenancy, performance, requirements completeness. The review found one
bug already fixed (the harness's Member role) and lists the gaps that
remain: production migration of existing audit rows, tightening the
harness's SLO gates to production-anchored values, Key Vault for the
Bicep's admin password, and a coverage re-measurement that the milestone's
new tests are excluded from.

The review is the maintainer's self-review, which the ROADMAP records
as the project's standing weakness. A second pair of eyes is the next
maintenance task.
@ivanvyd
Add ScopeVersion to TenantContext and broker
ce3dcf1 
The gap analysis (2.2) identified that LakeWright's hardcoded
TenantId.ToString() for external_value produces a token that always has
the same scope for a given tenant. A tenant whose data scope narrows keeps
seeing the old rows for the full 24-hour result-cache lifetime.

This commit adds the feature the gap calls out, the same way the gap
recommends it:

- TenantContext gains an optional . The Create method accepts the
  value and validates it: no , , or  (reserved in the claim format; the
   delimiter is the only delimiter that survives both the GUID body and the
  version value, verified live this session).
- The broker (DashboardTokenBroker.IssueAsync) composes
   when  is non-empty; otherwise the bare
  id. The isolation property the existing code defends — a caller can never choose
  the external_value freely — remains intact: the value comes from the tenant,
  which comes from the resolver, not from the request.
- The change is backward-compatible: when  is null,
  produces exactly the previous value.

This closes the highest-value design gap from the gap analysis, without
forcing VRM-specific policy into the library. It does not close 3.2,
3.4, or the .NET 10 / preview-release blockers, which remain open as
separate work.
@ivanvyd
Merge main into feature/audit-partition
88f5394 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>

# Conflicts:
#	src/LakeWright.Core/Tenancy/TenantContext.cs
#	src/LakeWright.Embedding/DashboardTokenBroker.cs
@ivanvyd
Make audit partitioning safe for existing databases
4779dad 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>
@ivanvyd
Merge latest main into feature/audit-partition
fdb6e62 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>

# Conflicts:
#	tests/LakeWright.TenantIsolation.Tests/PostgresFixture.cs
@ivanvyd
Merge remote-tracking branch 'origin/main' into feature/audit-partition
40d6f2d
@ivanvyd
Harden audit partition maintenance lifecycle
3953169 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>
@ivanvyd
Close audit partition migration safety gaps
7aef8de 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>
@ivanvyd
Merge remote-tracking branch 'origin/main' into feature/audit-partition
7bd8e58 
Signed-off-by: Ivan Vydrin <ivan.vydrin.99@gmail.com>
@ivanvyd
Harden audit partition topology validation
1bc418f
@ivanvyd ivanvyd changed the title Partition audit_events by month (ADR 0016) Add safe audit-event partition maintenance Sep 1, 2026
@ivanvyd
ivanvyd merged commit c5ffa09 into main Sep 1, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@ivanvyd