feat: record durable run history in a ClickHouse table

CHANGELOG.md

@@ -17,6 +17,54 @@ expose.
 
 ### Added
 
+- **A durable run log in ClickHouse**, `<COLLECTTABLEPREFIX><disk>_log`, written
+  beside the auxiliary table and **never truncated**.
+
+  Why: pod logs are not a record. The kubelet rotates container output (10Mi
+  over 5 files by default), so `kubectl logs` cannot return the beginning of a
+  long run, and `ttlSecondsAfterFinished` deletes the Job and its pods along
+  with everything they printed. A cleanup that reclaimed terabytes left no
+  evidence of what it did once that window closed — the reported symptom was
+  simply "the job does not have all the log output".
+
+  ClickHouse is the sink rather than a volume or the bucket: the connection,
+  credentials and grants already exist, a `readOnlyRootFilesystem` container
+  cannot write a file, and an object written under `S3PATH` would be listed by
+  the *next* collect, found absent from `system.remote_data_paths`, and become
+  a deletion candidate — s3gc would garbage-collect its own logs.
+
+  Rows are self-describing evidence, not just text: `run_id`, `phase`, `event`,
+  `message`, running `objects`/`bytes`, and the scope the run was pointed at
+  (bucket, prefix, disk, cluster, dry-run, ClickHouse host, container
+  hostname). Events are phase start, throttled collect progress, per-sample
+  start, **per-delete-batch checkpoint**, finish with attempt totals, warnings
+  and errors. `S3GC_RUNID` defaults to a generated timestamped id and the Job
+  template passes `JOB_NAME`, so a row traces back to the Job that wrote it.
+
+  Three constraints, each with a test, because this is bookkeeping attached to
+  an irreversible operation:
+
+  1. **Writes go on `ch_writer`, never `ch_client`.** `do_use()` holds
+     `ch_client`'s session for the whole anti-join stream, and a second query on
+     a held session is `SESSION_IS_LOCKED` (373) — the 0.6.0 defect, which
+     would now fire mid-delete at the worst possible moment.
+  2. **A logging failure never fails the run.** One failure disables the run log
+     for the remainder of the process rather than retrying every batch, and a
+     missing `CREATE TABLE` grant degrades to stdout only with one warning.
+  3. **Messages are redacted** through the existing `LogFormatter._filter`
+     before insert, so a credential cannot reach a table that outlives the run.
+
+  Opt out with `--runlog false` / `S3GC_RUNLOG_FLAG=false`, or `RUNLOG=false`
+  in the renderer. Default on: durability is the point.
+
+- **Collect now reports progress at INFO**, throttled to every 100 000 objects.
+  Per-batch progress was logged at DEBUG only, so a multi-hour collect over
+  millions of objects emitted about four lines at `--verbose` — while `--debug`
+  emits one line *per object*, which on a large bucket exceeds the kubelet's
+  rotation limit and destroys the beginning of its own output. The operator's
+  real choice was "almost nothing" or "too much to retrieve".
+
+
 - **Regression tests pinning the deletion scope itself.** The suite already
   proved that a candidate list is deleted, batched and checkpointed correctly,
   but nothing proved the list contained only orphans: the ClickHouse fake

README.md

@@ -283,6 +283,31 @@ are in [deploy/kubernetes/README.md](deploy/kubernetes/README.md). A guarded
 collect, dry-run, and delete in one Job and still requires an explicit delete
 confirmation.
 
+## Durable run history
+
+Each run appends structured events to `<collecttableprefix><disk>_log` in
+ClickHouse, beside the auxiliary table: phase start, collect progress,
+per-sample start, one row per confirmed delete batch, and a closing total, each
+carrying the scope the run was pointed at. The table is never truncated.
+
+This exists because pod logs are ephemeral — the kubelet rotates them and a
+deleted Job takes them with it — so a completed cleanup would otherwise leave no
+evidence of what it removed. ClickHouse is the sink because the connection and
+grants already exist; writing the log into the bucket would make the *next*
+collect see it as an orphan and delete it.
+
+```sql
+SELECT event_time, phase, event, objects, bytes, message
+FROM   <collecttableprefix><disk>_log
+WHERE  run_id = '<run-id>'
+ORDER BY event_time;
+```
+
+`--runid` labels the run (Kubernetes Jobs use the Job name automatically);
+`--runlog false` turns the table off. It needs the same `CREATE TABLE` grant as
+the auxiliary table, and a missing grant degrades to stdout only rather than
+failing the run.
+
 ## Testing
 
 Run all isolated unit tests:

TODO.md

@@ -21,3 +21,12 @@ forward-looking backlog.
 - [ ] Quote `--useafter` as a SQL string literal (strict `xfail` in the suite).
 - [ ] Consider re-checking cluster topology per sample, not once per run, so a
   replica lost mid-run cannot widen the deletion scope.
+- [ ] `print()` output is block-buffered because the image sets no
+  `PYTHONUNBUFFERED` and stdout is a pipe under Kubernetes. Logger records are
+  flushed per line, but the bare prints — including the closing `s3gc: OK` —
+  are lost when `activeDeadlineSeconds` fires and the kubelet sends SIGTERM.
+  The durable run log now covers the evidence case; this remains a stdout
+  fidelity gap, and it also makes the `dev-automation` phase markers interleave
+  wrongly against the shell's unbuffered `echo`.
+- [ ] Consider a `TTL` on the run-log table. Volume is a few hundred rows per
+  run so it is not urgent, but it grows without bound across many cleanups.

deploy/kubernetes/README.md

@@ -56,7 +56,7 @@ Create or provision a dedicated ClickHouse user for `s3gc`, then grant it:
 
 ```sql
 GRANT SELECT ON system.*                        TO s3gc;  -- remote_data_paths, one, disks, tables
-GRANT SELECT, INSERT, CREATE TABLE ON <db>.*    TO s3gc;  -- the auxiliary table
+GRANT SELECT, INSERT, CREATE TABLE ON <db>.*    TO s3gc;  -- auxiliary + run-log tables
 ```
 
 ### Values that vary per cluster, and bite when wrong
@@ -136,6 +136,49 @@ Any failed stage stops the Job and later stages do not run; successful delete
 batches remain checkpointed. Do not use this phase for customer or production
 work because it removes the manual dry-run approval gate.
 
+## Durable run history
+
+Pod logs are **not** a record. The kubelet rotates container output, so
+`kubectl logs` cannot return the start of a long run, and
+`ttlSecondsAfterFinished` deletes the Job and its pods along with everything
+they printed. `kubectl logs -f` is for watching, not for evidence.
+
+So each run also appends to `<COLLECTTABLEPREFIX><S3DISKNAME>_log` in
+ClickHouse, beside the auxiliary table. That table is **never truncated**, and
+`S3GC_RUNID` is set to the Job name, so a row traces back to the Job that wrote
+it long after the pod is gone.
+
+What a delete actually did, on the same replica-pinned `CHHOST`:
+
+```sql
+SELECT event_time, phase, event, objects, bytes, message
+FROM   <db>.<prefix><disk>_log
+WHERE  run_id = '<job-name>'
+ORDER BY event_time;
+```
+
+How far a *failed* delete got before it died, which is what tells you whether to
+start a replacement delete Job:
+
+```sql
+SELECT max(objects) AS deleted, max(bytes) AS reclaimed
+FROM   <db>.<prefix><disk>_log
+WHERE  run_id = '<job-name>' AND event = 'checkpoint';
+```
+
+Every phase of a cleanup, newest first:
+
+```sql
+SELECT run_id, min(event_time) AS started, max(event_time) AS ended,
+       anyIf(message, event = 'error') AS error
+FROM   <db>.<prefix><disk>_log
+GROUP BY run_id ORDER BY started DESC;
+```
+
+Set `RUNLOG=false` to opt out. A missing `CREATE TABLE` grant degrades to
+stdout only with one warning rather than failing the run, and a run-log write
+that fails mid-delete disables the log instead of stopping the deletion.
+
 ## Safety
 
 - Delete checks the local cluster macro and expected replica count before S3

deploy/kubernetes/example.env

@@ -48,3 +48,9 @@ TTL_SECONDS_AFTER_FINISHED=604800
 MEMORY_REQUEST=1Gi
 MEMORY_LIMIT=4Gi
 VERBOSE=true
+# Record durable run history in <COLLECTTABLEPREFIX><disk>_log in ClickHouse.
+# Pod logs are rotated by the kubelet and deleted with the Job, so this table is
+# the only evidence of what a cleanup did once TTL_SECONDS_AFTER_FINISHED
+# expires. Needs the same CREATE TABLE grant as the auxiliary table; a missing
+# grant degrades to stdout only instead of failing the run.
+RUNLOG=true

deploy/kubernetes/job.yaml.tmpl

@@ -96,3 +96,9 @@ spec:
               value: "${ACTIVE_DEADLINE_SECONDS}"
             - name: S3GC_VERBOSE_FLAG
               value: "${VERBOSE}"
+            - name: S3GC_RUNLOG_FLAG
+              value: "${RUNLOG}"
+            # The Job name is the run id, so a run-log row traces back to the
+            # Job that wrote it after the pod and its logs are gone.
+            - name: S3GC_RUNID
+              value: "${JOB_NAME}"

deploy/kubernetes/render.py

@@ -45,7 +45,9 @@
 # Optional keys and their defaults. An empty value renders no environment
 # variable at all, because s3gc parses S3GC_USETOTAL as an integer and would
 # reject an empty string.
-OPTIONAL = {"USETOTAL": ""}
+# RUNLOG defaults on: the durable run-log table is the only record that
+# outlives the Job, whose pod and logs ttlSecondsAfterFinished deletes.
+OPTIONAL = {"USETOTAL": "", "RUNLOG": "true"}
 # Environment variables dropped from the manifest when they render empty.
 OPTIONAL_ENV = ("S3GC_USETOTAL",)
 JOB_NAME_RE = re.compile(r"^[a-z0-9]([-a-z0-9]*[a-z0-9])?$")
@@ -98,6 +100,8 @@ def validate(values: dict[str, str]) -> None:
         raise ValueError("VERBOSE must be true or false")
     if values["ORDER_BY_OBJPATH"] not in {"true", "false"}:
         raise ValueError("ORDER_BY_OBJPATH must be true or false")
+    if values["RUNLOG"] not in {"true", "false"}:
+        raise ValueError("RUNLOG must be true or false")
     if values["USETOTAL"] and (
         not values["USETOTAL"].isdigit() or int(values["USETOTAL"]) < 1
     ):