Skip to content

Commit e70b775

Browse files
authored
bench docs: name the script that refreshes the cache (#211)
vortex-data/vortex#11192 moved the cache revalidate out of scripts/post-ingest.py into scripts/refresh-bench-site.py, which a reusable workflow runs once per CI run. Update the contract, the architecture docs, the secrets runbook, and two code comments to name the new caller and describe when it runs and how it handles failure. No behaviour changes. Signed-off-by: Will Manning <will@willmanning.io>
1 parent 4172386 commit e70b775

8 files changed

Lines changed: 45 additions & 27 deletions

File tree

‎docs/bench/CONTRACT.md‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,11 @@ and are unchanged by this repository; this repo owns the ingest *contract*, the
1212
read service.
1313

1414
> **Scope.** There is one ingest path: the monorepo's `scripts/post-ingest.py --postgres` writes
15-
> emitter records directly into the hosted RDS Postgres and then pings this repo's read service
16-
> to flush its cache. The retired v3 HTTP ingest path (`POST /api/ingest` into a Rust server) is
17-
> described in [`docs/legacy.md`](legacy.md); the record shapes on the wire are unchanged
18-
> from that era, which is why the producer flag is still spelled `--gh-json-v3`.
15+
> emitter records directly into the hosted RDS Postgres. Then, once per CI run, the monorepo's
16+
> `scripts/refresh-bench-site.py` pings this repo's read service to flush its cache. The retired
17+
> v3 HTTP ingest path (`POST /api/ingest` into a Rust server) is described in
18+
> [`docs/legacy.md`](legacy.md); the record shapes on the wire are unchanged from that era, which
19+
> is why the producer flag is still spelled `--gh-json-v3`.
1920
2021
## Versioning: `SCHEMA_VERSION`
2122

@@ -80,8 +81,9 @@ The `reopen` mode does not clear the OS page cache.
8081
8182
## The ingest path — direct Postgres write + cache revalidate
8283

83-
The monorepo's `scripts/post-ingest.py --postgres` runs in benchmark CI (required, not
84-
best-effort — a failure fails the bench job and alerts):
84+
The monorepo's `scripts/post-ingest.py --postgres` runs in each benchmark CI job and does steps 1
85+
and 2 (required, not best-effort — a failure fails the bench job and alerts). The monorepo's
86+
`scripts/refresh-bench-site.py` does step 3, once per CI run:
8587

8688
1. **Commit upsert:** the writer builds the commit row from `git show` and upserts it
8789
(`ON CONFLICT (commit_sha) DO UPDATE`) before applying any record.
@@ -96,6 +98,14 @@ best-effort — a failure fails the bench job and alerts):
9698
`Authorization: Bearer $BENCH_REVALIDATE_TOKEN`, to mark the Next.js Data Cache stale so
9799
it refreshes against freshly written data: the next read serves the previous data once
98100
while each entry refreshes in the background, and later reads get the new data.
101+
The reusable workflow `.github/workflows/refresh-bench-site.yml` runs the script in its own
102+
job, after the ingest jobs of a `develop-bench.yml`, `nightly-bench.yml`, or
103+
`commit-metadata.yml` run finish. In `develop-bench.yml` and `nightly-bench.yml` it runs even
104+
when some benchmark jobs fail. The script does nothing when `BENCH_SITE_BASE_URL` or the
105+
token is unset. It retries a transport error, `429`, or `5xx` up to three times (after 5, 15,
106+
and 45 seconds). A revalidate that still fails, or fails with another status such as `401`,
107+
fails the refresh job but not the ingest jobs. After a successful revalidate, the script
108+
warms `/` and each group's default `n=100` bundle. A failed warm request only logs a warning.
99109

100110
### `measurement_id`: the frozen hash contract
101111

‎docs/bench/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,8 @@ the site reads straight from it.
3636
│ benchmark jobs │
3737
│ vortex-bench ─▶ JSONL records │
3838
│ scripts/post-ingest.py │
39+
│ once per run, after the jobs │
40+
│ scripts/refresh-bench-site.py │
3941
└───────┬──────────────────┬───────┘
4042
│ upsert rows │ POST /api/revalidate
4143
│ (direct SQL) │ (flush the read cache)

‎docs/bench/architecture/data-pipeline.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -88,9 +88,10 @@ every CI ingest run fails. Full procedure in
8888

8989
`post-ingest.py --postgres` (required in monorepo benchmark CI) writes the
9090
records **directly to RDS** as the least-privilege `bench_ingest` IAM role
91-
(`INSERT … ON CONFLICT (measurement_id) DO UPDATE`), then pings
92-
`POST {BENCH_SITE_BASE_URL}/api/revalidate` to flush the Next.js read cache.
93-
Each JSONL file applies in one transaction. The retired v3 HTTP path
91+
(`INSERT … ON CONFLICT (measurement_id) DO UPDATE`). Each JSONL file applies in
92+
one transaction. After the ingest jobs of a CI run finish, the monorepo's
93+
`scripts/refresh-bench-site.py` sends one `POST {BENCH_SITE_BASE_URL}/api/revalidate`
94+
to flush the Next.js read cache. The retired v3 HTTP path
9495
(`POST /api/ingest` into the Rust server) is described in
9596
[`../legacy.md`](../legacy.md).
9697

‎docs/bench/architecture/performance.md‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -61,24 +61,27 @@ app, all sharing the `bench-data` tag.
6161

6262
The backstop bounds staleness; `POST /api/revalidate`
6363
([`src/app/(bench)/bench/api/revalidate/route.ts`](../../../src/app/(bench)/bench/api/revalidate/route.ts))
64-
*removes* it. On a successful ingest the emitter calls it and it runs
64+
*removes* it. After the ingest jobs of a benchmark CI run finish, the monorepo's
65+
`scripts/refresh-bench-site.py` calls it once, and it runs
6566
`revalidateTag('bench-data', 'max')`, which marks the entries stale rather than
6667
dropping them: the next read of each serves the previous data once and refreshes
6768
it in the background, and the read after that gets the fresh data, instead of
6869
waiting out the 24-hour cap. No read waits on the database because of an ingest;
6970
new results can take about one more CDN cycle (five minutes) to show. Dropping
7071
the entries instead (`{ expire: 0 }`) failed in production: every fill then
71-
shared one advisory lock, the emitter revalidates after every benchmark job
72+
shared one advisory lock, the emitter then revalidated after every benchmark job
7273
(about once a minute during a develop run), and the reads that waited on the
7374
lock ran out of retries. A failed render of `/` then stayed in the CDN for five
7475
minutes. Each entry now fills under its own lock (see
75-
[read-path.md](read-path.md#database-connection)). The
76+
[read-path.md](read-path.md#database-connection)), and the monorepo now
77+
revalidates once per run, not once per job. The
7678
endpoint **fails closed** — a missing `BENCH_REVALIDATE_TOKEN` is a `503`, never
77-
a silent accept — and the post-ingest hook is **best-effort**: every failure in
78-
it is caught, logged, and swallowed so a cache-refresh problem can never change
79-
an ingest's exit code. That is why the whole path is safe to ship before the
80-
emitter cutover wires the token (it is inert until both `BENCH_SITE_BASE_URL`
81-
and the token are set).
79+
a silent accept. The refresh runs in its own CI job, apart from the ingest jobs,
80+
so a cache-refresh problem can never change an ingest's exit code. The script
81+
retries a transport error, `429`, or `5xx` up to three times. A revalidate that
82+
still fails fails only the refresh job. The refresh is inert until both
83+
`BENCH_SITE_BASE_URL` and the token are set, which is why the path was safe to
84+
ship before the emitter cutover wired the token.
8285

8386
### Keeping the function and its connections warm
8487

‎docs/bench/architecture/read-path.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,8 +68,8 @@ by two independent layers, so the common case is cheap and ingest can flush both
6868
are never cached.
6969

7070
**Freshness propagation.** The primary mechanism is
71-
`POST /api/revalidate` (token-gated; `src/app/(bench)/bench/api/revalidate/route.ts`): on a
72-
successful ingest, `scripts/post-ingest.py` calls it and it runs
71+
`POST /api/revalidate` (token-gated; `src/app/(bench)/bench/api/revalidate/route.ts`): once a
72+
benchmark CI run's ingest jobs finish, `scripts/refresh-bench-site.py` calls it, and it runs
7373
`revalidateTag('bench-data', 'max')`, the stale-while-revalidate profile: the
7474
next read of each Layer 1 entry serves the previous data once and refreshes the
7575
entry in the background, so no page waits on the database because of an ingest,

‎docs/bench/runbooks/deploy-secrets-setup.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -142,17 +142,17 @@ team's secret store. **Never paste a value into this document.**
142142
| `BENCH_DB_POOL_MAX` | Max connections in the pg pool (default `8`; omit to use default) | Existing monorepo Vercel project env |
143143
| `BENCH_DB_IDLE_TIMEOUT_MS` | Pool idle-connection timeout in ms (default `5000`; positive integer; omit to use default) | Existing monorepo Vercel project env |
144144
| `BENCH_DB_CONNECTION_TIMEOUT_MS` | New-connection and pool-slot timeout in ms (default `5000`; positive integer; omit to use default) | Existing monorepo Vercel project env |
145-
| `BENCH_REVALIDATE_TOKEN` | Bearer token for `POST /api/revalidate` — must match the value the monorepo's emitter caller (`post-ingest.py`) sends | Existing monorepo Vercel project env; coordinate with the monorepo caller (see note below) |
145+
| `BENCH_REVALIDATE_TOKEN` | Bearer token for `POST /api/revalidate` — must match the value the monorepo's caller (`refresh-bench-site.py`) sends | Existing monorepo Vercel project env; coordinate with the monorepo caller (see note below) |
146146

147-
> **`BENCH_REVALIDATE_TOKEN` coordination:** this token authenticates the monorepo's `post-ingest.py`
147+
> **`BENCH_REVALIDATE_TOKEN` coordination:** this token authenticates the monorepo's `refresh-bench-site.py`
148148
> revalidation caller against `POST /api/revalidate`. Setting it on the new project only makes the new
149149
> project ACCEPT a matching bearer — it does **not** redirect the emitter's traffic here. The emitter
150150
> POSTs to whatever base URL it is configured with (currently the monorepo-owned project's domain), so
151151
> until that target URL is repointed at this project (at the Phase-5 DNS cutover, or by updating the
152152
> emitter's configured URL), revalidation traffic continues to hit the **old** project — that is
153153
> expected during the Phase-4 setup window. Use the SAME token value the monorepo caller sends; if you
154154
> rotate it, update it in both places: this Vercel project and the monorepo's secret that feeds
155-
> `post-ingest.py`.
155+
> `refresh-bench-site.py`.
156156
157157
> **`BENCH_DATA_TAG` is not env-configurable:** the Next.js Data Cache tag is a compile-time constant
158158
> (`export const BENCH_DATA_TAG = 'bench-data'` in `src/bench/lib/data-cache.ts`), imported directly by

‎src/app/(bench)/bench/api/revalidate/route.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,9 @@ import { BENCH_DATA_TAG } from '@/bench/lib/data-cache';
1010

1111
/**
1212
* `POST /api/revalidate` marks the [`BENCH_DATA_TAG`] Data Cache entries stale, so
13-
* they refresh against freshly ingested data (see below). `scripts/post-ingest.py`
14-
* calls this after a successful Postgres write. Auth is a bearer token compared
13+
* they refresh against freshly ingested data (see below). The monorepo's
14+
* `scripts/refresh-bench-site.py` calls this once per benchmark CI run, after the
15+
* run's Postgres writes finish. Auth is a bearer token compared
1516
* in constant time against `BENCH_REVALIDATE_TOKEN`; a missing env var fails
1617
* closed with `503` so an unconfigured deployment never silently accepts
1718
* unauthenticated revalidation. The response is never CDN-cached.

‎src/bench/lib/data-cache.ts‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,16 +33,17 @@ export const BENCH_DATA_TAG = 'bench-data';
3333

3434
/**
3535
* Backstop revalidation interval (seconds) for every cached read. The
36-
* post-ingest revalidate hook is the primary freshness mechanism; this bound
36+
* revalidate hook that runs after each benchmark CI run is the primary
37+
* freshness mechanism; this bound
3738
* caps staleness at twenty-four hours if that hook ever fails to fire, so the
3839
* layer degrades to bounded staleness rather than serving stale data forever.
3940
*
4041
* The window is one day rather than one hour because this is a low-traffic
4142
* site: a longer backstop keeps the default last-100 window warm across
4243
* overnight idle gaps, so a CDN miss reads this Data Cache instead of paying
4344
* the multi-second cold database fill. The freshness cost is bounded by the
44-
* revalidate hook, which marks the tag stale on every ingest once its env is
45-
* set.
45+
* revalidate hook, which marks the tag stale after every benchmark CI run once
46+
* its env is set.
4647
*/
4748
export const DATA_CACHE_BACKSTOP_SECONDS = 86400;
4849

0 commit comments

Comments
 (0)