-
Notifications
You must be signed in to change notification settings - Fork 523
feat(cli): add inspect db toast-sizes command #6528
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Changes from all commits
f970296
36c3ee0
746fda8
5ce4286
463eb9e
f7f6e49
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # db-toast-sizes | ||
|
|
||
| This command displays TOAST table sizes and dead chunk counts for every user table that has a TOAST relation. When a column value exceeds ~2 kB (TEXT, JSONB, bytea), Postgres stores it out-of-line in a companion TOAST table. Autovacuum runs on the TOAST table independently from the main heap, so it can accumulate dead chunks even when the parent table looks healthy by its own dead-tuple count. | ||
|
|
||
| A table that appears fine by `vacuum-stats` or `bloat` alone can still have significant TOAST bloat that wastes disk space and slows reads. High `TOAST Dead %` values indicate that autovacuum is not keeping up with the TOAST table and a manual `VACUUM` may be needed. | ||
|
|
||
| ``` | ||
| TABLE │ TOTAL SIZE │ HEAP SIZE │ TOAST SIZE │ TOAST LIVE CHUNKS │ TOAST DEAD CHUNKS │ TOAST DEAD % │ LAST AUTOVACUUM │ LAST VACUUM | ||
| ────────────────────────┼────────────┼───────────┼────────────┼───────────────────┼───────────────────┼──────────────┼────────────────────┼───────────── | ||
| public.documents │ 4200 MB │ 800 MB │ 3400 MB │ 250000 │ 18000 │ 6.7 │ 2024-03-01 04:12 │ | ||
| public.media_assets │ 890 MB │ 120 MB │ 770 MB │ 80000 │ 200 │ 0.2 │ 2024-03-02 01:45 │ | ||
| public.messages │ 340 MB │ 280 MB │ 60 MB │ 95000 │ 0 │ 0.0 │ 2024-03-02 03:10 │ | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ import { inspectDbTableStatsCommand } from "./table-stats/table-stats.command.ts | |
| import { inspectDbTotalIndexSizeCommand } from "./total-index-size/total-index-size.command.ts"; | ||
| import { inspectDbTotalTableSizesCommand } from "./total-table-sizes/total-table-sizes.command.ts"; | ||
| import { inspectDbTrafficProfileCommand } from "./traffic-profile/traffic-profile.command.ts"; | ||
| import { inspectDbToastSizesCommand } from "./toast-sizes/toast-sizes.command.ts"; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ NIT · The new imports do not follow the established ordering in either the command registry or sibling command files. Evidence: db.command.ts:22-27 places toast-sizes after unused-indexes rather than before the total-* imports in the otherwise grouped ordering. toast-sizes.command.ts:1-4 places its handler import last, while every other sibling imports its handler immediately after Effect's Command import. Suggested fix: Move the registry import before total-index-size and move the handler import to the second line of toast-sizes.command.ts. |
||
| import { inspectDbUnusedIndexesCommand } from "./unused-indexes/unused-indexes.command.ts"; | ||
| import { inspectDbVacuumStatsCommand } from "./vacuum-stats/vacuum-stats.command.ts"; | ||
|
|
||
|
|
@@ -42,6 +43,7 @@ export const inspectDbCommand = Command.make("db").pipe( | |
| inspectDbVacuumStatsCommand, | ||
| inspectDbTableStatsCommand, | ||
| inspectDbTrafficProfileCommand, | ||
| inspectDbToastSizesCommand, | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ NIT · Adding toast-sizes raises the inspect-db leaf count to 26, leaving four comments that still say 25. Evidence: apps/cli/src/commands/inspect/db/db.command.ts:32-59 registers 26 leaves. Stale counts remain at inspect-db-command.ts:10 and :40, db.layers.ts:13, and db.layers.unit.test.ts:9. Suggested fix: Change the four counts from 25 to 26. |
||
| inspectDbCacheHitCommand, | ||
| inspectDbIndexUsageCommand, | ||
| inspectDbTotalIndexSizeCommand, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,6 +19,7 @@ import { outliersSpec } from "./outliers/outliers.query.ts"; | |
| import { replicationSlotsSpec } from "./replication-slots/replication-slots.query.ts"; | ||
| import { roleStatsSpec } from "./role-stats/role-stats.query.ts"; | ||
| import { tableStatsSpec } from "./table-stats/table-stats.query.ts"; | ||
| import { toastSizesSpec } from "./toast-sizes/toast-sizes.query.ts"; | ||
| import { trafficProfileSpec } from "./traffic-profile/traffic-profile.query.ts"; | ||
| import { vacuumStatsSpec } from "./vacuum-stats/vacuum-stats.query.ts"; | ||
|
|
||
|
|
@@ -248,11 +249,27 @@ const cases: ReadonlyArray<Case> = [ | |
| }, | ||
| expect: ["public", "100", "50", "12.0", "1:1 (Balanced)"], | ||
| }, | ||
| { | ||
| spec: toastSizesSpec, | ||
| params: "schemas1", | ||
| row: { | ||
| name: "public.events", | ||
| total_size: "120 kB", | ||
| heap_size: "80 kB", | ||
| toast_size: "40 kB", | ||
| toast_live_chunks: 1200, | ||
| toast_dead_chunks: 80, | ||
| toast_dead_pct: "6.3", | ||
| last_autovacuum: "2025-01-15 03:00", | ||
| last_vacuum: "", | ||
| }, | ||
| expect: ["public.events", "120 kB", "40 kB", "1200", "80", "6.3", "2025-01-15 03:00"], | ||
| }, | ||
|
Comment on lines
+253
to
+267
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 MINOR ·
Evidence: apps/cli-e2e/src/tests/inspect.e2e.test.ts:31-45 lists 13 active subcommands and drives success and connection-failure subprocess tests at lines 66-90. apps/cli-e2e/fixtures/pg contains matching fixtures for those 13 commands but none for toast-sizes. The added integration case exercises Suggested fix: Add a toast-sizes fixture and entry to the cli-e2e |
||
| ]; | ||
|
|
||
| describe("inspect db specs (per-subcommand correctness)", () => { | ||
| it("covers all 13 active subcommands", () => { | ||
| expect(cases).toHaveLength(13); | ||
| it("covers all 14 active subcommands", () => { | ||
| expect(cases).toHaveLength(14); | ||
| }); | ||
|
|
||
| for (const testCase of cases) { | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| import { Command } from "effect/unstable/cli"; | ||
| import { INSPECT_DB_FLAGS, inspectDbCommandHandler } from "../inspect-db-command.ts"; | ||
| import { inspectDbRuntimeLayer } from "../db.layers.ts"; | ||
| import { inspectDbToastSizes } from "./toast-sizes.handler.ts"; | ||
|
|
||
| export const inspectDbToastSizesCommand = Command.make("toast-sizes", INSPECT_DB_FLAGS).pipe( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 MINOR · The new command lacks a docs default override, so its published Evidence: apps/cli/src/commands/inspect/db/inspect-db-command.ts:21-24 declares the primitive default as false, while apps/cli/src/commands/inspect/db/inspect-query.ts:193-196 defaults execution to linked. apps/cli/src/docs/docs-spec.ts:254 falls back to the primitive default, and apps/cli/src/docs/docs-spec.tables.ts:167-179 overrides every other active inspect-db command but not toast-sizes. Suggested fix: Add |
||
| Command.withDescription( | ||
| "Displays TOAST table sizes and dead chunk counts for every user table that has overflow storage. " + | ||
| "Autovacuum runs on TOAST relations independently, so dead chunks can accumulate even when the " + | ||
| "main heap looks healthy.", | ||
| ), | ||
|
Comment on lines
+7
to
+11
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ NIT · The command's five-sentence help description is substantially longer than every sibling inspect-db description. Evidence: toast-sizes.command.ts:7-13 contains five sentences. The other 25 command files each use a single-sentence description; examples include vacuum-stats.command.ts:7 and traffic-profile.command.ts:10-12. Suggested fix: Use a concise one-sentence CLI description and retain the detailed explanation in the docs overlay. |
||
| Command.withShortDescription("Show TOAST table sizes and dead chunk counts"), | ||
| Command.withHandler(inspectDbCommandHandler(inspectDbToastSizes)), | ||
| Command.provide(inspectDbRuntimeLayer("toast-sizes")), | ||
| ); | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| import { makeInspectDbHandler } from "../inspect-query.ts"; | ||
| import { toastSizesSpec } from "./toast-sizes.query.ts"; | ||
|
|
||
| export const inspectDbToastSizes = makeInspectDbHandler(toastSizesSpec, "inspect.db.toast-sizes"); |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| import { | ||
| inspectFloat1, | ||
| inspectInt, | ||
| inspectPlainText, | ||
| inspectText, | ||
| type InspectQuerySpec, | ||
| } from "../inspect-query.ts"; | ||
| import { INTERNAL_SCHEMAS, likeEscapeSchema } from "../inspect-schemas.ts"; | ||
|
|
||
| const SQL = ` | ||
| SELECT | ||
| FORMAT('%I.%I', n.nspname, main.relname) AS name, | ||
| pg_size_pretty(pg_total_relation_size(main.oid)) AS total_size, | ||
| pg_size_pretty(pg_relation_size(main.oid)) AS heap_size, | ||
| pg_size_pretty(pg_total_relation_size(main.reltoastrelid)) AS toast_size, | ||
| COALESCE(ts.n_live_tup, 0) AS toast_live_chunks, | ||
| COALESCE(ts.n_dead_tup, 0) AS toast_dead_chunks, | ||
| COALESCE( | ||
| round(100.0 * ts.n_dead_tup / nullif(ts.n_live_tup + ts.n_dead_tup, 0), 1), | ||
| 0.0 | ||
| ) AS toast_dead_pct, | ||
| COALESCE(to_char(ts.last_autovacuum, 'YYYY-MM-DD HH24:MI'), '') AS last_autovacuum, | ||
| COALESCE(to_char(ts.last_vacuum, 'YYYY-MM-DD HH24:MI'), '') AS last_vacuum | ||
| FROM pg_class main | ||
| JOIN pg_namespace n ON n.oid = main.relnamespace | ||
| LEFT JOIN pg_stat_all_tables ts ON ts.relid = main.reltoastrelid | ||
| WHERE main.relkind = 'r' | ||
| AND main.reltoastrelid <> 0 | ||
| AND pg_total_relation_size(main.reltoastrelid) > 0 | ||
| AND NOT n.nspname LIKE ANY($1) | ||
| ORDER BY pg_total_relation_size(main.reltoastrelid) DESC`; | ||
|
|
||
| export const toastSizesSpec: InspectQuerySpec = { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ⚪ NIT ·
Evidence: toast-sizes.query.ts:32 declares the spec without JSDoc. Each of the other 13 active query files has a JSDoc immediately before its exported Suggested fix: Add a short JSDoc describing the report and its notable size and timestamp semantics. |
||
| name: "toast-sizes", | ||
| sql: SQL, | ||
| params: () => [likeEscapeSchema(INTERNAL_SCHEMAS)], | ||
| headers: [ | ||
| "Table", | ||
| "Total Size", | ||
| "Heap Size", | ||
| "TOAST Size", | ||
| "TOAST Live Chunks", | ||
| "TOAST Dead Chunks", | ||
| "TOAST Dead %", | ||
| "Last Autovacuum", | ||
| "Last Vacuum", | ||
| ], | ||
| project: (row) => [ | ||
| inspectText(row["name"]), | ||
| inspectText(row["total_size"]), | ||
| inspectText(row["heap_size"]), | ||
| inspectText(row["toast_size"]), | ||
| inspectInt(row["toast_live_chunks"]), | ||
| inspectInt(row["toast_dead_chunks"]), | ||
| inspectFloat1(row["toast_dead_pct"]), | ||
| inspectPlainText(row["last_autovacuum"]), | ||
| inspectPlainText(row["last_vacuum"]), | ||
| ], | ||
| }; | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
⚪ NIT ·
documentation· source: claudeThe documentation incorrectly describes the approximately 2 kB TOAST threshold as applying to an individual column value rather than the row tuple.
Evidence: docs/supabase/inspect/db-toast-sizes.md:3 says Postgres moves a value out-of-line when that value exceeds approximately 2 kB. PostgreSQL instead applies the threshold to the tuple and compresses or moves eligible attributes until it fits.
Suggested fix: Explain that when a row exceeds the threshold, PostgreSQL compresses and/or moves eligible variable-length attributes out of line.