Add tenant-safe billing cost attribution

docs/README.md

@@ -12,6 +12,7 @@
 | Contribute | [CONTRIBUTING.md](../CONTRIBUTING.md) and [Testing isolation](guides/testing-isolation.md) |
 | Know what is next, and what nobody has checked | [ROADMAP.md](../ROADMAP.md) |
 | Deploy the Databricks side | [Deploying Databricks](guides/deploying-databricks.md) |
+| Attribute tenant job cost in currency | [Billing cost attribution](guides/billing-cost-attribution.md) |
 | Cut a release | [Releasing](guides/releasing.md) |
 | Assess it for security or compliance | [Threat model](security/threat-model.md), [SOC 2 mapping](compliance/soc2-mapping.md), [Data handling](compliance/data-handling.md) |
 

docs/compatibility.md

@@ -8,7 +8,8 @@ surrounding prose sounds.
 Last updated 2026-09-01 for the 1.0.1 release. The core Databricks statement and Jobs paths were
 re-run against the active development workspace, and the bundle was validated there. The proxy is
 verified against a real application database; the Bicep template compiles against the public
-schema. The real currency path remains blocked on a metastore-admin grant.
+schema. The currency path is implemented and locally verified, but its system-table read remains
+blocked on a metastore-admin grant in the verification workspace.
 A `Documented` row is not promoted to `Verified` by a stable version number; the matrix
 records the work that has been done, not a claim about future work.
 
@@ -63,6 +64,7 @@ request, package-consumer, publication and cleanup commands are in the
 | Capability | Status | Date | Evidence |
 |---|---|---|---|
 | Elapsed-time cost attribution (per-tenant, per-kind DBU) | **Documented** | 2026-08-29 | `OperationCostAttribution` against a real Postgres; the aggregation runs in `EXTRACT(EPOCH FROM (CompletedAt - ClaimedAt))` and is bounded by `ClaimedAt` to `CompletedAt` to exclude in-flight work. ADR 0012. |
+| Billing cost attribution (per-tenant DBU and effective list-price currency) | **Documented** | 2026-09-01 | Local tests cover indexed distinct tenant-owned run selection in PostgreSQL, the 500-run/one-query budget, bound workspace/run/window parameters, report/price overlap proration, corrections, malformed rows, polling deadline and best-effort cancellation. The live system-table read is not verified because the development identity lacks the grant. [Runbook](guides/billing-cost-attribution.md), ADR 0012. |
 | Bicep reference deployment template compiles | **Documented** | 2026-08-29 | `az bicep build --file infra/azure-container-apps/main.bicep` against the public schema. No deploy has been run. ADR 0014. |
 | OpenTelemetry export via the sample's opt-in pipeline | **Documented** | 2026-08-29 | The sample's `Program.cs` subscribes to `LakeWright.Multitenancy` when `Lakewright:OpenTelemetry:Enabled=true`. Vendor-specific wiring is the adopter's. ADR 0013. |
 | Entra ID token accepted as a Databricks bearer token (user principal) | **Verified** | 2026-09-01 | `LiveDatabricksTests`; previously [spike 01](planning/spike-01-statement-execution.md) |
@@ -121,10 +123,10 @@ request, package-consumer, publication and cleanup commands are in the
 
 ## Known gaps
 
-- No per-tenant cost attribution **in currency**. The elapsed-time proxy in
-  `OperationCostAttribution` ships, labelled `CostSource.Proxy`; a real billing read remains
-  blocked on a metastore-admin grant on `system.billing.usage` — see T5 in
-  [the threat model](security/threat-model.md) and ADR 0012.
+- Billing attribution is not live-verified in the development workspace. The implementation ships
+  as an opt-in, but the development identity still lacks access to `system.billing.usage` and
+  `system.billing.list_prices`; the proxy remains the default. See the
+  [billing runbook](guides/billing-cost-attribution.md), T5 in the threat model and ADR 0012.
 - No reference deployment *executed*. The Bicep template in `infra/azure-container-apps/`
   compiles; the workflow in `.github/workflows/deploy-azure.yml` is in place; no one has run a
   deploy with it, so the ingress half of encryption in transit and the managed identity path in

docs/decisions/0012-elapsed-time-cost-attribution.md

@@ -1,4 +1,4 @@
-# ADR 0012: Cost attribution as an interface, with the elapsed-time proxy as the first implementation
+# ADR 0012: Cost attribution from an elapsed-time proxy or correlated billing usage
 
 Status: accepted
 Date: 2026-08-29
@@ -15,19 +15,49 @@ That leaves two questions for the codebase. What does the cost API look like, an
 
 **The first implementation, `OperationCostAttribution` in `LakeWright.Multitenancy`, reads the application database.** It sums `EXTRACT(EPOCH FROM (CompletedAt - ClaimedAt))` per kind, weights by the configured DBU/hour, and labels the result `CostSource.Proxy`. Only terminal-state operations are counted, because an open-ended duration is not a cost number. The aggregation runs in Postgres rather than the application: a 100k-row operations table pulled across the wire to sum in memory is the kind of "small data" that is small until it isn't.
 
-**The interface exists so a real billing-table read replaces this implementation without changing the call sites.** The `CostSource` discriminator tells the caller which one ran, and `WarehouseSku` is null on a billing read because the rate comes from billing data rather than configuration.
+**A billing implementation is split at the existing dependency boundary.**
+`BillingCostAttribution` in `LakeWright.Multitenancy` selects tenant-owned job run ids and their
+operation kinds from PostgreSQL. `IBillingUsageReader` in `LakeWright.Core` is the typed seam;
+`DatabricksBillingUsageReader` in `LakeWright.Databricks` reads only those run ids from
+`system.billing.usage`. `LakeWright.AspNetCore` composes both. Multitenancy therefore does not
+reference the Databricks integration, and PostgreSQL is never assumed to be visible to Databricks
+SQL.
+
+**The billing query is fixed SQL with bound values.** It filters both `workspace_id` and
+`usage_metadata.job_run_id`, plus timestamp and `usage_date` bounds. PostgreSQL projects distinct
+run ownership through a tenant/time partial covering index and stops at 501 rows. At most 500 run
+ids enter one bound system-table query; a larger report returns `REPORT_TOO_LARGE` and HTTP 422
+rather than multiplying account-wide scans. Correlation happens in application code. A billing
+row for an id not selected from the tenant's operations is rejected rather than ignored.
+
+**Currency is explicit and additive.** `TenantCostSummary` and `CostByKind` retain their original
+constructors and gain init-only `EstimatedListCost` collections. Each `CurrencyAmount` keeps its
+currency code beside its amount; unlike currencies are never added. Cost is calculated from
+`system.billing.list_prices.pricing.effective_list.default`. A usage row is split at report and
+price-validity boundaries; its quantity is prorated by each overlap and that same quantity feeds
+both DBUs and cost. This is estimated effective list-price cost, not an adopter's negotiated invoice
+amount. The query sums every billing record, including negative correction rows, before the
+application aggregates by operation kind. Kinds are ordered by DBUs and then name, never by adding
+unlike currency amounts.
 
 **Configuration belongs to the adopter, not the library.** A product running more than one warehouse SKU picks one as the proxy, or moves to a billing read. `CostAttributionOptions` carries the SKU and DBU/hour rate; the section is bound by `AddLakeWrightCostAttribution` rather than read by `AddLakeWright`, so a contributor working on the application without a workspace never hits a validation failure on a value they have no opinion about.
 
 **The endpoint is opt-in via `MapLakeWrightCost`, behind the Viewer policy, with a 31-day window cap.** A customer-facing usage page that lets a tenant ask for a multi-year range is asking Postgres to sum a range nobody actually wants.
 
+**Billing statement work has an overall deadline.** `PollingTimeoutSeconds` defaults to 120.
+Caller cancellation, deadline expiry, and poll transport failures all trigger a five-second
+best-effort statement cancellation; cancellation failure never replaces the original result.
+
 ## Consequences
 
 **The threat model updates from "partly mitigated" to "mitigated with a proxy."** The proxy is documented as such: the per-tenant report is "elapsed compute time on the configured warehouse SKU" and not currency, and the dollar number an operator would want is hours late rather than actionable in flight. The ceiling in `OperationWorker:MaxInFlightPerTenant` remains the control that acts in time; this one reports afterwards.
 
-**`system.billing.usage` is now a real extension point rather than a sentence in the threat model.** A product that gets the grant wires an `ICostAttribution` against the API, returns `CostSource.Billing`, and the rest of the codebase reads it the same way. No second interface, no second endpoint, no migration.
+**`system.billing.usage` is now a shipping opt-in rather than only an extension point.** A product
+with the system-table grants calls `AddLakeWrightBillingCostAttribution` after the base Databricks
+registration. Workspaces without those grants keep the proxy. Billing records can arrive hours
+after a run, so the billing report is eventually consistent and an absent row is not replaced with
+proxy data.
 
 **A test pins the property that the library's own instruments never carry a tenant id.** Per-tenant totals come from `operations` and `audit_events` rather than from the metrics, which is the property the cardinality-bomb rule was written to protect. A future change that adds a `tenant` or `tenantid` tag to a metric call site fails the build with the offending line.
 
 **`CostAttribution` joins the v0.2 milestone rather than v0.1.** v0.1 was the eight-week milestone whose definition of done did not name this. The published version is `0.1.2-preview.1`; the next published version with cost attribution is `0.2.0-preview.1`, which carries a breaking-change note for the `Operation` rows the implementation reads (none, today, but documented for the next maintainer).
-

docs/guides/billing-cost-attribution.md

@@ -0,0 +1,78 @@
+# Billing cost attribution
+
+The default `OperationCostAttribution` is deliberately a DBU proxy. Use the billing reader only
+when the application identity can read the Databricks billing system tables and the product needs
+currency-denominated effective list cost.
+
+## Prerequisites
+
+- The Databricks identity registered by `AddLakeWrightDatabricks` can use
+  `system.billing` and select from `system.billing.usage` and
+  `system.billing.list_prices`.
+- `DatabricksBilling:WorkspaceId` is the workspace whose Lakeflow run ids are stored in this
+  application's `operations.ExternalId` column.
+- Operation processing uses the LakeWright worker, so `ExternalId` is the Lakeflow
+  `job_run_id`. Statement ids and job ids are different identifiers and are not accepted.
+
+System billing rows are account-wide and typically arrive after the workload finishes. The reader
+always filters `workspace_id`, job run id, timestamp and `usage_date`; a report can therefore be
+empty while recent records are still being delivered.
+
+## Registration
+
+```csharp
+builder.Services.AddLakeWright(builder.Configuration);
+builder.Services.AddLakeWrightDatabricks(builder.Configuration);
+builder.Services.AddLakeWrightBillingCostAttribution(builder.Configuration);
+```
+
+```json
+{
+  "Databricks": {
+    "WorkspaceUrl": "https://adb-....azuredatabricks.net",
+    "WarehouseId": "..."
+  },
+  "DatabricksBilling": {
+    "WorkspaceId": "...",
+    "PollIntervalMilliseconds": 250,
+    "PollingTimeoutSeconds": 120
+  }
+}
+```
+
+`AddLakeWrightBillingCostAttribution` replaces the proxy registration. Keep
+`AddLakeWrightCostAttribution` instead in environments without the grants.
+
+The response retains the existing DBU fields. `EstimatedListCost` is a collection of
+`CurrencyAmount` values on both the summary and each operation-kind row. The amount uses
+`pricing.effective_list.default`; it does not include a private negotiated discount and must not be
+presented as an invoice total. `ElapsedSeconds` is zero for billing rows because the billing table
+reports quantities and usage intervals, not the operation wall-clock value used by the proxy.
+
+Usage rows that cross either report-window or price-validity boundaries are prorated by their
+overlap. The same prorated quantity feeds both DBUs and effective list cost, and a price change
+inside one usage row contributes one segment at each effective price. The endpoint never adds
+amounts in unlike currencies when ordering operation kinds; it orders by DBUs, then kind.
+
+One report is limited to 500 distinct tenant-owned job runs and issues one billing-system query.
+HTTP 422 with code `REPORT_TOO_LARGE` means the caller must narrow the window. This prevents a
+high-volume tenant from turning one request into repeated scans of the account-wide billing table.
+A statement that remains pending past `PollingTimeoutSeconds` is cancelled best-effort and returns
+the transient code `POLL_TIMEOUT`.
+
+## Live verification
+
+Run this only in the non-production workspace whose id is configured above.
+
+1. Start one known LakeWright operation and wait for its `ExternalId` job run to finish.
+2. Wait for the corresponding `system.billing.usage` record to arrive. Databricks documents a
+   typical delay of up to 12 hours for original records.
+3. Query the cost endpoint for a window that contains the run. Confirm `Source` is `Billing`, the
+   run contributes once to `Operations`, and `EstimatedListCost` carries the expected currency.
+4. Compare the returned DBUs and effective list cost with a direct, read-only query over the same
+   workspace id, job run id and window.
+5. Remove the system-table grants from a disposable verification principal and confirm the endpoint
+   returns HTTP 502 with `PERMISSION_DENIED`; restore the grant afterwards.
+
+No test data needs to be written to the system tables. Do not use a production workspace merely to
+obtain a billing row.

docs/security/threat-model.md

@@ -84,12 +84,11 @@ caps how many operations one tenant can have running at once, across every worke
 ceiling on the compute a runaway loop can buy before anyone notices. Warehouse auto-stop bounds
 the idle half. Evidence: `OperationClaimTests.A_tenant_at_its_ceiling_is_skipped_rather_than_failed`.
 
-The reporting half is `ICostAttribution` (ADR 0012), with the elapsed-time proxy as its first
-implementation. The proxy sums `EXTRACT(EPOCH FROM (CompletedAt - ClaimedAt))` per kind and
-weights it by the configured warehouse SKU's DBU/hour. The result is `CostSource.Proxy` and
-is documented as a proxy rather than a currency read. The interface exists so a real
-billing-table read replaces this implementation without changing the call sites; the
-`CostSource` discriminator tells the caller which one ran.
+The reporting half is `ICostAttribution` (ADR 0012). The default elapsed-time proxy sums
+`EXTRACT(EPOCH FROM (CompletedAt - ClaimedAt))` per kind and weights it by the configured
+warehouse SKU's DBU/hour. The opt-in billing implementation instead returns DBUs and effective
+list-price currency amounts from the Databricks system tables; the `CostSource` discriminator
+tells the caller which one ran.
 
 Nothing here caps the *cost of one query*. A single operation against a large warehouse is bounded
 only by the run timeout — which cancels the run rather than merely abandoning it, so the timeout
@@ -102,16 +101,23 @@ The tenant does not reach the compute in a form billing can see. `TenantScopedJo
 `lakewright_tenant_id` as a *job parameter*, and Databricks attributes usage in
 `system.billing.usage` by `custom_tags`, which come from the job or cluster definition, not from
 per-run parameters. Tagging per run is not available on `RunNow`, and a job per tenant does not
-scale. So attribution has to go the other way: join `system.billing.usage` to a run id, and join
-that run id to `operations.ExternalId`, which this project already stores. The tenant identity
-lives in our database, not in theirs, and that is the correct place for it.
-
-Reading those tables needs a grant this project does not have. Querying `system.billing.usage` as
+scale. So attribution goes the other way. PostgreSQL first selects `operations.ExternalId` for the
+resolved tenant. The Databricks query receives only those job run ids as bound parameters and
+also filters the configured `workspace_id`; application code joins the returned rows to operation
+kinds. PostgreSQL is never named in Databricks SQL. A provider response containing any run id not
+in the tenant-owned set fails the report instead of being attributed. The PostgreSQL selection is
+index-backed and stops after 501 distinct runs; more than 500 returns HTTP 422, so one request can
+issue at most one account-wide billing query. Pending statements also have an overall deadline and
+are cancelled best-effort on deadline, transport failure, or caller cancellation.
+
+Reading those tables needs a grant the default development identity does not have. Querying `system.billing.usage` as
 the workspace identity returns `INSUFFICIENT_PERMISSIONS: User does not have USE SCHEMA on Schema
-'system.billing'`. It is a metastore-admin grant, so the prerequisite for cost attribution in
-currency is an administrative decision rather than a feature. The proxy is the alternative until
-that grant is made; a product with the grant registers its own `ICostAttribution`, returns
-`CostSource.Billing`, and the rest of the system reads it the same way.
+'system.billing'`. It is an administrative decision, so the proxy remains the default. A product
+with access to both `system.billing.usage` and `system.billing.list_prices` opts in with
+`AddLakeWrightBillingCostAttribution`. The code path is covered locally with fixed-query,
+window and price-boundary proration, malformed-row, correction, ownership, query-budget, polling
+and cancellation tests; the system-table grants and live response shape remain workspace
+verification steps.
 
 ### T6. Denial of service against the operation queue
 

samples/Signalboard/Program.cs

@@ -29,8 +29,8 @@
 
 // Cost attribution: opt-in via configuration, like the Databricks clients. The sample ships the
 // elapsed-time proxy because it costs nothing to run and makes the cost endpoint answerable
-// without a live workspace. A product wired to its own billing table replaces this registration
-// with its own ICostAttribution.
+// without a live workspace. A product with the system-table grants replaces this call with
+// AddLakeWrightBillingCostAttribution; see docs/guides/billing-cost-attribution.md.
 builder.Services.AddLakeWrightCostAttribution(builder.Configuration);
 
 // OpenTelemetry: opt-in via configuration. The library publishes plain System.Diagnostics

src/LakeWright.AspNetCore/CostEndpoints.cs

@@ -76,7 +76,38 @@ private static async Task<IResult> GetCostAsync(
             });
         }
 
-        var summary = await cost.ResolveAsync(tenant, effectiveFrom, effectiveUntil, cancellationToken);
-        return Results.Ok(summary);
+        try
+        {
+            var summary = await cost.ResolveAsync(
+                tenant,
+                effectiveFrom,
+                effectiveUntil,
+                cancellationToken);
+            return Results.Ok(summary);
+        }
+        catch (BillingUsageException exception)
+        {
+            if (exception.Code == "REPORT_TOO_LARGE")
+            {
+                return Results.Problem(
+                    statusCode: StatusCodes.Status422UnprocessableEntity,
+                    title: "Billing report is too large.",
+                    detail: $"Narrow the window to at most {BillingUsageLimits.MaxJobRunsPerReport} distinct job runs.",
+                    extensions: new Dictionary<string, object?>
+                    {
+                        ["code"] = exception.Code,
+                        ["maxJobRuns"] = BillingUsageLimits.MaxJobRunsPerReport
+                    });
+            }
+
+            return Results.Problem(
+                statusCode: StatusCodes.Status502BadGateway,
+                title: "Billing usage is unavailable.",
+                extensions: new Dictionary<string, object?>
+                {
+                    ["code"] = exception.Code,
+                    ["transient"] = exception.IsTransient
+                });
+        }
     }
 }