Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
87 changes: 61 additions & 26 deletions .agents/skills/audit-copilot-workarounds/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,73 @@
---
name: audit-copilot-workarounds
description: Use periodically to verify each documented workaround is still
needed against current Copilot upstream. Inventories drift, dispatches
parallel cluster audits, runs live probes, and produces deletion commits
with experimental justification.
description: Use periodically to verify each Copilot workaround against the
current upstream. Inventories provider registrations and reference URLs,
dispatches parallel cluster audits, runs live probes, and produces focused
deletion or maintenance commits with experimental justification.
---

# Audit Copilot Workarounds

Workarounds rot. Re-validate them against current Copilot upstream.
Workarounds and pinned wire mimicry rot. Revalidate them against the current
Copilot upstream.

## Flow
## Build the inventory

1. Inventory drift between `index.ts` registrations and AGENTS.md
"Data Plane Workarounds".
2. Dispatch parallel read-only audits, one per source/target × API cluster.
3. Loop further agent rounds until remaining open questions are only
"needs live probe" or "needs human decision".
4. Run live probes for the former.
5. Land deletion + doc commits. Hand the human the rest.
The provider code and the reference URLs beside each workaround are the
inventory; there is no separate documentation list to reconcile.

1. Start from
`packages/provider-copilot/src/interceptors/{chat-completions,messages,responses}/index.ts`
and `packages/provider-copilot/src/defaults.ts`. Follow every registered
interceptor and default-enabled shim to its implementation and tests.
2. Sweep the rest of `packages/provider-copilot/src` for non-pricing reference
URLs and for vendor constants, thresholds, timeouts, retries, and pinned wire
values that require a reference but may be missing one. Pricing citations
belong to `fetching-models-pricing`; everything else remains in this audit.
3. Include provider-level request/result shaping and catalog shaping even when
they are not interceptors. In particular, inspect `provider.ts`,
`fetch-models.ts`, `known-models.ts`, `model-selection.ts`, and
`merge-claude-variants.ts` together with their imports and tests.
4. Include the authentication fingerprint and management/data-plane behavior in
`auth.ts`, plus Responses item identity and replay handling rooted at
`interceptors/responses/item-id-membrane.ts`. Follow adjacent carrier and
compaction modules rather than assuming the interceptor registry contains the
whole workaround.
5. Record each item's owning module, reference URLs, affected source and target
APIs, models, account scope, default flag state, tests, and exit condition:
delete an obsolete workaround, refresh pinned mimicry, or retain it with
current evidence.

## Audit flow

1. Group the inventory by source API, target API, and behavior so independent
clusters can be investigated without overlapping edits.
2. Dispatch parallel read-only audits, one per cluster. Recheck the cited
upstream or prior-art source, inspect current Copilot behavior, and record the
exact code path that would be deleted or refreshed.
3. Continue audit rounds until every open question requires either a live probe
or a human policy decision.
4. Run the required live probes, then land each proven deletion or maintenance
update with its tests and reference cleanup. Hand unresolved policy decisions
to the human.

## Extra constraints

- **Live probes follow `probing-copilot`** — credential discovery, token
exchange, headers, and direct upstream calls all live there. Don't ask the
human for credentials and don't route probes through our gateway.
- **Full-matrix evidence.** Test every applicable model from `GET /models`,
on every account in D1 (different account types may diverge). One model on
one account is never enough to delete.
- **One workaround per deletion commit.** Never bundle.
- **Each deletion commit message must contain the live experiment
conclusion** that justified it: which models tested, which values,
exact upstream error text when relevant, and the originating commit
sha being reverted.
- **When a policy value (threshold, floor, retry count) has no official
upstream basis, the comment must say so explicitly** in addition to
citing prior-art permalinks.
exchange, headers, proxy fallback order, and direct upstream calls all live
there. Do not ask the human for credentials and do not route probes through
Floway.
- **Full-matrix evidence.** Test every applicable model from `GET /models` on
every account in D1; different accounts can diverge. One model on one account
is never enough to delete a workaround.
- **Source references are leads, not proof.** A still-valid URL explains why a
workaround exists; only current upstream behavior proves whether it remains
necessary.
- **One workaround per deletion commit.** Never bundle independent removals.
- **Each deletion commit message must contain the live experiment conclusion**
that justified it: accounts and models tested, values exercised, exact
upstream error text when relevant, and the originating commit SHA being
reverted.
- **When a policy value has no official upstream basis, say so in code.**
Thresholds, floors, timeouts, retry counts, and pinned fingerprints must
identify an empirical or prior-art basis and include the relevant permalink.
88 changes: 60 additions & 28 deletions .agents/skills/backfill-model-pricing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,39 +5,71 @@ description: Write or rewrite usage.unit_price for a selected slice of live D1 u

# Backfill Model Pricing

`usage` stores one row per
`(key_id, model, upstream, model_key, hour, pricing_selector, dimension)`.
`tokens` is the count and `unit_price` is the request-time USD-per-million-
token rate snapshot. `pricing_selector` is canonical selector JSON; `{}`
is the base coordinate.
`usage` stores one metric row per unique
`(key_id, model, COALESCE(upstream, ''), model_key, hour, pricing_selector, metric)`.
`quantity` is a canonical non-negative decimal string. `unit_price` is either
NULL or a canonical non-negative decimal string containing USD per one base
unit of that metric. `pricing_selector` is canonical selector JSON; `{}` is the
Base coordinate.

`BILLING_METRICS` in `packages/protocols/src/common/pricing.ts` owns the
complete metric domain, and `BillingMetric` is derived from it. Read that array
before every operation and enumerate the metrics present in the
selected database slice; do not maintain another metric list in this procedure.
The repository read path rejects stored metric values outside that domain.

Realized cost is the sum of `quantity * unit_price` for priced metric rows. Both
operands are decimal strings in storage, and there is no additional scaling
step. Aggregation skips NULL-price rows: cost is NULL only when no metric row was
priced, while a non-NULL cost may still be partial when other metric rows remain
unpriced.

## Procedure

1. Announce the environment. Default to production (`--remote`).
2. Establish the exact model, upstream, hour range, timezone, dimensions, and
write mode:
2. Before planning or running an UPDATE, re-read the current implementations in
`packages/gateway/src/repo/sql.ts` (`SqlUsageRepo` and usage row assembly),
`packages/gateway/src/repo/types.ts` (the usage contracts), and
`packages/gateway/src/control-plane/token-usage/aggregate.ts` (cost
aggregation). They are the authority if this procedure and the runtime ever
diverge.
3. Establish the exact model, upstream, hour range, timezone, metrics, and write
mode:
- fill only rows where `unit_price IS NULL`; or
- overwrite the selected range.
3. If intent is incomplete, show enabled upstreams and grouped NULL-rate rows
by `(upstream, model_key, pricing_selector, dimension)`, including count
and `MIN/MAX(hour)`. Do not guess.
4. Read the current provider rate source or the upstream's
4. If intent is incomplete, show enabled upstreams and grouped NULL-price rows
by `(upstream, model_key, pricing_selector, metric)`, including count and
`MIN/MAX(hour)`. Do not guess.
5. Read the current provider rate source or the upstream's
`config_json.models[].pricing`. Resolve one `ModelPricing` per
`(upstream, model_key)`.
5. Match the stored `pricing_selector` exactly against
`ModelPricing.entries` using canonical selector JSON.
- Current runtime selector misses are stored as `{}` with Base rates.
- A historical non-Base selector absent from today's catalog indicates
catalog drift; stop and investigate rather than guessing its old rates.
- Read only `entry.rates[dimension]`.
- A missing dimension is unpriced; there is no cache, image, or other
field-by-field fallback.
6. Preview the affected count and representative rows.
7. Execute one UPDATE per exact
`(slice, pricing_selector, dimension)`. Include
`unit_price IS NULL` only in fill mode.
8. Re-query every slice and report the selector, dimension, rate, rows updated,
and remaining NULL count.
6. Match the stored `pricing_selector` exactly against `ModelPricing.entries`
using canonical selector JSON.
- An exact selector hit uses that entry. A selector miss in a catalog with a
Base entry is recorded as `{}` with the whole Base vector.
- A non-Base selector on an unpriced row is ordinary when no `ModelPricing`
existed: runtime facts form the selector before rate lookup, and it is
retained when no Base rates exist. It is not catalog drift by itself.
- A priced sibling row for the same `(upstream, model_key)` proves that a
catalog existed. If such a slice also contains an unpriced non-Base selector
absent from today's catalog, stop and investigate historical catalog drift.
Without a priced sibling, resolve today's catalog normally but never infer
historical rates.
- Read only the evaluated `entry.rates[metric]`; those values are already USD
per base metric unit. Never transcribe a numeric literal from a provider
`pricing.ts` into `unit_price`.
- A missing metric is unpriced; there is no cache, image, audio, rerank, or
other field-by-field fallback.
7. Preview the affected count and representative rows, including the current
and proposed decimal-string `unit_price`.
8. Execute one UPDATE per exact `(slice, pricing_selector, metric)`. Include
`unit_price IS NULL` only in fill mode, preserve NULL upstream matching with
`COALESCE(upstream, '')`, and bind the new rate as a decimal string.
9. Re-query every slice and report the selector, metric, rate, rows updated, and
remaining NULL count per metric. Compare those NULL counts with the expected
metric set; a non-NULL aggregate cost does not prove the slice is fully
priced. Independently validate decimal-string multiplication on
representative rows.

Use the local Wrangler dependency and read the D1 database name from
`wrangler.jsonc`. Never ask the human for credentials already available to
Expand All @@ -49,8 +81,8 @@ Wrangler.
- Do not write a JSON rate vector into `unit_price`; it is one scalar.
- Do not map an obsolete selector to a newer “closest” threshold.
- Leave rows NULL when the current catalog has no exact entry or explicit
dimension rate.
- Realized cost is `SUM(tokens * unit_price) / 1e6`; validate each scalar
before writing.
metric rate.
- Validate decimal-string multiplication without converting through JavaScript
numbers or SQL floating-point arithmetic.
- Writing today's documented rate into historical rows is intentional unless
the human explicitly supplies price-at-the-time data.
71 changes: 55 additions & 16 deletions .agents/skills/fetching-models-pricing/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,21 @@ Maintain the notional per-token rate cards in:
These providers are subscription-backed or self-hosted. Floway records
notional API-equivalent value so the usage dashboard remains comparable.

`ModelPricing.entries[].rates` stores decimal-string USD prices per one base
`BillingMetric` unit. The ten-member `BILLING_METRICS` array in
`packages/protocols/src/common/pricing.ts` owns the complete metric domain, and
`BillingMetric` is derived from it. Read the array rather than copying its
members into this procedure; each provider table may price only the defensible
subset for a model.

## Procedure

1. Fetch the provider's live catalog and diff its ids against the table's
string and RegExp keys. Record new, retired, and renamed models.
2. Find a defensible rate source for every new id:
- Prefer the model vendor's first-party API.
- For open weights with no vendor API, use the cheapest credible commodity
host that publishes the required dimensions.
host that publishes the required metrics.
- For retired versions, use a permalink or dated archive from when that
version was current.
3. Cross-check at least two sources. models.dev remains useful as an independent
Expand All @@ -36,25 +43,53 @@ notional API-equivalent value so the usage dashboard remains comparable.

OpenRouter prices below first-party rates are usually mirror-host prices,
not the canonical vendor rate.
4. Author one `ModelPricing` with `modelPricing` and `pricingEntry`:
4. Author pricing with the token-rate conversion helpers and decimal strings:

```ts
modelPricing(
pricingEntry({ input: 2.5, input_cache_read: 0.25, output: 15 }),
pricingEntry(
{ input: 5, input_cache_read: 0.5, output: 22.5 },
{ inputTokens: { operator: 'gt', value: 272000 } },
),
)
import {
modelPricing,
tokenBasePricing,
tokenPricingEntry,
type PriceVector,
} from '@floway-dev/protocols/common';

const EXAMPLE_BASE_RATES = {
input_tokens: '1',
input_cache_read_tokens: '0.1',
output_tokens: '10',
} satisfies PriceVector;

const EXAMPLE_PRIORITY_RATES = {
input_tokens: '2',
input_cache_read_tokens: '0.2',
output_tokens: '20',
} satisfies PriceVector;

export const BASE_ONLY_PRICING = tokenBasePricing(EXAMPLE_BASE_RATES);

export const TIERED_PRICING = modelPricing(
tokenPricingEntry(EXAMPLE_BASE_RATES),
tokenPricingEntry(EXAMPLE_PRIORITY_RATES, { serviceTier: 'priority' }),
);
```

Every entry is one exact selector coordinate plus explicit USD-per-million-
token rates. Follow these invariants:
The numbers above are placeholders. Published token rate cards are normally
USD per million tokens. `tokenPricingEntry` and `tokenBasePricing` apply the
existing `perMillionTokenRates` conversion, so their resulting `PriceVector`
values are USD per base token. Do not divide manually or pass number literals.
Follow `packages/provider-codex/src/pricing.ts` for a complete production
example instead of copying a rate vector into this skill.

`collectModelPricingIssues` in
`packages/protocols/src/common/pricing.ts` enforces the structural invariants
below, including Base count, matching rate metrics, selector uniqueness, and
threshold-operator consistency. The source-quality rules still require
human judgment.

- Declare exactly one Base entry without a selector.
- Give every entry the same rate dimensions as Base.
- Give every entry the same metrics as Base.
- Never merge entries or inherit individual cache/image rates from another
dimension. A dimension absent from Base is unpriced everywhere.
metric. A metric absent from Base is unpriced everywhere.
- Treat `serviceTier` as an open-string equality coordinate.
- Treat `inputTokens` `gt` / `gte` thresholds as whole-request bands, not
marginal token buckets.
Expand All @@ -71,11 +106,12 @@ notional API-equivalent value so the usage dashboard remains comparable.
is serialized inside cached `ProviderModel` rows; a mismatch makes every
older row cold before TTL evaluation.
6. Add boundary tests for exact ids, aliases, dated releases, RegExp coverage,
threshold edges, and Base fallback through `priceRequest`.
threshold edges, Base fallback, and the per-base-unit result through
`priceRequest`.
7. Run all affected provider tests, typecheck, lint, and the full test suite.
8. If an existing rate changed, use `backfill-model-pricing` for the intended
historical usage slice. Catalog revisioning changes future snapshots; it
does not rewrite recorded unit prices.
does not rewrite recorded `unit_price` values.

## Catalog Revision Policy

Expand All @@ -92,7 +128,10 @@ ineligible.
## Provider Identity

- Copilot usage stores raw variant suffixes such as `-high`, `-xhigh`, and
`-1m` in `model_key`; its pricing lookup normalizes them to the public id.
`-1m` in `model_key`. Its pricing table is keyed by the public id that survives
variant merging: catalog projection merges the raw variants first, and
`pricingForCopilotPublicModelId` is a plain table match over anchored keys that
normalizes nothing itself.
- Claude Code resolves pricing from the dated raw upstream id before catalog
aliases are merged into public ids.
- Codex and Ollama use the raw upstream slug directly.
Expand Down
Loading