OpenZeppelin · dylankilkenny · Jul 27, 2026
@@ -75,6 +75,32 @@ Symbiotic provides the shared security layer:
 | Storage | `operator/src/storage/` | redb key-value store for messages, Merkle trees, and submissions |
 | Crypto | `operator/src/crypto/` | Merkle tree construction, leaf hashing, signing message encoding |
 
+## Production Topology
+
+The starter kit runs everything on one machine; production needs a distributed layout. The parts that matter:
+
+### Attestation API availability
+
+For the Chainlink CCV path, executors fetch attestations from the URLs the verifier advertises on-chain (`getStorageLocations()`). **A single attestation node is a single point of failure that stalls the lane**: messages keep verifying, but executors cannot fetch attestations, so nothing executes until the node returns.
+
+Every operator node serves the same attestation data (`GET /verifications`) — all of them hold the messages and the aggregated quorum signatures. Production options, in order of preference:
+
+1. **Load balancer over multiple operator nodes** — advertise one stable URL, health-check `/healthz`, route to any live operator.
+2. **Multiple URLs on-chain** — `getStorageLocations()` returns a list; advertise several operators directly and let executors fail over.
+3. **One designated HA node** — acceptable only with real redundancy behind that name (failover, monitoring).
+
+The advertised URL is on-chain state, not config: rotate it with the owner-gated `updateStorageLocations` and treat DNS you can repoint as more operable than raw hosts.
+
+### Where aggregation happens
+
+Individual operators do not serve partial signatures. Each operator signs the Merkle root through its Symbiotic relay sidecar; the relay network aggregates BLS signatures across operators, and every operator then stores the same aggregated result. That is why any single operator can answer attestation queries — availability of the *API* is your concern; signature aggregation already tolerates individual signer downtime as long as quorum voting power signs.
+
+### Availability assumptions to engineer for
+
+- The CCIP indexer polls your attestation API — sustained unavailability means unexecuted (not lost) messages, recoverable once service returns.
+- Operators must reach source-chain RPC (event ingestion + finality gating) and the relay sidecars; run operators across failure domains.
+- The OZ Monitor→operator webhook path is at-least-once, not exactly-once; operators de-duplicate. Monitor outages are recovered by block backfill on restart — bound your `max_past_blocks` accordingly.
+
 ## Adding a New Provider
 
 1. Implement the `Provider` trait in `operator/src/provider/<provider>.rs`.

@@ -14,6 +14,24 @@ This template supports the Symbiotic CCV path only. The Chainlink auxiliary deve
 
 </Callout>
 
+## On-Chain Architecture: Resolver + Verifier
+
+The CCV identity that CCIP references is split across two contracts:
+
+- **Resolver** — Chainlink's stock `VersionedVerifierResolver`, deployed via Chainlink's `CREATE2Factory` so it has the **same address on every supported chain**. This is the permanent address: it is what OnRamp/OffRamp configuration and the CCIP indexer register, and it never changes. It holds two registries: version prefix → inbound verifier implementation, and destination chain selector → outbound implementation.
+- **Verifier implementation** — `SymbioticVerifier`, which inherits Chainlink's `BaseVerifier` and adds Symbiotic BLS quorum verification against `Settlement`. Implementations are versioned and replaceable.
+
+Every verifier result starts with a 4-byte version tag (`0x1a75bd93` for v1, derived from `keccak256("SymbioticCCV 1.0.0")`). Signers commit to `keccak256(versionTag ‖ messageId)`, so the tag both routes the message to the right implementation and prevents a signature produced for one implementation from being replayed against another.
+
+### Upgrading the verifier
+
+Deploying verification logic v2 never changes the address integrators use:
+
+1. Deploy the new `SymbioticVerifier` with a new version tag and configure its lanes.
+2. Register it on the resolver: `applyInboundImplementationUpdates([{tag_v2, v2}])` — v1 and v2 now coexist, and in-flight messages signed under v1 still resolve to v1.
+3. Switch outbound per lane: `applyOutboundImplementationUpdates([{selector, v2}])` — new messages carry v2's tag.
+4. After v1 traffic drains, optionally retire v1's tag by registering it to `address(0)`.
+
 ## Message Flow
 
 ```mermaid
@@ -25,7 +43,8 @@ sequenceDiagram
     participant Relay as Symbiotic Relay (BLS)
     participant Relayer as OZ Relayer
     participant OffRamp as OffRamp (Dest)
-    participant CCV as SymbioticCCV (Dest)
+    participant Resolver as Resolver (Dest)
+    participant Verifier as SymbioticVerifier (Dest)
 
     App->>OnRamp: sendMessage()
     OnRamp-->>Monitor: CCIPMessageSent event
@@ -35,18 +54,58 @@ sequenceDiagram
     Relay-->>Operators: aggregated signature
     Operators->>Relayer: OffRamp.execute calldata
     Relayer->>OffRamp: execute(message, ccvs, verifierResults)
-    OffRamp->>CCV: verifyMessage(message, messageId, verifierResults)
-    CCV-->>OffRamp: verified
+    OffRamp->>Resolver: getInboundImplementation(verifierResults[:4])
+    Resolver-->>OffRamp: SymbioticVerifier (by version tag)
+    OffRamp->>Verifier: verifyMessage(message, messageId, verifierResults)
+    Verifier-->>OffRamp: verified (RMN curse check, OffRamp auth via Router, BLS quorum)
     OffRamp-->>OffRamp: emit MessageExecuted(messageId)
 ```
 
+## Verification Gas Budget
+
+Each lane's `gasForVerification` (set via `applyRemoteChainConfigUpdates`, returned by `getFee`) must cover the full `verifyMessage` call. Its cost scales with the validator set in three ways:
+
+- **proof handling in the verifier** — the quorum proof embeds the entire validator set (64 bytes per validator), and the current `SymbioticVerifier` implementation copies the proof byte-by-byte before handing it to `Settlement`. At large sets this copy dominates everything else;
+- **signature verification** (`SigVerifierBlsBn254Simple`) — hashes the full validator-set data against the committed set, then pays a key decompression plus curve addition for each validator that did *not* sign, plus one BLS pairing (~113k, fixed);
+- **fixed base** — `BaseVerifier` ramp/RMN checks and `Settlement` epoch bookkeeping.
+
+Measured end to end with the repo's gas harness (real `SymbioticVerifier` → `Settlement` hop → real `SigVerifierBlsBn254Simple`, synthetic equal-power validator sets):
+
+```bash
+cd contracts && forge test --match-contract VerificationGas -vv
+```
+
+| Validators | All sign | 33% non-signers (by count) |
+|-----------:|---------:|---------------------------:|
+| 3   | 292k | 296k |
+| 10  | 407k | 421k |
+| 25  | 657k | 693k |
+| 50  | 1.09M | 1.16M |
+| 100 | 2.00M | 2.17M |
+
+The 3-validator row is within ~5% of what Chainlink measured on the staging deployment (299k–312k across live messages), so treat the table as representative. Interpolate for your set size and add a **safety margin (recommend 20%)**. The staging config value of `400000` sits *below* the measured cost of a 10-validator set (407k before margin) — it is sized for the current small staging set, not a constant. (Do not benchmark against the unit tests: `SymbioticVerifier.t.sol` stubs `Settlement`, so its gas numbers exclude all of the above.)
+
+Two caveats when sizing the non-signer allowance:
+
+- Quorum is enforced on **voting power, not head count**. With unequal stake weights, far more than a third of validators (by count) can abstain while quorum still passes — and each absent validator costs decompression + addition. Derive your worst case from the actual power distribution; the table's second column is the equal-power case.
+- **Re-evaluate whenever the validator set grows** — operator registration changes the set automatically at epoch boundaries, so an under-provisioned value does not fail immediately; it fails when the set outgrows it, and messages then stall at verification. Alert on validator-set size, not on failures.
+
+Because per-message cost grows roughly linearly with the set (~18k gas per validator end to end in the current implementation), Symbiotic's ZK verification mode — near-constant gas regardless of set size — becomes the cheaper option well before very large sets. If your set is growing past a few dozen validators, evaluate the switch; it deploys as a verifier-implementation upgrade (see above — the resolver address does not change).
+
+## Recommended Verifier Composition (Token Issuers)
+
+**Run the Chainlink committee verifier alongside the Symbiotic verifier as your default.** Two independent CCVs give defense in depth out of the box: a message executes only when both verifiers attest, so compromising either the committee or the Symbiotic validator set alone is not sufficient to forge a message.
+
+Configure both CCVs (the Chainlink committee verifier plus this resolver's address) in your token pool / app CCV list. Symbiotic-only operation is supported, but treat it as a **conscious, documented opt-out** — you are removing an independent verification layer, and you should record that decision and its rationale in your own security documentation.
+
+Token issuers can also **supply their own token as the collateral securing the verifier network** (self-securing collateral). This needs nothing CCV-specific: create a Symbiotic vault, deposit the asset, and have operators opt in — standard [Symbiotic vault creation](https://docs.symbiotic.fi/), with your token as the staked asset backing the validator set's economic security.
+
 ## Contracts and Code
 
 ### Contracts
 
-- `contracts/src/ccv/SymbioticCCV.sol`
-- `contracts/src/ccv/interfaces/`
-- `contracts/src/ccv/libraries/`
+- `contracts/src/chainlink/SymbioticVerifier.sol` — verifier implementation (inherits Chainlink `BaseVerifier`)
+- Resolver + factory: stock contracts from `@chainlink/contracts-ccip` (`VersionedVerifierResolver`, `CREATE2Factory`) — deployed from the package, no local source
 - `contracts/src/symbiotic/Settlement.sol`
 - `contracts/src/symbiotic/KeyRegistry.sol`
 - `contracts/src/symbiotic/Driver.sol`
@@ -77,16 +136,17 @@ Address resolution order:
 1. `CCV_*` environment variables
 2. `deployments/<env>.json`
 
+The resolver address is the exception: it has no environment override and is always read from `deployments/<env>.json` (`chainlinkCcv.resolver`), matching the CREATE2 determinism guarantee.
+
 Available overrides:
 
 | Variable | Description |
 |----------|-------------|
-| `CCV_SOURCE_ADDRESS` | SymbioticCCV on source chain |
-| `CCV_DEST_ADDRESS` | SymbioticCCV on destination chain |
 | `CCV_SOURCE_ONRAMP_ADDRESS` | Source OnRamp-compatible contract |
 | `CCV_DEST_OFFRAMP_ADDRESS` | Destination OffRamp submit target |
 | `CCV_SOURCE_CHAIN_SELECTOR` | Override the source CCIP chain selector |
 | `CCV_DEST_CHAIN_SELECTOR` | Override the destination CCIP chain selector |
+| `CCV_STORAGE_LOCATION_URIS` | Comma-separated attestation API base URLs advertised on-chain (required for non-local deploys) |
 
 Local and testnet are supported (testnet uses the `testnet-ccv` environment: `config/environments/testnet-ccv.json` and `deployments/testnet-ccv.json`, run via `ENV=testnet-ccv`); mainnet is not yet supported for CCV. `make watch` only succeeds once destination `MessageExecuted(messageId)` is observed.
 

@@ -16,8 +16,15 @@ Send one test message through the active provider.
 make send
 make send MSG="test message"
 make send ENV=testnet MSG="hello"
+make send ENV=local-ccv EXECUTOR=0x... FINALITY=safe
 ```
 
+| Variable | Description |
+|----------|-------------|
+| `MSG` | Message payload (default: "hello") |
+| `EXECUTOR` | Executor address encoded into the message (CCV local mock send only). Operators self-execute only when this matches their configured executor address. |
+| `FINALITY` | Source-finality requirement encoded into the message (CCV local mock send only): `finalized` (default), `safe`, or a block count `N`. |
+
 ### `make watch`
 
 Watch a previously sent message until the destination path succeeds.
@@ -44,12 +51,15 @@ Send a message, then watch it to completion.
 make e2e
 make e2e MSG="custom message"
 make e2e ENV=testnet MSG="hello" TIMEOUT=180
+make e2e ENV=local-ccv EXECUTOR=0x... TIMEOUT=180
 ```
 
+`EXECUTOR` and `FINALITY` pass through exactly as in `make send`.
+
 LayerZero starter OApp addresses are published under:
 
 ```text
-deployments/<env>.json -> layerzero.oapp.{source,destination}
+deployments/<env>.json -> {source,destination}.layerzero.exampleApp
 ```
 
 ## Direct xtask Usage
@@ -134,6 +144,16 @@ LayerZero operators also expose:
 | `POST /api/v1/layerzero/proof` | Return Merkle proofs for processed messages |
 | `POST /api/v1/layerzero/verify` | Verify a Merkle proof in test workflows |
 
+### Chainlink CCV Verification Endpoint
+
+CCV operators expose the attestation API advertised on-chain via `getStorageLocations()` (host ports 3001-3003):
+
+```bash
+curl "http://localhost:3001/verifications?messageID=0x..."
+```
+
+`messageID` is repeatable (up to 20 per request). Responses carry Chainlink-CCV-shaped verifier results whose `ccv_data` is prefixed with the verifier version tag. Requests with no `messageID`, more than 20, or a malformed id return HTTP 400.
+
 ## Webhook Template Requirements
 
 OZ Monitor trigger templates live under `config/templates/oz-monitor/triggers/` and are copied into `generated/<env>/oz-monitor/triggers/`.

@@ -6,7 +6,7 @@ Deploying and operating the stack beyond local development.
 
 <Callout type="warn">
 
-Testnet supports both providers: `layerzero` (`config/environments/testnet.json`, `ENV=testnet`) and `chainlink_ccv` (`config/environments/testnet-ccv.json`, `ENV=testnet-ccv`). The workflow below uses `ENV=testnet` (LayerZero); for the CCV path substitute `ENV=testnet-ccv`. Mainnet CCV is not yet supported.
+Testnet supports both providers: `layerzero` (`config/environments/testnet.json`, `ENV=testnet`) and `chainlink_ccv` (`config/environments/testnet-ccv.json`, `ENV=testnet-ccv`). The workflow below uses `ENV=testnet` (LayerZero); for the CCV path substitute `ENV=testnet-ccv`, but note the `make e2e` step is LayerZero-only (it drives the starter `ExampleOApp`) — CCV runs are confirmed with `make watch` instead (see [Chainlink CCV](/symbiotic/chainlink-ccv#configuration)). Mainnet CCV is not yet supported.
 
 </Callout>
 
@@ -63,12 +63,59 @@ KEYSTORE_PASSPHRASE=<relayer keystore passphrase>   # used by OZ Relayer at runt
 
 Operator and deployer passphrases are resolved by `xtask` via the `signers` config. `KEYSTORE_PASSPHRASE` is read directly by the OZ Relayer container to decrypt relayer keystores at runtime.
 
+CCV environments (`ENV=testnet-ccv`) additionally need the reserved factory-deployer key:
+
+```bash
+cargo xtask --env testnet-ccv generate-signer --name ccv-factory-deployer
+```
+
+```bash
+CCV_FACTORY_DEPLOYER_PASSPHRASE=<ccv-factory-deployer keystore passphrase>
+CCV_STORAGE_LOCATION_URIS=<comma-separated attestation API base URLs>
+```
+
+The factory-deployer key must be freshly generated and stay at nonce 0 on every target chain until the deploy — it anchors the resolver's CREATE2 address parity. Fund it just enough for one factory deploy per chain.
+
 ## Operational Notes
 
 - Local uses `make chains && make deploy && make start`; testnet uses `make deploy ENV=testnet` plus `make start ENV=testnet`.
 - Testnet uses `docker-compose.yml` only, not the local overlay.
 - Canonical deployment addresses always live in `deployments/<env>.json`.
 
+## CCV Deterministic Deployment (CREATE2)
+
+The CCV resolver must have the **same address on every chain** — the CCIP indexer registers that one address, and identical addresses give automatic discovery when you add a chain. The deployment flow that guarantees this:
+
+1. **Deploy Chainlink's `CREATE2Factory` once per chain from a reserved key at nonce 0.** CREATE addresses depend only on (deployer, nonce), so a dedicated key that has never transacted produces the same factory address everywhere. Locally this is the `ccv-factory-deployer` signer (Anvil account 9); on testnet/production, generate a dedicated key, use it for nothing else, and fund it just enough for one deploy per chain. The factory constructor takes an allowlist of addresses permitted to claim CREATE2 addresses through it.
+2. **Deploy the resolver through the factory**: `deployResolver` loads Chainlink's *published* resolver creation bytecode from the npm package, calls `factory.createAndTransferOwnership(code, salt, owner)`, asserts the result matches `factory.computeAddress`, and completes ownership with `acceptOwnership()` from the owner. A deploy without the accept step is not finished — the resolver would still have a pending owner.
+3. Deploy the `SymbioticVerifier` (plain CREATE — its address doesn't need parity), register it on the resolver, and configure lanes.
+
+Non-local CCV deployments also require the real RMN contract per chain: set `chainlinkCcip.rmn` in the environment config (`RMNRemote` on CCIP staging — discoverable from `OnRamp.getStaticConfig()`). The verifier's constructor requires it, and fork tests (`make test-fork`) read it from `SOURCE_CCIP_RMN_ADDRESS` / `DEST_CCIP_RMN_ADDRESS` in `.env.testnet`.
+
+### Address-affecting inputs
+
+The resolver address is a pure function of these — pin them and never change them casually:
+
+| Input | Where pinned | Notes |
+|-------|--------------|-------|
+| Factory deployer key + nonce 0 | `ccv-factory-deployer` signer per environment | Determines the factory address |
+| Resolver salt | `RESOLVER_SALT` in `contracts/script/chainlink/DeployCCV.s.sol` (`keccak256("symbiotic.ccv.versioned-verifier-resolver.v1")`) | |
+| Resolver creation bytecode | `@chainlink/contracts-ccip@2.0.0` published artifact (`bytecode/v2_0_0/versioned_verifier_resolver.bin`) | Using the published artifact makes the address independent of local compiler settings |
+
+The resolver **owner** is *not* address-affecting (the stock resolver has no constructor arguments), so ownership can differ per chain without breaking parity.
+
+Validate determinism any time with `make validate-resolver-address` — it deploys the full flow on two throwaway Anvil chains and asserts identical addresses and correct ownership.
+
+### Attestation storage locations
+
+`CCV_STORAGE_LOCATION_URIS` (comma-separated) is a **required** deploy input for the verifier: it becomes the on-chain `getStorageLocations()` answer that executors use to fetch attestations. There is no placeholder fallback — an empty value fails the deploy. Point it at the operator attestation API (each operator serves `GET /verifications`; put a load balancer or an HA node in front for production — see [Architecture](/symbiotic/architecture#production-topology)). Rotate later with the owner-gated `updateStorageLocations`.
+
+### CCIP indexer onboarding
+
+Each new protocol deployment is a per-deployment onboarding with Chainlink: the CCIP indexer must learn (a) your resolver address and (b) your attestation API endpoint before messages get automated execution. Messages verify without this step — but nobody executes them. Coordinate with your Chainlink contact (or Symbiotic/OpenZeppelin on your behalf) when standing up a new deployment, and again only when adding chains if you skipped deterministic addresses.
+
+Onboarding is **per protocol deployment, not per validator set**. Operator registration and deregistration rotate the active validator set inside the same `Settlement` and verifier contracts at epoch boundaries — the contracts, the resolver address, and the attestation API endpoint all stay the same as the set evolves. The indexer never needs re-onboarding for validator churn; keeping the advertised endpoint reachable across it is your responsibility (see [Production Topology](/symbiotic/architecture#production-topology)).
+
 ## Sidecar Image Compatibility
 
 The Symbiotic relay sidecar (`symbioticfi/relay`) is still in release-candidate phase. The pinned version in `docker-compose.yml` and the `RELAY_IMAGE` default in `xtask/src/genesis.rs` must stay in sync. Known states observed during integration: