[docs] Add CRDB survivability configuration

docs/architecture/index.md

@@ -48,9 +48,12 @@ See [Additional requirements](../operations/pooling.md#additional-requirements).
 
 One of the primary design considerations of the DSS is to be very resilient to
 failures.  This resiliency is obtained primarily from the behavior of the
-underlying CockroachDB database technology and how we configure it.  The diagram
-below shows the result of failures (bringing a node down for maintenance, or
-having an entire USS go down) from different starting points, assuming 3 replicas.
+underlying CockroachDB database technology and how we configure it. To meet these
+survivability objectives, see [CockroachDB Pool Survivability](../operations/survivability-crdb.md) for how to configure
+the CockroachDB pool with replica placement constraints.
+
+The diagram below shows the result of failures (bringing a node down for maintenance,
+or having an entire USS go down) from different starting points, assuming 3 replicas.
 
 ![Survivability diagram](../assets/generated/survivability_3x2.png)
 

docs/operations/.nav.yml

@@ -3,6 +3,7 @@ nav:
   - "Certificates management (Yugabyte)": certificates-management.md
   - "Pooling (Yugabyte)": pooling.md
   - "Pooling (CockroachDB)": pooling-crdb.md
+  - "Pool Survivability (CockroachDB)": survivability-crdb.md
   - "Monitoring": monitoring.md
   - "Health checks": healthchecks.md
   - "Migrations": migrations.md

docs/operations/pooling-crdb.md

@@ -124,7 +124,8 @@ following those instructions.
   `--certs-dir` and `--ca-key` flags.
     - Do not specify `--insecure`
 - The ordering of the `--locality` flag keys must be the same across all
-  CockroachDB nodes in the cluster.
+  CockroachDB nodes in the cluster. See [CockroachDB Pool Survivability](survivability-crdb.md)
+  for details on setting up these values and configuring replication zone constraints.
 - All DSS instances in the same cluster must point their ntpd at the same NTP
   Servers.
   [CockroachDB recommends](https://www.cockroachlabs.com/docs/stable/recommended-production-settings.html#considerations)
@@ -210,6 +211,11 @@ to scale down the Statefulset and that removes the last node first (ex:
     1. If there are unhealthy nodes please investigate and fix them so that the
        ranges can return to a healthy state
 
+1. Identify whether the replication zone constraints should be adapted
+   accordingly. This might be the case e.g. when all nodes of an USS are leaving
+   the pool. See [CockroachDB Pool Survivability](survivability-crdb.md) for
+   details on setting up these values.
+
 1. Identify the node id we intend to decommission from the previous commands
    then decommission them. The following command assumes that `cockroachdb-0` is
    not targeted for decommission otherwise select a different instance to

docs/operations/survivability-crdb.md

@@ -0,0 +1,67 @@
+# CockroachDB Pool Survivability
+
+This document describes how to meet the survivability and high-availability objectives of a DSS pool running on CockroachDB (as described in the [Survivability section of the Architecture Overview](../architecture/index.md#Survivability)).
+
+For example, in a standard 3-USS pool running 2 CockroachDB nodes per USS (6 nodes total):
+* If we do not restrict where data replicas are placed, CockroachDB might place two of the three replicas of a range on nodes belonging to a single USS.
+* If that USS suffers an outage or goes down for maintenance, we would lose two replicas at once. This breaks the Raft quorum (since only 1 of 3 replicas remains active), causing that range to become unavailable.
+* To prevent this and meet survivability objectives, we must configure the CockroachDB pool to place at least one replica per USS. This guarantees that even if a full USS goes down, 2 out of 3 replicas remain active on the other USSs, maintaining quorum and ensuring uninterrupted DSS operations.
+
+To achieve this, we:
+1. Start each CockroachDB node with a specific locality embedding its USS and node identifier.
+2. Configure replication zones using the CockroachDB `ALTER RANGE` SQL command to enforce placement constraints across the pool.
+
+---
+
+## 1. Setting Node Locality
+
+When starting each CockroachDB node, you must configure its `--locality` flag to specify which USS and node it represents. The locality must use first at least the `uss` key, and optionally second the `node` key.
+
+!!! danger "Ordering Constraint"
+    The ordering of the `--locality` flag keys must be exactly the same across all CockroachDB nodes in the cluster (e.g., `uss` first, then `node`). Mixing the order (e.g., `node` then `uss` on some nodes) will cause CockroachDB to treat them as incompatible locality hierarchies and fail to apply constraints correctly.
+
+### Flag Format
+
+```shell
+--locality=uss=<uss_id>,node=<node_id>
+# or
+--locality=uss=<uss_id>
+```
+
+Where:
+* `<uss_id>` is a unique identifier for the USS organization (e.g., `uss1`, `uss2`, `uss3`).
+* `<node_id>` is a unique identifier for the node within that USS (e.g., `node-0`, `node-1`).
+
+### Example Configuration for a 3-USS Pool
+
+* **USS 1 (uss1)**:
+  * Node 0: `--locality=uss=uss1,node=node-0`
+  * Node 1: `--locality=uss=uss1,node=node-1`
+* **USS 2 (uss2)**:
+  * Node 0: `--locality=uss=uss2,node=node-0`
+  * Node 1: `--locality=uss=uss2,node=node-1`
+* **USS 3 (uss3)**:
+  * Node 0: `--locality=uss=uss3,node=node-0`
+  * Node 1: `--locality=uss=uss3,node=node-1`
+
+
+## 2. Configuring Replication Constraints (`ALTER RANGE`)
+
+By default, CockroachDB automatically distributes replicas to optimize resource usage and load. To enforce the "one replica per USS" survivability rule, you must manually define replication zone constraints using the `ALTER RANGE default CONFIGURE ZONE` SQL command.
+
+The `default` range is the cluster-wide catch-all. Any database or table created within the DSS (including the RID and SCD tables) that does not have its own specific zone configuration will inherit these default placement constraints.
+
+### The `ALTER RANGE` SQL Statement
+
+To configure a 3-replica cluster where exactly one replica lives on each of the three USSs:
+
+```sql
+ALTER RANGE default CONFIGURE ZONE USING
+  num_replicas = 3,
+  constraints = '{"+uss=uss1": 1, "+uss=uss2": 1, "+uss=uss3": 1}';
+```
+
+### Explanation of the Parameters:
+* `num_replicas = 3`: Tells CockroachDB to keep 3 copies of each range.
+* `constraints`: A JSON object specifying per-replica constraints. Do note that specifying this might not always be necessary depending on your deployment.
+    * `"+uss=uss1": 1` tells CockroachDB that exactly one replica must be located on nodes matching the locality `uss=uss1`.