diff --git a/docs/architecture/concepts/failure-domains.md b/docs/architecture/concepts/failure-domains.md index 67127475..02f14c61 100644 --- a/docs/architecture/concepts/failure-domains.md +++ b/docs/architecture/concepts/failure-domains.md @@ -9,9 +9,14 @@ distribution unit, or an availability zone. When failure domains are enabled, si journal copies, and failover paths across the domains so that the loss of one entire domain does not interrupt the availability of the cluster. -Failure domains are identified by a non-negative integer chosen by the operator. Simplyblock does not detect the -physical topology itself: every storage node is explicitly tagged with the id of the domain it belongs to when it -is added to the cluster. +Failure domains are named by the operator — `RACK1`, `AZ2`, `HOST1`. Simplyblock does not detect the physical +topology itself: every storage node is explicitly tagged with the label of the domain it belongs to when it is +added to the cluster. The first node added with a given label creates the domain; every later node naming the same +label joins it. + +Internally each label maps to a cluster-unique integer id, which is what placement and the data plane key off. +Operators do not need to manage those ids: they appear only in low-level logs and in the `failure_domain` field of +the API, which keeps its integer type for compatibility. !!! important Failure-domain support is a deploy-time decision. It is enabled when the storage cluster is created and cannot @@ -66,7 +71,7 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr !!! note Balance is counted in physical hosts, not storage nodes. On multi-socket hosts running two storage nodes, both - nodes count as one host and must carry the same failure-domain id. Dedicated secondary nodes are not counted + nodes count as one host and must carry the same failure-domain label. Dedicated secondary nodes are not counted toward the balance. ## Failure Domains and Erasure Coding Schemes @@ -85,7 +90,7 @@ its loss would break the journal quorum. ## Domain Membership Is Immutable A host's failure domain cannot be changed while the host is part of the cluster. Moving a host between domains -requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain id. This +requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This prevents accidental topology changes that would silently invalidate the placement of existing data. ## Recovery Behavior diff --git a/docs/non-kubernetes/operations/failure-domains.md b/docs/non-kubernetes/operations/failure-domains.md index 57eccf2f..774c089f 100644 --- a/docs/non-kubernetes/operations/failure-domains.md +++ b/docs/non-kubernetes/operations/failure-domains.md @@ -31,22 +31,39 @@ plane. ## Tagging Storage Nodes -On a failure-domain cluster, every storage node must be added with a failure-domain id (a non-negative integer -identifying the rack, cabinet, or availability zone). All nodes in the same physical fault group share the same id. +On a failure-domain cluster, every storage node must be added with a failure-domain label naming the rack, +cabinet, or availability zone it sits in. All nodes in the same physical fault group share the same label. -```bash title="Add storage nodes with failure-domain tags" -# Rack A (domain 0) -{{ cliname }} storage-node add-node --failure-domain 0 +```bash title="Add storage nodes with failure-domain labels" +# Rack A +{{ cliname }} storage-node add-node --failure-domain RACK1 -# Rack B (domain 1) -{{ cliname }} storage-node add-node --failure-domain 1 +# Rack B +{{ cliname }} storage-node add-node --failure-domain RACK2 ``` +The first node added with a given label creates the domain; every later node naming the same label joins it. There +is no separate command to declare a domain up front. + The tag is mandatory on failure-domain clusters and must be omitted on clusters without the feature. Both mismatches are rejected with an explanatory error. -All storage nodes on the same physical host must carry the same failure-domain id. On multi-socket hosts with two -storage nodes, both nodes belong to the host's domain. +All storage nodes on the same physical host must carry the same failure-domain label. On multi-socket hosts with +two storage nodes, both nodes belong to the host's domain. + +### Label Syntax + +A label starts with a letter, followed by up to 31 letters, digits, `_` or `-`: `RACK1`, `AZ2`, `DC-EU-WEST_1`, +`HOST1`. Labels are case-insensitive — `rack1`, `Rack1` and `RACK1` are the same domain — and are stored +upper-cased. Anything else is rejected before the node is touched. + +Choose labels that match how the datacenter is actually described, so that a node list reads like the floor plan. + +!!! note + Internally, each label maps to a cluster-unique integer id that placement and the data plane key off. For + backward compatibility, an all-digits value passed to `--failure-domain` is still interpreted as that internal + id rather than as a label, so existing scripts and automation keep working unchanged. New deployments should + use labels. The assigned domains are shown in the node list once at least one node carries a tag: @@ -54,13 +71,38 @@ The assigned domains are shown in the node list once at least one node carries a {{ cliname }} storage-node list ``` +The **Failure Domain** column shows the label. A cluster that has not been through +[label initialization](#labels-on-existing-clusters) yet, or a domain created by passing an internal id directly, +shows the id instead. + +## Labels on Existing Clusters + +Clusters deployed before labels existed identify their domains by internal id only. The label registry is +initialized by the regular cluster update: + +```bash title="Initialize labels on an existing cluster" +{{ cliname }} cluster update +``` + +Every domain in service is given a derived name — `FD0`, `FD1`, … — and every physical label becomes `HOST1`, +`HOST2`, …. These are placeholders that make the existing topology addressable by name; they do not guess at +intent. A rack the datacenter calls `RACK7` should be renamed afterward. + +Initialization is idempotent and safe to repeat: an id that already carries a label is left untouched, so a rename +survives later updates. If the derived name is already taken by a *different* id — for example, a domain an +operator named `FD3` by hand — that id is left unnamed and a warning is logged rather than renaming anything. + +!!! note + This initializes the *names* of existing domains. It does not enable the failure-domain feature on a cluster + that was created without `--enable-failure-domain`; that still requires a redeployment. + ## Activation Requirements Activating a freshly assembled failure-domain cluster enforces the following rules: | Rule | Enforcement | |-----------------------------------------------|-------------------------------------------------------------------------------------| -| Every node carries a failure-domain id | Hard: activation fails | +| Every node carries a failure-domain label | Hard: activation fails | | A host does not span two domains | Hard: activation fails | | At least two distinct domains exist | Hard: activation fails | | All domains hold an equal number of hosts | Hard: activation fails | @@ -77,7 +119,7 @@ chunk. The default of `--ha-jm-count` is 3 for single-parity clusters, so it mus ```bash title="Add a node with four journal copies" {{ cliname }} storage-node add-node \ - --failure-domain 0 --ha-jm-count 4 + --failure-domain RACK1 --ha-jm-count 4 ``` With three copies and two domains, one domain would hold two copies, and losing that domain would break the @@ -91,7 +133,7 @@ Once the cluster holds data, topology changes are admitted only if the failure d added to any domain. The next host must then go to a different domain. - No domain may drop below two hosts. - Adding another storage node slot on an already-member host (multi-socket systems) is balance-neutral and always - admitted, as long as the host keeps its original domain id. + admitted, as long as the host keeps its original domain label. Violating additions and removals are refused up front, before any data is moved. @@ -102,7 +144,7 @@ failover paths: ```bash title="Expand the cluster by one node" {{ cliname }} storage-node add-node \ - --failure-domain --expansion + --failure-domain --expansion ``` On failure-domain clusters, the expansion planner inserts the newcomer into the existing host rotation at a @@ -123,12 +165,12 @@ primary. If no such node exists, the removal is refused. ## Moving a Host Between Domains -A host's failure domain is immutable. Re-adding a host or one of its node slots with a different domain id is +A host's failure domain is immutable. Re-adding a host or one of its node slots with a different domain label is rejected. To move a host: 1. Remove the node with `{{ cliname }} storage-node remove`. 2. Restore the domain balance if necessary. -3. Re-add the node with the new `--failure-domain` id. +3. Re-add the node with the new `--failure-domain` label. ## Behavior During Outages