Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions docs/architecture/concepts/failure-domains.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,14 @@ distribution unit, or an availability zone. When failure domains are enabled, si
journal copies, and failover paths across the domains so that the loss of one entire domain does not interrupt
the availability of the cluster.

Failure domains are identified by a non-negative integer chosen by the operator. Simplyblock does not detect the
physical topology itself: every storage node is explicitly tagged with the id of the domain it belongs to when it
is added to the cluster.
Failure domains are named by the operator — `RACK1`, `AZ2`, `HOST1`. Simplyblock does not detect the physical
topology itself: every storage node is explicitly tagged with the label of the domain it belongs to when it is
added to the cluster. The first node added with a given label creates the domain; every later node naming the same
label joins it.

Internally each label maps to a cluster-unique integer id, which is what placement and the data plane key off.
Operators do not need to manage those ids: they appear only in low-level logs and in the `failure_domain` field of
the API, which keeps its integer type for compatibility.

!!! important
Failure-domain support is a deploy-time decision. It is enabled when the storage cluster is created and cannot
Expand Down Expand Up @@ -66,7 +71,7 @@ a same-domain secondary path, and its tertiary path is still guaranteed to be cr

!!! note
Balance is counted in physical hosts, not storage nodes. On multi-socket hosts running two storage nodes, both
nodes count as one host and must carry the same failure-domain id. Dedicated secondary nodes are not counted
nodes count as one host and must carry the same failure-domain label. Dedicated secondary nodes are not counted
toward the balance.

## Failure Domains and Erasure Coding Schemes
Expand All @@ -85,7 +90,7 @@ its loss would break the journal quorum.
## Domain Membership Is Immutable

A host's failure domain cannot be changed while the host is part of the cluster. Moving a host between domains
requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain id. This
requires removing the node, restoring the domain balance, and re-adding it with the new failure-domain label. This
prevents accidental topology changes that would silently invalidate the placement of existing data.

## Recovery Behavior
Expand Down
72 changes: 57 additions & 15 deletions docs/non-kubernetes/operations/failure-domains.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,36 +31,78 @@ plane.

## Tagging Storage Nodes

On a failure-domain cluster, every storage node must be added with a failure-domain id (a non-negative integer
identifying the rack, cabinet, or availability zone). All nodes in the same physical fault group share the same id.
On a failure-domain cluster, every storage node must be added with a failure-domain label naming the rack,
cabinet, or availability zone it sits in. All nodes in the same physical fault group share the same label.

```bash title="Add storage nodes with failure-domain tags"
# Rack A (domain 0)
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> --failure-domain 0 <FURTHER_OPTIONS>
```bash title="Add storage nodes with failure-domain labels"
# Rack A
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> --failure-domain RACK1 <FURTHER_OPTIONS>

# Rack B (domain 1)
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> --failure-domain 1 <FURTHER_OPTIONS>
# Rack B
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> --failure-domain RACK2 <FURTHER_OPTIONS>
```

The first node added with a given label creates the domain; every later node naming the same label joins it. There
is no separate command to declare a domain up front.

The tag is mandatory on failure-domain clusters and must be omitted on clusters without the feature. Both
mismatches are rejected with an explanatory error.

All storage nodes on the same physical host must carry the same failure-domain id. On multi-socket hosts with two
storage nodes, both nodes belong to the host's domain.
All storage nodes on the same physical host must carry the same failure-domain label. On multi-socket hosts with
two storage nodes, both nodes belong to the host's domain.

### Label Syntax

A label starts with a letter, followed by up to 31 letters, digits, `_` or `-`: `RACK1`, `AZ2`, `DC-EU-WEST_1`,
`HOST1`. Labels are case-insensitive — `rack1`, `Rack1` and `RACK1` are the same domain — and are stored
upper-cased. Anything else is rejected before the node is touched.

Choose labels that match how the datacenter is actually described, so that a node list reads like the floor plan.

!!! note
Internally, each label maps to a cluster-unique integer id that placement and the data plane key off. For
backward compatibility, an all-digits value passed to `--failure-domain` is still interpreted as that internal
id rather than as a label, so existing scripts and automation keep working unchanged. New deployments should
use labels.

The assigned domains are shown in the node list once at least one node carries a tag:

```bash title="List storage nodes with their failure domains"
{{ cliname }} storage-node list
```

The **Failure Domain** column shows the label. A cluster that has not been through
[label initialization](#labels-on-existing-clusters) yet, or a domain created by passing an internal id directly,
shows the id instead.

## Labels on Existing Clusters

Clusters deployed before labels existed identify their domains by internal id only. The label registry is
initialized by the regular cluster update:

```bash title="Initialize labels on an existing cluster"
{{ cliname }} cluster update <CLUSTER_ID>
```

Every domain in service is given a derived name — `FD0`, `FD1`, … — and every physical label becomes `HOST1`,
`HOST2`, …. These are placeholders that make the existing topology addressable by name; they do not guess at
intent. A rack the datacenter calls `RACK7` should be renamed afterward.

Initialization is idempotent and safe to repeat: an id that already carries a label is left untouched, so a rename
survives later updates. If the derived name is already taken by a *different* id — for example, a domain an
operator named `FD3` by hand — that id is left unnamed and a warning is logged rather than renaming anything.

!!! note
This initializes the *names* of existing domains. It does not enable the failure-domain feature on a cluster
that was created without `--enable-failure-domain`; that still requires a redeployment.

## Activation Requirements

Activating a freshly assembled failure-domain cluster enforces the following rules:

| Rule | Enforcement |
|-----------------------------------------------|-------------------------------------------------------------------------------------|
| Every node carries a failure-domain id | Hard: activation fails |
| Every node carries a failure-domain label | Hard: activation fails |
| A host does not span two domains | Hard: activation fails |
| At least two distinct domains exist | Hard: activation fails |
| All domains hold an equal number of hosts | Hard: activation fails |
Expand All @@ -77,7 +119,7 @@ chunk. The default of `--ha-jm-count` is 3 for single-parity clusters, so it mus

```bash title="Add a node with four journal copies"
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
--failure-domain 0 --ha-jm-count 4 <FURTHER_OPTIONS>
--failure-domain RACK1 --ha-jm-count 4 <FURTHER_OPTIONS>
```

With three copies and two domains, one domain would hold two copies, and losing that domain would break the
Expand All @@ -91,7 +133,7 @@ Once the cluster holds data, topology changes are admitted only if the failure d
added to any domain. The next host must then go to a different domain.
- No domain may drop below two hosts.
- Adding another storage node slot on an already-member host (multi-socket systems) is balance-neutral and always
admitted, as long as the host keeps its original domain id.
admitted, as long as the host keeps its original domain label.

Violating additions and removals are refused up front, before any data is moved.

Expand All @@ -102,7 +144,7 @@ failover paths:

```bash title="Expand the cluster by one node"
{{ cliname }} storage-node add-node <CLUSTER_ID> <SN_CTR_ADDR> <MGT_IF> \
--failure-domain <FD_ID> --expansion <FURTHER_OPTIONS>
--failure-domain <FD_LABEL> --expansion <FURTHER_OPTIONS>
```

On failure-domain clusters, the expansion planner inserts the newcomer into the existing host rotation at a
Expand All @@ -123,12 +165,12 @@ primary. If no such node exists, the removal is refused.

## Moving a Host Between Domains

A host's failure domain is immutable. Re-adding a host or one of its node slots with a different domain id is
A host's failure domain is immutable. Re-adding a host or one of its node slots with a different domain label is
rejected. To move a host:

1. Remove the node with `{{ cliname }} storage-node remove`.
2. Restore the domain balance if necessary.
3. Re-add the node with the new `--failure-domain` id.
3. Re-add the node with the new `--failure-domain` label.

## Behavior During Outages

Expand Down
Loading