Erasure Coding

In cluster mode, Alarik protects object data with Reed-Solomon erasure coding rather than storing whole copies. Every object is split into k data shards plus m parity shards, one shard per node across the top k + m HRW-ranked nodes, and can be rebuilt from any k of those k + m shards. The durability of tolerating m simultaneous node losses, at roughly (k + m) / k storage overhead (about 1.3-1.5x for common parameters) instead of the 3x a three-copy scheme costs.

This is not an opt-in side mode or a per-bucket choice: in cluster mode it is the storage engine for every object, of every size, in every bucket - including zero-byte objects and delete markers. There is no size threshold below which objects are stored differently.

Configuration

The k and m parameters are cluster-wide deployment settings, configured once via environment variables that every node shares. They default to k = 4, m = 2, which requires a six-node cluster.

CLUSTER_EC_DATA_SHARDS=4

CLUSTER_EC_PARITY_SHARDS=2

CLUSTER_EC_SCRUB_INTERVAL_HOURS=168

Both are validated at boot: a nonsensical value (below the minimum, non-numeric, or a combined k + m above 255 - the mathematical ceiling of the underlying GF(256) Reed-Solomon code) fails startup with a clear error rather than booting into a broken placement scheme. Every node in the cluster must use the same k/m - they describe a single, shared encoding.

Choose k and m for your cluster size

A cluster needs at least k + m active nodes to accept writes at all - there is nowhere to place a full stripe otherwise. A write attempted on a cluster that has shrunk below k + m active nodes fails with a clear 503 ServiceUnavailable (Erasure coding requires at least N active cluster nodes), rather than silently writing a stripe with missing shards.

This makes k + m the single most important thing to get right for your deployment. The default 4 + 2 needs six nodes; a smaller cluster must lower k/m to match, or every write will fail.

Higher k lowers storage overhead but means every read must gather from more nodes; higher m buys more fault tolerance at the cost of more parity storage. Pick k + m no larger than your node count, and leave headroom (k + m strictly less than your node count) if you want writes to keep succeeding while a node is down for maintenance.

How reads work

Unlike full replication, no single node ever holds a whole object - every read gathers shards from across the responsible nodes and reconstructs. Alarik's encoding is systematic, so the first k shards are the original object split k ways: the common case (all shards healthy) is "fetch k data shards and concatenate," with no matrix math. A genuine Reed-Solomon reconstruction runs only when a data shard is missing or fails its checksum, rebuilding exactly the missing piece from the surviving shards (data or parity).

Every shard carries a per-stripe SHA-256 checksum, so a silently-corrupted shard is detected and treated as missing - reconstructed from the others rather than served as-is.

If fewer than k shards can be gathered (too many nodes down at once), the read returns 503 ServiceUnavailable - never a 404, which would falsely tell a client the object was deleted, and never a corrupt or truncated body.

Ranged reads (an HTTP Range header) are fully supported and return 206 Partial Content: only the stripes the requested byte range actually covers are gathered and reconstructed, so fetching a small slice of a large object doesn't pay to rebuild the whole thing.

Bit-rot scrubbing and read-repair

Two mechanisms keep shards healthy over time:

• Read-repair happens automatically on every read. If a read finds a shard missing or checksum-failed, the object is still served (reconstructed from survivors), and the damaged or absent shard is rebuilt from healthy survivors in the background, off the response path. Frequently-read objects therefore self-heal on access, with no operator action.
• Background scrubbing covers everything reads don't touch. Each node periodically re-verifies the checksums of the shards it physically holds and rebuilds any it finds corrupt - the defense against a shard that silently rots on disk while never being read. It's deliberately gentle (one shard at a time, with a pause between), so a full pass trickles rather than saturating disk I/O. Interval is set by CLUSTER_EC_SCRUB_INTERVAL_HOURS (default weekly; 0 disables it).

A scrub can also be triggered immediately across the whole cluster - after a suspected disk problem, say - via POST /admin/cluster/erasure-coding/scrub.

Reconstruction and rebalancing

When membership changes - a node joins, or one is drained - each object's k + m shards are redistributed so every shard index lands on the node the current ranking assigns to it. A shard whose home node has permanently left, with no surviving copy, is reconstructed from k healthy survivors and placed on its new owner. This is a distinct, tracked operational event from an ordinary shard move; the console surfaces it separately (see Monitoring). Reconstruction is also what heals corrupt or missing shards found by read-repair and scrubbing.

As with full-replication rebalancing, a node that crashes or is killed - rather than being explicitly drained - does not trigger reconstruction on its own. After confirming it's gone for good, drain it or trigger a resync to rebuild whatever shards it was holding onto the remaining nodes.

Writes and quorum

A write is coordinated by the object's top-ranked (rank-0) node - the one node allowed to assign shard indices to nodes, so two nodes can never race to produce conflicting placements. It encodes the object into k + m shards, places its own and fans the rest out to the other responsible nodes, and waits for a quorum of shards to be durably acknowledged before returning success. A shard that doesn't land in time gets a durable, retried delivery task, exactly like replication's catch-up - the difference is that a missing shard is reconstructed from survivors at delivery time, since no single node holds a spare whole copy to re-push.

Because coordination is pinned to rank-0, erasure-coded writes do not use the capacity-aware coordinator hand-off that plain replication does - shard placement is a hard requirement, not a soft preference.

Monitoring

The admin console's Cluster page (Admin → Cluster) shows the configured k + m and, when there's shard-repair work outstanding, a Shard Repair by Reason card. A reconstruct there means a shard was lost and is being rebuilt from survivors - a genuine durability event, surfaced distinctly from ordinary shard moves and catch-up. The same data is available programmatically:

• GET /admin/cluster/erasure-coding/status - the configured k/m, the write quorum, and pending/failed shard-repair counts by reason.
• GET /admin/cluster/erasure-coding/tasks - the drill-down: which shard of which object is stuck, targeting which node, and why.
• POST /admin/cluster/erasure-coding/scrub - trigger an immediate cluster-wide bit-rot scrub on demand.

Legacy data and coexistence

If you enable a build with erasure coding on a cluster that already held objects written under an older, full-replication version of Alarik, that existing data stays in its original three-copy format indefinitely - there is no in-place migration. Both formats coexist correctly on the same cluster: reads, deletes, copies, and rebalancing all detect which format an object uses and handle it accordingly. New writes are always erasure-coded.

Scope and limitations

• k and m are fixed cluster-wide constants, not configurable per bucket.
• The cluster must have at least k + m active nodes to accept writes; below that, writes fail with a clear 503 rather than degrading placement.
• The storage engine is never exposed on the S3 wire protocol - every object still reports the standard STANDARD storage class, so strict SDK clients see nothing unusual.
• Placement has no rack/zone awareness - it treats every node as an independent failure domain, so m protects against m independent node failures, not a correlated rack/zone outage.