Horizontal Scaling

A single Alarik instance is a complete, self-contained deployment - nothing in this guide is required to get started, and there's no database of any kind to provision, ever. Cluster mode is a strict opt-in on top of that: it shards and replicates object data (and Alarik's own control-plane metadata - buckets, users, access keys, policies, webhook/replication config, cluster membership itself) across multiple nodes, so any node can serve any request regardless of which node originally stored the data.

Enabling Cluster Mode

Two variables turn a node into a cluster member:

CLUSTER_NODE_ADDRESS=http://10.0.1.5:8080

CLUSTER_SECRET=YOUR_SECURE_CLUSTER_SECRET

Both must be set together - a node started with only one fails to start with a clear error rather than booting half-configured.

CLUSTER_SEED_NODES=http://10.0.1.5:8080,http://10.0.1.6:8080

CLUSTER_MIN_FREE_PERCENT=10

CLUSTER_METADATA_EC_DATA_SHARDS=1
CLUSTER_METADATA_EC_PARITY_SHARDS=2

CLUSTER_METADATA_REPLICA_COUNT=2

CLUSTER_METADATA_TOMBSTONE_GRACE_DAYS=7

Every node keeps an in-memory cache of buckets, users, access keys, policies, cluster membership, and the rest of the control plane, invalidated in real time via an HTTP broadcast between nodes - a change made through any node (a new bucket, a revoked access key, an updated policy, a node joining or leaving) propagates to every other node within milliseconds, without polling.

Two background passes back that up, both designed to cost the same whether the cluster has four nodes or a thousand. Membership is refreshed by exchanging views with a small random sample of peers rather than polling everybody, so information still spreads quickly while the traffic per node stays flat as the cluster grows. The full control-plane reload - the safety net for a dropped broadcast - first compares a short per-collection fingerprint with each peer and only transfers records when one of them has actually changed, so an idle cluster does no real work. Neither pass is load-bearing for correctness: they exist to close the gap left by a broadcast that never arrived.

How it works

Alarik stores everything - object data and its own control-plane metadata alike - through the same storage engine, under a hidden internal namespace for metadata. There's no separate database server: back up Storage/ on each node and you've backed up everything, including users, buckets, and policies.

The two are stored differently on purpose. Object data is erasure-coded: split into k data + m parity shards, so it takes the durability of replication at a fraction of the size. Control-plane metadata is replicated: each of its responsible nodes holds a complete copy, so any single one of them can answer "does this bucket exist" or "is this credential valid" on its own, without contacting anybody. Metadata is kilobytes, so whole copies cost nothing - and it means a node restarting, draining, or briefly slow can never make the control plane unreadable.

How object placement works

Every object's key is hashed together with each node's identity to deterministically rank every node for that key, using rendezvous (HRW) hashing - deterministic, coordination-free placement where adding or removing a node only moves the objects whose ownership set actually changed, not a large contiguous chunk of the keyspace. The top-ranked nodes become that object's responsible nodes; the highest-ranked (rank-0) is its primary/coordinator.

Ownership is computed over every registered node, not just the ones answering right now, and that distinction is deliberate. A node that is briefly unreachable keeps its keys: reads are served by its co-replicas, writes destined for it are queued and replayed when it returns, and the data stays exactly where it was written. Nothing "takes over" for it.

Deriving ownership from live nodes instead would mean every restart, GC pause or network blip silently reassigns part of the keyspace - and then reassigns it back - so records stop being where the current placement says they are, and the cluster spends its time moving data rather than serving it. Ownership therefore changes only when you deliberately change the cluster: a node joining, or an operator draining or removing one.

Any node can front any client request. The node a load balancer happens to hit authenticates the request exactly as it always does, then either serves it locally (if it's one of the object's responsible nodes) or forwards the already-authenticated request to a responsible peer, which replies as if it had received the request directly. From a client's perspective, every node in the cluster behaves identically.

Alarik's own control-plane metadata (buckets, users, access keys, policies, cluster membership, ...) is ranked the same way, under a reserved internal namespace - but stored as whole copies rather than split into shards, so any single responsible node can answer a control-plane read on its own. See How it works.

Writes: quorum and catch-up

A write is coordinated by one of the object's responsible nodes and returns success only once a quorum of its pieces is durably acknowledged - the data shards (plus a parity shard of slack when parity allows) for an erasure-coded object. A piece that doesn't ack in time - a slow or briefly-down node - still gets a durable, retried delivery task; the client was already told "success" based on quorum, so this is pure catch-up, not something the client waits on.

Unique names under concurrency

Bucket names, usernames and access-key values must stay unique cluster-wide, even when two clients race to claim the same one through different nodes at the same instant. With no shared database to enforce a unique constraint, Alarik uses a quorum claim. Before a "create only if this name is free" operation runs, the coordinating node must reserve the name on a majority of the nodes responsible for it; each node grants at most one live reservation per name. Two racing creates therefore can't both win a majority - one succeeds, the other is rejected with the normal "already exists" error, exactly as a unique index in a database would behave. The reservation is short-lived and released as soon as the create finishes, so it costs a single extra round trip on creates only - reads, ordinary writes and deletes are unaffected.

Automatic rebalancing

An explicit membership change - a node joining, or an operator draining it - triggers an automatic rebalance walk on every remaining node. Each node reconciles what it physically holds against current placement:

• Copy: objects it's still responsible for get pushed to any newly-responsible peer that may not have a copy yet.
• Reclaim: objects it's no longer responsible for get deleted locally, but only once every new owner is confirmed to actually have a copy - never before. This includes historical object versions and delete markers, not just current objects.

This is fully automatic for ordinary, explicit membership changes - no manual trigger needed. The same walk redistributes shards to their new owners, and a shard whose home node is permanently gone is reconstructed from surviving shards rather than copied.

A manual resync is also the right call for a node that was down long enough to exceed the replication outbox's retry limit, past the point where ordinary catch-up delivery would ever revisit it.

Capacity-aware placement

Every node reports its own free disk space on its regular heartbeat. When a node's free space drops below CLUSTER_MIN_FREE_PERCENT, it's treated as "near-full" for new-write coordination only: if a near-full node would otherwise coordinate a write (because it's one of the object's responsible nodes), it instead hands coordination off to whichever other responsible node has the most free space.

This is a soft, deliberately narrow preference, not capacity-based rebalancing:

• Placement itself (which nodes are responsible for a key) never changes because of capacity - only which of the already-responsible nodes happens to coordinate a given write.
• A near-full node's existing data is never evicted or moved automatically. To free up space on a specific node, drain it as usual.
• If every responsible node for a key is near-full, the write proceeds locally anyway - a write is never refused for capacity reasons.

Adding a node

Point a new node at the running cluster with CLUSTER_SEED_NODES (and the same CLUSTER_SECRET) and start it - nothing else is required. On boot it registers itself, learns the full membership from a seed, and pulls the existing control-plane metadata, so it can serve any request immediately. Joining shifts HRW ranks, so the new node becomes responsible for a share of existing keys; the automatic rebalance walk on the other nodes then migrates that share onto it in the background, and new writes are placed with the new node included from the moment it joins. No manual step, no downtime, no re-sharding of the whole keyspace - only the objects whose ownership set actually changed move.

Draining a node

To take a node out of service without losing data, drain it rather than just stopping the process. Draining excludes the node from new placement immediately and kicks off a rebalance walk to migrate its data onto the remaining nodes - watch rebalance status drain to zero pending/failed tasks before actually stopping it.

Once the data has moved off and you're about to stop the process for good, decommission the node. That flips it from draining to removed, which is what tells the rest of the cluster to stop reaching out to its address - otherwise every metadata listing keeps probing a node that will never answer. A node that later comes back under the same identity simply re-registers itself as active, so this is safe to do the moment the drain finishes.

On Kubernetes

The Helm chart runs Alarik as a StatefulSet, which lines up with the design almost exactly: no external database means nothing else stateful to run, any node serves any request so a plain Service load-balances with no session affinity, and stable placement means a rescheduled pod resumes owning its keys instead of reshuffling the keyspace. Each pod gets its own PersistentVolume and a stable DNS name that becomes its CLUSTER_NODE_ADDRESS, and the /livez//readyz probes gate traffic so the Service never routes to a pod that is still converging. Rolling updates work out of the box - the same stable placement and erasure-coding quorum that tolerate a drain also tolerate a one-pod-at-a-time restart.

Scaling up is just kubectl scale (or raising replicaCount): new pods join, and the existing nodes migrate a share of the data onto them automatically.

Scaling down must not be a bare kubectl scale. Deleting a pod removes the shards it held without migrating them first, leaving the objects they belonged to under-replicated. Use the scale-down.sh helper shipped with the chart instead - it drains the highest-ordinal pods, waits for each one's data to fully migrate off, decommissions it, and only then reduces the replica count:

./scale-down.sh <release-name> <target-replicas> [namespace]

A drained pod's PersistentVolumeClaim is retained by Kubernetes (StatefulSets never delete them), so scaling back up reuses the same volume - the script prints the exact commands to delete those PVCs once you're sure you won't scale back up.

Listing and scanning

Because placement is per-key (not range-sharded), no single node knows in advance which keys under a prefix live where. Bucket-wide operations - ListObjectsV2, ListObjectVersions, ListMultipartUploads, and the check that a bucket is empty before DeleteBucket is allowed - transparently fan out to every active node and merge the results, so they behave identically whether the cluster has one node or ten.

Console

The admin console's Cluster page (Admin → Cluster) shows live node health, a storage distribution chart across nodes, pending/failed replication counts, and a placement browser for inspecting exactly which nodes hold a given bucket's objects. Drain and manual resync are available as actions from the same page. See the admin cluster API reference to drive the same operations programmatically.

The Dashboard page is different in kind: its process metrics (CPU, memory, traffic, local disk) describe only the node that happened to answer the request, not the cluster as a whole - it's labeled with that node's address in cluster mode to make this explicit. For the aggregate picture across every node, use the Cluster page instead.

Example deployment

A minimal 3-node cluster - no database anywhere. alarik-2 and alarik-3 seed their initial membership view from alarik-1; once joined, all three learn about each other and stay in sync via the same broadcast every other control-plane change uses.

services:
  alarik-1:
    image: ghcr.io/achtungsoftware/alarik:latest
    restart: unless-stopped
    ports:
      - "8081:8080"
    environment:
      - API_BASE_URL=http://localhost:8081
      - JWT=YOUR_SECURE_JWT_KEY
      - CLUSTER_NODE_ADDRESS=http://alarik-1:8080
      - CLUSTER_SECRET=YOUR_SECURE_CLUSTER_SECRET
      # Erasure coding sized for this 3-node cluster (the default 4+2 needs 6 nodes).
      - CLUSTER_EC_DATA_SHARDS=2
      - CLUSTER_EC_PARITY_SHARDS=1
    volumes:
      - alarik-1-storage:/app/Storage

  alarik-2:
    image: ghcr.io/achtungsoftware/alarik:latest
    restart: unless-stopped
    ports:
      - "8082:8080"
    environment:
      - API_BASE_URL=http://localhost:8082
      - JWT=YOUR_SECURE_JWT_KEY
      - CLUSTER_NODE_ADDRESS=http://alarik-2:8080
      - CLUSTER_SECRET=YOUR_SECURE_CLUSTER_SECRET
      - CLUSTER_SEED_NODES=http://alarik-1:8080
      - CLUSTER_EC_DATA_SHARDS=2
      - CLUSTER_EC_PARITY_SHARDS=1
    volumes:
      - alarik-2-storage:/app/Storage
    depends_on:
      - alarik-1

  alarik-3:
    image: ghcr.io/achtungsoftware/alarik:latest
    restart: unless-stopped
    ports:
      - "8083:8080"
    environment:
      - API_BASE_URL=http://localhost:8083
      - JWT=YOUR_SECURE_JWT_KEY
      - CLUSTER_NODE_ADDRESS=http://alarik-3:8080
      - CLUSTER_SECRET=YOUR_SECURE_CLUSTER_SECRET
      - CLUSTER_SEED_NODES=http://alarik-1:8080
      - CLUSTER_EC_DATA_SHARDS=2
      - CLUSTER_EC_PARITY_SHARDS=1
    volumes:
      - alarik-3-storage:/app/Storage
    depends_on:
      - alarik-1

volumes:
  alarik-1-storage:
  alarik-2-storage:
  alarik-3-storage:

Point a load balancer (or your S3 client, for local testing) at any of the three nodes - there's no "primary" to pick, any node handles any request.

Scope and limitations

• Object data is erasure-coded by default; k/m are fixed cluster-wide constants, not configurable per bucket, and the cluster needs at least k + m active nodes to accept writes.
• Placement has no rack/zone awareness - it treats every node as equally likely to fail independently.
• Inter-node traffic isn't encrypted; run it on a private network.
• In-progress multipart uploads aren't replicated while the upload is happening - only the final completed object goes through quorum replication. A node holding an in-progress upload that crashes mid-upload loses that upload; the client needs to restart it.
• A crashed (not explicitly drained) node doesn't trigger automatic rebalancing - see Automatic rebalancing above. An operator needs to drain it or trigger a manual resync once it's confirmed gone.
• Capacity awareness only ever redirects new-write coordination among a key's already-responsible nodes - see Capacity-aware placement above. It doesn't rebalance data off a full node, and doesn't help if the whole cluster is near-full, not just one node.