Replication

Continuously copy a bucket's objects to another S3-compatible endpoint.

Alarik can continuously copy a bucket's objects to another S3-compatible endpoint - another Alarik instance, or any service that speaks the S3 API - for disaster recovery or multi-site durability. Configuration is split into two parts: a target (the remote destination and its credentials) and one or more rules (which objects, referencing a target).

Replication is configured per bucket by the bucket owner, under Replication in the console's object browser, or via the replication targets and rules APIs.

Targets and rules

A target is a remote destination: an endpoint, a destination bucket, and its own access key/secret credentials. A bucket can have up to 4 targets, each identified by a server-assigned id (surfaced as an ARN, arn:alarik:replication:::{id}, in the S3 ?replication XML response).

A rule references a target and controls what gets replicated to it:

  • An optional key prefix filter - empty means the whole bucket.
  • replicateDeletes - whether deletes are mirrored to the target. Defaults to false.
  • replicateExisting - whether objects that existed before the rule was created are eligible for replication via resync. Defaults to false.
  • synchronous - whether writes wait for delivery before responding to the client. Defaults to false. See Synchronous vs. asynchronous replication.

A bucket can have up to 4 rules. Deleting a target that's still referenced by a rule automatically disables that rule, rather than leaving it pointing at nothing.

Both replicateDeletes and replicateExisting are opt-in, never a silent default. Replicating deletes to a remote copy is exactly the kind of thing that shouldn't happen by accident.

Versioning requirement

A bucket's versioning must be Enabled before it can have replication rules. Without versioning, a fast overwrite immediately followed by a write can race with replication and silently apply out of order; with versioning, every replicated write and delete refers to one exact, immutable object version, so ordering never matters.

How it works

When an object is written or deleted, Alarik enqueues one replication task per matching, enabled rule and delivers it asynchronously by default - the request that triggered it (a PutObject, a DELETE, ...) is never slowed down by replication, and tasks survive restarts and target outages (they're persisted until delivered). A rule can opt into synchronous delivery instead - see below.

Alarik replicates using genuine, SigV4-signed S3 API requests (PutObject / multipart upload for large objects, DeleteObject) built with the same signing library real AWS SDKs use, so any S3-compatible target - Alarik itself, or AWS - works as a destination.

Version ids are assigned independently by every S3 endpoint - Alarik has no way to force a chosen version id onto the target the way AWS's internal replication protocol does. Practically, this means replication mirrors the current object: a plain PutObject/DeleteObject against the target, letting its own versioning status decide the outcome. Permanently deleting one specific historical version (via an explicit versionId on a DELETE, or lifecycle's NoncurrentVersionExpiration) has no meaningful equivalent on a remote target and is never replicated.

Delete replication

With replicateDeletes enabled, a delete of the current object - whether that creates a delete marker (versioned bucket) or permanently removes it (unversioned) - is mirrored to the target as a plain delete, so the target's own versioning status decides whether that becomes a delete marker or a permanent removal there too.

Existing-object replication (resync)

Enabling replicateExisting on a rule doesn't replicate anything by itself - it only makes the rule eligible for a resync, an explicit "replicate everything currently in this bucket that matches" trigger. Use it after adding a rule to an already-populated bucket, or after fixing a target that was misconfigured.

Synchronous vs. asynchronous replication

By default, replication is asynchronous: PutObject/DeleteObject responds as soon as the local write is durable, and the copy to the target happens afterward, on its own schedule. This is the fastest and most available option - a slow or unreachable target never affects the request that triggered replication.

Setting synchronous: true on a rule makes the local write wait for delivery to that rule's target before responding to the client (up to a 20-second internal timeout), so a client that gets a successful response knows the object already exists on the target too - useful for tooling that writes and then immediately expects to read the same object from the replica.

The local write itself is never blocked or failed by a synchronous rule - only delayed. If delivery fails or times out, Alarik falls back to the normal async outbox automatically and the client still gets a successful response; the failure only shows up in task health and as extra latency on that request. A synchronous rule trades latency for a same-request delivery guarantee - it does not trade away the reliability of the underlying object write.

Mixing rules is fine: a bucket can have some synchronous and some asynchronous rules across its (up to 4) targets. Synchronous rules for the same write are attempted concurrently with each other, not one after another, so having more than one doesn't multiply the added latency.

Retries and reliability

Delivery is at-least-once. A task is only removed once its transfer succeeds; anything else (an unreachable target, a missing destination bucket, a transport error) is retried with exponential backoff (starting at ~1 minute, capped at 1 hour) for up to 8 attempts, then dead-lettered and retained for 7 days for inspection before being purged.

Editing or deleting a target never changes what an already-queued task does - each task snapshots its own endpoint and credentials at enqueue time, so it keeps working (or keeps failing, and reporting why) independently of later configuration changes.

Task health

The Recent Tasks section of the console's Replication modal (and the list tasks API) shows every currently-pending or dead-lettered task for a bucket - the object key, target, attempt count, and, once something has failed, the reason. Successful tasks aren't listed; a row only exists while retrying.

A dead-lettered task (state: failed, retries exhausted) can be requeued from the console with Retry, or via the retry API.

Private and internal targets

Because Alarik itself makes the outbound request, target endpoints are a potential SSRF vector, exactly like webhook URLs. Endpoints that resolve to private, loopback, or link-local addresses can therefore only be configured by administrators.

S3 API compatibility

GET /:bucket?replication returns the configuration as an S3 ReplicationConfiguration document (rules appear with a Destination pointing at an arn:alarik:replication:::{id} ARN, since Alarik's targets aren't real AWS ARNs). DELETE /:bucket?replication clears the whole configuration. Configuring replication through the S3 PUT ?replication operation is not supported - it returns 501 NotImplemented, because that XML shape has no field for target credentials, which Alarik's model requires. Use the console or the internal API instead.

Scope

Alarik ships one-way (active-passive) replication: a bucket replicates to one or more targets, and nothing flows back. Running the same setup in both directions gives you active-active replication yourself - just be aware of the well-known hazard of two sides racing to create conflicting delete markers for the same key.

Webhooks

Receive HTTP notifications when objects are created or removed.