Operations

This page is for running the store: what you can configure, what durability you get, and what happens on a crash.

Configuring the server

The server resolves its settings from, in increasing precedence, built-in defaults, a TOML file passed with --config, TEPHRA__* environment variables, and the command-line flags. The flags carry only --bind, --data-dir, and --log; everything else lives in the file or the environment. Each config key maps to an environment variable by upper-casing it and joining the path with a double underscore, so writer.max_batch_bytes is TEPHRA__WRITER__MAX_BATCH_BYTES and the nested server.keepalive.idle_secs is TEPHRA__SERVER__KEEPALIVE__IDLE_SECS. The tracing filter is the exception: set it with --log, the log key, or the TEPHRA_LOG environment variable (single underscore), and it falls back to info.

Every value below is the built-in default, so an empty config behaves exactly like no config. The full annotated file ships as tephra.example.toml.

bind = "127.0.0.1:9000"
data_dir = "tephra-data"

[segment]
size = 268435456           # 256 MiB, the segment file size including its header

[writer]
queue_capacity = 16384     # bounded request queue; a full queue blocks the caller
max_batch_records = 2048   # most requests folded into one group commit
max_batch_bytes = 8388608  # 8 MiB byte budget per batch
tips_window = 1000000      # recent-position window for the durable tips (memory bound only)
condition_force_scan = false

[read]
scan_bias = 4              # index only when the range is >= 4x the estimated result

[server]
max_frame_len = 16777216            # 16 MiB

[server.reads]
batch_events = 1024
batch_bytes = 524288                # 512 KiB
worker_threads = 0                  # shared read pool size; 0 = one worker per logical CPU

[server.subscriptions]
wait_tick_ms = 250
max_concurrent = 64                 # live subscriptions per connection (excess is rejected)

[server.backpressure]
max_inflight_per_conn = 256         # per-conn budget, applied separately to appends and reads
frame_queue_depth = 256             # buffered bulk read frames before backpressure (control lane is separate)

[server.limits]
max_connections = 1024              # total concurrent connections; 0 = unlimited

[server.keepalive]
idle_secs = 60
interval_secs = 15

[server.timeouts]
incomplete_frame_secs = 30          # slow-loris trickle defence; 0 disables
handshake_secs = 0                  # first-frame deadline; 0 disables (default, for pooling clients)
idle_secs = 0                       # idle-connection reaper; 0 disables (default)

[metrics]
# bind = "127.0.0.1:9100"           # Prometheus /metrics on its own port; omitted disables it

The 256 MiB segment size is the server’s default. There is no library-level default: an embedded caller passes a size to SegmentConfig::new, which sets max_record_len to a quarter of it and reserves 64 bytes for the segment header.

Configuring the embedded engine

Embedding uses three config structs directly. SegmentConfig (segment size, max record length, header size) has no Default. WriterConfig carries the same writer fields as the TOML above, plus verify_tips, a paranoid cross-check that is never operator-settable and stays off. ReadConfig is the single scan_bias dial. The planner only ever changes which correct path runs, so scan_bias cannot change a result, only its speed.

Transport security and authentication

Every connection opens with a Hello: the client announces its protocol version and, when authenticating, a bearer token, and the server replies with an acknowledgement or an error before it serves any request. The protocol version is the single compatibility gate; a version the server does not support is rejected outright rather than inferred from field presence.

TLS is server-authenticated and off by default. Set a PEM certificate chain and its private key together to serve TLS 1.3 (rustls over the ring provider, no TLS 1.2, no renegotiation); a plaintext client is then rejected at the handshake. Setting only one of the pair is a startup error.

[tls]
cert = "server.crt"
key = "server.key"

Authentication is bearer tokens, also off by default. Each [[auth.tokens]] entry is a table (so scopes can be added later without a config change), and any configured token authenticates a connection. List several to rotate without downtime: add the new token, roll clients over, then drop the old.

[[auth.tokens]]
token = "a-long-random-secret"
[[auth.tokens]]
token = "the-next-secret-during-rotation"

A token is a secret, so it must not cross an unencrypted hop: the server refuses to start when tokens are set without TLS, unless you opt in with auth.allow_insecure = true for a deployment that terminates TLS at a proxy or mesh in front of Tephra. A missing or wrong token fails the Hello, so the connection is refused before any request is served.

On the client, Client::connect_tls(addr, server_name, config) verifies the server certificate, where config comes from tephra_client::tls::config_with_native_roots() or config_with_custom_ca(path) for a self-signed certificate. Pass a token with the *_with variants (connect_with, connect_tls_with); the async client carries it on AsyncClientConfig::auth_token, and every socket in its control-plus-bulk pool authenticates independently.

Authorization (per-token read-only versus read-write, or tag and type scoping) is not built yet: a valid token today grants full access. The token config is a table so that lands additively.

Durability

A batch of appends is made durable in one fsync. The writer appends every data record, then a control record marking the end of the batch, then syncs once. One fsync per batch, not per event, which is why batching is the throughput dial (see the table below).

A batch is committed if, and only if, every record from the previous commit point onward passes its CRC and the run ends in a valid commit marker. A trailing marker alone is not enough: fsync gives no ordering guarantee within a flush, so recovery validates the whole run, not just its terminator.

Crash recovery and startup

The log is the source of truth; the indexes are derived. That asymmetry decides what recovery does.

On startup the store scans the last log segment forward from the last known-good point. A clean log opens as is. A log whose tail is a torn, uncommitted batch opens with that tail rolled back, and the discarded byte and position range are logged: the store tells you it recovered, and by how much. A log with corruption it cannot explain as a torn tail refuses to open rather than serve a wrong prefix. These three outcomes are kept distinct on purpose.

A corrupt index is not fatal. Because an index is derived, a corrupt or missing index segment is rebuilt by replaying its log segment, never a refusal to open. A corrupt log is the opposite: it is the source of truth, so the store refuses rather than guess. The active segment’s in-memory index is always rebuilt by a scan on startup, so a durable-but-unindexed tail after a crash is covered.

The LOCK file

A data directory takes one writer at a time. Opening it read-write takes an advisory lock on a LOCK file in the directory and holds it for as long as the store is open, so a second writer fails fast with an error naming the file and the holding process, instead of two writers recovering the same segment and interleaving appends into it.

Three things worth knowing operationally:

• A leftover LOCK file blocks nothing. The kernel releases the lock however the process exits, including a SIGKILL or a panic. There is never a stale lock to clear by hand, and deleting the file does not release a live lock.
• Two stores on one directory conflict even inside one process, which is usually a handle that was not dropped.
• Spawning subprocesses is safe. The lock is a POSIX record lock, owned by the process rather than by a file descriptor, so a forked child does not inherit it and cannot leave you unable to reopen your own directory.
• A filesystem that cannot lock is tolerated, with a warning. Some network mounts do not implement record locking. Refusing to start there would be worse than the warning, so the store logs and continues, and keeping one writer becomes yours to enforce. Only that case is tolerated: any other locking failure is reported rather than opening unprotected.

Running a second process against a live store

Follower (see Embedded) opens a data directory read-only and tracks the writer from another process. It creates nothing, deletes nothing, writes no index file, and takes no lock, so it works against a live writer, on a directory it cannot write, and on a read-only mount. A follower never blocks the writer, and a writer never blocks a follower.

This is sound because of how the format is already built. Nothing is mutated in place or deleted, so a committed byte never changes. Each batch ends in a commit marker, and a follower applies the same rule crash recovery does: a run counts only if every record in it validates by CRC and the run terminates in that marker. Indexes are derived, so an index file read while the writer is rewriting it fails its checksum and is rebuilt in memory. What a follower exposes is therefore always a committed prefix: gap-free, duplicate-free, and only growing.

The limits are worth stating plainly:

• A follower is not a durability oracle. An event becomes visible when its commit marker reaches the page cache, which is just before the writer’s fsync returns. Power loss in that window erases events a follower has already served. Do not let a follower be the last word on whether something happened; treat it as an asynchronous replica. Clients appending through the writer are unaffected, since append returns only after the fsync.
• A follower lags by up to one poll interval plus its scan.
• Same host only. Visibility relies on the writer’s writes reaching a shared page cache. That holds for a local filesystem; it does not hold for NFS or SMB.
• Cost at open. Sealed segments whose index file the writer has already written load directly. Anything missing or corrupt is rebuilt by scanning that segment, and a follower never persists the result, so a store whose index was never written costs a full log read per follower start.

Backpressure

The writer’s request queue is bounded (queue_capacity). When it is full, append blocks the calling thread until there is room. Backpressure is not an error and not a dropped write: it is the caller slowing to the rate the store can make durable. Batch size grows automatically as fsync latency rises, because a slower disk lets more requests accumulate between syncs.

The network server adds per-connection bounds. A connection serves its requests concurrently and out of order, with responses demultiplexed by request id. server.backpressure.max_inflight_per_conn is applied separately to appends and reads. Appends: this many may await their durable reply before the reader backpressures (an append never blocks behind reads, so it cannot strand a cancel for them). Reads: this many run concurrently, plus this many more may queue for a slot without ever blocking the reader, and a read past both is rejected, so a read never strands a cancel behind it. (A saturated append budget, or a client that has stopped reading its socket, still applies ordinary backpressure that can pause the reader.) Response frames leave on two lanes: small control frames (append acks, stats, errors) are prioritised so they never queue behind a multi-megabyte read (head-of-line blocking), and server.backpressure.frame_queue_depth bounds the buffered bulk read/subscription frames toward a slow client, applying backpressure to the workers producing them. Subscriptions are budgeted separately by server.subscriptions.max_concurrent, and one over the limit is rejected with an error rather than blocking the connection.

Reads themselves run on a single shared, server-wide pool of reusable worker threads rather than a thread spawned per request, so a warm read pays no per-request thread-creation cost. server.reads.worker_threads sets the pool size; 0 means one worker per logical CPU, which suits short, CPU-bound reads. A slow read still runs off the request loop, so it never blocks a connection’s other in-flight requests.

Connection limits and reaping

server.limits.max_connections caps how many connections the server serves at once across all clients. Each connection costs several OS threads (a reader, a writer, an append pump, and one per live subscription), so the cap bounds total server resources independently of any per-connection budget. A connection accepted over the cap is closed immediately, before a request is read; 0 removes the cap as an explicit opt-out.

Three timeouts under [server.timeouts] reap a connection that stops making progress, each measured in seconds and each disabled by 0:

• incomplete_frame_secs bounds how long a partial request frame may take to finish once its first byte has arrived. It defends against a slow-loris trickle, which a per-read socket timeout misses because that resets on every byte. It only ever touches a partial frame in flight, so a connection idling silently at a frame boundary is untouched. On by default at 30 seconds.
• handshake_secs bounds how long a freshly accepted connection may take to send its first complete frame. Off by default, because a pooling client (such as the async client’s idle bulk sockets) legitimately opens a connection and sends nothing until its first read.
• idle_secs reaps a connection that has no request in flight, no live subscription, and has been silent for that long. Off by default, for the same pooling reason. In-flight requests and live subscriptions count as activity, so a long-lived subscription is never reaped.

TCP keepalive is separate and always on: server.keepalive.idle_secs and interval_secs set when probing begins and how often it repeats, so a silently dead peer (a yanked cable, a crashed client) is detected without waiting on the OS default of roughly two hours.

Monitoring

Three mechanisms report on a running server.

--healthcheck runs the binary as a client: it connects to the configured bind address, completes the Hello handshake (with the first configured token, and over TLS pinning the configured certificate when TLS is on), issues one stats request, and exits 0 if healthy or 1 if not. It opens no store and starts no listener, which suits a container HEALTHCHECK or a readiness probe.

The stats request is itself a client call. Client::stats() returns a point-in-time snapshot: the event, segment, and on-disk-byte counts, the uptime, the live connection and subscription counts, the configured connection cap, and the monotonic totals of connections refused at the cap and reaped on a timeout. It is the same data the healthcheck reads.

A Prometheus /metrics endpoint is served on its own port when metrics.bind is set (omitted by default). It is separate from the main data listener, so it can be exposed to an internal scraping network without opening the data port.

Shutdown

The server shuts down gracefully on SIGINT (Ctrl-C) and SIGTERM, the signal docker stop, systemd, and Kubernetes send. The accept loop stops, connections parked on a read are unblocked, and the writer thread is signalled and joined, so no committed batch is left unflushed. A second signal, which means an operator gave up waiting on a stuck shutdown, forces an immediate exit (code 143) rather than being swallowed. Shutdown is deterministic: the graceful path returns once the writer thread has joined.

Group commit, measured

The one dial that moves write throughput is batch size, because it sets how many events share a single fsync. These are Tephra’s own numbers on one machine, not a comparison, and every append carries a Dynamic Consistency Boundary condition (one tag, one type), so this is the guarded write path rather than a raw insert.

Batch size	Throughput	p50 latency	p99 latency
1	6,464 events/s	2.4 ms	4.9 ms
64	192,794 events/s	5.2 ms	8.2 ms
512	796,724 events/s	9.7 ms	19.7 ms

That table is at 16 concurrent writers. Concurrency is a second dial: at 64 writers the batch-512 figure rises to 984,881 events/s, and unbatched conditional appends scale from about 1,000 at a single writer to 32,738 at 128 writers with p50 latency holding near 3.8 ms, because each writer adds requests to the same group commit rather than contending for a lock.

Conditions: Hetzner CCX, AMD EPYC-Milan, 4 cores / 8 threads, 32 GB RAM, Ubuntu 26.04, ext4 on an SSD, measured fsync latency about 1.15 ms average. Tephra built from source at the 256 MiB default segment size, in a container with a 4 GB memory limit. 256-byte events, 15-second runs. These come from the benchmark harness, not from memory. Run it yourself on the storage you will deploy on: a tmpfs makes fsync almost free and these numbers meaningless.