Embedded

Running the server and talking to it with the client is the common path, and most of this site assumes it. You can also link the engine (tephra) directly into a Rust process and skip the wire entirely. This page is the whole of what that changes.

When to embed

Embed when the code making decisions and the store live in the same process and you want no network hop: a single-process service, a command-line tool, a test harness. You give up the things the server provides: remote access over TCP, multiple client processes against one store, and the language-agnostic protobuf protocol. You gain the removal of serialisation and a socket from the write path.

What you do not gain is a second writer. The engine is one logical writer per store whether you embed it or run the server, so embedding is not a way around the single-writer constraint. It is the same store, reached directly.

Open the store and start the coordinator

SegmentSet::open opens (or creates) a log directory, and WriteCoordinator::start starts the single writer thread and hands back a cloneable WriteHandle. SegmentConfig and SegmentSet are re-exported at the crate root.

// Open (or create) a log directory and start the single-writer coordinator.
// SegmentConfig and SegmentSet are re-exported at the crate root.
let set =
    SegmentSet::open(dir.path(), SegmentConfig::new(256 * 1024 * 1024)).expect("open store");
let (coordinator, handle) =
    WriteCoordinator::start(set, WriterConfig::default()).expect("start coordinator");

There is no library default segment size: you pass one to SegmentConfig::new. The 256 MiB figure quoted elsewhere is the server’s default, not the engine’s.

Append

The engine’s Event is a packed, zero-copy codec type, distinct from the client’s owned Event. It is built from &EventType, &Tags, and a payload, and the in-memory layout is the on-disk layout, so the events you build are the bytes that get written.

// The engine's packed Event is built from &EventType, &Tags, and an opaque payload.
let ty = EventType::new("CourseOpened").expect("type");
let tags = Tags::new([Tag::new("course:c1").expect("tag")]).expect("tags");
let event = Event::new(&ty, &tags, br#"{"course":"c1","seats":30}"#).expect("encode");

// Guard the append so it fails if any event already carries course:c1 (a uniqueness guard).
let guard = AppendCondition::new(Query::item(QueryItem::with_tags(
    Tags::new([Tag::new("course:c1").expect("tag")]).expect("tags"),
)));
let range = handle.append(vec![event], Some(guard)).expect("append");

append blocks until the batch is durable and returns the position range. Under the async cargo feature there is also an append_async; it is feature-gated and off by default.

Read

Reads run on the caller’s own thread over a snapshot the writer publishes at each commit. The read handle’s read returns Reads, a lending iterator: it yields a borrow of its own buffer per item, which is why it is consumed with while let rather than a for loop.

// Reads run on the caller's own thread over a snapshot published at each commit. Reads is a
// lending iterator, so it is consumed with `while let`, not a `for` loop.
let query = Query::item(QueryItem::with_tags(
    Tags::new([Tag::new("course:c1").expect("tag")]).expect("tags"),
));
let mut reads = handle.read(&query, Position::ZERO, None);
let mut count = 0usize;
while let Some(item) = reads.next() {
    let seq = item.expect("decode");
    println!("{} {}", seq.position, seq.event.event_type());
    count += 1;
}

Reads::next is not std::iter::Iterator::next. That is deliberate: a standard iterator cannot yield a borrow of itself per item, and yielding an owned value would force an allocation and a copy on the highest-volume read path. The client’s read stream, by contrast, is a plain Iterator, because it hands back owned events decoded off the wire.

Read backwards

read_back is the newest-first dual of read: it returns matching events in descending position order. Where read takes after (an exclusive lower bound), read_back takes before (an exclusive upper bound), so Position::MAX starts at the durable tip. It is the natural fit for an event-explorer UI that shows recent events first.

// Append a little history so there is something to page through.
for seats in [40u32, 50] {
    let payload = format!(r#"{{"course":"c1","seats":{seats}}}"#);
    let more = Event::new(&ty, &tags, payload.as_bytes()).expect("encode");
    handle.append(vec![more], None).expect("append");
}

// Read newest-first: `read_back` yields matching events in descending position order.
// `before` is an exclusive upper bound, so `Position::MAX` starts at the durable tip. Pair it
// with a `limit` and pass the oldest position of a page back as the next `before` to drive an
// event explorer one newest-first page at a time.
let mut newest_first = Vec::new();
let mut reads = handle.read_back(&query, Position::MAX, Some(10));
while let Some(item) = reads.next() {
    let seq = item.expect("decode");
    newest_first.push(seq.position);
}

Pagination is symmetric to a forward read: pass a limit for the page size, then set the next before to the oldest position the page returned, and the pages tile the whole history with no gap and no duplicate at any seam. The read is still pinned to the watermark at call time, and it stays fast without reading the whole log backward: it reads records in windows and only reverses their order within each window, so a page near the tip touches only a page’s worth of data. See Architecture for how the reverse scan preserves read-ahead.

Follow a live store read-only

A data directory takes one writer at a time, and that is enforced: a read-write open holds an advisory lock on a LOCK file for as long as the SegmentSet lives, so a second writer is refused instead of quietly corrupting the log.

A follower is the other way to attach. Follower::open opens the same directory read-only and tracks whatever the writer commits. It creates nothing, deletes nothing, writes no index file, opens every file read-only, and takes no lock, so it runs against a live writer, on a directory it has no write permission for, or on a read-only mount.

// Open the same directory read-only. This creates nothing, deletes nothing, writes
// nothing, and takes no lock, so it works while another process holds the write lock and
// even on a read-only mount. An empty or absent directory is an error rather than an
// empty store, so a follower that starts first should retry.
let follower = Follower::open(
    dir.path(),
    FollowerConfig::new(SegmentConfig::new(256 * 1024 * 1024)),
)
.expect("open follower");

An empty or missing directory is an error rather than an empty store. A follower that raced the writer’s very first segment would otherwise report “no events” for a store that is merely not ready yet, so start the writer first, or retry.

refresh advances to the writer’s current committed prefix and returns the new tip. Reads then go through the ordinary ReadHandle, so queries and backward reads behave exactly as they do against a writer.

// `refresh` advances to whatever the writer has committed and returns the new tip. What a
// follower exposes is always a committed prefix: gap-free, duplicate-free, and only
// growing. Reads then go through an ordinary ReadHandle.
let tip = follower.refresh().expect("refresh");
let query = Query::item(QueryItem::with_tags(
    Tags::new([Tag::new("course:c1").expect("tag")]).expect("tags"),
));
let reader = follower.reader();
let mut reads = reader.read(&query, Position::ZERO, None);
let mut seen = Vec::new();
while let Some(item) = reads.next() {
    seen.push(item.expect("decode").position);
}

For subscriptions, let a background poller do the advancing. A subscription is already just repeated reads off an advancing tip, so nothing about it changes.

// A background poller advances the follower on an interval, which is what lets a
// subscription tail a writer in another process. Dropping the poller stops the thread and
// closes the follower, so any parked subscription ends instead of hanging.
let follower = Arc::new(follower);
let poller = follower.poll_every(Duration::from_millis(10));
let mut subscription = follower.reader().subscribe(query.clone(), Position::ZERO);
let batch = subscription.poll_batch().expect("poll");

What a follower guarantees, and what it does not

Every position a follower exposes belongs to a batch that satisfied the commit rule: every record in it validated by CRC and the run ended in a valid commit marker. So what you read is always a prefix, gap-free, duplicate-free and only growing. You never see a torn record, a rolled-back batch, or a half-written index.

Three caveats matter before you build on one:

• It is not a durability oracle. A batch becomes visible when its commit marker reaches the page cache, which is just before the writer’s fsync returns. A power loss in that window erases events a follower has already read. Treat it like an asynchronous replica: do not let it be the last word on whether something happened. The writer’s own append is unaffected, since it returns only after the fsync.
• It lags by up to one poll interval plus the scan.
• It is same-host only. Visibility depends on the writer’s writes reaching a shared page cache, which holds for a local filesystem and not for a network mount.

In the same process, prefer the coordinator’s own ReadHandle: it shares the writer’s snapshot with no lag and no second scan. A follower is for when the writer is somewhere else.

Shutdown

WriteCoordinator::shutdown signals the writer thread, joins it, and returns the SegmentSet. Dropping the coordinator does the same. Either way the writer thread is joined before the process exits, so no committed batch is left unflushed.

For the field-by-field configuration surface, durability semantics, and what recovery does on restart, see Operations. For why the engine is shaped this way, see Architecture.