Projections

Felix stores everything in one kind of thing: an append-only log of records, split into shards, each shard owned by one broker and replicated to others. A semantic is a way of reading that log. Streams, caches and queues are three readings of the same bytes, not three subsystems.

This page says what each one stores, what it keeps in memory, and what it rebuilds from the log — and cites the test behind every claim it makes. A claim here without a citation is a claim nothing checks, which is the failure this page exists to prevent. scripts/check_doc_evidence.py verifies that every test named below still exists.

The three markers are the whole idea. A stream, a queue and a cache are not three stores — they are three ways of pointing into one log, and they run at the same time over the same bytes:

a stream marker sweeps every record, in the order it was written;
a queue marker covers the same records but trails, because it moves only as records are acknowledged;
a cache is not a single position at all. It is one marker per key, each jumping to that key’s newest record and leaving the older ones behind.

Which is why only the cache treats an older record as dead. Compaction reclaims those; a stream reading the same log still returns them, until retention removes the segment they live in. Nothing is copied into a second store, and no reading can disturb another.

Every semantic gets the same four things from the log, and none of them reimplements any of it:

Durability. Records are fsynced according to the stream’s policy, and a record acknowledged as durable is on disk.
Recovery. A torn tail is repaired at startup; interior corruption refuses to start rather than silently losing acknowledged records.
Placement and routing. Every shard of every stream and cache has one leader, chosen by rendezvous hashing. A broker that receives a request for a shard it does not lead forwards it or refuses.
Replication. The leader ships records at their offsets; a follower checks each batch begins at its tail.

a_cache_value_survives_the_loss_of_its_owner — a cache shard is replicated by the machinery that replicates a stream, because it is the same log.

Streams: read the log forward

The plainest reading. A subscriber holds an offset and moves through the log in order.

In the log: the records, as published.
In memory: per-subscriber queues with an overflow policy, and the offset each subscriber has reached.
Rebuilt from the log: nothing needs to be. A subscriber’s position is supplied by the client, so a broker holds no derived state a restart could lose.

Ordering is per shard. Two records in one shard are delivered in the order they were written; two records in different shards have no order between them, because they are different logs.

a_publish_to_a_non_owner_is_still_forwarded — a publish entering any broker reaches the shard’s owner.

Caches: read the log through a key index

A cache write is an append. A cache read is an index lookup followed by a read at that offset.

In the log: one record per put and per delete, each carrying the key.
In memory: an index of key to the offset of its latest record, plus the bytes each record occupies, so compaction knows when it is worth running.
Rebuilt from the log: the whole index. It is never read from disk, and it catches up to the log’s tail whenever it is behind — which is what lets a follower be promoted and serve records that arrived by replication rather than through a put.

Compaction rewrites the live set and drops superseded and expired records. It appends the live set at the tail rather than renumbering from zero, so an offset names the same record for the life of the shard.

a_cache_survives_a_restart — the index is rebuilt from the log. the_index_catches_up_with_records_appended_behind_it — records that arrive without going through a put are still found. compaction_does_not_rewind_the_offset_space — an offset keeps its meaning. a_deleted_key_stays_deleted_across_a_restart — a tombstone is a record like any other.

Queues: read the log through a shared cursor

A consumer group is a position in the log shared by its consumers, plus the bookkeeping for what is currently handed out.

In the log: nothing of the group’s own. The records are the stream’s.
On disk, beside the log: the group’s committed position, and the offsets it has given up on. Both are themselves key-to-latest-value projections — the same reading the cache is, on their own roots.
In memory: what is handed out and to when, what is owed, and how many times each unsettled record has been tried.

The in-memory part is deliberately not durable. A leader that dies loses it and the group resumes from its committed position, so those records are delivered again — which is at-least-once, the guarantee a queue offers anyway.

a_position_survives_a_restart — the committed position is durable. an_offset_in_flight_is_not_handed_out_again — one consumer holds a record at a time. a_lapsed_claim_is_handed_out_again — a consumer that stops answering does not hold a record for ever. the_cursor_does_not_advance_over_a_gap — the position moves only over a contiguous run of acknowledgements. a_record_is_given_up_on_after_the_attempt_bound — one poison record does not stop the queue. only_the_shard_owner_serves_a_group — two brokers cannot both hand out the same records. a_group_position_survives_a_leader_failover — a promoted leader resumes where the group had got to. a_dead_letter_survives_a_leader_failover — and it still lists what that group gave up on.

Both of those travel together. The offsets a group abandoned are one log per stream shard, with the group folded into the entry key exactly as the cursors are, so they ship beside the cursors on the same replica set at the same generation. A promoted leader lists what its groups set aside, and an operator’s redrive works there.

What a queue does not promise

Order. A shared cursor gives it up the moment two consumers hold adjacent records, and a redelivery reorders regardless of how many consumers there are. A queue in Felix preserves the order records were handed out in, and nothing about the order they are finished in.

A consumer per group. A group is bound to the shard the caller names, and nothing assigns shards across a group’s consumers. Scaling one past a single consumer per shard is the application’s job today.

Exactly-once. A record can be delivered twice: after a claim lapses, after a leader is lost, or after a redrive. Every consumer has to be able to see the same record again.

Where this stops being true

The claims above are the ones with tests. These are the gaps, listed so nothing here has to be read as covering them:

Retention outranks a group. A record trimmed before a group reached it is skipped, and the group moves past. A retention window shorter than a group is allowed to fall behind loses work.
A cache declares no consistency level. A stream chooses Leader or Quorum; a cache write is acknowledged by its leader, so losing that leader between the acknowledgement and the ship loses the write.
Dead letters recorded under the old layout do not travel. Shipping them became possible when the list moved to one log per stream shard, which is the unit the replication driver already walks; the earlier layout kept one log per (stream, group), a set it cannot enumerate, because a group appears whenever a consumer names one. Entries written under that earlier layout are still read and still discardable, but they were never shipped — so on a shard carrying them, a failover keeps the cursor and forgets those particular set-aside records.