Project Structure

Understanding Felix’s repository layout, crate organization, and architectural conventions.

Repository Overview

felix/
├── crates/              # Rust crates (libraries)
├── services/            # Runnable services (binaries)
├── docs/                # Design and architecture docs
├── docs-site/           # User documentation (Astro Starlight)
├── scripts/             # Build and automation scripts
├── docker/              # Docker configuration
├── .github/             # GitHub Actions workflows
├── githooks/            # Git hooks for development
├── Cargo.toml           # Workspace manifest
├── Cargo.lock           # Dependency lock file
├── Taskfile.yml         # Task runner configuration
├── deny.toml            # Dependency audit rules
└── rust-toolchain.toml  # Rust version specification

Crates Directory

The crates/ directory contains all library crates following a modular architecture:

crates/
├── felix-broker/        # Broker core (pub/sub, cache, fanout)
├── felix-wire/          # Wire protocol and framing
├── felix-transport/     # QUIC transport abstraction
├── felix-storage/       # Storage layer (ephemeral + durable)
├── felix-client/        # Client SDK
├── felix-common/        # Shared types and utilities
├── felix-router/        # Region-aware routing
├── felix-authz/         # Authentication and authorization
└── felix-conformance/   # Wire protocol conformance tests

Core Crates

felix-broker

Purpose: Broker core logic for pub/sub, cache and consumer groups.

Responsibilities:

• Stream registry and subscription management
• Event fanout and batching
• Cache storage with TTL
• Consumer groups: claims, redelivery, dead letters
• Backpressure and flow control
• Connection lifecycle management

Key modules:

• broker.rs: The Broker aggregate, construction, and the publish/subscribe data path
• registry.rs: Tenant / namespace / stream / cache registries
• stream_state.rs: Per-stream subscriber registry, publish snapshot, and replay log
• subscription.rs: Subscriber-facing receive handles and the unregister guard
• delivery.rs: Shared delivery batches and subscriber queue-depth accounting
• commit_order.rs: CommitSequencer, which holds a publish behind the ones that took their offsets before it
• durable.rs: The DurableStorage / StreamLog seam between the broker and a shard’s log
• replication.rs: Leader-side shipping and follower-side acceptance of committed records
• consumer_groups.rs: A group’s durable cursor, a key → latest-value projection over its own log
• group_delivery.rs: GroupTracker — in-flight claims, the visibility timeout, attempt counts, and the contiguous-run advance
• group_reader.rs: Joins the stream’s log, the cursor and the tracker into poll / ack / nack
• dead_letters.rs: Offsets a group gave up on, stored as pointers into the stream’s log rather than copies
• keys.rs: Map keys plus their borrowed lookup twins
• config.rs / error.rs / telemetry.rs: Capacity defaults and queue policy, BrokerError, cfg-gated metrics shims

Everything public is re-exported from lib.rs, so downstream code addresses these types as felix_broker::<Name> regardless of which module defines them.

Dependencies:

• felix-wire: Protocol framing
• felix-transport: QUIC abstraction
• felix-storage: Data persistence
• felix-common: Shared types

felix-wire

Purpose: Wire protocol definition and frame encoding/decoding.

Responsibilities:

• Frame type definitions
• Binary batch encoding/decoding
• Protocol versioning
• Frame validation

Key modules:

• frame.rs: Protocol constants, FrameHeader, and Frame
• message.rs: The Message enum and its JSON codec
• text.rs: Hand-rolled zero-copy JSON writer for the publish-batch hot path
• binary.rs: Binary batch codec for publish and event batches
• error.rs / base64_serde.rs: Wire Error type, base64 serde adapters

Key types:

• Frame / FrameHeader: Top-level frame and its 12-byte header
• Message: The v1 message enum carried in JSON control frames
• binary::PublishBatch / binary::EventBatch: Binary batch frame formats

frame, message, and error items are re-exported at the crate root; text and binary are addressed through their module paths (felix_wire::binary::…).

Protocol layers:

1. Envelope: Version, type, length
2. Binary frames: Zero-copy fast paths

felix-transport

Purpose: QUIC transport abstraction and connection pooling.

Responsibilities:

• QUIC client/server setup
• Connection lifecycle
• Stream management
• TLS certificate handling
• Flow control configuration

Key types:

• QuicClient: Client-side connection
• QuicServer: Server-side listener
• StreamPool: Connection pooling
• QuicConfig: Transport configuration

Based on: quinn (QUIC implementation)

felix-storage

Purpose: Storage layer abstraction for ephemeral and durable data.

Responsibilities:

• Ephemeral in-memory storage
• Durable WAL and log segments
• TTL management
• Retention policies
• Compaction for the cache and counter logs; stream logs are not compacted

Storage types:

• EphemeralStore: In-memory with TTL
• DurableStore: persistent log-structured segment store
• CacheStore: Key-value with expiration

felix-client

Purpose: Rust client SDK for Felix.

Responsibilities:

• Publish API
• Subscribe API
• Cache operations (put/get)
• Connection management
• Stream pooling
• Error handling

Key types:

• Client: Main client interface
• Publisher: Publishing handle
• Subscription: Subscription handle
• InProcessClient: Embedded testing client
• ClientConfig: Client configuration

Example usage:

use felix_client::{Client, ClientConfig};
use felix_wire::AckMode;
use std::net::SocketAddr;

let quinn = quinn::ClientConfig::with_platform_verifier();
let config = ClientConfig::optimized_defaults(quinn);
let addr: SocketAddr = "127.0.0.1:5000".parse()?;
let client = Client::connect(addr, "localhost", config).await?;
let publisher = client.publisher().await?;
publisher
    .publish("tenant", "namespace", "stream", b"data".to_vec(), AckMode::None)
    .await?;

Supporting Crates

felix-common

Purpose: Shared types and utilities used across crates.

Contents:

• types.rs: Common type definitions
• error.rs: Error types
• ids.rs: ID types (TenantId, StreamId, etc.)
• config.rs: Configuration types
• time.rs: Time utilities

Principle: Minimal dependencies, stable API.

felix-router

Purpose: Region-aware routing and locality policies.

Responsibilities:

• Region topology
• Locality-based routing
• Cross-region bridge configuration
• Request routing logic

Future: Control plane integration for dynamic routing.

felix-authz

Purpose: Authentication and authorization.

Responsibilities:

• Token-based auth (OIDC exchange + Felix JWTs)
• RBAC policies and permission matching
• Tenant isolation enforcement
• Broker-to-broker mTLS, when certificates are configured

felix-conformance

Purpose: Wire protocol conformance test suite.

Responsibilities:

• Test vector validation
• Cross-implementation testing
• Protocol regression tests

Usage:

cargo run -p felix-conformance

Services Directory

The services/ directory contains runnable binaries:

services/
├── broker/              # Broker service
│   ├── src/
│   │   ├── main.rs      # Broker entrypoint
│   │   ├── config.rs    # Configuration loading
│   ├── Cargo.toml
│   └── README.md        # Performance profiles
└── controlplane/        # Control plane service

demos/
├── broker/              # Demo binaries for the broker crate
│   ├── simple_pubsub_demo.rs
│   ├── cache_demo.rs
│   ├── latency_demo.rs
│   ├── notifications_demo.rs
│   └── orders_demo.rs
├── rbac-live/            # End-to-end RBAC mutation demo crate
│   └── src/main.rs
└── cross_tenant_isolation/  # End-to-end tenant isolation demo crate
    └── src/main.rs

Broker Service

Location: services/broker/

Entrypoint: src/main.rs

Responsibilities:

• Load configuration from env/YAML
• Initialize broker runtime
• Start QUIC listener
• Expose metrics endpoint
• Handle graceful shutdown

Demo binaries (see demos/broker/):

• simple_pubsub_demo.rs: Pub/sub demonstration
• cache_demo.rs: Cache benchmark
• latency_demo.rs: Latency measurement tool
• notifications_demo.rs: Multi-tenant notifications workflow demo
• orders_demo.rs: Orders/payments pipeline demo
• rbac-live/: Live RBAC policy change demo (control plane + broker + token exchange)
• cross_tenant_isolation/: Cross-tenant isolation demo (Postgres + control plane + broker)

Documentation

Design Docs (docs/)

Architecture and design documentation:

docs/
├── architecture.md      # System architecture overview
├── demos.md             # Demo catalog and run instructions
├── protocol.md          # Wire protocol specification
├── control-plane.md     # Control plane design
├── semantics.md         # Delivery semantics
├── design.md            # Product design notes
├── broker-config.md     # Broker configuration
├── client-config.md     # Client configuration
├── todos.md             # The original MVP checklist (historical)
└── assets/              # Diagrams and images
    └── logo.PNG

Purpose: Technical design for contributors.

User Docs (docs-site/)

User-facing documentation (Astro Starlight):

docs-site/
├── src/
│   ├── content/
│   │   └── docs/
│   │       ├── index.md
│   │       ├── getting-started/
│   │       ├── architecture/
│   │       ├── api/
│   │       ├── features/
│   │       ├── deployment/
│   │       ├── reference/
│   │       └── development/
│   └── content.config.ts
├── astro.config.mjs     # Site and navigation configuration
└── package.json

Purpose: End-user guides and API references.

Build:

cd docs-site
npm install
npm run dev

Scripts Directory

Automation and utility scripts:

scripts/
└── perf/                # Performance benchmarking
    ├── run_latency_matrix.py
    ├── normalize_and_aggregate.py
    ├── make_charts.py
    ├── render_markdown_snippets.py
    └── presets.yml      # Benchmark configurations

Docker Directory

Docker build configuration:

docker/
├── broker.Dockerfile           # Multi-stage broker build
├── controlplane.Dockerfile     # Control plane build
├── prometheus/
│   ├── prometheus.yml          # Prometheus config
│   └── prometheus.Dockerfile
└── otel-collector/
    ├── config.yml              # OTEL config
    └── otel-collector.Dockerfile

GitHub Workflows

CI/CD configuration:

.github/
├── workflows/
│   ├── ci.yml           # Main CI pipeline
│   └── coverage.yml     # Code coverage
└── dependabot.yml       # Dependency updates

Configuration Files

Cargo.toml (Workspace)

Purpose: Define workspace and shared dependencies.

[workspace]
members = [
    "crates/*",
    "services/*",
]
resolver = "2"

[workspace.dependencies]
tokio = { version = "1.35", features = ["full"] }
anyhow = "1.0"
# ... shared dependencies

rust-toolchain.toml

Purpose: Pin Rust version for consistency.

[toolchain]
channel = "1.97.1"
components = ["rustfmt", "clippy"]

deny.toml

Purpose: Configure cargo-deny for dependency auditing.

Checks:

• Security vulnerabilities
• License compliance
• Banned crates
• Duplicate dependencies

Taskfile.yml

Purpose: Define common development tasks.

Tasks: build, test, lint, fmt, coverage, demos, etc.

Naming Conventions

Crate Names

• Library crates: felix-<component> (e.g., felix-broker)
• Binary crates: Service name (e.g., broker)
• All lowercase, hyphen-separated

Module Structure

crate_root/
├── lib.rs              # Public API (library)
├── main.rs             # Entrypoint (binary)
├── module_name.rs      # Single-file module
└── module_name/        # Multi-file module
    ├── mod.rs          # Module root
    ├── submodule.rs
    └── tests.rs        # Module tests

File Naming

• Snake_case: my_module.rs
• Tests: mod_tests.rs or tests/
• Binaries: bin/my_app.rs

Type Naming

• PascalCase: Structs, enums, traits (BrokerConfig, FrameType)
• snake_case: Functions, methods, variables (publish_event, config_value)
• SCREAMING_SNAKE_CASE: Constants (DEFAULT_PORT, MAX_BATCH_SIZE)

Dependency Management

Dependency Categories

Core dependencies:

• tokio: Async runtime
• quinn: QUIC implementation
• serde: Serialization
• anyhow/thiserror: Error handling

Development dependencies:

• serial_test: Test isolation
• tempfile: Temporary files in tests
• criterion: Benchmarking

Dependency Rules

1. Minimize dependencies: Only add when necessary
2. Pin versions: Use exact versions in workspace
3. Audit regularly: Run cargo-deny check
4. No unmaintained crates: Check maintenance status
5. License compliance: Only Apache-2.0 / MIT

Adding Dependencies

# Add to workspace
cargo add --workspace <crate>

# Add to specific crate
cargo add -p felix-broker <crate>

# Add dev dependency
cargo add --dev <crate>

Testing Structure

Test Organization

Unit tests: Inline in source files

#[cfg(test)]
mod tests {
    use super::*;
    // Tests here
}

Integration tests: tests/ directory

crate/
  tests/
    integration_test.rs
    common/
      mod.rs           # Shared test utilities

Conformance tests: Separate crate (felix-conformance)

Test Naming

• Functions: test_<what_it_does>
• Modules: tests or <module>_tests
• Files: integration_test.rs, e2e_test.rs

Build Artifacts

Target Directory

target/
├── debug/              # Debug builds
│   ├── broker          # Binary
│   ├── deps/           # Dependencies
│   └── build/          # Build scripts
├── release/            # Release builds
└── doc/                # Generated docs

Cargo Cache

~/.cargo/
├── registry/           # Downloaded crate sources
├── git/                # Git dependencies
└── bin/                # Installed binaries

Development Environment

Recommended Setup

Editor: VS Code with rust-analyzer

Extensions:

• rust-analyzer: IntelliSense
• CodeLLDB: Debugging
• Better TOML: TOML syntax

.vscode/settings.json:

{
    "rust-analyzer.cargo.features": "all",
    "rust-analyzer.checkOnSave.command": "clippy"
}

Git Hooks

# Install pre-commit hook
cp githooks/pre-commit .git/hooks/
chmod +x .git/hooks/pre-commit

Pre-commit hook:

• Format check
• Clippy warnings
• Run tests

Code Organization Principles

Modularity

• Small, focused crates: Each crate has a single purpose
• Clear boundaries: Minimal cross-crate dependencies
• Public API: pub means “someone outside this crate uses this.” Everything else is pub(crate), and the unreachable_pub lint (enforced workspace-wide, promoted to an error by CI’s -D warnings) catches drift.

Module style

One style throughout: a module foo is foo.rs with its submodules in foo/ — never foo/mod.rs. Clippy’s mod_module_files lint enforces it workspace-wide. The single exception is tests/common/mod.rs, which is the standard Cargo pattern for helpers shared between integration-test binaries.

Dependency versions

Shared dependencies are declared once in the root [workspace.dependencies] and inherited with dep = { workspace = true }; a member adds features on top when it needs them. A version bump is one edit, and two crates cannot drift onto different versions of the same dependency.

Layering

┌─────────────────────────┐
│   Services (binaries)   │
├─────────────────────────┤
│    Application Layer    │
│  (broker, client, etc)  │
├─────────────────────────┤
│     Protocol Layer      │
│   (wire, transport)     │
├─────────────────────────┤
│    Foundation Layer     │
│  (common, storage)      │
└─────────────────────────┘

Dependency Direction

Dependencies flow downward:

• Services depend on applications
• Applications depend on protocol
• Protocol depends on foundation
• Foundation has minimal dependencies

Never: Lower layers depend on upper layers.

Future Structure Changes

New capabilities may add crates, language SDKs, and deployment packaging (Helm charts, operators). The core structure — crates/ for libraries, services/ for binaries, demos outside the workspace — will remain stable. Placeholder crates are not kept around: a crate exists when something uses it.

Next Steps

• Contributing: Contributing Guide
• Building: Building & Testing
• Architecture: System Design