rustka

A Kafka-style event streaming broker, built from scratch in Rust using only the standard library. No external crates.

It stores an append-only, replayable log of events on disk, splits topics into partitions, serves many clients at once over TCP, and lets consumer groups track their own position so they can resume after a restart.

The core idea

Kafka is one simple thing: an append-only log that many programs write to and read from over the network.

producers  ->  [ 0 | 1 | 2 | 3 | 4 | ... ]  ->  consumers
 (append)        the log (never edited)          (read by offset)

• Producers only add new records to the end. Old records are never changed.
• Each record gets a permanent number called an offset (0, 1, 2, ...).
• A consumer just remembers "I have read up to offset N" and continues from there.
• Reading a record does not delete it, so many consumers can read the same log at their own speed.

Everything else in this project is machinery around that one idea.

Features

• Append-only commit log stored on disk
• Segmented log files with an on-disk index for fast lookups by offset
• Automatic segment rollover when a segment gets too big
• Size-based retention that deletes old segment files
• Topics split into partitions, with key-based routing (same key always goes to the same partition)
• A binary wire protocol over TCP with length-prefixed framing
• A concurrent server (one thread per connection, shared state behind a lock)
• Consumer groups with committed offsets for at-least-once delivery
• A CLI to run the broker and to produce, fetch, and consume

How a request flows

client CLI  --TCP-->  server  -->  broker  -->  topic  -->  partition log  -->  segment files
 (produce)            (framing)    (routes)    (hashes     (rollover +          (.log + .index
                                                key)        retention)           on disk)

The server reads a framed request, hands it to the broker, and the broker calls into the storage layer. The response travels back the same way.

Project layout

File	What it does
src/record.rs	A single record (offset, key, value) and how to turn it into bytes and back
src/segment.rs	One segment: a .log file plus a .index file, with seek-based reads
src/log.rs	A partition log made of many segments: rollover, read routing, retention
src/topic.rs	A topic made of many partition logs, plus the key-to-partition hashing
src/offset_store.rs	Committed consumer offsets, stored as their own log
src/protocol.rs	The wire format: request and response types, encode, decode, and framing
src/broker.rs	The registry of topics and offsets; turns a request into a response
src/server.rs	The TCP server: accepts connections and serves them concurrently
src/client.rs	A small client library used by the CLI and tests
src/main.rs	The CLI: serve, produce, fetch, consume

On-disk layout

data/
  orders/                  a topic
    0/                     partition 0
      00000000000000000000.log     records
      00000000000000000000.index   offset -> byte position
    1/
      ...
    2/
      ...
  __offsets/               committed consumer offsets (its own log)
    00000000000000000000.log
    00000000000000000000.index

Each segment file is named by its base offset (the offset of its first record). There is no central metadata file. The broker rebuilds its state on startup just by listing the directory and reading the small index files.

Build and run

You need Rust and Cargo.

cargo build
cargo test      # runs the full test suite

Start the broker

cargo run -- serve

It listens on 127.0.0.1:9092 (the same default port as real Kafka).

You can also pass an optional max segment size and retention limit in bytes:

cargo run -- serve 1000000 50000000

Produce a message

In another terminal:

cargo run -- produce orders user-1 "placed order"

The topic is created automatically the first time it is used. The broker prints the partition and offset the message landed at.

Fetch a single message

cargo run -- fetch orders 2 0

That reads topic orders, partition 2, offset 0.

Consume as a group

cargo run -- consume analytics orders 2

This reads topic orders, partition 2, as the consumer group analytics. It starts from the group's committed offset, prints every new record, and commits its position as it goes. Run it again and it only sees new messages, because the broker remembered where the group left off. A different group name reads the same partition from the start, with its own position.

The wire protocol

Every message on the wire is length-prefixed so the reader always knows where it ends:

[u32 total_length][ payload bytes ]

Requests:

Produce      [op=1][topic][key][value]
Fetch        [op=2][topic][partition][offset]
Commit       [op=3][group][topic][partition][offset]
OffsetFetch  [op=4][group][topic][partition]

Responses:

Produced   [tag=1][partition][offset]
Record     [tag=2][offset][key][value]
NotFound   [tag=3]
Error      [tag=4][message]
CommitOk   [tag=5]
Offset     [tag=6][offset]

All numbers are big-endian. Strings and byte fields are length-prefixed the same way records are.

Design notes

• Why segments instead of one big file. Reads stay fast because we seek straight to a record instead of scanning. Startup stays fast because we read a small index instead of the whole log. Old data can be dropped by deleting whole segment files, which is cheap. A single giant file could not do any of these well.
• Why length-prefixed framing. TCP is a raw stream of bytes with no message boundaries. Framing puts the boundaries back so two requests never blur together.
• Why a lock around the broker. Many client threads share one broker. A mutex makes concurrent writes safe, and Rust will not even compile shared mutation without it. The lock is held only while handling one request, never while waiting on the network, so idle clients cost nothing.
• Why offsets are stored as a log. Committed offsets do not need a database. Each commit is appended to a log, and on startup we replay it and keep the last value for each key. This mirrors how real Kafka uses its internal __consumer_offsets topic.

Known limitations

This is a learning project, so some things are simplified on purpose:

• One global lock, so requests are handled one at a time inside the broker. Real Kafka uses finer locks per partition.
• A dense index (one entry per record). Real Kafka uses a sparse index to save space.
• No replication across machines, so there is one copy of the data.
• If a consumer's committed offset points at data that retention already deleted, the consumer is not reset to the new earliest offset yet.
• The partition count for a topic is fixed once created.

Tests

cargo test

The suite covers record encoding, segment reads, log rollover and retention, partition routing, the wire protocol, the broker, real end-to-end traffic over TCP, concurrent producers, and consumer group resume.