Tier 2 Storage Engine

Kafka Storage Internals — Segments & Sparse Indexing

Append-only .log segments, memory-mapped sparse .index files, binary search & zero-copy sendfile

01 / 12
P
Producer Batching
Writes messages to Kafka:
Msg (Offset 109)40 B
Msg (Offset 110)40 B
Msg (Offset 111)40 B
Msg (Offset 112)40 B
in: Events (IN:2) out: Batch (OUT:2)
B
Kafka Broker (Partition clicks-0) Active
...0100 Closed
.index (Sparse)
OffsetPos (B)
1000
103120
106240
.log (Sequential)
Offset 100Pos: 0 B | Sz: 40B
Offset 101Pos: 40 B | Sz: 40B
Offset 102Pos: 80 B | Sz: 40B
Offset 103Pos: 120 B | Sz: 40B
Offset 104Pos: 160 B | Sz: 40B
Offset 105Pos: 200 B | Sz: 40B
Offset 106Pos: 240 B | Sz: 40B
Offset 107Pos: 280 B | Sz: 40B
Offset 108Pos: 320 B | Sz: 40B
...0109 Active
.index (Sparse)
OffsetPos (B)
1090
.log (Sequential)
Offset 109Pos: 0 B | Sz: 40B
Index Acc: 40 B / 120 B
in: Produce/Fetch (IN:2) out: Pagecache (OUT:2)
C
Consumer Polling
Consumed offsets:
History empty
in: Zero-Copy (IN:2) out: Offset (OUT:2)
100%