A persistent, fsync-durable binary stack backed by a single file.
Every write — push, pop, and the optional set/atomic operations —
performs a durable sync before returning, so data survives a process crash
or unclean shutdown. On macOS, fcntl(F_FULLFSYNC) is used instead of
fdatasync to flush the drive's hardware write cache, which plain
fdatasync does not guarantee.
A 32-byte file header stores a magic number, a committed-length sentinel, and a write-in-progress journal. On reopen, an interrupted in-place write is replayed or rolled back and any header/size mismatch is repaired automatically — no user intervention required.
On Unix, open acquires an exclusive advisory flock; on
Windows, LockFileEx is used instead. Both prevent two processes from
concurrently corrupting the same stack file.
The optional atomic feature adds compound read-modify-write and
compare-and-swap operations, including a generator-driven get_batched_gen for
multi-step reads, process_gen for multi-step read and writes, and set_batched
/ inplace_gen for committing several in-place writes as one crash-atomic unit.
Independent of
features, lock_up_to lets a prefix of the file be marked permanently immutable
for lock-free reads, and an optional in-memory cache (open_cached) can
mirror that region for even faster, syscall-free access.
Other optional Cargo features layer on more: set adds in-place writes to
existing bytes, and alloc adds typed sub-allocators (linear, first-fit,
slab, etc.) over the payload.
**Minimal dependencies (libc on Unix, windows-sys on Windows).
Upgrading from 0.2.x? See docs/MIGRATION_0.4.0.md for a step-by-step migration guide.
Warning: bstack files must only be opened through this crate or a compatible implementation that understands the file format, header protocol, and locking semantics. Reading or writing the file with raw tools (
dd,xxd, customopen(2)calls, etc.) while aBStackinstance is live, or manually editing the header fields, can silently corrupt the committed-length sentinel or bypass the advisory lock. The authors make no guarantees about the behaviour of the crate — including freedom from data loss or logical corruption — when the file has been accessed outside of this crate's controlled interface.
use bstack::BStack;
let stack = BStack::open("log.bin")?;
// push appends bytes and returns the starting logical offset.
let off0 = stack.push(b"hello")?; // 0
let off1 = stack.push(b"world")?; // 5
assert_eq!(stack.len()?, 10);
// peek reads from a logical offset to the end.
assert_eq!(stack.peek(off1)?, b"world");
// get reads an arbitrary half-open logical byte range.
assert_eq!(stack.get(3, 8)?, b"lowor");
// pop removes bytes from the tail and returns them.
assert_eq!(stack.pop(5)?, b"world");
assert_eq!(stack.len()?, 5);impl BStack {
/// Open or create a stack file at `path`.
/// Acquires an exclusive flock on Unix, or LockFileEx on Windows.
/// Validates the header and performs crash recovery on existing files.
pub fn open(path: impl AsRef<Path>) -> io::Result<Self>;
/// Append `data` and durable-sync. Returns the starting logical offset.
/// An empty slice is valid and a no-op on disk.
pub fn push(&self, data: impl AsRef<[u8]>) -> io::Result<u64>;
/// Append `n` zero bytes and durable-sync. Returns the starting logical offset.
/// `n = 0` is valid and a no-op on disk.
pub fn extend(&self, n: u64) -> io::Result<u64>;
/// Remove and return the last `n` bytes, then durable-sync.
/// `n = 0` is valid. Errors if `n` exceeds the current payload size.
pub fn pop(&self, n: u64) -> io::Result<Vec<u8>>;
/// Remove the last `buf.len()` bytes and write them into `buf`, then durable-sync.
/// An empty buffer is a valid no-op. Errors if `buf.len()` exceeds the current payload size.
/// Prefer this over `pop` when a buffer is already available to avoid an extra allocation.
pub fn pop_into(&self, buf: &mut [u8]) -> io::Result<()>;
/// Discard the last `n` bytes without reading or returning them, then durable-sync.
/// `n = 0` is valid and is a no-op. Errors if `n` exceeds the current payload size.
/// Prefer this over `pop` when the removed bytes are not needed, to avoid any allocation or copy.
pub fn discard(&self, n: u64) -> io::Result<()>;
/// Overwrite `data` bytes in place starting at logical `offset`.
/// Never changes the file size; errors if the write would exceed the
/// current payload. Requires the `set` feature.
#[cfg(feature = "set")]
pub fn set(&self, offset: u64, data: impl AsRef<[u8]>) -> io::Result<()>;
/// Overwrite `n` bytes with zeros in place starting at logical `offset`.
/// Never changes the file size; errors if the write would exceed the
/// current payload. `n = 0` is a no-op. Requires the `set` feature.
#[cfg(feature = "set")]
pub fn zero(&self, offset: u64, n: u64) -> io::Result<()>;
/// Fill `count` copies of `pattern` in place starting at logical `offset`
/// (the general form of `zero`). Never changes the file size; errors if the
/// fill would exceed the current payload. An empty `pattern` or `count = 0`
/// is a no-op. Requires the `set` feature.
#[cfg(feature = "set")]
pub fn repeat(&self, offset: u64, pattern: impl AsRef<[u8]>, count: u64) -> io::Result<()>;
/// Atomically cut `n` bytes off the tail then append `buf`.
/// Combines discard + push under a single write lock. Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn atrunc(&self, n: u64, buf: impl AsRef<[u8]>) -> io::Result<()>;
/// Pop `n` bytes off the tail then append `buf`; returns the removed bytes.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn splice(&self, n: u64, buf: impl AsRef<[u8]>) -> io::Result<Vec<u8>>;
/// Pop `old.len()` bytes into `old` then append `new`.
/// Buffer-reuse variant of `splice`. Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn splice_into(&self, old: &mut [u8], new: impl AsRef<[u8]>) -> io::Result<()>;
/// Append `buf` only if the current payload size equals `s`; returns whether it did.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn try_extend(&self, s: u64, buf: impl AsRef<[u8]>) -> io::Result<bool>;
/// Discard `n` bytes only if the current payload size equals `s`; returns whether it did.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn try_discard(&self, s: u64, n: u64) -> io::Result<bool>;
/// Append `n` zero bytes only if the current payload size equals `s`; returns whether it did.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn try_extend_zeros(&self, s: u64, n: u64) -> io::Result<bool>;
/// Atomically read `buf.len()` bytes at `offset` and overwrite them with `buf`;
/// returns the old contents. Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn swap(&self, offset: u64, buf: impl AsRef<[u8]>) -> io::Result<Vec<u8>>;
/// Atomic swap via a caller-supplied buffer: on return `buf` holds the old bytes.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn swap_into(&self, offset: u64, buf: &mut [u8]) -> io::Result<()>;
/// Compare-and-exchange: if the bytes at `offset` match `old`, overwrite with `new`.
/// Returns `true` if the exchange was performed. Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn cas(&self, offset: u64, old: impl AsRef<[u8]>, new: impl AsRef<[u8]>) -> io::Result<bool>;
/// Read the tail `n` bytes, pass them to `f`, write back whatever `f` returns as the new tail.
/// The file may grow or shrink. `n = 0` is valid. Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn replace<F>(&self, n: u64, f: F) -> io::Result<()>
where F: FnOnce(&[u8]) -> Vec<u8>;
/// Read `[start, end)`, pass the bytes to `f` for in-place mutation, write them back.
/// File size never changes. `start == end` is a valid no-op.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn process<F>(&self, start: u64, end: u64, f: F) -> io::Result<()>
where F: FnOnce(&mut [u8]);
/// Atomically swap two non-overlapping byte regions of length `n` under one write lock.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn cross_exchange(&self, a: u64, b: u64, n: u64) -> io::Result<()>;
/// Run a sequence of dependent reads, optionally followed by a single write, under
/// one held write lock. `f` is called in a loop and drives the sequence through
/// `BStackGenOp::{Read, Len, Write, Swap, Push, Pop, Discard, Atrunc, Splice}`; at
/// most one of `Write`/`Swap`/`Push`/`Pop`/`Discard`/`Atrunc`/`Splice` is permitted
/// and ends the sequence.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn process_gen<'a, F>(&self, f: F) -> io::Result<()>
where F: FnMut() -> Option<BStackGenOp<'a>>;
/// Commit several non-overlapping in-place writes as one crash-atomic unit.
/// Each `(offset, data)` overwrites `[offset, offset + data.len())`; either all
/// apply or none do. Empty `data` is ignored, overlapping writes are rejected,
/// and the file size never changes.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn set_batched<I, D>(&self, writes: I) -> io::Result<()>
where I: IntoIterator<Item = (u64, D)>, D: AsRef<[u8]>;
/// Run dependent reads interleaved with multiple in-place writes, committing
/// every write together at the end via the multi-write journal. `f` is called
/// in a loop over `BStackGenOp::{Read, Write, Len}` (size-changing ops and
/// `Swap` are rejected); `Write`s accumulate rather than ending the sequence,
/// later writes override earlier overlapping ones, and `Read`s see the
/// batch-so-far content. `f` receives the previous op's `io::Result` (an
/// erroring op is simply not recorded); `None` commits and ends.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn inplace_gen<'a, F>(&self, f: F) -> io::Result<()>
where F: FnMut(io::Result<()>) -> Option<BStackGenOp<'a>>;
/// Copy `n` bytes from `from` to `to` under one write lock. Regions may overlap.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn copy(&self, from: u64, to: u64, n: u64) -> io::Result<()>;
/// Write `b_buf` at `b_offset` only if bytes at `a_offset` equal `a_expected`.
/// Returns `Some(old_b)` on success, `None` if the condition was not met.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn eq_crds(&self, a_offset: u64, a_expected: impl AsRef<[u8]>,
b_offset: u64, b_buf: impl AsRef<[u8]>) -> io::Result<Option<Vec<u8>>>;
/// Like `eq_crds` but writes when region A does NOT match `a_expected`.
/// Returns `Some(old_b)` on success, `None` if the condition was not met.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn ne_crds(&self, a_offset: u64, a_expected: impl AsRef<[u8]>,
b_offset: u64, b_buf: impl AsRef<[u8]>) -> io::Result<Option<Vec<u8>>>;
/// Like `eq_crds` with a bitmask applied to the comparison of region A.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn masked_eq_crds(&self, a_offset: u64, mask: impl AsRef<[u8]>,
a_expected: impl AsRef<[u8]>, b_offset: u64,
b_buf: impl AsRef<[u8]>) -> io::Result<Option<Vec<u8>>>;
/// Like `ne_crds` with a bitmask applied to the comparison of region A.
/// Requires the `set` and `atomic` features.
#[cfg(all(feature = "set", feature = "atomic"))]
pub fn masked_ne_crds(&self, a_offset: u64, mask: impl AsRef<[u8]>,
a_expected: impl AsRef<[u8]>, b_offset: u64,
b_buf: impl AsRef<[u8]>) -> io::Result<Option<Vec<u8>>>;
/// Copy all bytes from `offset` to the end of the payload.
/// `offset == len()` returns an empty Vec.
pub fn peek(&self, offset: u64) -> io::Result<Vec<u8>>;
/// Fill `buf` with exactly `buf.len()` bytes starting at logical `offset`.
/// An empty buffer is a valid no-op.
/// Prefer this over `peek` when a buffer is already available to avoid an extra allocation.
pub fn peek_into(&self, offset: u64, buf: &mut [u8]) -> io::Result<()>;
/// Copy bytes in the half-open logical range `[start, end)`.
/// `start == end` returns an empty Vec.
pub fn get(&self, start: u64, end: u64) -> io::Result<Vec<u8>>;
/// Fill `buf` with bytes from the half-open logical range `[start, start + buf.len())`.
/// An empty buffer is a valid no-op.
/// Prefer this over `get` when a buffer is already available to avoid an extra allocation.
pub fn get_into(&self, start: u64, buf: &mut [u8]) -> io::Result<()>;
/// Read multiple byte ranges under a single read lock; ranges may overlap.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn get_batched<I>(&self, ranges: I) -> io::Result<Vec<Vec<u8>>>
where I: IntoIterator<Item = std::ops::Range<u64>>;
/// Like `get_batched` but reads into caller-supplied `(offset, buf)` pairs.
/// Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn get_batched_into<'a, I>(&self, bufs: I) -> io::Result<()>
where I: IntoIterator<Item = (u64, &'a mut [u8])>;
/// Like `get_batched_into` but the caller supplies a generator closure yielding
/// `(offset, buf)` pairs; `None` ends the batch. Requires the `atomic` feature.
#[cfg(feature = "atomic")]
pub fn get_batched_gen<'a, F>(&self, f: F) -> io::Result<()>
where F: FnMut() -> Option<(u64, &'a mut [u8])>;
/// Current payload size in bytes (excludes the 32-byte header).
pub fn len(&self) -> io::Result<u64>;
/// Current locked length. `0` means no bytes are locked.
/// Bytes in `[0, locked_len())` are permanently immutable for the lifetime of this open file.
pub fn locked_len(&self) -> u64;
/// Extend the locked region to cover `[0, n)`. Monotonically growing: `n` must be
/// ≥ the current locked length and ≤ the current payload length. After this call,
/// reads to `[0, n)` are lock-free on Unix and Windows, and writes/shrinks that
/// would touch `[0, n)` return `InvalidInput`.
pub fn lock_up_to(&self, n: u64) -> io::Result<()>;
/// Open a `BStack` and immediately lock the first `n` bytes.
/// Convenience for the common pattern where the locked region is known up front.
pub fn open_locked_up_to(path: impl AsRef<Path>, n: u64) -> io::Result<Self>;
/// Create a `BStackReader` positioned at the start of the payload.
pub fn reader(&self) -> BStackReader<'_>;
/// Create a `BStackReader` positioned at `offset` bytes into the payload.
pub fn reader_at(&self, offset: u64) -> BStackReader<'_>;
}
// BStack and &BStack both implement std::io::Write (each write = one push + durable_sync).
// BStackReader implements std::io::Read + std::io::Seek + From<&BStack>.BStack and &BStack implement [std::io::Write]; each write call
forwards to push (atomic append + durable sync). flush is a no-op.
Wrap in BufWriter to batch many small writes into a single push.
use std::io::Write;
let mut stack = BStack::open("log.bin")?;
stack.write_all(b"hello")?;
std::io::copy(&mut std::io::Cursor::new(b"world"), &mut stack)?;[BStackReader] wraps a &BStack with a cursor and implements
[std::io::Read] and [std::io::Seek]. Multiple readers can coexist
concurrently with each other and with peek/get calls.
use std::io::{Read, SeekFrom, Seek};
use bstack::{BStack, BStackReader};
let stack = BStack::open("log.bin")?;
let mut reader = stack.reader(); // from the start
let mut mid = stack.reader_at(6); // from offset 6
let mut buf = [0u8; 5];
reader.read_exact(&mut buf)?;BStack maintains an in-memory monotonically growing partition boundary
called the locked region. Bytes in [0, locked_len()) are declared
permanently immutable for the lifetime of the open file.
The locked length starts at 0 on every open and is not persisted to
disk — the file format is unchanged. Callers extend the boundary by
calling lock_up_to (or open and lock in one step with
open_locked_up_to). It can only grow; attempts to shrink it return
InvalidInput.
Opening with open_cached (or open_locked_up_to_cached) enables an
in-memory mirror of the locked region: each lock_up_to call reads the
newly locked bytes from disk into a heap buffer, and subsequent reads whose
range falls entirely within the cached region are served with no syscall.
The trade-off is that lock_up_to becomes significantly more expensive on
cached stacks (it must read up to n bytes from disk before returning).
get/get_intofast-path reads. Calls whose range lies entirely within the locked region bypass the internalRwLock.- On non-cached stacks (Unix/Windows), the read is lock-free and uses
pread(2)(Unix) orReadFile+OVERLAPPED(Windows). - On cached stacks (all platforms), the read is served from the
in-memory buffer under a
Mutex(so RwLock-free, but not lock-free). The locked length remains a sufficient upper bound, so no extra payload-size check is needed on this path.
- On non-cached stacks (Unix/Windows), the read is lock-free and uses
- Write protection.
set,zero,repeat,swap,swap_into,cas,process,atrunc,splice,splice_into, andreplacereturnInvalidInputif their target overlaps the locked region. - Shrink protection.
pop,pop_into,discard, andtry_discardreturnInvalidInputif they would shrink the payload below the locked length.
Callers that never call lock_up_to see no behavioural change — every
read and write path adds only an uncontended atomic load and a comparison.
use bstack::BStack;
// 64-byte metadata header read by many threads, never modified after first write.
let stack = BStack::open_locked_up_to("meta.bin", 64)?;
assert_eq!(stack.locked_len(), 64);
// Reads of the metadata bypass the rwlock on Unix and Windows.
let header = stack.get(0, 64)?;
// Writes into the locked region are rejected.
assert!(stack.pop(stack.len()? - 60).is_err()); // would shrink below lockedlock_up_to(n) takes the write lock and publishes the new boundary with a Release store. Locked-region fast-path readers Acquire-load the boundary; a stale load safely falls through to the rwlock path. Writers re-check under the write lock and cannot race against an in-flight lock_up_to. On cached stacks this fast path is available on all platforms via the cache Mutex; on non-cached stacks the lock-free path is Unix/Windows-only.
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Pointer identity. Two values are equal iff they are the same instance. No two distinct BStack values in one process can refer to the same file. |
Hash |
Hashes the instance address — consistent with pointer-identity equality. |
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Equal when both the BStack pointer and the cursor offset match. |
Hash |
Hashes (BStack pointer, offset). |
PartialOrd / Ord |
Ordered by BStack instance address, then by cursor offset. |
BStackRange — raw (offset, len) coordinate pair, no backing reference.
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Compares (offset, len). |
Hash |
Hashes (offset, len). |
PartialOrd / Ord |
Ordered by offset, then len. |
From<[u8; 16]> for BStackRange / From<BStackRange> for [u8; 16] |
Serialises/deserialises [offset_le8 ‖ len_le8] for on-disk storage. |
BStackOwnedSlice<'a, A> — ownership handle carrying &'a A. Non-Copy, non-Clone.
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Compares (offset, len). The allocator reference is not compared. |
Hash |
Hashes (offset, len). |
PartialOrd / Ord |
Ordered by offset, then len. |
BStackSlice<'a> — borrowed I/O view carrying &'a BStack. Non-Copy, Clone.
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Compares (offset, len). The stack reference is not compared. |
Hash |
Hashes (offset, len). |
PartialOrd / Ord |
Ordered by offset, then len. |
| Trait | Semantics |
|---|---|
PartialEq / Eq |
Equal when the underlying slice (offset + len) and cursor position both match. |
Hash |
Hashes (slice, cursor). |
PartialOrd / Ord |
Ordered by absolute payload position (slice.start() + cursor), then slice.len(). |
Reader and writer are also cross-comparable: PartialEq and PartialOrd are defined between
BStackSliceReader and BStackSliceWriter using the same (abs_pos, len) key, so the two cursor
types can be mixed in sorted collections. Both also implement PartialEq<BStackSlice> (cursor
position is ignored for that comparison).
Enables compound read-modify-write operations that hold the write lock across what would otherwise be separate calls, providing thread-level atomicity and crash-safe ordering.
[dependencies]
bstack = { version = "0.4", features = ["atomic"] }
# Combined set + atomic unlocks swap, swap_into, and cas:
bstack = { version = "0.4", features = ["set", "atomic"] }atrunc,splice,splice_into— atomic discard+push / pop+push tail replacement.try_extend,try_discard,try_extend_zeros— size-checked, optimistic append/discard.replace(n, f)— popnbytes, pass tof, push back the returned tail.get_batched,get_batched_into,get_batched_gen— read multiple (possibly dependent) ranges under one read lock.swap,swap_into,cas(requiresset) — atomic read-modify-write / compare-and-swap of a single region.process,process_gen(requiresset) — in-place mutation, or a dependent read/write sequence ending in at most oneWrite/Swap/Push/Pop/Discard/Atrunc/Splice.set_batched,inplace_gen(requiresset) — commit several non-overlapping in-place writes as one crash-atomic unit (a batch, or a generator that also reads the batch-so-far state).cross_exchange,copy(requiresset) — swap or copy two regions under one write lock.eq_crds,ne_crds,masked_eq_crds,masked_ne_crds(requiresset) — cross-region compare-and-swap, with==/!=/masked variants.
See API for full signatures and details.
Enables BStack::set(offset, data) (in-place overwrite), BStack::zero(offset, n) (zero-fill in place), and BStack::repeat(offset, pattern, count) (fill with a repeated pattern — the general form of zero). None changes the file size or the committed-length header.
[dependencies]
bstack = { version = "0.4", features = ["set"] }Enables the region-management layer on top of BStack: BStackAllocator, BStackBulkAllocator, BStackUninitAllocator, BStackOwnedSliceAllocator, BStackAllocError, BStackBulkAllocError, BStackRange, BStackOwnedSlice, BStackSlice, BStackSliceReader, LinearBStackAllocator, and GhostTreeBstackAllocator. Combined with set, also enables BStackSliceWriter, FirstFitBStackAllocator, SlabBStackAllocator, CheckedSlabBStackAllocator, BStackByteVec, and BStackByteVecIter.
[dependencies]
bstack = { version = "0.4", features = ["alloc"] }
# In-place slice writes (BStackSliceWriter) also need `set`:
bstack = { version = "0.4", features = ["alloc", "set"] }A fixed 32-byte header precedes the payload:
bytes field
───── ─────
0 .. 8 magic[8]
8 .. 16 clen — committed payload length (u64 LE)
16 .. 24 wip_ptr — write-in-progress journal target (u64 LE)
24 .. 32 wip_aux — write-in-progress journal mode (u64 LE)
32 .. payload — push 0, push 1, … concatenated
magic— 8 bytes:BSTK+ major(1 B) + minor(1 B) + patch(1 B) + reserved(1 B). This version writesBSTK\x00\x04\x00\x00(0.4.0).openaccepts any 0.4.x file (first 6 bytesBSTK\x00\x04) and rejects a different major or minor as incompatible. Legacy0.1.xfiles can be upgraded in place withBStack::migrate.clen— little-endianu64recording the last successfully committed payload length. Updated on everypushandpopbefore the durable sync.wip_ptr/wip_aux— the write-in-progress journal that makes in-place mutations crash-atomic.wip_ptris the physical offset a crashed in-place write is replayed into (0when idle);wip_auxnames the mode (verbatim replay, repeated pattern, a disjoint copy replayed from its still-intact source, or a length-changing tail replace whose new committed length recovery derives from the file size and direction). A batch of several in-place writes (set_batched/inplace_gen) uses theMultiWritesentinel inwip_auxwithwip_ptrleft0, staging the writes as[s | e | data]blocks pastclen. Interpreted by recovery onopen.
All user-visible offsets (returned by push, accepted by peek/get) are
logical — 0-based from the start of the payload region (file byte 32).
In-place same-length writes — set, zero, repeat, swap, swap_into,
cas, copy, cross_exchange, process, set_batched, inplace_gen, and the
crds family — never change the payload length and are each crash-atomic,
committing by one of three strategies (recovered on the next open):
- Aligned-block write — a target within one power-fail-atomic block is
committed by a single
write+ sync; no journal is armed. - Write-in-progress journal — otherwise: stage a backup past
clen→ sync → armwip_ptr→ sync → write in place → sync → clearwip_ptr→ sync →ftruncate.zero/repeatstage only[count | pattern];cross_exchangestages one region and commits at a single atomicwip_ptrflip; moves and fills stream through a bounded buffer. - Multi-write journal —
set_batchedandinplace_gencommit several non-overlapping in-place writes as one unit: stage every[s | e | data]block pastclen→ sync → arm theMultiWritesentinel (wip_ptrstays0, so it never collides with a single-region journal) → sync → replay each block in place → sync → disarm →ftruncate. A batch of one write falls back to the single-write strategies above.
Below, commit denotes whichever strategy applies to the bytes written; anything before it is read/compare/callback work under the lock.
| Operation | Sequence |
|---|---|
push |
lseek(END) → write(data) → lseek(8) → write(clen) → sync |
extend |
lseek(END) → set_len(new_end) → lseek(8) → write(clen) → sync |
pop, pop_into |
lseek → read → ftruncate → lseek(8) → write(clen) → sync |
discard |
ftruncate → lseek(8) → write(clen) → sync |
set (feature) |
commit data |
zero, repeat (feature) |
commit the repeated pattern (the journal stages only [count | pattern]) |
atrunc (atomic) |
dispatch on shape: truncation → ftruncate → commit clen; append → set_len(new_end) → write(buf) → sync → commit clen; same-length → commit buf in place; length change → splice journal (stage new tail past the payload → arm SpliceGrow/SpliceShrink → replay into place → atomically commit clen' + disarm → truncate, sync at each barrier) |
splice, splice_into (atomic) |
lseek(tail) → read(n) → (then as atrunc) |
try_extend (atomic) |
size check → conditional push sequence |
try_discard (atomic) |
size check → conditional discard sequence |
try_extend_zeros (atomic) |
size check → conditional extend(n) sequence |
swap, swap_into (set+atomic) |
read old bytes → commit buf |
cas (set+atomic) |
read → compare → conditional commit of new |
process (set+atomic) |
read(start..end) → (callback) → commit the buffer |
process_gen (set+atomic) |
closure-driven reads (and Len queries), ending in at most one mutating step — Write commits; Swap uses the exchange journal (as cross_exchange); Push/Pop/Discard/Atrunc/Splice behave as their standalone forms |
set_batched (set+atomic) |
validate + reject overlap → multi-write journal: stage every [s | e | data] block past clen → arm the MultiWrite sentinel (wip_ptr stays 0) → replay each block in place → disarm → ftruncate (sync at each barrier); a lone effective write takes the ordinary single-write commit |
inplace_gen (set+atomic) |
closure-driven reads (each overlaid with the batch-so-far edits) interleaved with accumulated Writes (later overrides earlier on overlap); on None the pending edits commit together via the multi-write journal (as set_batched) |
replace (atomic) |
lseek(tail) → read(n) → (callback) → (then as atrunc) |
cross_exchange (set+atomic) |
read(a), read(b) → exchange journal: stage a → arm at a → write b→a → flip wip_ptr to b → write a→b → disarm → ftruncate (sync at each barrier) |
copy (set+atomic) |
same-location → no-op; single-block dest → commit; overlapping → stream source→tail→dest (Set journal); disjoint → copy journal (stage only [src | n], arm Copy, stream source→dest; recovery replays from the untouched source) |
eq_crds, ne_crds (set+atomic) |
read(a) → compare → conditional commit of b_buf |
masked_eq_crds, masked_ne_crds (set+atomic) |
read(a) → mask+compare → conditional commit of b_buf |
peek, peek_into, get, get_into, get_batched, get_batched_into, get_batched_gen |
pread(2) on Unix; ReadFile+OVERLAPPED on Windows; lseek → read elsewhere |
durable_sync on macOS issues fcntl(F_FULLFSYNC). Unlike fdatasync,
this flushes the drive controller's write cache, providing the same "barrier
to stable media" guarantee that fsync gives on Linux. Falls back to
sync_data if the device does not support F_FULLFSYNC.
durable_sync on Linux / other Unix calls sync_data (fdatasync).
durable_sync on Windows calls sync_data, which maps to
FlushFileBuffers. This flushes the kernel write-back cache and waits for
the drive to acknowledge, providing equivalent durability to fdatasync.
Push rollback: if the write or sync fails, a best-effort ftruncate and
header reset restore the pre-push state.
On the next open, recovery first checks the write-in-progress journal
(wip_ptr); if disarmed, it reconciles the committed length against the file
size:
| Condition | Cause | Recovery |
|---|---|---|
wip_ptr != 0, wip_aux = Set |
in-place set/swap/cas/copy/cross_exchange crashed mid-commit |
replay the staged tail into [wip_ptr, …), disarm, truncate |
wip_ptr != 0, wip_aux = Repeat |
zero/repeat crashed mid-fill |
write count copies of the staged pattern, disarm, truncate |
wip_ptr != 0, wip_aux = Copy |
disjoint copy crashed mid-copy |
replay move_chunked(src → wip_ptr) from the untouched source (tail stages only [src | n]), disarm, truncate |
wip_ptr != 0, wip_aux = SpliceGrow/SpliceShrink |
length-changing atrunc/splice/splice_into/replace crashed mid-replace |
derive clen' from the file size and direction, replay the staged new tail, commit clen' while disarming, truncate |
wip_ptr != 0, wip_aux unrecognized |
mode armed by a newer build | roll back: disarm, truncate to 32 + clen |
wip_ptr == 0, wip_aux = MultiWrite |
set_batched/inplace_gen batch crashed after all blocks were staged |
replay each staged [s | e | data] block into [s, e), disarm, truncate (a corrupt tail rolls back, applying nothing) |
wip_ptr == 0, file_size − 32 > clen |
partial tail write (crashed before header update) | truncate to 32 + clen, durable-sync |
wip_ptr == 0, file_size − 32 < clen |
partial truncation (crashed before header update) | set clen = file_size − 32, durable-sync |
Each replay is idempotent (the staged tail is immutable), so a crash during recovery is safe to re-run. No caller action is required; recovery is transparent.
On Unix, open calls flock(LOCK_EX | LOCK_NB) on the file. If another
process already holds the lock, open returns immediately with
io::ErrorKind::WouldBlock. The lock is released when the BStack is
dropped.
On Windows, open calls LockFileEx with
LOCKFILE_EXCLUSIVE_LOCK | LOCKFILE_FAIL_IMMEDIATELY covering the entire
file range. The same WouldBlock semantics apply (ERROR_LOCK_VIOLATION
maps to io::ErrorKind::WouldBlock in Rust). The lock is released when the
BStack is dropped.
Both
flock(Unix) andLockFileEx(Windows) are advisory and per-process. They protect against concurrentBStack::opencalls across well-behaved processes, not against raw file access.
BStack wraps the file in a RwLock. The committed payload length is also
cached in memory and kept in sync with the on-disk header by every
write-lock-held operation, so len/is_empty can be answered under the read
lock without any File::metadata syscall.
| Operation | Lock (Unix / Windows) | Lock (other) |
|---|---|---|
push, extend, pop, pop_into, discard |
write | write |
set, zero, repeat (feature) |
write | write |
atrunc, splice, splice_into, try_extend (atomic) |
write | write |
try_discard(s, n > 0) (atomic) |
write | write |
try_discard(s, 0) (atomic) |
read | read |
try_extend_zeros (atomic) |
write | write |
swap, swap_into, cas (set+atomic) |
write | write |
process, process_gen (set+atomic) |
write | write |
set_batched, inplace_gen (set+atomic) |
write | write |
replace (atomic) |
write | write |
cross_exchange, copy (set+atomic) |
write | write |
eq_crds, ne_crds (set+atomic) |
write | write |
masked_eq_crds, masked_ne_crds (set+atomic) |
write | write |
peek, peek_into, get, get_into |
read | write |
get_batched, get_batched_into, get_batched_gen (atomic) |
read | write |
len |
read | read |
On Unix and Windows, peek, peek_into, get, and get_into use a
cursor-safe positional read (pread(2) / read_exact_at on Unix; ReadFile
with OVERLAPPED via seek_read on Windows) that does not modify the shared
file-position cursor. Multiple concurrent calls to any of these methods can
therefore run in parallel. Any in-progress push, pop, or pop_into still
blocks all readers via the write lock, so readers always observe a consistent,
committed state.
On other platforms a seek is required; peek, peek_into, get, and
get_into fall back to the write lock and reads serialise.
Unlike get_batched_gen, which only ever takes the read lock, process_gen
and inplace_gen always take the write lock — even for sequences that turn
out to be read-only and end in None — because the closure may decide, only
after seeing earlier reads, to mutate; the lock must therefore be acquired before
the first read so the whole sequence runs as one indivisible step.
- No record framing. The file stores raw bytes; the caller must track how many bytes each logical record occupies.
- Push rollback is best-effort. A failure during rollback is silently
swallowed; crash recovery on the next
openwill repair the state. - No
O_DIRECT. Writes go through the page cache; durability relies ondurable_sync, not cache bypass. - Single file only. There is no WAL, manifest, or secondary index.
- Multi-process lock is advisory.
flock(Unix) andLockFileEx(Windows) protect well-behaved processes but not raw file access.
The fault-injection feature lets tests make BStack I/O fail on demand, to exercise error-handling and rollback paths that a successful push/pop/realloc/dealloc sequence can never reach. Implement FaultPolicy — fn next_fault(&self, op: &'static str, seq: u64) -> Option<io::Error> — and arm it with BStack::with_fault_policy(policy) (at construction) or set_fault_policy(Some(policy)) / fault_policy() (arm, re-arm, or disarm mid-test). Every I/O method then consults the policy once, after validating its arguments, so validation errors always take precedence over an injected fault; under atomic, concurrent operations share one per-stack sequence counter for reproducible, seedable schedules.
The whole mechanism is gated on all(debug_assertions, feature = "fault-injection"): it is off by default, and a --release build carries none of it — no struct field, no per-call branch — so release performance is unaffected.
The alloc feature adds typed region management over a BStack payload.
The alloc feature provides three distinct handle types for different roles:
| Type | Carries | Copy | I/O | Alloc ops |
|---|---|---|---|---|
BStackRange |
nothing | yes | no | no |
BStackOwnedSlice<'a, A> |
&'a A |
no | via view | yes |
BStackSlice<'a> |
&'a BStack |
no | yes | no |
BStackOwnedSlice is non-Copy and non-Clone: an allocation has exactly one owner. Obtaining an I/O view via as_slice() or as_slice_mut() ties the view's lifetime to the borrow of the owned slice, preventing it from outliving the handle. BStackSlice is non-Copy so that write*(&mut self) provides single-writer exclusivity; it is Clone for explicit second views.
A trait for types that own a BStack and manage contiguous byte regions
within its payload. Implementors must provide:
pub trait BStackAllocator: Sized {
type Error: fmt::Debug + fmt::Display;
// All built-in allocators set Allocated<'a> = BStackOwnedSlice<'a, Self>.
// Custom allocators may use a richer type that implements Into<BStackOwnedSlice<'a, Self>>.
type Allocated<'a>: Into<BStackOwnedSlice<'a, Self>> where Self: 'a;
fn stack(&self) -> &BStack;
fn into_stack(self) -> BStack;
fn alloc(&self, len: u64) -> Result<Self::Allocated<'_>, Self::Error>;
// On failure, realloc/dealloc return a BStackAllocError carrying the
// surviving allocation (see below), so a failed operation never leaks it.
fn realloc<'a>(&'a self, handle: Self::Allocated<'a>, new_len: u64)
-> Result<Self::Allocated<'a>, BStackAllocError<'a, Self>>;
// Default no-op; override for free-list allocators:
fn dealloc<'a>(&'a self, handle: Self::Allocated<'a>)
-> Result<(), BStackAllocError<'a, Self>> { Ok(()) }
// Delegation helpers:
fn len(&self) -> io::Result<u64>;
fn is_empty(&self) -> io::Result<bool>;
}realloc and dealloc consume the handle by value, but a failed resize or free
almost always leaves a valid allocation behind — either the original region is
untouched, or a new region is fully committed. Because BStackOwnedSlice's
Drop is a no-op, dropping that handle on the error path would silently leak the
region, so both methods return it instead:
pub struct BStackAllocError<'a, A: BStackAllocator> {
pub source: A::Error,
/// `Some` — the region survived and is owned by the caller again
/// (implementations return this whenever possible); `None` — the allocation
/// was genuinely lost mid-operation (recoverable only via crash recovery).
pub handle: Option<A::Allocated<'a>>,
}It implements Display (delegating to source) and std::error::Error, so ?
works within functions that return it. Converting out to a bare Self::Error
discards the recovered handle, so that step is deliberately explicit — decide
whether to retry, fall back, or free the allocation first:
// Give up and surface just the error (drops the recovered handle):
let resized = alloc.realloc(handle, new_len).map_err(|e| e.source)?;
// Or recover the handle and retry / fall back:
let resized = match alloc.realloc(handle, new_len) {
Ok(h) => h,
Err(e) => {
let original = e.handle.expect("region survived the failed realloc");
// ... retry with a different size, or alloc.dealloc(original), etc.
}
};A convenience bound for the common case of a BStackAllocator whose handle type is BStackOwnedSlice and whose error type is io::Error:
// Compact form:
A: BStackOwnedSliceAllocator
// Equivalent verbose form:
A: 'static + BStackAllocator<Error = io::Error>,
for<'a> A: BStackAllocator<Allocated<'a> = BStackOwnedSlice<'a, A>>,All built-in allocators implement BStackOwnedSliceAllocator.
An extension trait for BStackAllocator that adds two atomic bulk methods. "Atomic" means either the operation succeeds completely or the backing store is left entirely unchanged.
pub trait BStackBulkAllocator: BStackAllocator {
fn alloc_bulk(&self, lengths: impl AsRef<[u64]>)
-> Result<Vec<Self::Allocated<'_>>, Self::Error>;
// On failure, returns the handles it did not free (every handle, for an
// atomic implementation) rather than leaking them.
fn dealloc_bulk<'a>(&'a self, handles: impl IntoIterator<Item = Self::Allocated<'a>>)
-> Result<(), BStackBulkAllocError<'a, Self>>;
}BStackBulkAllocError<'a, A> is the bulk analogue of BStackAllocError: it
carries source: A::Error plus handles: Vec<A::Allocated<'a>>, the handles
still owned by the caller after a failed bulk free.
An opt-in extension trait for BStackAllocator that adds uninitialised
variants of alloc and realloc. alloc guarantees a zero-initialised region
and realloc zero-fills newly added bytes when growing; that guarantee costs a
write, since a region pulled from a free list may hold leftover bytes that the
allocator must scrub first. Callers that immediately overwrite the whole region
(for example, write-ing a serialized record right after alloc) have no use
for the zero-fill. These methods let them skip it.
pub trait BStackUninitAllocator: BStackAllocator {
fn alloc_uninit(&self, len: u64) -> Result<Self::Allocated<'_>, Self::Error>;
fn realloc_uninit<'a>(&'a self, handle: Self::Allocated<'a>, new_len: u64)
-> Result<Self::Allocated<'a>, BStackAllocError<'a, Self>>;
}The bytes in a region returned by alloc_uninit, or in the newly added portion
of one returned by realloc_uninit, are unspecified: they may be zero, or
may be leftover bytes from a previous allocation that occupied the same on-disk
space. They are always valid to read — no undefined behavior, unlike
MaybeUninit<u8> in memory — but callers must not rely on their value until
they have written to the region themselves. Existing bytes are preserved exactly
as realloc; only newly added bytes are left unspecified.
Implementing the trait is optional and signals that the allocator actually has a
cheaper uninitialised path. The savings are concentrated in the free-list-reuse
path, where a previously-occupied block is handed back without being scrubbed.
Allocators for which zero-fill is already free — an always-extend bump allocator
(the freshly extended tail is already zero via set_len on a sparse file), or
one that scrubs blocks eagerly on free — gain nothing and may either implement
the trait as a thin wrapper around alloc/realloc or not implement it at all.
The ownership handle for one allocation. Returned by alloc, consumed by realloc and dealloc. Non-Copy, non-Clone — exactly one owner per region.
Key methods on BStackOwnedSlice:
| Method | Description |
|---|---|
start() / end() / len() |
Coordinate accessors |
as_slice<'s>(&'s self) -> BStackSlice<'s> |
Shared read view (lifetime tied to &self) |
as_slice_mut<'s>(&'s mut self) -> BStackSlice<'s> |
Exclusive write view |
to_range() |
Convert to a BStackRange for serialisation |
A borrowed I/O view carrying &'a BStack directly. Obtained from BStackOwnedSlice::as_slice[_mut]() or constructed via unsafe { BStackSlice::from_raw_parts(stack, offset, len) }.
Key methods on BStackSlice:
| Method | Description |
|---|---|
read() |
Read the entire region into a new Vec<u8> |
read_into(buf) |
Read into a caller-supplied buffer |
read_range(start, end) |
Read a sub-range |
subslice(start, end) |
Narrow to a sub-range |
reader() / reader_at(offset) |
Cursor-based BStackSliceReader |
write(data) (feature set) |
Overwrite the beginning of the region |
write_range(start, data) (feature set) |
Overwrite a sub-range |
zero() / zero_range(start, n) (feature set) |
Zero the region or a sub-range |
A raw (offset, len) coordinate pair with no backing reference. Copy, serializable to/from [u8; 16] via to_bytes()/from_bytes(). Used for on-disk token storage.
A cursor-based reader over a BStackSlice. Implements io::Read and io::Seek.
BStackOwnedSlice<'a, A> borrows the allocator for 'a. Views obtained via as_slice[_mut]() have a shorter lifetime tied to the borrow of the owned slice, preventing them from outliving the handle that owns the region.
use bstack::{BStack, BStackAllocator, LinearBStackAllocator};
let alloc = LinearBStackAllocator::new(BStack::open("data.bstack")?);
let mut slice = alloc.alloc(128)?; // reserve 128 zero bytes
let data = slice.as_slice().read()?; // read them back
// dealloc returns the handle inside its error on failure; `.map_err(|e| e.source)`
// surfaces just the io::Error.
alloc.dealloc(slice).map_err(|e| e.source)?; // release (tail → O(1) discard)
let stack = alloc.into_stack(); // reclaim the BStackFor detailed on-disk layouts, allocation policies, coalescing rules, crash consistency guarantees, and thread safety analysis for each allocator, see algos/ALLOCATOR.md.
Bump allocator — regions appended sequentially to the tail. dealloc on a
non-tail slice is a no-op; realloc returns Unsupported for non-tail slices.
Without atomic: Send only. With atomic: Send + Sync via
try_extend/try_discard.
Doubly-linked intrusive free list with first-fit placement and immediate
coalescing. On-disk header flags trigger a linear recovery scan on the next
open after an unclean shutdown. Without atomic: Send only. With
atomic: Send + Sync via an internal Mutex serialising free-list mutations.
AVL tree keyed on (size, address); best-fit; zero per-allocation overhead.
Also implements BStackBulkAllocator. Without atomic: Send only. With
atomic: Send + Sync via an internal Mutex.
Fixed block_size slab; singly-linked free list; zero per-block overhead.
Constructors: new(stack, block_size) for a fresh stack, open(stack) to
reattach. Without atomic: Send only. With atomic: Send + Sync with
no allocator-level lock (uses BStack::process_gen / cross_exchange).
Like SlabBStackAllocator but each block has an 8-byte overhead tag that
catches double-frees immediately and allows full recovery after a crash.
Constructor takes data_size (usable bytes per block; physical = data_size + 8).
open runs recover() automatically. Without atomic: Send only. With
atomic: Send + Sync (same lock-free strategy as SlabBStackAllocator).
A growable byte (u8) vector backed by a BStack allocation, mirroring the core Vec<u8> API.
A general typed vector over arbitrary Copy types requires a sound POD/byte-castable bound and is planned for a future release.
[dependencies]
bstack = { version = "0.4", features = ["alloc", "set"] }┌──────────────────────┬──────────────────────┬────────────────────────┐
│ len (8 B, LE u64) │ cap (8 B, LE u64) │ elements: [u8; cap] │
└──────────────────────┴──────────────────────┴────────────────────────┘
byte 0 byte 8 byte 16
The header is re-read from disk on every call, so the (len, cap) metadata is
recoverable after a crash by reconstructing the handle from the raw block via
BStackByteVec::from_raw_block.
- Growth:
pushreallocates tomax(cap × 2, 4)bytes whenlen == cap. New space is zero-initialised byBStack::extend. - Readback helper:
read_bytesloads all logical bytes into a RustVec<u8>. - Zeroing on removal:
popdecrementslenbefore zeroing the vacated slot;truncatewrites the newlenbefore zeroing removed slots in a singleBStackSlice::zero_rangecall. Deallocation zeroing is delegated to the allocator. - Iterator:
BStackByteVecIterborrows the vec immutably for its lifetime (preventing concurrent mutation) and yieldsio::Result<u8>per byte, reading from disk on demand.
use bstack::{BStack, BStackByteVec, LinearBStackAllocator};
let alloc = LinearBStackAllocator::new(BStack::open("buf.bstack")?);
let mut v: BStackByteVec<_> = BStackByteVec::new(&alloc)?;
v.push(b'A')?;
v.push(b'B')?;
v.push(b'C')?;
assert_eq!(v.len()?, 3);
assert_eq!(v.get(1)?, Some(b'B'));
assert_eq!(v.pop()?, Some(b'C'));
let all = v.read_bytes()?;
println!("{}", String::from_utf8_lossy(&all));
alloc.dealloc(v.into_raw_block()).map_err(|e| e.source)?;