Phase 2: .theos v2 binary format

.github/workflows/ci.yml

@@ -11,23 +11,38 @@ jobs:
       fail-fast: false
       matrix:
         compiler:
-          - { cc: gcc, cxx: g++, flags: "" }
-          # libstdc++ 13 hides std::expected from Clang 18, so Clang builds against libc++
-          - { cc: clang, cxx: clang++, flags: "-stdlib=libc++" }
+          - { cc: gcc, cxx: g++ }
+          # Clang 19 is the first Clang whose concepts level unlocks std::expected in libstdc++ 13
+          - { cc: clang, cxx: clang++-19 }
     name: linux-${{ matrix.compiler.cc }}
     runs-on: ubuntu-24.04
     steps:
       - uses: actions/checkout@v4
-      - name: Install libc++
+      - name: Install Clang 19
         if: matrix.compiler.cc == 'clang'
-        run: sudo apt-get update && sudo apt-get install -y libc++-18-dev libc++abi-18-dev
+        run: sudo apt-get update && sudo apt-get install -y clang-19
       - name: Configure
-        run: cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=${{ matrix.compiler.cxx }} "-DCMAKE_CXX_FLAGS=${{ matrix.compiler.flags }}"
+        run: cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=${{ matrix.compiler.cxx }}
       - name: Build
         run: cmake --build build
       - name: Test
         run: ctest --test-dir build --output-on-failure
 
+  fuzz:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install Clang 19
+        run: sudo apt-get update && sudo apt-get install -y clang-19 libclang-rt-19-dev
+      - name: Build fuzzers
+        run: |
+          cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CXX_COMPILER=clang++-19 -DTHEOS_FUZZ=ON -DTHEOS_BUILD_TESTS=OFF
+          cmake --build build
+      - name: Fuzz the binary decoder (60 s)
+        run: ./build/fuzz/decode_fuzzer -max_total_time=60 -max_len=8192 fuzz/corpus/decode
+      - name: Fuzz the parser (60 s)
+        run: ./build/fuzz/parse_fuzzer -max_total_time=60 -max_len=8192 fuzz/corpus/parse
+
   windows-mingw:
     runs-on: windows-latest
     defaults:

.gitignore

@@ -48,3 +48,11 @@ Thumbs.db
 Desktop.ini
 $RECYCLE.BIN/
 /build*/
+!/tests/golden/**/*.theos
+!/fuzz/corpus/**/*.theos
+
+# libFuzzer artifacts
+crash-*
+leak-*
+timeout-*
+oom-*

CLAUDE.md

@@ -7,8 +7,9 @@
 - **Run Binary:** `./build/theosc`
 
 ## Coding Standards & C++20 Style Guide
-- **Language Standard:** Strict C++23 (`std::expected`, `std::span`, `std::unique_ptr`, `constexpr`, concepts). GCC 13+ / Clang 18+ / MinGW-w64 GCC 13+.
+- **Language Standard:** Strict C++23 (`std::expected`, `std::span`, `std::unique_ptr`, `constexpr`, concepts). GCC 13+ / Clang 19+ / MinGW-w64 GCC 13+ (Clang 18 cannot see `std::expected` in libstdc++ 13).
 - **Memory Safety:** Avoid raw `new`/`delete`. Use smart pointers and deterministic resource acquisition (RAII).
-- **Binary Packing:** All disk structs must use `#pragma pack(push, 1)` to prevent compiler padding misalignment when reading/writing raw binary streams.
+- **Binary Format:** `.theos` is read and written field by field with explicit little-endian encoding and bounds checks (`src/core/binary.cpp`); never `reinterpret_cast` file bytes into structs. Spec: `docs/spec/theos-v2.md`.
+- **Golden Files:** `THEOS_UPDATE_GOLDEN=1 ./build/tests/theos_tests "[golden]"` regenerates `tests/golden/**` outputs; review the diff before committing.
 - **Error Handling:** Use `std::expected` or exceptions with explicit error categories for file I/O and syntax parsing failures. Never leave parsing streams unchecked.
 - **Const Correctness:** Make member functions `const` wherever state is unmodified.

CMakeLists.txt

@@ -16,8 +16,16 @@ if(MINGW)
     add_link_options(-static -static-libstdc++ -static-libgcc)
 endif()
 
+# Fuzz builds instrument everything so the fuzzers see coverage inside the library
+option(THEOS_FUZZ "Build libFuzzer targets (Clang only)" OFF)
+if(THEOS_FUZZ)
+    add_compile_options(-fsanitize=fuzzer-no-link,address,undefined -fno-sanitize-recover=all)
+    add_link_options(-fsanitize=address,undefined)
+endif()
+
 add_library(theos_core STATIC
     src/core/ast.cpp
+    src/core/binary.cpp
     src/core/canon.cpp
     src/core/diagnostic.cpp
     src/core/driver.cpp
@@ -33,6 +41,10 @@ target_link_libraries(theos_core PRIVATE theos_warnings)
 add_executable(theosc src/cli/main.cpp)
 target_link_libraries(theosc PRIVATE theos_core theos_warnings)
 
+if(THEOS_FUZZ)
+    add_subdirectory(fuzz)
+endif()
+
 option(THEOS_BUILD_TESTS "Build the test suite" ON)
 if(THEOS_BUILD_TESTS)
     enable_testing()

README.md

@@ -15,7 +15,8 @@ Theos bridges this gap. It provides a deterministic pipeline to turn human-reada
 ## Pipeline
 
 ```
-.theo source ─► Lexer ─► Parser ─► AST ─► Validator ─► Mermaid exporter
+.theo source ─► Lexer ─► Parser ─► AST ─► Validator ─┬─► Mermaid exporter
+                                                      └─► .theos binary (encode / decode)
 ```
 
 Each claim in a `.theo` file carries natural-language text and a formula:
@@ -45,11 +46,27 @@ The full language is specified in [docs/spec/language.md](docs/spec/language.md)
 
 ---
 
+## The `.theos` Binary Format
+
+`theosc build` compiles a checked argument into a `.theos` file, specified in
+[docs/spec/theos-v2.md](docs/spec/theos-v2.md):
+
+- PNG-style signature (`89 54 48 45 4F 0D 0A 1A`) that detects text-mode corruption
+- Tagged, CRC-32-protected chunks; unknown ancillary chunks are skipped, unknown critical chunks rejected
+- Fixed-width little-endian records referencing each other by index, for O(1) access
+- One deduplicated string table and one expression table with shared sub-formulas
+- A reader that bounds-checks every offset and index, fuzzed with libFuzzer in CI
+
+---
+
 ## CLI Usage
 
 ```sh
-theosc check  examples/aquinas_first_way.theo
-theosc export examples/resurrection.theo --format mermaid -o resurrection.mmd
+theosc check   examples/aquinas_first_way.theo
+theosc export  examples/resurrection.theo --format mermaid -o resurrection.mmd
+theosc build   examples/resurrection.theo -o resurrection.theos
+theosc inspect resurrection.theos            # decoded argument
+theosc inspect resurrection.theos --chunks   # chunk table and hex view
 ```
 
 Diagnostics go to stderr as `file:line:col: error: message` with the source line underneath;
@@ -62,18 +79,19 @@ exit code 0 = success, 1 = invalid input, 2 = usage error.
 ```text
 theos-engine/
 ├── include/theos/   # Public headers
-├── src/core/        # Core library (lexer, parser, validator, exporter)
+├── src/core/        # Core library (lexer, parser, validator, exporter, binary format)
 ├── src/cli/         # theosc command-line tool
-├── docs/spec/       # Language specification
+├── docs/spec/       # Language and binary format specifications
 ├── examples/        # Sample .theo arguments
+├── fuzz/            # libFuzzer targets and seed corpora
 └── tests/           # Catch2 unit tests and golden files
 ```
 
 ---
 
 ## Building from Source
 
-Requires CMake 3.20+ and a C++23 compiler (GCC 13+, Clang 18+, or MinGW-w64 GCC 13+ via MSYS2).
+Requires CMake 3.20+ and a C++23 compiler (GCC 13+, Clang 19+, or MinGW-w64 GCC 13+ via MSYS2).
 Tests use Catch2, fetched automatically by CMake.
 
 ```sh
@@ -83,6 +101,9 @@ ctest --test-dir build --output-on-failure
 ./build/theosc
 ```
 
+Fuzzing (Clang 19+): `cmake -B build-fuzz -DCMAKE_CXX_COMPILER=clang++-19 -DTHEOS_FUZZ=ON -DTHEOS_BUILD_TESTS=OFF`,
+then `./build-fuzz/fuzz/decode_fuzzer fuzz/corpus/decode`.
+
 On Windows, run the same commands from an MSYS2 UCRT64 shell (add `-G "MinGW Makefiles"` or `-G Ninja` if needed).
 
 See [ROADMAP.md](ROADMAP.md) for where the project is going. Licensed under the [MIT License](LICENSE).

ROADMAP.md

@@ -259,4 +259,5 @@ Answers to the original open questions (2026-10-24) and decisions made while bui
 | Phase | Status |
 |-------|--------|
 | 0 Foundation | done |
-| 1 Language v2 | done — formulas are fully parsed already (needed to find where they end); the v1 binary format was removed here rather than in M2.2 because it was tied to the old AST; syntax errors stop at the first error, static-rule errors are all reported; Clang builds use libc++ because libstdc++ 13 hides `std::expected` from Clang 18 |
+| 1 Language v2 | done — formulas are fully parsed already (needed to find where they end); the v1 binary format was removed here rather than in M2.2 because it was tied to the old AST; syntax errors stop at the first error, static-rule errors are all reported; Clang CI uses Clang 19, the first Clang that can use `std::expected` from libstdc++ 13 |
+| 2 Binary v2 | done — formulas, proofs and objections are serialized here (M3.4 folded in, since the AST already carries them); padding must be zero so every single-byte corruption is detected (verified by flipping every byte); nesting depth is capped in parser (256) and decoder (512) |