ci: compile corpus modules serially under hard memory limits

.agents/skills/corpus-compile-workflow/SKILL.md

@@ -0,0 +1,34 @@
+---
+name: corpus-compile-workflow
+description: Generate corpus module dependency closures and verify all seven destination languages under fixed systemd cgroup limits.
+---
+
+Use scripts/check_module_codegen.py with -m MODULE, optional repeated -l LANGUAGE,
+--bin PATH, and a new --output directory. Defaults: 3500 MiB, swap disabled,
+600 seconds per command, serial compilation. Never bypass the cap after failure.
+Local runs require a working user systemd manager; CI uses --system and sudo.
+CI checks out PolyXML inside the corpus, so pass --core PolyXML/crates/polyxml-core.
+Otherwise the default is the sibling PolyXML checkout. The selected module and
+all dependencies are read from polyxml.toml; compile the full generated closure.
+
+Run python3 -m unittest discover -s tests -v to verify limits, error retention,
+and descendant cleanup on timeout. User-manager tests skip when unavailable;
+CI's real generation/compilation jobs enforce the system-manager path.
+Retain logs, results.json, metadata.json, and file counts when publishing evidence.
+Record exact compiler source commit and tool versions with local results.
+Do not treat syntax compilation as runtime XML conformance or skipped targets
+as full success. C# net8.0 build tests need the SDK/reference pack but no runtime
+execution. Missing toolchains are fatal unless --allow-missing is explicit.
+
+Rust chunking is opt-in (`split_units = true`, `chunk_size = 250` in the
+generated manifest), not automatic. Inspect emitted file counts and preserve
+this setting in regression tests; a monolithic run does not test the split
+strategy. Both strategies still share one cargo check crate.
+
+UCI's complete Rust codec/Serde check needs ~11.2 GiB process RSS. Use the
+12000 MiB documented CI/local budget with at least 1 GiB additional available
+headroom. Other heavy modules retain the 3500 MiB default. Rust compile checks
+disable debug info and incremental compilation; they keep codecs and Serde.
+Use --keep-going to collect every target outcome serially; any target failure
+still produces a nonzero final exit and a retained failures.json. Scheduled
+CI uses this mode. Never label failed targets as skipped or passing.

.github/workflows/ci.yml

@@ -6,6 +6,8 @@ on:
   pull_request:
     branches: [ main ]
   workflow_dispatch:
+  schedule:
+    - cron: "0 6 * * 1"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,9 +42,119 @@ jobs:
 
       - name: Compile PolyXML CLI (Release)
         run: |
-          cargo build --release -p polyxml-cli --manifest-path PolyXML/Cargo.toml
+          sudo systemd-run --scope -p MemoryMax=3500M -p MemorySwapMax=0 \
+            sudo -u "$USER" env "PATH=$PATH" timeout --kill-after=10s 600 \
+            cargo build --release -j 1 -p polyxml-cli --manifest-path PolyXML/Cargo.toml
           echo "POLYXML_BIN=$GITHUB_WORKSPACE/PolyXML/target/release/polyxml" >> $GITHUB_ENV
 
       - name: Run Schema Validation & Module Graph Verification
         run: |
-          python3 runner.py --bin "$POLYXML_BIN"
+          sudo systemd-run --scope -p MemoryMax=12000M -p MemorySwapMax=0 \
+            sudo -u "$USER" env "PATH=$PATH" timeout --kill-after=10s 600 \
+            python3 -u runner.py --bin "$POLYXML_BIN"
+
+  compile-representative:
+    name: Compile ${{ matrix.module }} / Python
+    if: github.event_name == 'push' || github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    strategy:
+      fail-fast: false
+      matrix:
+        module: [hl7_cda, ubl_invoice]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/checkout@v4
+        with:
+          repository: polyxml/PolyXML
+          path: PolyXML
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: PolyXML
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Verify harness limits and cleanup
+        run: CORPUS_TEST_SYSTEM=1 python3 -m unittest discover -s tests -v
+      - name: Build CLI under a hard memory cap
+        run: >-
+          sudo systemd-run --scope -p MemoryMax=3500M -p MemorySwapMax=0
+          sudo -u "$USER" env "PATH=$PATH" timeout --kill-after=10s 600
+          cargo build -j 1 -p polyxml-cli --manifest-path PolyXML/Cargo.toml
+      - name: Generate and compile dependency closure
+        run: >-
+          python3 scripts/check_module_codegen.py --system
+          --bin PolyXML/target/debug/polyxml --core PolyXML/crates/polyxml-core
+          -m "${{ matrix.module }}" -l python --output compile-results
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: compile-${{ matrix.module }}-python
+          path: compile-results
+
+  compile-heavy:
+    name: Compile ${{ matrix.module }} / all seven targets
+    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    timeout-minutes: 180
+    strategy:
+      fail-fast: false
+      max-parallel: 1
+      matrix:
+        include:
+          - module: defense_uci
+            memory: 12000
+          - module: hl7_cda
+            memory: 3500
+          - module: ubl_invoice
+            memory: 3500
+          - module: transit_netex
+            memory: 3500
+          - module: finance_fpml
+            memory: 3500
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/checkout@v4
+        with:
+          repository: polyxml/PolyXML
+          path: PolyXML
+      - uses: dtolnay/rust-toolchain@stable
+      - uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: PolyXML
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - uses: actions/setup-go@v5
+        with:
+          go-version: "1.24.x"
+      - uses: actions/setup-node@v4
+        with:
+          node-version: "22"
+      - run: npm install --global typescript@5
+      - uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: "25"
+      - uses: actions/setup-dotnet@v4
+        with:
+          dotnet-version: "8.0.x"
+      - run: sudo apt-get update && sudo apt-get install -y g++ time
+      - name: Verify harness limits and cleanup
+        run: CORPUS_TEST_SYSTEM=1 python3 -m unittest discover -s tests -v
+      - name: Build CLI under a hard memory cap
+        run: >-
+          sudo systemd-run --scope -p MemoryMax=3500M -p MemorySwapMax=0
+          sudo -u "$USER" env "PATH=$PATH" timeout --kill-after=10s 600
+          cargo build -j 1 -p polyxml-cli --manifest-path PolyXML/Cargo.toml
+      - name: Generate and compile dependency closure serially
+        run: >-
+          python3 scripts/check_module_codegen.py --system
+          --bin PolyXML/target/debug/polyxml --core PolyXML/crates/polyxml-core
+          -m "${{ matrix.module }}" --keep-going --memory-mib "${{ matrix.memory }}" --output compile-results
+      - uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: compile-${{ matrix.module }}-all
+          path: compile-results

README.md

@@ -33,8 +33,9 @@ Across open-source XSD generator trackers, certain widely adopted industry schem
 - **Multi-Megabyte Monolithic Schemas**: Massive schema files such as NATO UCI (8.0 MB single file) and HL7 FHIR (2.6 MB).
 
 The default validation run checks schema parsing and module ownership. A separate
-UCI check generates code and compiles five destination languages; Rust and C++
-large-module compilation are tracked in the linked PolyXML issues.
+parameterized check generates source and compiles seven destination languages
+serially under a hard cgroup memory limit. Scheduled/manual CI checks UCI, CDA,
+UBL Invoice, NeTEx, and FpML; ordinary PR CI checks CDA and UBL Python syntax.
 
 ---
 
@@ -89,7 +90,8 @@ polyxml-schema-corpus/
 │   └── w3c-xmlschema/         # Normative W3C XML Schema definition
 ├── polyxml.toml               # Multi-module workspace configuration (45 modules)
 ├── runner.py                  # Automated validation and benchmark harness
-├── scripts/check_uci_codegen.py # Bounded UCI generated-code compile check
+├── scripts/check_module_codegen.py # Parameterized bounded compile check
+├── scripts/check_uci_codegen.py # UCI compatibility entry point
 ├── LICENSE                    # MIT License
 ├── NOTICE                     # Standards attribution & third-party copyright notices
 └── README.md
@@ -99,19 +101,58 @@ polyxml-schema-corpus/
 
 ## ⚡ Usage
 
-### UCI generated-code compilation
+### Bounded generated-code compilation
 
-With Python 3.12+, Go, TypeScript (`tsc`), Java, and .NET 8 installed, run:
+Install Python 3.12+, Go, g++ (C++20), Java 25, TypeScript 5 (`tsc`),
+.NET 8 SDK, and Rust. Linux with cgroup v2 and a working systemd user manager
+is required locally; CI uses the system manager through passwordless sudo.
 
 ```bash
-python3 scripts/check_uci_codegen.py --bin ../PolyXML/target/debug/polyxml
+python3 scripts/check_module_codegen.py --bin ../PolyXML/target/debug/polyxml \
+  -m defense_uci --memory-mib 12000 --output /tmp/uci-compile-results
+python3 scripts/check_module_codegen.py --bin ../PolyXML/target/debug/polyxml \
+  -m ubl_invoice -l python --output /tmp/ubl-python-results
 ```
 
-The script generates the four-schema UCI module into a temporary directory,
-compiles one language at a time, and uses PolyXML's memory-cap wrapper when it
-is available. Select a target with `--lang python` (repeat the option for
-several targets). Full Rust and C++ UCI compilation is tracked in
-[PolyXML issue #92](https://github.com/polyxml/PolyXML/issues/92).
+The harness reads the selected module and its transitive dependencies from
+`polyxml.toml`, emits actual source, and checks one language at a time.
+Repeat `-l` / `--lang` to select targets; the default is all seven.
+`check_uci_codegen.py` remains a compatibility entry point selecting `defense_uci`
+and accepts the same options (including required `--output`).
+
+Every generation and compile command runs in a separate systemd scope with
+`MemoryMax=3500M` by default and `MemorySwapMax=0`. UCI Rust needs the larger
+12,000 MiB budget used by its scheduled/manual CI job; local full-UCI runs
+should pass `--memory-mib 12000` with sufficient headroom. The ten-minute timeout stops the entire
+scope, including compiler children. The cap must leave at least 1 GiB available
+for the host; no uncapped fallback is permitted. `--memory-mib` and `--timeout`
+allow deliberate budget changes. Rust checks explicitly enable `split_units = true` with 250-type chunks;
+chunking is opt-in in PolyXML. Rust checks disable debug info and incremental
+compilation to avoid unnecessary compiler memory overhead. Rust/Go builds use one worker; Java's heap
+is additionally limited to 2200 MiB. `--system` selects CI's system manager
+and runs compilers as the invoking user. Missing tools fail before generation;
+`--allow-missing` is for local partial runs only and records explicit skips.
+
+The output directory must be new. It retains manifests, generated file counts
+and bytes, full command logs, GNU time resource reports (peak process RSS),
+and machine-readable elapsed times/exit statuses. Generated source and build
+products are temporary. RSS is a process measurement, not total cgroup memory;
+the kernel cap covers the whole tree. These are syntax/compile checks, not XML
+runtime round trips. Any failure or timeout returns nonzero; a failed target
+stops that module run and remains visible in CI artifacts by default.
+`--keep-going` checks the remaining targets while retaining a nonzero final exit
+status; scheduled/manual CI uses it to collect all seven outcomes.
+
+PR/push CI checks `hl7_cda` and `ubl_invoice` Python. Weekly and manual CI
+runs all seven languages for `defense_uci`, `hl7_cda`, `ubl_invoice`,
+`transit_netex`, and `finance_fpml`, with heavy matrix jobs serialized.
+The original validation/dry-run job remains separate.
+
+Harness regression tests (including real user-cgroup limit and timeout checks):
+
+```bash
+python3 -m unittest discover -s tests -v
+```
 
 ### 1. Run the Validation & Benchmark Suite
 

polyxml.toml

@@ -173,6 +173,7 @@ depends_on = ["aviation_fixm"]
 
 [modules.finance_fpml]
 schemas = ["schemas/finance-fpml/fpml-main-5-12.xsd"]
+depends_on = ["iso15118_xmldsig"]
 
 [modules.geospatial_gml]
 schemas = ["schemas/transit-netex/gml/gml_extract_all_objects_v_3_2_1.xsd"]

results/2026-10-03/README.md

@@ -0,0 +1,68 @@
+# Bounded corpus checks — 2026-10-03
+
+Latest: UCI passes all seven targets locally at 12000 MiB using the compiler
+fixes published in PolyXML 031ab53. NeTEx Python passes at 3500 MiB after UPA
+source deduplication, module ownership, and quoted-docstring fixes. These checks
+verify compilation, not runtime XML conformance. Detailed outcomes and historical
+failures follow.
+
+## Initial checks at 3500 MiB
+
+Source: PolyXML 640bc40 (compiler code from v0.34.0), with C# fix a25aa49
+for the later C# rerun. Exact compiler hash and tool versions are in
+`verification.json`. Every generation/compile command
+used MemoryMax=3500M, MemorySwapMax=0, a 600-second timeout, and serial builds.
+The host had ~13.6 GiB available before checks; WSL swap remained unused.
+
+| Module | Target | Result |
+| --- | --- | --- |
+| hl7_cda | Python syntax | pass |
+| ubl_invoice + dependencies | Python syntax | pass |
+| defense_uci (exact four-schema module) | Python, Go, TypeScript, Java | pass |
+| defense_uci | C++20 syntax | pass, peak process RSS 2,753,596 KiB |
+| defense_uci | C# net8.0 build | pass after Equals property-name fix |
+| defense_uci | Rust cargo check, monolithic | cgroup OOM, 3.4 GiB reported scope peak |
+| defense_uci | Rust cargo check, split_units=true, 250-type chunks | cgroup OOM, 25 emitted files |
+| transit_netex + dependencies | Python generation | cgroup OOM (journal retained) |
+| finance_fpml | Python generation/syntax | pass after adding XML-signature module dependency |
+
+`corpus-uci-seven` includes the initial CS8866 failure; the later
+`corpus-uci-csharp-fixed` log records the successful build. The Rust run failed
+before C++ was reached, so C++ was checked independently. The initial FpML
+failure and successful dependency fix are retained separately. These runs
+predate automatic compiler SHA-256 recording in the final harness.
+
+Harness tests passed all five tests, including a real 64 MiB kernel cap with
+swap disabled, malformed-source failure retention, and timeout descendant
+cleanup. The sudo/system-manager path is wired into CI but cannot be tested
+locally because this machine requires interactive sudo authentication.
+CI must verify that path. Checks are syntax/compilation only, not XML runtime
+round trips; Python code is not imported during syntax checking.
+
+At this stage UCI Rust and NeTEx failed the original budget. The larger-machine
+follow-up below records the subsequent compiler fixes and deliberately chosen
+UCI budget; initial failures are retained rather than relabeled as passes.
+
+The follow-up `corpus-uci-rust-split` run explicitly enabled topological
+chunking (25 emitted files versus 3 for the monolithic run). It also hit the
+cgroup memory cap. Chunking is retained in the final harness, but this evidence
+does not support claiming that splitting alone resolves Rust compilation.
+
+## Larger-machine follow-up
+
+UCI passes all seven targets in `corpus-uci-all-12000-final`, with codecs and
+Serde enabled, Rust topological chunks, debug info disabled, and incremental
+compilation disabled. The complete run uses a 12000 MiB cap and 600-second
+per-command timeouts. The initial 11500 MiB Rust run exposed binary-field
+type errors before cgroup OOM; binary fields now preserve XML lexical strings,
+and the fixed Rust-only rerun passed at 11500 MiB. Scheduled UCI CI uses 12000
+MiB to provide extra margin; other modules retain the 3500 MiB default.
+
+NeTEx's original higher-budget run still exceeded 11500 MiB. Deduplicating
+identical UPA source documents fixes exponential duplicate retention in shared
+include graphs. The first deduplicated run used 2036360 KiB peak process RSS
+at 3500 MiB and reached an imported-helper ownership error, not an OOM. After
+that fix, Python compilation exposed a quoted-docstring syntax error, fixed
+with an AST regression covering both Python backends. Subsequent results below
+record the final outcomes. Intermediate cache-budget experiments were canceled
+and removed from the implementation because they hurt shared-include reuse.