can-telemetry-forge

Synthetic heavy-equipment telemetry, grounded in the J1939 standard — realistic predictive-maintenance data you can regenerate from a seed.

A generator of synthetic heavy-equipment CAN Bus telemetry for predictive-maintenance datasets. It models a realistically composed fleet — machinery mix, age curve and regional deployment — emitting correlated engine/sensor signals over time, injects a registry of labeled anomalies (obvious + joint/contextual outliers, stuck/drift/dropout sensor faults, and CAN-frame faults from a real J1939 frame encoder), derives a multi-mode failure label, and writes tidy tables (Parquet / CSV / DuckDB) ready for any downstream machine-learning or data-quality work.

The data is the product. The model that consumes it can be deliberately boring — the point is a dataset that is diverse, statistically credible, and fully reproducible, so the pipeline around it (training, monitoring, drift detection) has something real to work on.

The upstream half of a pair. Its companion forge-pdm-mlops is the ML-in-production system built on this data — train → registry → serve → drift → retrain, with a live interactive demo. --season is the knob that drives its drift story. Built the data engine, then the ML system over it.

---

Quickstart

pip install -e .
forge generate --out out/          # a full, reproducible PdM dataset
forge validate --data out/         # check it against the J1939 spec

Same config + seed → byte-identical output. More knobs:

forge generate --config configs/fleet.json --seed 42 --format duckdb
forge generate --season heatwave --out out/     # shift the whole fleet's distribution
forge generate --emit-raw-frames --out out/     # raw J1939 CAN frames as an artifact

The honesty boundary. 100% synthetic, clean-room — modelled from the public SAE J1939 standard and documented physics, never from a real log. The one real dataset that appears (a CC-BY public one) is used only to sanity-check distributions, fetched at run time, never committed. Roadmap F0–F6 complete (docs/ROADMAP.md).

---

Why it's credible (and clean-room)

The signal model is grounded in the publicly documented SAE J1939 standard — the heavy-duty CAN application layer — using its published PGN/SPN structure, value ranges, units and scaling. Each generated field carries its real SPN (engine speed 190, coolant temp 110, oil pressure 100, engine load 92, fuel rate 183, boost 102, EGT 173, DEF 1761, …), unit and operating range in the committed data dictionary. The relationships between signals (fuel rate tracking load and RPM, coolant temperature responding to ambient and load, EGT rising at altitude, vibration rising with terrain and wear) come from documented physics, not from any proprietary log — and their signs are asserted in tests so they can't silently drift.

• No real telemetry is shipped or used as a seed. A permissively-licensed public CAN/OBD dataset is used only to validate that the generated distributions look plausible — the Vehicle Energy Dataset (Kaggle, CC-BY 4.0), fetched at validation time and never committed (forge validate --dataset ved; see F4 and ADR-017).
• Reproducible by construction. Every dataset regenerates from a config file + a fixed seed. Same seed → identical data.
• Clean room. Built from the standard and known physics; contains no proprietary code or data.

What makes it different

A few realism choices set it apart from a generic random-signal generator (full rationale in docs/DECISIONS.md):

• Two-layer realism. A fleet-composition layer (operator → regions → contracts → units, with realistic machinery mix, age curve and units per contract) sits under the J1939 signal layer. Credibility comes from both, not from any single real log.
• CAN capability gated by model-year era. Older units only expose the SPNs their bus actually supported — unsupported signals are emitted as NULL, never zero. Structural missingness a downstream model has to handle, the way real mixed-age fleets behave.
• Multi-mode failures that progressively degrade. Distinct failure modes (overheating, oil starvation, bearing/mechanical wear) each build their own signal signature — and the signature ramps up over the hours before the event (coolant climbs, oil pressure sags, vibration rises), the way a real machine degrades, not a flag that flips out of nowhere (ADR-020). Without that drift the pre-failure rows are indistinguishable from healthy ones; with it the target is genuinely learnable per-row (a downstream LightGBM goes from ≈ 0.55 to ≈ 0.82 ROC-AUC) and per-mode evaluable.
• Environment that wears the machine. Per-region thermal + wear + terrain / road-quality modifiers (grounded in public data) shift signal baselines and accelerate failure hazards — the seam a future drift demo shifts.
• Real J1939 frames under the hood. Each bus signal has its true byte/bit frame layout; a frame-level encoder/decoder lets the Tier-3 CAN-frame faults (byte corruption, stale re-sends, error/not-available codes, truncated DLCs) be injected at the byte layer and decoded back — exactly what a receiver sees — with the raw frames optionally emitted as a can_frames artifact.

What it generates

The pipeline below is live end to end:

config (fleet, regions, climate, terrain, season, anomaly rates, resolution, seed)
   └─► fleet composition  operator → contracts → units (model, build year, duty)
        └─► signal model   J1939-grounded per-signal generators + correlations,
        │                  gated by capability era (unsupported SPN → NULL)
        └─► fleet sim      units over time at configurable resolution; thermal /
             │             wear / terrain modifiers per region
             └─► faults     registry of labeled defects → anomaly_type: obvious ·
                  │         joint/contextual outliers · sensor stuck / drift / dropout ·
                  │         CAN-frame corrupt / stale / error-indicator / truncated
                  └─► label  multi-mode failure_within_h (overheat / oil / bearing)
                       └─► tidy tables → Parquet / CSV / DuckDB + data dictionary

The entry point is a single command:

pip install -e .

# Generate the bundled default fleet (~134 units × 90 days) to Parquet:
forge generate --out out/

# Or point at a config, pick a seed, format and a smaller window:
forge generate --config configs/fleet.json --seed 42 --format duckdb \
  --days 30 --resolution 5min --out out/

# Tier-2: apply a seasonal anomaly (the knob a future drift demo shifts):
forge generate --season heatwave --out out/

# Tier-3: also emit the byte-level corrupted J1939 frames as a side table:
forge generate --emit-raw-frames --out out/

This writes the tidy readings table — signals plus the labels failure_within_h / failure_mode and anomaly_type / anomaly_signal (with an is_outlier rollup) so every injected defect is recoverable — plus units / vehicle_classes / equipment_models / regions / contracts dimension tables, a manifest.json (provenance + run parameters + per-type defect counts + the run's season), and a generated dataset_dictionary.md. With --emit-raw-frames it also writes an opt-in can_frames table of the byte-level corrupted J1939 frames behind each CAN-frame fault. The default config in configs/fleet.json is a fictional international operator whose regions are pinned to cited public climate-type + road-roughness sources (see docs/DATA_DESIGN.md §6) — never any private data. Everything the CLI does is callable as a library (config → sim.simulate → io.write_dataset), so the future MLOps repo can import it directly.

Validating the data (F4)

forge validate checks that the generated distributions are plausible and emits a self-contained Markdown report. It runs two offline reference adapters by default — so it needs no network and is reproducible by anyone:

• in_spec — every value sits inside its documented SAE J1939 range.
• golden — per-signal summary stats match a pinned, recomputed reference run (catches silent drift in the generator itself — nothing is committed).

# Offline plausibility report to stdout (or --report report.md):
forge validate --seed 42

# Opt into a real-data comparison (Vehicle Energy Dataset, CC-BY 4.0):
pip install -e '.[validate]'
forge validate --seed 42 --dataset ved --report report.md

# Point at a different Kaggle VED mirror if you like:
forge validate --dataset ved --ved-handle owner/slug

The optional ved adapter overlaps the shared engine channels (engine RPM, engine load) against the Vehicle Energy Dataset (Kaggle, CC-BY 4.0), fetched at run time and never committed. VED is light-vehicle OBD-II, so the overlap is a plausibility sanity-check on shared channels, not an equivalence claim — stated plainly in the report. A live run gives a histogram-intersection overlap of ~0.48 (engine RPM) / ~0.51 (engine load) against 200k VED rows. CI never requests it and the offline checks always stand on their own (rationale in ADR-017).

Auth for --dataset ved. The fetch uses the classic Kaggle REST endpoint with your legacy API credentials — put a kaggle.json (Kaggle → Settings → API → Create Legacy API Key) at ~/.kaggle/kaggle.json. The dataset handle is configurable (--ved-handle / FORGE_VED_HANDLE), so a moved/renamed mirror is a flag change, not a code edit. (Where TLS is terminated, e.g. some managed antivirus, install pip-system-certs so Python trusts the system certificate store.)

Data tiers

Richness is sequenced so the repo ships fast and grows on a roadmap (full spec in docs/DATA_DESIGN.md):

• Tier 1 (MVP): one fleet with realistic composition, core J1939 signals gated by capability era, a multi-mode failure label, deliberate bad data and obvious labeled outliers.
• Tier 2 (shipped in F5): six contrasting public-grounded regions/contracts, a catalog of equipment models with distinct reliability + signature profiles (per-mode hazard multipliers, baseline offsets, optional capability floor), and configurable seasons (heatwave / cold_snap / wet_season) — the knob a future drift demo shifts.
• Tier 3 (shipped): joint/contextual outliers and stuck/drift/dropout sensor faults (F3); plus CAN-frame faults — byte corruption, stale re-sends, J1939 error/not-available codes, and truncated DLCs — injected at the byte layer by a real frame-level encoder/decoder and decoded back into the table (F6).

Roadmap

Phase	What
F0	Foundations & runnable skeleton (package, CLI, CI) ✅
F1	J1939-grounded signal model + data dictionary ✅
F2	Fleet simulator + Parquet/CSV/DuckDB writers — Tier 1 ships (MVP) ✅
F3	Labeled anomaly & sensor-fault injection (declarative injector registry) ✅
F4	Distribution validation vs a license-checked public dataset (offline in-spec/golden + opt-in CC-BY VED overlap) ✅
F5	Diversity (Tier 2): more regions + equipment models + seasons ✅
F6	CAN-frame faults (Tier 3) + the frame-level encoder they need ✅

See docs/ROADMAP.md for objectives and definitions of done, and docs/DECISIONS.md for the design rationale (ADRs).

Project context

This is a public, clean-room portfolio project — part of a pair: a future MLOps project will consume this generator as its data source (experiment tracking, model registry, serving, drift monitoring on top of the telemetry produced here).

License

MIT © 2026 Jorge Ribeiro