Design and decisions
Thadam’s design is record-first: decisions are accepted as ADRs, evidence comes from spike reports, and only accepted decisions are built on. This page indexes the design corpus; it links rather than copies, so the repository stays the source of truth. Links to repository documents point at the main branch on GitHub, because the design records are not part of this published site.
Top-level design documents
| Document | What it is |
|---|---|
| ARCHITECTURE.md | The current-state architecture of the MVP; supersedes earlier, partly contradictory plans. |
| product.md | Product requirements and scope: vision, personas, use cases, roadmap. (Amended in places by the decision records below.) |
| Readiness tracker | Per-component implementation status and the remaining readiness blockers. |
| Decision register | The maintainer’s decision register (scope, defaults, ADR acceptances), dated. |
| Product amendments | Scope amendments to product.md (Part A applied; Parts B and C proposed). |
| Version inventory | Pinned versions of formats, dependencies, and tooling. |
Architecture decision records
Naming: NNNN-kebab-title.md; status lifecycle Draft → Proposed → Accepted → Superseded (process: ADR-0000, itself Proposed). Only the maintainer accepts an ADR.
| ADR | Status | Decision |
|---|---|---|
| 0000 — ADR process | Proposed | How decisions are recorded, accepted, and revisited. |
| 0001 — Canonical event model | Accepted | Typed events: time domains, quality, identity, extended values, shape-shared records. |
| 0002 — Connector contract | Accepted | Pull Source with liveness/cancel-safety clauses; split BatchSink/SegmentSink; epoch-scoped acks. |
| 0003 — Execution and process model | Accepted | Single process, per-sink chains, capture-first WAL. |
| 0004 — Durable buffer | Draft | Custom segmented WAL, fsync policy, on_full. Draft until the WAL hardware re-run; format versioned but not frozen. |
| 0006 — MQTT client | Accepted | rumqttc behind thadam-mqtt, wrapper rules, in-process TLS relay, required broker settings, EMQX 5.8.x pin. |
| 0007 — ROS 2 strategy | Draft | MCAP/rosbag2 offline first, live transport chosen by the ROS 2 live spike. |
| 0008 — ClickHouse sink | Draft (P1) | Batch sink, dedup probe, benchmarked client choice; outside the core v0.1 reliability gate. |
| 0011 — Delivery guarantees and claims register | Accepted | Per-hop guarantees (on_full, ack modes, deduplication, commit-then-ship, memory bounds), and the claims register that gates all public wording. |
| 0012 — Config format and YAML crate | Accepted | serde-saphyr behind thadam-config, strict booleans, parse budget, rejected YAML features. |
| 0013 — Parquet segment layout | Accepted | Wide tables per dataset, field IDs, finalize sequence, keys, catalog. |
| 0018 — Filter and expression language | Accepted | Structured YAML predicates; CEL reserved; expr: reserved. |
| 0020 — Dependency security and toolchain policy | Accepted | MSRV 1.89 (thadam-iceberg 1.94), pinned toolchain, rustls+ring, license allow-list, cargo deny bans. |
| 0005 Internal representation | Reserved | Folded into ADR-0001. |
| 0009 Object-store architecture | Reserved | Split across ADR-0011 (commit-then-ship) and ADR-0013 (keys); ARCHITECTURE section 8. |
| 0010 Python SDK boundary | Reserved | Post-MVP. |
| 0014 Time and clock semantics | Reserved | Folded into ADR-0001 and ADR-0013. |
| 0015 — Recording and replay format | Draft | WAL retention window; MCAP is export/import only; thadam export --mcap keeps original encodings and records gaps and provenance. |
| 0016 — Naming and publishing | Accepted | Name kept (no trademark search done); the CLI is the thadam crate; 23 crates go to crates.io and the Python package to PyPI. |
| 0017 Plugin mechanism | Reserved | Compile-time features now; out-of-process later; Rust dylibs rejected. |
| 0019 Transform semantics | Reserved | Folded into ADR-0011 and ARCHITECTURE section 7. |
Spike reports
Spikes are time-boxed experiments whose reports feed ADRs; the code is throwaway, the reports are evidence.
| Report | Feeds | Question answered |
|---|---|---|
| WAL spike | ADR-0004 | Custom segmented log vs SQLite vs redb: correctness under kill -9/LazyFS/torn tails, write amplification, recovery time, fsync behavior. |
| MQTT spike | ADR-0006, ADR-0011 | Ack-window throughput model, liveness under backpressure, poll cancel-safety, payload-stripped acks, persistent sessions, broker limits (Mosquitto 2.1.2, EMQX 5.8.6). |
| ROS 2 live spike | ADR-0007 | Live ROS 2 transport over rmw_zenoh (zenoh 1.10.1), key expressions, 33-byte publisher attachments, and transient_local late-join queries on Jazzy and Lyrical. |
| Sparkplug B and OPC UA PubSub spike | Phase 3 | Sparkplug B spBv1.0 protobuf decoding, BIRTH/DEATH alias resolution, and OPC UA PubSub evaluation. |
| Record representation | ADR-0001 | Memory layout of the canonical record (shape-shared positional records behind an opaque Record). |
| ADR-0012 YAML spike | ADR-0012 | YAML crate selection (serde-saphyr), strictness and budget measurements. |
| ADR-0018 filter spike | ADR-0018 | Structured predicates vs an expression language (CEL evaluator cost on ARM). |
A caution from the readiness tracker: every spike ran on an arm64 laptop (the WAL and MQTT spikes added a Linux VM in Docker with unknown flush semantics). Ratios and macro differences transfer; absolute numbers do not, and re-runs on reference ARM and x86 hardware are required before the results back any published claim.