Implemented history writer

This commit is contained in:
Martino Ferrari
2026-06-14 14:09:51 +02:00
parent f25bd7f08e
commit dd7dd22cb0
27 changed files with 1537 additions and 104 deletions
+69 -6
View File
@@ -21,6 +21,7 @@ Wire protocols: [Protocol.md](Protocol.md) (UDPS, source → hub) and
| `SignalRingBuffer.h` | Per-signal (t,v) ring: monotonic write counter, `ReadSince` cursor reads, binary-search `ReadRange` |
| `TriggerEngine.{h,cpp}` | Trigger FSM (IDLE/ARMED/COLLECTING/TRIGGERED), edge detection per decoded sample |
| `UDPSourceStats.h` | 512-entry cycle/frag/byte rings → avg/std/min/max, rate, 20-bin histogram |
| `HistoryWriter.{h,cpp}` | Disk-backed circular history: per-signal `.shist` files, `WriteTick`, `ReadRange` (binary search via `pread`), disk space monitoring |
| `WSServer.{h,cpp}` | RFC 6455 server: handshake, framing, per-client write mutex, broadcast/unicast |
| `WSFrame.h`, `SHA1.h`, `Base64.h` | Header-only WS plumbing, shared with the ImGui client |
| `LTTB.h` | Largest-Triangle-Three-Buckets decimation |
@@ -124,6 +125,66 @@ Sources = {
Sources from `SourcesFile` are loaded after the static `Sources` block;
sources added at runtime via WS `addSource` get ids `s1, s2, …`.
### History configuration
An optional `+History` block enables disk-backed circular storage
(`HistoryWriter`). The `+` prefix is MARTe2 `StandardParser` syntax for a
child node; the hub looks for both `+History` and `History` as the node name.
```
+History = {
Directory = "/data/streamhub_history" // required
DurationHours = 1 // hours of data to retain per signal (default 1)
Decimation = 10 // keep every Nth sample (default 1)
FlushIntervalSec = 5 // header flush period in seconds (default 5)
MinDiskFreeMB = 500 // pause writes below this threshold (default 500)
}
```
Per-signal file capacity is computed at source CONFIG time:
`capacity = ceil(DurationHours × 3600 × samplingRate / Decimation)`, minimum
1000 pairs.
### `.shist` binary file format
Each signal gets one file: `<Directory>/<sourceId>/<signalName>.shist`.
The file size is fixed at creation (`64 + capacity × 16` bytes) and never grows.
```
Offset Size Field
0 4 Magic: "SHR1"
4 4 uint32 version (1)
8 4 uint32 capacity (max pairs)
12 4 uint32 head (next write position, 0-based, wraps at capacity)
16 4 uint32 count (valid entries, ≤ capacity)
20 4 uint32 decimation
24 8 float64 tOldest (Unix seconds)
32 8 float64 tNewest (Unix seconds)
40 24 reserved (zero-padded to 64 bytes)
64 … data: capacity × 16 bytes (float64 time + float64 value per pair)
```
Data is written at the `head` position and wraps circularly. The oldest valid
entry is at logical index `(head + capacity count) % capacity`. All I/O
uses `pwrite`/`pread` (no `mmap`), so concurrent reads from WS threads are
safe without locking.
Headers are flushed to disk every `FlushIntervalSec` seconds and on shutdown.
On restart, if an existing file has matching magic, version and capacity, it is
reopened — head/count/time bounds are restored from the on-disk header.
### History query path
`historyZoom` requests (see [StreamHub-API.md](StreamHub-API.md)) call
`HistoryWriter::ReadRange` which performs binary search over the circular file
using `pread` to locate the `[t0, t1]` window, then copies matching pairs.
If the result exceeds the requested `n`, LTTB decimation is applied (same
`LTTBDecimate` as in-memory zoom).
Both the web SPA and ImGui client issue `historyZoom` in parallel with regular
`zoom` and merge the results: history covers the older part of the visible
window, the in-memory ring covers the recent part.
## 7. Build & test
```bash
@@ -147,12 +208,14 @@ make -f Makefile.gcc test
`./run_e2e_test.sh` builds everything, launches the demo MARTe2 application
(`Test/Configurations/streamhub_demo.cfg`: 3 UDPStreamers — multicast scalars,
FirstSample/LastSample arrays, FullArray + uint64 ns time array) plus a
StreamHub on port 8095, then runs the Go WS client `Test/E2E/streamhub` which
verifies: `sources`/`config` events, ≥10 binary v1 pushes with wall-clock and
strictly monotonic time on all streams, `stats` shape, a `zoom` round-trip
(reqId echo, unicast), and a complete trigger cycle (setTrigger → arm →
binary v2 capture → triggered → disarm). Logs land in
`/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
StreamHub on port 8095 (with history enabled in `/tmp/streamhub_e2e_history`),
then runs the Go WS client `Test/E2E/streamhub` which verifies:
`sources`/`config` events, ≥10 binary v1 pushes with wall-clock and strictly
monotonic time on all streams, `stats` shape, a `zoom` round-trip (reqId echo,
unicast), `historyInfo` broadcast (enabled, duration, decimation, signal count),
a `historyZoom` round-trip (reqId echo, signal data), and a complete trigger
cycle (setTrigger → arm → binary v2 capture → triggered → disarm). Logs land
in `/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
When changing the WS protocol, update **in lockstep**: this hub, the Go hub
(`Common/Client/go/wshub`), the browser SPA (`Client/udpstreamer/static`), the