Implemented history writer
This commit is contained in:
@@ -21,6 +21,7 @@ Wire protocols: [Protocol.md](Protocol.md) (UDPS, source → hub) and
|
||||
| `SignalRingBuffer.h` | Per-signal (t,v) ring: monotonic write counter, `ReadSince` cursor reads, binary-search `ReadRange` |
|
||||
| `TriggerEngine.{h,cpp}` | Trigger FSM (IDLE/ARMED/COLLECTING/TRIGGERED), edge detection per decoded sample |
|
||||
| `UDPSourceStats.h` | 512-entry cycle/frag/byte rings → avg/std/min/max, rate, 20-bin histogram |
|
||||
| `HistoryWriter.{h,cpp}` | Disk-backed circular history: per-signal `.shist` files, `WriteTick`, `ReadRange` (binary search via `pread`), disk space monitoring |
|
||||
| `WSServer.{h,cpp}` | RFC 6455 server: handshake, framing, per-client write mutex, broadcast/unicast |
|
||||
| `WSFrame.h`, `SHA1.h`, `Base64.h` | Header-only WS plumbing, shared with the ImGui client |
|
||||
| `LTTB.h` | Largest-Triangle-Three-Buckets decimation |
|
||||
@@ -124,6 +125,66 @@ Sources = {
|
||||
Sources from `SourcesFile` are loaded after the static `Sources` block;
|
||||
sources added at runtime via WS `addSource` get ids `s1, s2, …`.
|
||||
|
||||
### History configuration
|
||||
|
||||
An optional `+History` block enables disk-backed circular storage
|
||||
(`HistoryWriter`). The `+` prefix is MARTe2 `StandardParser` syntax for a
|
||||
child node; the hub looks for both `+History` and `History` as the node name.
|
||||
|
||||
```
|
||||
+History = {
|
||||
Directory = "/data/streamhub_history" // required
|
||||
DurationHours = 1 // hours of data to retain per signal (default 1)
|
||||
Decimation = 10 // keep every Nth sample (default 1)
|
||||
FlushIntervalSec = 5 // header flush period in seconds (default 5)
|
||||
MinDiskFreeMB = 500 // pause writes below this threshold (default 500)
|
||||
}
|
||||
```
|
||||
|
||||
Per-signal file capacity is computed at source CONFIG time:
|
||||
`capacity = ceil(DurationHours × 3600 × samplingRate / Decimation)`, minimum
|
||||
1000 pairs.
|
||||
|
||||
### `.shist` binary file format
|
||||
|
||||
Each signal gets one file: `<Directory>/<sourceId>/<signalName>.shist`.
|
||||
The file size is fixed at creation (`64 + capacity × 16` bytes) and never grows.
|
||||
|
||||
```
|
||||
Offset Size Field
|
||||
0 4 Magic: "SHR1"
|
||||
4 4 uint32 version (1)
|
||||
8 4 uint32 capacity (max pairs)
|
||||
12 4 uint32 head (next write position, 0-based, wraps at capacity)
|
||||
16 4 uint32 count (valid entries, ≤ capacity)
|
||||
20 4 uint32 decimation
|
||||
24 8 float64 tOldest (Unix seconds)
|
||||
32 8 float64 tNewest (Unix seconds)
|
||||
40 24 reserved (zero-padded to 64 bytes)
|
||||
64 … data: capacity × 16 bytes (float64 time + float64 value per pair)
|
||||
```
|
||||
|
||||
Data is written at the `head` position and wraps circularly. The oldest valid
|
||||
entry is at logical index `(head + capacity − count) % capacity`. All I/O
|
||||
uses `pwrite`/`pread` (no `mmap`), so concurrent reads from WS threads are
|
||||
safe without locking.
|
||||
|
||||
Headers are flushed to disk every `FlushIntervalSec` seconds and on shutdown.
|
||||
On restart, if an existing file has matching magic, version and capacity, it is
|
||||
reopened — head/count/time bounds are restored from the on-disk header.
|
||||
|
||||
### History query path
|
||||
|
||||
`historyZoom` requests (see [StreamHub-API.md](StreamHub-API.md)) call
|
||||
`HistoryWriter::ReadRange` which performs binary search over the circular file
|
||||
using `pread` to locate the `[t0, t1]` window, then copies matching pairs.
|
||||
If the result exceeds the requested `n`, LTTB decimation is applied (same
|
||||
`LTTBDecimate` as in-memory zoom).
|
||||
|
||||
Both the web SPA and ImGui client issue `historyZoom` in parallel with regular
|
||||
`zoom` and merge the results: history covers the older part of the visible
|
||||
window, the in-memory ring covers the recent part.
|
||||
|
||||
## 7. Build & test
|
||||
|
||||
```bash
|
||||
@@ -147,12 +208,14 @@ make -f Makefile.gcc test
|
||||
`./run_e2e_test.sh` builds everything, launches the demo MARTe2 application
|
||||
(`Test/Configurations/streamhub_demo.cfg`: 3 UDPStreamers — multicast scalars,
|
||||
FirstSample/LastSample arrays, FullArray + uint64 ns time array) plus a
|
||||
StreamHub on port 8095, then runs the Go WS client `Test/E2E/streamhub` which
|
||||
verifies: `sources`/`config` events, ≥10 binary v1 pushes with wall-clock and
|
||||
strictly monotonic time on all streams, `stats` shape, a `zoom` round-trip
|
||||
(reqId echo, unicast), and a complete trigger cycle (setTrigger → arm →
|
||||
binary v2 capture → triggered → disarm). Logs land in
|
||||
`/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
|
||||
StreamHub on port 8095 (with history enabled in `/tmp/streamhub_e2e_history`),
|
||||
then runs the Go WS client `Test/E2E/streamhub` which verifies:
|
||||
`sources`/`config` events, ≥10 binary v1 pushes with wall-clock and strictly
|
||||
monotonic time on all streams, `stats` shape, a `zoom` round-trip (reqId echo,
|
||||
unicast), `historyInfo` broadcast (enabled, duration, decimation, signal count),
|
||||
a `historyZoom` round-trip (reqId echo, signal data), and a complete trigger
|
||||
cycle (setTrigger → arm → binary v2 capture → triggered → disarm). Logs land
|
||||
in `/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
|
||||
|
||||
When changing the WS protocol, update **in lockstep**: this hub, the Go hub
|
||||
(`Common/Client/go/wshub`), the browser SPA (`Client/udpstreamer/static`), the
|
||||
|
||||
Reference in New Issue
Block a user