Implemented history writer
This commit is contained in:
Martino Ferrari
@@ -101,6 +101,27 @@ Every transition is broadcast as a `triggerState` event.
|
||||
- `signals` — comma-separated full keys; absent → all signals of all sources.
|
||||
- The reply is **unicast** to the requesting client only.
|
||||
|
||||
### `historyZoom`
|
||||
|
||||
```json
|
||||
{"type":"historyZoom","reqId":42,"t0":1765360000.0,"t1":1765370000.0,
|
||||
"n":2400,"signals":"s1:Sine,s2:Wave"}
|
||||
```
|
||||
- Reads from **disk-backed history** instead of the in-memory ring buffer.
|
||||
Identical semantics to `zoom` but queries the `.shist` files written by
|
||||
`HistoryWriter`.
|
||||
- `reqId`, `t0`/`t1`, `n`, `signals` — same meaning as `zoom`.
|
||||
- If history is not enabled, the reply contains `"error":"history not enabled"`.
|
||||
- Reply is **unicast** (same shape as `zoom` reply, but `"type":"historyZoom"`).
|
||||
|
||||
### `historyInfo`
|
||||
|
||||
```json
|
||||
{"type":"historyInfo"}
|
||||
```
|
||||
Request the hub to send a `historyInfo` event (unicast). Also sent automatically
|
||||
on client connect.
|
||||
|
||||
### `setMaxPoints`
|
||||
|
||||
```json
|
||||
@@ -164,6 +185,34 @@ trigger has fired.
|
||||
`t` is serialised with `%.17g` (full float64 precision — required for
|
||||
µs windows at Unix-epoch magnitudes), `v` with `%.9g`.
|
||||
|
||||
### `historyInfo`
|
||||
|
||||
Sent on client connect (if history is enabled) and on `historyInfo` command:
|
||||
|
||||
```json
|
||||
{"type":"historyInfo","enabled":true,"durationHours":1.0,"decimation":10,
|
||||
"signals":{
|
||||
"scalar:Sine1":{"t0":1765360000.0,"t1":1765370000.0,"count":360000,"capacity":360000},
|
||||
"scalar:Sine2":{"t0":1765360000.0,"t1":1765370000.0,"count":360000,"capacity":360000}}}
|
||||
```
|
||||
- `enabled` — `true` if the `+History` config block is present and valid.
|
||||
- `durationHours` — configured history duration.
|
||||
- `decimation` — samples-to-disk decimation factor (1 = every sample).
|
||||
- `signals` — per-signal metadata keyed by `"sourceId:signalName"`:
|
||||
- `t0`/`t1` — oldest/newest timestamp stored on disk (Unix seconds).
|
||||
- `count` — number of valid entries currently in the circular file.
|
||||
- `capacity` — total capacity of the circular file.
|
||||
|
||||
### `historyZoom` (reply)
|
||||
|
||||
Same shape as `zoom` reply, but `"type":"historyZoom"`:
|
||||
|
||||
```json
|
||||
{"type":"historyZoom","reqId":42,"signals":{
|
||||
"s1:Sine":{"t":[1765360000.1230000,"…"],"v":[0.123456789,"…"]}}}
|
||||
```
|
||||
If history is not enabled: `{"type":"historyZoom","error":"history not enabled"}`.
|
||||
|
||||
### `maxPointsUpdated`
|
||||
|
||||
```json
|
||||
|
||||
@@ -21,6 +21,7 @@ Wire protocols: [Protocol.md](Protocol.md) (UDPS, source → hub) and
|
||||
| `SignalRingBuffer.h` | Per-signal (t,v) ring: monotonic write counter, `ReadSince` cursor reads, binary-search `ReadRange` |
|
||||
| `TriggerEngine.{h,cpp}` | Trigger FSM (IDLE/ARMED/COLLECTING/TRIGGERED), edge detection per decoded sample |
|
||||
| `UDPSourceStats.h` | 512-entry cycle/frag/byte rings → avg/std/min/max, rate, 20-bin histogram |
|
||||
| `HistoryWriter.{h,cpp}` | Disk-backed circular history: per-signal `.shist` files, `WriteTick`, `ReadRange` (binary search via `pread`), disk space monitoring |
|
||||
| `WSServer.{h,cpp}` | RFC 6455 server: handshake, framing, per-client write mutex, broadcast/unicast |
|
||||
| `WSFrame.h`, `SHA1.h`, `Base64.h` | Header-only WS plumbing, shared with the ImGui client |
|
||||
| `LTTB.h` | Largest-Triangle-Three-Buckets decimation |
|
||||
@@ -124,6 +125,66 @@ Sources = {
|
||||
Sources from `SourcesFile` are loaded after the static `Sources` block;
|
||||
sources added at runtime via WS `addSource` get ids `s1, s2, …`.
|
||||
|
||||
### History configuration
|
||||
|
||||
An optional `+History` block enables disk-backed circular storage
|
||||
(`HistoryWriter`). The `+` prefix is MARTe2 `StandardParser` syntax for a
|
||||
child node; the hub looks for both `+History` and `History` as the node name.
|
||||
|
||||
```
|
||||
+History = {
|
||||
Directory = "/data/streamhub_history" // required
|
||||
DurationHours = 1 // hours of data to retain per signal (default 1)
|
||||
Decimation = 10 // keep every Nth sample (default 1)
|
||||
FlushIntervalSec = 5 // header flush period in seconds (default 5)
|
||||
MinDiskFreeMB = 500 // pause writes below this threshold (default 500)
|
||||
}
|
||||
```
|
||||
|
||||
Per-signal file capacity is computed at source CONFIG time:
|
||||
`capacity = ceil(DurationHours × 3600 × samplingRate / Decimation)`, minimum
|
||||
1000 pairs.
|
||||
|
||||
### `.shist` binary file format
|
||||
|
||||
Each signal gets one file: `<Directory>/<sourceId>/<signalName>.shist`.
|
||||
The file size is fixed at creation (`64 + capacity × 16` bytes) and never grows.
|
||||
|
||||
```
|
||||
Offset Size Field
|
||||
0 4 Magic: "SHR1"
|
||||
4 4 uint32 version (1)
|
||||
8 4 uint32 capacity (max pairs)
|
||||
12 4 uint32 head (next write position, 0-based, wraps at capacity)
|
||||
16 4 uint32 count (valid entries, ≤ capacity)
|
||||
20 4 uint32 decimation
|
||||
24 8 float64 tOldest (Unix seconds)
|
||||
32 8 float64 tNewest (Unix seconds)
|
||||
40 24 reserved (zero-padded to 64 bytes)
|
||||
64 … data: capacity × 16 bytes (float64 time + float64 value per pair)
|
||||
```
|
||||
|
||||
Data is written at the `head` position and wraps circularly. The oldest valid
|
||||
entry is at logical index `(head + capacity − count) % capacity`. All I/O
|
||||
uses `pwrite`/`pread` (no `mmap`), so concurrent reads from WS threads are
|
||||
safe without locking.
|
||||
|
||||
Headers are flushed to disk every `FlushIntervalSec` seconds and on shutdown.
|
||||
On restart, if an existing file has matching magic, version and capacity, it is
|
||||
reopened — head/count/time bounds are restored from the on-disk header.
|
||||
|
||||
### History query path
|
||||
|
||||
`historyZoom` requests (see [StreamHub-API.md](StreamHub-API.md)) call
|
||||
`HistoryWriter::ReadRange` which performs binary search over the circular file
|
||||
using `pread` to locate the `[t0, t1]` window, then copies matching pairs.
|
||||
If the result exceeds the requested `n`, LTTB decimation is applied (same
|
||||
`LTTBDecimate` as in-memory zoom).
|
||||
|
||||
Both the web SPA and ImGui client issue `historyZoom` in parallel with regular
|
||||
`zoom` and merge the results: history covers the older part of the visible
|
||||
window, the in-memory ring covers the recent part.
|
||||
|
||||
## 7. Build & test
|
||||
|
||||
```bash
|
||||
@@ -147,12 +208,14 @@ make -f Makefile.gcc test
|
||||
`./run_e2e_test.sh` builds everything, launches the demo MARTe2 application
|
||||
(`Test/Configurations/streamhub_demo.cfg`: 3 UDPStreamers — multicast scalars,
|
||||
FirstSample/LastSample arrays, FullArray + uint64 ns time array) plus a
|
||||
StreamHub on port 8095, then runs the Go WS client `Test/E2E/streamhub` which
|
||||
verifies: `sources`/`config` events, ≥10 binary v1 pushes with wall-clock and
|
||||
strictly monotonic time on all streams, `stats` shape, a `zoom` round-trip
|
||||
(reqId echo, unicast), and a complete trigger cycle (setTrigger → arm →
|
||||
binary v2 capture → triggered → disarm). Logs land in
|
||||
`/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
|
||||
StreamHub on port 8095 (with history enabled in `/tmp/streamhub_e2e_history`),
|
||||
then runs the Go WS client `Test/E2E/streamhub` which verifies:
|
||||
`sources`/`config` events, ≥10 binary v1 pushes with wall-clock and strictly
|
||||
monotonic time on all streams, `stats` shape, a `zoom` round-trip (reqId echo,
|
||||
unicast), `historyInfo` broadcast (enabled, duration, decimation, signal count),
|
||||
a `historyZoom` round-trip (reqId echo, signal data), and a complete trigger
|
||||
cycle (setTrigger → arm → binary v2 capture → triggered → disarm). Logs land
|
||||
in `/tmp/streamhub_e2e_{marte,hub}.log`. Exit 0 iff every check passes.
|
||||
|
||||
When changing the WS protocol, update **in lockstep**: this hub, the Go hub
|
||||
(`Common/Client/go/wshub`), the browser SPA (`Client/udpstreamer/static`), the
|
||||
|
||||
@@ -98,7 +98,78 @@ client and uses full-rate ring data (not the decimated live stream).
|
||||
On capture, the trigger view shows all plotted signals relative to the trigger
|
||||
instant (marker at t = 0) over `[-pre, +post]`.
|
||||
|
||||
## 5. Troubleshooting
|
||||
## 5. History (disk-backed storage)
|
||||
|
||||
StreamHub can store signal data to disk, allowing you to browse hours of past
|
||||
data even after the in-memory ring buffers have wrapped.
|
||||
|
||||
### Enabling history
|
||||
|
||||
Add a `+History` block to the hub configuration:
|
||||
|
||||
```
|
||||
Hub = {
|
||||
WSPort = 8090
|
||||
+History = {
|
||||
Directory = "/data/streamhub_history"
|
||||
DurationHours = 1
|
||||
Decimation = 10
|
||||
FlushIntervalSec = 5
|
||||
MinDiskFreeMB = 200
|
||||
}
|
||||
Sources = { … }
|
||||
}
|
||||
```
|
||||
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `Directory` | *(required)* | Root directory for `.shist` files. Created automatically. Each source gets a subdirectory; each signal a separate file. |
|
||||
| `DurationHours` | 1 | How many hours of data to keep per signal. The file size is fixed at creation: `duration × samplingRate / decimation × 16 bytes`. |
|
||||
| `Decimation` | 1 | Write every N-th sample (1 = no decimation). Useful for high-rate signals where full-rate disk writes are impractical. |
|
||||
| `FlushIntervalSec` | 5 | How often file headers are flushed to disk (data is written immediately via `pwrite`). |
|
||||
| `MinDiskFreeMB` | 500 | Writing pauses when free disk space drops below this threshold and resumes when space is available again. |
|
||||
|
||||
### Disk usage
|
||||
|
||||
Each sample on disk is 16 bytes (8 bytes timestamp + 8 bytes value). Example:
|
||||
|
||||
- 10 signals at 1 ksps, decimation 10, duration 1 hour:
|
||||
`10 × (1000/10) × 3600 × 16 bytes = 57.6 MB`
|
||||
- 10 signals at 1 Msps, decimation 1000, duration 1 hour:
|
||||
`10 × (1e6/1000) × 3600 × 16 bytes = 576 MB`
|
||||
|
||||
Files are pre-allocated at creation and never grow. The directory structure is:
|
||||
|
||||
```
|
||||
/data/streamhub_history/
|
||||
├── scalar/
|
||||
│ ├── Sine1.shist
|
||||
│ └── Sine2.shist
|
||||
├── med/
|
||||
│ ├── Ch1.shist
|
||||
│ └── Ch2.shist
|
||||
└── fast/
|
||||
├── Ch3.shist
|
||||
└── Ch4.shist
|
||||
```
|
||||
|
||||
### Using history in clients
|
||||
|
||||
When history is enabled, both the web SPA and ImGui client show a **HIST**
|
||||
indicator. History data is fetched transparently:
|
||||
|
||||
- **Zoom**: when you zoom or pan into a time range that extends beyond the
|
||||
in-memory ring buffer, the client issues a `historyZoom` request in parallel
|
||||
with the regular `zoom` request and merges the results — history data fills
|
||||
the older part of the window, ring data the recent part.
|
||||
- **Status**: the web SPA displays a history badge in the status bar showing
|
||||
the configured duration and decimation factor. The ImGui client shows
|
||||
"HIST" in the plot header when history data is being displayed.
|
||||
|
||||
History files survive hub restarts. If a `.shist` file exists with a matching
|
||||
capacity, the hub reopens it and continues writing where it left off.
|
||||
|
||||
## 6. Troubleshooting
|
||||
|
||||
| Symptom | Check |
|
||||
|---------|-------|
|
||||
@@ -111,7 +182,7 @@ instant (marker at t = 0) over `[-pre, +post]`.
|
||||
Hub-side logs go to stdout; the demo scripts write
|
||||
`/tmp/streamhub_e2e_{marte,hub}.log`.
|
||||
|
||||
## 6. Further reading
|
||||
## 7. Further reading
|
||||
|
||||
- [StreamHub-API.md](StreamHub-API.md) — WebSocket protocol (write your own client)
|
||||
- [StreamHub-Developer.md](StreamHub-Developer.md) — internals, build, tests
|
||||
|
||||
Reference in New Issue
Block a user