Implemented history writer

This commit is contained in:
Martino Ferrari
2026-06-14 14:09:51 +02:00
parent f25bd7f08e
commit dd7dd22cb0
27 changed files with 1537 additions and 104 deletions
+73 -2
View File
@@ -98,7 +98,78 @@ client and uses full-rate ring data (not the decimated live stream).
On capture, the trigger view shows all plotted signals relative to the trigger
instant (marker at t = 0) over `[-pre, +post]`.
## 5. Troubleshooting
## 5. History (disk-backed storage)
StreamHub can store signal data to disk, allowing you to browse hours of past
data even after the in-memory ring buffers have wrapped.
### Enabling history
Add a `+History` block to the hub configuration:
```
Hub = {
WSPort = 8090
+History = {
Directory = "/data/streamhub_history"
DurationHours = 1
Decimation = 10
FlushIntervalSec = 5
MinDiskFreeMB = 200
}
Sources = { … }
}
```
| Key | Default | Description |
|-----|---------|-------------|
| `Directory` | *(required)* | Root directory for `.shist` files. Created automatically. Each source gets a subdirectory; each signal a separate file. |
| `DurationHours` | 1 | How many hours of data to keep per signal. The file size is fixed at creation: `duration × samplingRate / decimation × 16 bytes`. |
| `Decimation` | 1 | Write every N-th sample (1 = no decimation). Useful for high-rate signals where full-rate disk writes are impractical. |
| `FlushIntervalSec` | 5 | How often file headers are flushed to disk (data is written immediately via `pwrite`). |
| `MinDiskFreeMB` | 500 | Writing pauses when free disk space drops below this threshold and resumes when space is available again. |
### Disk usage
Each sample on disk is 16 bytes (8 bytes timestamp + 8 bytes value). Example:
- 10 signals at 1 ksps, decimation 10, duration 1 hour:
`10 × (1000/10) × 3600 × 16 bytes = 57.6 MB`
- 10 signals at 1 Msps, decimation 1000, duration 1 hour:
`10 × (1e6/1000) × 3600 × 16 bytes = 576 MB`
Files are pre-allocated at creation and never grow. The directory structure is:
```
/data/streamhub_history/
├── scalar/
│ ├── Sine1.shist
│ └── Sine2.shist
├── med/
│ ├── Ch1.shist
│ └── Ch2.shist
└── fast/
├── Ch3.shist
└── Ch4.shist
```
### Using history in clients
When history is enabled, both the web SPA and ImGui client show a **HIST**
indicator. History data is fetched transparently:
- **Zoom**: when you zoom or pan into a time range that extends beyond the
in-memory ring buffer, the client issues a `historyZoom` request in parallel
with the regular `zoom` request and merges the results — history data fills
the older part of the window, ring data the recent part.
- **Status**: the web SPA displays a history badge in the status bar showing
the configured duration and decimation factor. The ImGui client shows
"HIST" in the plot header when history data is being displayed.
History files survive hub restarts. If a `.shist` file exists with a matching
capacity, the hub reopens it and continues writing where it left off.
## 6. Troubleshooting
| Symptom | Check |
|---------|-------|
@@ -111,7 +182,7 @@ instant (marker at t = 0) over `[-pre, +post]`.
Hub-side logs go to stdout; the demo scripts write
`/tmp/streamhub_e2e_{marte,hub}.log`.
## 6. Further reading
## 7. Further reading
- [StreamHub-API.md](StreamHub-API.md) — WebSocket protocol (write your own client)
- [StreamHub-Developer.md](StreamHub-Developer.md) — internals, build, tests