Martino FerrariandClaude Opus 4.6
50d85363a3
Add TCPStreamer/TCPStreamerClient (TCP transport behind the UDPSServerI/UDPSClientI seam), an optional single in-process TCPStreamService serving many streams over worker threads, multiplexed SUBSCRIBE (comma list, stream index in fragmentIdx), and TCPStreamJoin, which merges several streams into one sample-and-hold stream paced by a master. StreamHub takes a comma list of streams over one connection. Includes GTests, E2E scenarios s58-s73 and docs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
87 lines
4.2 KiB
Markdown
87 lines
4.2 KiB
Markdown
# TCPStreamService: joining streams (multiplex + merge)
|
|
|
|
Date: 2026-11-06. Extends `2026-11-05-tcp-streamer-design.md` section 7.
|
|
|
|
## Goal
|
|
|
|
Let one consumer take several service streams through a single, cheaper path.
|
|
Two independent mechanisms were requested ("both"):
|
|
|
|
1. **Multiplexed connection**: one TCP socket carries N streams, each keeping its
|
|
own CONFIG, rate and timeline.
|
|
2. **Merged stream**: the service builds one new stream whose CONFIG holds the
|
|
signals of several source streams, emitted at the pace of a master source
|
|
with sample-and-hold of the others.
|
|
|
|
## 1. Multiplexed connection
|
|
|
|
### Wire
|
|
- `SUBSCRIBE` (type 5) payload becomes a comma separated list of stream names
|
|
(`"fast,slow"`). One name is the old behaviour. Names may not contain `,`.
|
|
Limits: 64 names, 4096 payload bytes, no duplicates.
|
|
- The index of a stream is its position in the list.
|
|
- Every server-to-client frame carries the stream index in `fragmentIdx`
|
|
(`totalFragments` stays 1). Single-name connections always send 0, so
|
|
existing clients are unchanged.
|
|
- Any unknown name closes the whole connection (the client retries), exactly
|
|
like the single-name case.
|
|
|
|
### Server
|
|
- `TCPSConn` holds up to 64 stream attachments instead of one.
|
|
- A stream's frame is built once; `Push` copies it into each subscriber's queue
|
|
and stamps that connection's own index into `fragmentIdx`.
|
|
- `TCPSFrameQueue` accounts per stream: `ClientQueueDepth` DATA frames *per
|
|
stream*, drop-oldest of the same stream only, and a CONFIG only discards the
|
|
stale frames of its own stream. A fast stream can therefore never evict a
|
|
slow stream's frames.
|
|
- `Flush` batches queued frames into one `writev()` (up to 16 frames), which is
|
|
the optimisation for the many-small-frames case; it also applies to
|
|
single-stream connections.
|
|
- If a stream unregisters, every connection attached to it is closed.
|
|
|
|
### Client
|
|
- `TCPSClient` accepts `StreamName = "a,b,c"`. It demultiplexes by
|
|
`fragmentIdx` to one listener per stream and tracks the counter gap per
|
|
stream.
|
|
- `TCPSClient::GetStream(i)` returns a `UDPSClientI` facade for stream `i`.
|
|
Facades reference-count `Start()`/`Stop()` of the shared connection, so
|
|
existing consumers (StreamHub sessions) work unchanged.
|
|
- `TCPStreamer` / `TCPStreamerClient` DataSources are single-stream: a comma in
|
|
`StreamName` is rejected there.
|
|
|
|
### StreamHub
|
|
- A source with `Transport="tcp"` and `StreamName="a,b,c"` creates one session
|
|
per stream, all sharing one `TCPSClient`. Session labels are
|
|
`<Label>_<stream>` (just the stream name when no label is given).
|
|
|
|
## 2. Merged stream (`TCPStreamJoin`)
|
|
|
|
```
|
|
+Join = { Class = TCPStreamJoin Service = "Svc" StreamName = "all"
|
|
Sources = "fast,slow" Master = "fast" }
|
|
```
|
|
- Root-level object, like the service. It taps the source streams
|
|
(`TCPSStreamTapI`), and registers its own output stream with the service once
|
|
every source is registered. Losing a source unregisters the output.
|
|
- CONFIG: concatenated descriptors, names `<stream>_<signal>` (<= 63 chars),
|
|
`timeSignalIdx` rebased, publish mode Strict, common `hrtFrequency` (sources
|
|
with different frequencies are refused).
|
|
- DATA (one per master frame): `[master timestamp][source0 body][source1 body]...`
|
|
where a body is that source's last DATA payload without its 8-byte
|
|
timestamp. Nothing is emitted until every source has delivered one DATA.
|
|
- Only Strict and Decimate sources can be merged (Accumulate batches have
|
|
incompatible lengths) and a source body whose size changes without a CONFIG
|
|
change is ignored. Held values of a slower source are stamped with the
|
|
master's timestamp unless that source uses a time signal.
|
|
- Locking: join mutex, then the output stream mutex, then connection `qMutex`.
|
|
Notifications from the service happen outside its topology lock.
|
|
- No extra thread: the merged frame is built on the master producer's thread.
|
|
|
|
## Testing
|
|
- GTest: queue per-stream accounting + writev partial writes; service mux
|
|
(index tagging, fairness, unknown name, duplicate name, comma rejection);
|
|
client demux/gaps/facades; join (config merge, sample-and-hold, refusal
|
|
cases).
|
|
- E2E: a chain scenario with a multiplexed hub source (`tcp_service` + comma
|
|
list) and one with a merged stream.
|