0 Fork 0

Files

T

Martino FerrariandClaude Opus 4.6 50d85363a3 feat(tcp): TCPStreamer DataSources, in-process TCPStreamService, stream multiplexing and TCPStreamJoin

Add TCPStreamer/TCPStreamerClient (TCP transport behind the UDPSServerI/UDPSClientI
seam), an optional single in-process TCPStreamService serving many streams over
worker threads, multiplexed SUBSCRIBE (comma list, stream index in fragmentIdx),
and TCPStreamJoin, which merges several streams into one sample-and-hold stream
paced by a master. StreamHub takes a comma list of streams over one connection.
Includes GTests, E2E scenarios s58-s73 and docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-11-06 13:41:27 +01:00

4.2 KiB

Raw Blame History

TCPStreamService: joining streams (multiplex + merge)

Date: 2026-11-06. Extends 2026-11-05-tcp-streamer-design.md section 7.

Goal

Let one consumer take several service streams through a single, cheaper path. Two independent mechanisms were requested ("both"):

Multiplexed connection: one TCP socket carries N streams, each keeping its own CONFIG, rate and timeline.
Merged stream: the service builds one new stream whose CONFIG holds the signals of several source streams, emitted at the pace of a master source with sample-and-hold of the others.

1. Multiplexed connection

Wire

SUBSCRIBE (type 5) payload becomes a comma separated list of stream names ("fast,slow"). One name is the old behaviour. Names may not contain ,. Limits: 64 names, 4096 payload bytes, no duplicates.
The index of a stream is its position in the list.
Every server-to-client frame carries the stream index in fragmentIdx (totalFragments stays 1). Single-name connections always send 0, so existing clients are unchanged.
Any unknown name closes the whole connection (the client retries), exactly like the single-name case.

Server

TCPSConn holds up to 64 stream attachments instead of one.
A stream's frame is built once; Push copies it into each subscriber's queue and stamps that connection's own index into fragmentIdx.
TCPSFrameQueue accounts per stream: ClientQueueDepth DATA frames per stream, drop-oldest of the same stream only, and a CONFIG only discards the stale frames of its own stream. A fast stream can therefore never evict a slow stream's frames.
Flush batches queued frames into one writev() (up to 16 frames), which is the optimisation for the many-small-frames case; it also applies to single-stream connections.
If a stream unregisters, every connection attached to it is closed.

Client

TCPSClient accepts StreamName = "a,b,c". It demultiplexes by fragmentIdx to one listener per stream and tracks the counter gap per stream.
TCPSClient::GetStream(i) returns a UDPSClientI facade for stream i. Facades reference-count Start()/Stop() of the shared connection, so existing consumers (StreamHub sessions) work unchanged.
TCPStreamer / TCPStreamerClient DataSources are single-stream: a comma in StreamName is rejected there.

StreamHub

A source with Transport="tcp" and StreamName="a,b,c" creates one session per stream, all sharing one TCPSClient. Session labels are <Label>_<stream> (just the stream name when no label is given).

2. Merged stream (`TCPStreamJoin`)

+Join = { Class = TCPStreamJoin Service = "Svc" StreamName = "all"
          Sources = "fast,slow" Master = "fast" }

Root-level object, like the service. It taps the source streams (TCPSStreamTapI), and registers its own output stream with the service once every source is registered. Losing a source unregisters the output.
CONFIG: concatenated descriptors, names <stream>_<signal> (<= 63 chars), timeSignalIdx rebased, publish mode Strict, common hrtFrequency (sources with different frequencies are refused).
DATA (one per master frame): [master timestamp][source0 body][source1 body]... where a body is that source's last DATA payload without its 8-byte timestamp. Nothing is emitted until every source has delivered one DATA.
Only Strict and Decimate sources can be merged (Accumulate batches have incompatible lengths) and a source body whose size changes without a CONFIG change is ignored. Held values of a slower source are stamped with the master's timestamp unless that source uses a time signal.
Locking: join mutex, then the output stream mutex, then connection qMutex. Notifications from the service happen outside its topology lock.
No extra thread: the merged frame is built on the master producer's thread.

Testing

GTest: queue per-stream accounting + writev partial writes; service mux (index tagging, fairness, unknown name, duplicate name, comma rejection); client demux/gaps/facades; join (config merge, sample-and-hold, refusal cases).
E2E: a chain scenario with a multiplexed hub source (tcp_service + comma list) and one with a merged stream.

4.2 KiB Raw Blame History