fix(udps): stop packets being dated from an earlier time base
Reported as samples sporadically carrying a previous packet's timestamp: holes on one side of the stream and collisions on the other, in both the Go and the MARTe2 receiver. That it appeared in both is what located it -- the shared cause is upstream of either client. Four independent defects, all of which end in a packet's values being placed at a time that is not theirs. Reassembly slot exhaustion (the "Reassembly slots full; evicting oldest" flood). Chunk size was learnt only from fragment 0, so an out-of-order burst destroyed a packet whose bytes had all arrived and left the slot occupied until the 2 s GC. Slots were keyed on the counter alone, but DATA and CONFIG number independently, so equal counters merged the two streams. The 32-byte received-mask covered 256 of the 512 fragments the client accepts, so a duplicate above 255 was counted as new and the packet was delivered with a hole of stale bytes in it. And one datagram was read per Execute(), which cannot drain a fast producer. Fixed with a pendingTail deferral, (counter, type) keying, a 64-byte mask, a 256-datagram drain, counter-age slot reclamation, and a 1 Hz aggregated warning in place of the per-eviction flood. UDPStreamer dropping whole Accumulate batches. EventSem::ResetWait is Reset-then-Wait, so a Post() landing while the sender thread was inside ServiceClients()/SendData() was destroyed by the next Reset. The batch was then skipped with dataReady false, readyFill was never cleared, and the following flush overwrote it: an entire run of RT cycles never reached the wire. The record of pending work now lives in the buffers rather than in the semaphore edge, which also removes up to UDPS_DATA_WAIT_MS of latency; genuine backpressure overwrites are counted and reported. Against the unfixed code the new test sees 2999/3000 batches never consumed. Period inflation after loss. Accumulated scalars carry no SamplingRate, so the receiver derives dt from the sender-clock gap -- but dividing it by the previous packet's sample count is only right while nothing is lost. One loss doubles the reported period, which spreads a batch a full batch past its own end and into the range the next packet claims. That is the hole and the collision, exactly. Inferring the cycle count from the estimate's own period is not a way out: it has a stable fixed point wherever gap/dt is an integer, so a real rate change locks it at the old one for good (AccumDtGTest.FollowsSustainedRateChange). The packet counter removes the ambiguity, so all three receivers now order on it: a DATA packet that does not advance the counter is dropped rather than delivered, because its values are older than data already handed over. Ordering is on the signed difference so it survives the uint32 wrap, and the sequence resets on reconnect, where the producer's counter restarts independently of ours. The loss count that falls out of the same delta feeds the period estimate as cycles = prevN * (1 + lost), which reduces exactly to gap/prevN when nothing is lost and therefore still tracks a genuine rate change. UDPSClient::AcceptDataCounter (C++), udpsprotocol.SequenceGate (Go), decode_data (C). The C client's existing gap counter was wrap-unsafe and let a stale packet rewind last_counter, which made every subsequent gap wrong; it uses the same code now. Docs/Protocol.md gains an Ordering DATA section stating the requirement for any receiver, including ones outside this repository. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Martino Ferrari
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
f334995865
commit
fbae7d712c
@@ -225,3 +225,8 @@ TEST(UDPStreamerGTest, TestExecute_MulticastConnectDataDisconnect) {
|
||||
UDPStreamerTest test;
|
||||
ASSERT_TRUE(test.TestExecute_MulticastConnectDataDisconnect());
|
||||
}
|
||||
|
||||
TEST(UDPStreamerGTest, TestAccumulate_EveryPublishedCycleReachesTheWire) {
|
||||
UDPStreamerTest test;
|
||||
ASSERT_TRUE(test.TestAccumulate_EveryPublishedCycleReachesTheWire());
|
||||
}
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
#include "RealTimeApplication.h"
|
||||
#include "Sleep.h"
|
||||
#include "StandardParser.h"
|
||||
#include "UDPSClient.h"
|
||||
#include "UDPStreamer.h"
|
||||
#include "UDPStreamerTest.h"
|
||||
|
||||
@@ -1845,3 +1846,298 @@ bool UDPStreamerTest::TestExecute_MulticastConnectDataDisconnect() {
|
||||
ObjectRegistryDatabase::Instance()->Purge();
|
||||
return ok;
|
||||
}
|
||||
|
||||
/*---------------------------------------------------------------------------*/
|
||||
/* Accumulate publication continuity */
|
||||
/*---------------------------------------------------------------------------*/
|
||||
|
||||
/* Four float64 scalars, no quantisation: 32 wire bytes per RT cycle.
|
||||
* With MaxPayloadSize = 60 the accumulate header (8 B HRT + 4 B count) leaves
|
||||
* room for exactly one cycle, so the size condition flushes on every single
|
||||
* Synchronise() — the maximum number of hand-offs to the sender thread, each
|
||||
* one a chance for a promoted batch to be skipped. */
|
||||
#define ACC_FUNCTIONS_BLOCK \
|
||||
" +Functions = {\n" \
|
||||
" Class = ReferenceContainer\n" \
|
||||
" +Writer = {\n" \
|
||||
" Class = UDPStreamerTestOutputGAM\n" \
|
||||
" OutputSignals = {\n" \
|
||||
" A = {\n" \
|
||||
" DataSource = Streamer\n" \
|
||||
" Type = float64\n" \
|
||||
" }\n" \
|
||||
" B = {\n" \
|
||||
" DataSource = Streamer\n" \
|
||||
" Type = float64\n" \
|
||||
" }\n" \
|
||||
" C = {\n" \
|
||||
" DataSource = Streamer\n" \
|
||||
" Type = float64\n" \
|
||||
" }\n" \
|
||||
" D = {\n" \
|
||||
" DataSource = Streamer\n" \
|
||||
" Type = float64\n" \
|
||||
" }\n" \
|
||||
" }\n" \
|
||||
" }\n" \
|
||||
" }\n"
|
||||
|
||||
static const MARTe::char8 *const ACC_CFG_CONTINUITY =
|
||||
"+Test = {\n"
|
||||
" Class = RealTimeApplication\n"
|
||||
ACC_FUNCTIONS_BLOCK
|
||||
" +Data = {\n"
|
||||
" Class = ReferenceContainer\n"
|
||||
" +Streamer = {\n"
|
||||
" Class = UDPStreamer\n"
|
||||
" Port = 44680\n"
|
||||
" MaxPayloadSize = 60\n"
|
||||
" PublishingMode = Accumulate\n"
|
||||
" MinRefreshRate = 1000.0\n"
|
||||
" Signals = {\n"
|
||||
" A = {\n"
|
||||
" Type = float64\n"
|
||||
" }\n"
|
||||
" B = {\n"
|
||||
" Type = float64\n"
|
||||
" }\n"
|
||||
" C = {\n"
|
||||
" Type = float64\n"
|
||||
" }\n"
|
||||
" D = {\n"
|
||||
" Type = float64\n"
|
||||
" }\n"
|
||||
" }\n"
|
||||
" }\n"
|
||||
HF_TAIL_BLOCK;
|
||||
|
||||
namespace {
|
||||
|
||||
/** Cycles driven by TestAccumulate_EveryPublishedCycleReachesTheWire. */
|
||||
static const MARTe::uint32 ACC_CONTINUITY_CYCLES = 3000u;
|
||||
|
||||
/**
|
||||
* @brief Records which RT cycles reached the wire, and how often.
|
||||
*
|
||||
* The test stamps signal A with the cycle index before every Synchronise(),
|
||||
* and the config is sized so each Accumulate batch carries exactly one cycle.
|
||||
* The payload is [8 B HRT][4 B numSamples][A][B][C][D], so A of the single
|
||||
* slot sits at offset 12 and identifies the cycle unambiguously.
|
||||
*
|
||||
* Counting distinct cycles (rather than summing numSamples) is what makes this
|
||||
* able to tell a lost publication from a re-sent one: a sender that never
|
||||
* consumes its ready buffer emits the right *number* of packets while
|
||||
* repeating a stale batch, which shows up here as duplicates plus missing
|
||||
* cycles instead of a clean tally.
|
||||
*/
|
||||
class AccumRampRecorder: public MARTe::UDPSClientListener {
|
||||
public:
|
||||
AccumRampRecorder() :
|
||||
packets(0u), duplicates(0u), malformed(0u) {
|
||||
mux.Create();
|
||||
for (MARTe::uint32 i = 0u; i < ACC_CONTINUITY_CYCLES; i++) {
|
||||
seen[i] = false;
|
||||
}
|
||||
}
|
||||
|
||||
virtual void OnUDPSData(const MARTe::uint8 *payload, MARTe::uint32 payloadSize) {
|
||||
MARTe::uint32 n = 0u;
|
||||
MARTe::float64 v = 0.0;
|
||||
if (payloadSize >= 20u) {
|
||||
(void) MARTe::MemoryOperationsHelper::Copy(&n, &payload[8], 4u);
|
||||
(void) MARTe::MemoryOperationsHelper::Copy(&v, &payload[12], 8u);
|
||||
}
|
||||
(void) mux.FastLock();
|
||||
packets++;
|
||||
if ((payloadSize < 20u) || (n != 1u)) {
|
||||
malformed++;
|
||||
}
|
||||
else {
|
||||
MARTe::uint32 idx = static_cast<MARTe::uint32>(v);
|
||||
if ((static_cast<MARTe::float64>(idx) != v) || (idx >= ACC_CONTINUITY_CYCLES)) {
|
||||
malformed++;
|
||||
}
|
||||
else if (seen[idx]) {
|
||||
duplicates++;
|
||||
}
|
||||
else {
|
||||
seen[idx] = true;
|
||||
}
|
||||
}
|
||||
mux.FastUnLock();
|
||||
}
|
||||
|
||||
MARTe::uint32 DistinctCycles() {
|
||||
(void) mux.FastLock();
|
||||
MARTe::uint32 n = 0u;
|
||||
for (MARTe::uint32 i = 0u; i < ACC_CONTINUITY_CYCLES; i++) {
|
||||
if (seen[i]) {
|
||||
n++;
|
||||
}
|
||||
}
|
||||
mux.FastUnLock();
|
||||
return n;
|
||||
}
|
||||
|
||||
MARTe::uint32 Packets() {
|
||||
(void) mux.FastLock();
|
||||
MARTe::uint32 n = packets;
|
||||
mux.FastUnLock();
|
||||
return n;
|
||||
}
|
||||
|
||||
MARTe::uint32 Duplicates() {
|
||||
(void) mux.FastLock();
|
||||
MARTe::uint32 n = duplicates;
|
||||
mux.FastUnLock();
|
||||
return n;
|
||||
}
|
||||
|
||||
MARTe::uint32 Malformed() {
|
||||
(void) mux.FastLock();
|
||||
MARTe::uint32 n = malformed;
|
||||
mux.FastUnLock();
|
||||
return n;
|
||||
}
|
||||
|
||||
private:
|
||||
MARTe::FastPollingMutexSem mux;
|
||||
bool seen[ACC_CONTINUITY_CYCLES];
|
||||
MARTe::uint32 packets;
|
||||
MARTe::uint32 duplicates;
|
||||
MARTe::uint32 malformed;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
bool UDPStreamerTest::TestAccumulate_EveryPublishedCycleReachesTheWire() {
|
||||
using namespace MARTe;
|
||||
|
||||
/* One-cycle batches every 200 us: ~5000 small packets/s, which the sender
|
||||
* thread handles comfortably. The period has to be this short because a
|
||||
* wake-up can only be swallowed while the sender is mid-send; at 1 ms the
|
||||
* sender is always back in its wait before the next Synchronise() and the
|
||||
* defect never fires at all. */
|
||||
const uint32 CYCLES = ACC_CONTINUITY_CYCLES;
|
||||
static const float64 CYCLE_SEC = 200e-6;
|
||||
|
||||
/* Tolerance, as a fraction of CYCLES, for cycles that never reach the wire.
|
||||
* It is not zero: this is an ordinary userspace thread on a general-purpose
|
||||
* kernel, so it can occasionally be descheduled past a 200 us slot, and the
|
||||
* last batch may still be in the accumulation buffer when the loop ends.
|
||||
* It is small because the defect this guards against is not marginal — a
|
||||
* sender that decides what to send from the semaphore edge fails to consume
|
||||
* essentially every batch (~100% here), so a 1% ceiling separates the two
|
||||
* regimes with three orders of magnitude to spare. */
|
||||
const uint32 MAX_LOST = CYCLES / 100u;
|
||||
|
||||
ReferenceT<RealTimeApplication> app = LoadApplication(ACC_CFG_CONTINUITY);
|
||||
bool ok = app.IsValid();
|
||||
if (ok) {
|
||||
ok = (app->PrepareNextState("State1") == ErrorManagement::NoError);
|
||||
}
|
||||
Sleep::MSec(50u);
|
||||
|
||||
AccumRampRecorder counter;
|
||||
UDPSClient client;
|
||||
ReferenceT<UDPStreamer> ds;
|
||||
if (ok) {
|
||||
ConfigurationDatabase clientCfg;
|
||||
ok = clientCfg.Write("ServerAddr", "127.0.0.1");
|
||||
ok = ok && clientCfg.Write("Port", 44680u);
|
||||
ok = ok && clientCfg.Write("SilenceTimeout", 0.0f);
|
||||
ok = ok && clientCfg.Write("KeepAliveInterval", 0u);
|
||||
client.SetListener(&counter);
|
||||
ok = ok && client.Initialise(clientCfg);
|
||||
ok = ok && client.Start();
|
||||
}
|
||||
|
||||
/* Wait for the CONNECT to register on the streamer side. */
|
||||
if (ok) {
|
||||
ds = ObjectRegistryDatabase::Instance()->Find("Test.Data.Streamer");
|
||||
ok = ds.IsValid();
|
||||
}
|
||||
if (ok) {
|
||||
uint32 waited = 0u;
|
||||
while ((waited < 3000u) && !ds->IsClientConnected()) {
|
||||
Sleep::MSec(20u);
|
||||
waited += 20u;
|
||||
}
|
||||
ok = ds->IsClientConnected();
|
||||
}
|
||||
|
||||
/* Signal A carries the cycle index, so every packet identifies exactly
|
||||
* which RT cycle produced it. Synchronise() snapshots the DataSource
|
||||
* memory, so writing straight into it is equivalent to a GAM having
|
||||
* produced the value. */
|
||||
float64 *sigA = NULL_PTR(float64 *);
|
||||
if (ok) {
|
||||
void *addr = NULL_PTR(void *);
|
||||
ok = ds->GetSignalMemoryBuffer(0u, 0u, addr);
|
||||
sigA = reinterpret_cast<float64 *>(addr);
|
||||
ok = ok && (sigA != NULL_PTR(float64 *));
|
||||
}
|
||||
|
||||
/* Drive the RT cycles. */
|
||||
if (ok) {
|
||||
for (uint32 i = 0u; (i < CYCLES) && ok; i++) {
|
||||
*sigA = static_cast<float64>(i);
|
||||
ok = ds->Synchronise();
|
||||
Sleep::Sec(CYCLE_SEC);
|
||||
}
|
||||
}
|
||||
|
||||
/* Let the last packets drain. */
|
||||
Sleep::MSec(300u);
|
||||
|
||||
uint32 distinct = counter.DistinctCycles();
|
||||
uint32 packets = counter.Packets();
|
||||
uint32 duplicates = counter.Duplicates();
|
||||
uint32 malformed = counter.Malformed();
|
||||
uint32 dropped = (ds.IsValid()) ? ds->GetDroppedPublications() : 0u;
|
||||
|
||||
if (ok) {
|
||||
ok = (malformed == 0u);
|
||||
if (!ok) {
|
||||
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
|
||||
"%u of %u DATA packets did not carry exactly one "
|
||||
"decodable cycle index.", malformed, packets);
|
||||
}
|
||||
}
|
||||
if (ok) {
|
||||
/* A cycle that never arrives is a hole in the consumer's time series. */
|
||||
ok = (distinct + MAX_LOST) >= CYCLES;
|
||||
if (!ok) {
|
||||
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
|
||||
"Accumulate lost cycles: %u of %u reached the wire "
|
||||
"in %u packets (%u duplicates, %u publications "
|
||||
"overwritten before being sent).",
|
||||
distinct, CYCLES, packets, duplicates, dropped);
|
||||
}
|
||||
}
|
||||
if (ok) {
|
||||
/* A cycle that arrives twice means the sender re-sent a ready buffer it
|
||||
* had already transmitted, which lands the same samples on the receiver
|
||||
* under two different time bases. */
|
||||
ok = (duplicates == 0u);
|
||||
if (!ok) {
|
||||
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
|
||||
"%u of %u DATA packets repeated a cycle already sent.",
|
||||
duplicates, packets);
|
||||
}
|
||||
}
|
||||
if (ok) {
|
||||
/* Same ceiling from the producer's side: it sees the overwrite directly
|
||||
* and does not depend on the packet reaching the loopback socket. */
|
||||
ok = (dropped <= MAX_LOST);
|
||||
if (!ok) {
|
||||
REPORT_ERROR_STATIC(ErrorManagement::FatalError,
|
||||
"%u of %u publications were overwritten before the "
|
||||
"sender thread took them.", dropped, CYCLES);
|
||||
}
|
||||
}
|
||||
|
||||
(void) client.Stop();
|
||||
ObjectRegistryDatabase::Instance()->Purge();
|
||||
return ok;
|
||||
}
|
||||
|
||||
@@ -234,6 +234,16 @@ public:
|
||||
* @brief Tests full TCP CONNECT → CONFIG → DATA via multicast → DISCONNECT on loopback.
|
||||
*/
|
||||
bool TestExecute_MulticastConnectDataDisconnect();
|
||||
|
||||
/**
|
||||
* @brief Tests that Accumulate publishes every RT cycle it batches.
|
||||
* @details Drives 600 cycles at a rate the sender thread trivially keeps up
|
||||
* with, and sums the numSamples field of every DATA packet that arrives.
|
||||
* A batch promoted to the ready buffer but never sent — because the wake-up
|
||||
* announcing it was swallowed — shows up here as missing cycles, which a
|
||||
* consumer sees as a hole in the time series.
|
||||
*/
|
||||
bool TestAccumulate_EveryPublishedCycleReachesTheWire();
|
||||
};
|
||||
|
||||
#endif /* UDPSTREAMERTEST_H_ */
|
||||
|
||||
Reference in New Issue
Block a user