Martino Ferrari
270 lines
13 KiB
Markdown
270 lines
13 KiB
Markdown
# System Architecture
|
|
|
|
## 1. Overview
|
|
|
|
The MARTe2 Debug Suite is a C++ server library that instruments running MARTe2 applications
|
|
without modifying their source code. It exposes an abstract `DebugServiceI` interface that
|
|
decouples the broker injection layer from the communication transport, allowing multiple
|
|
transport backends to coexist (currently TCP/UDP and HTTP/SSE).
|
|
|
|
```
|
|
┌────────────────────────────────────────────────────────────────────┐
|
|
│ MARTe2 Application │
|
|
│ ┌──────────┐ ┌───────────────────────┐ ┌──────────────────┐ │
|
|
│ │ GAM(s) │ │ DebugBrokerWrapper │ │ FastScheduler │ │
|
|
│ │ │◄──│ (Registry-patched) │ │ (unmodified) │ │
|
|
│ └──────────┘ └──────────┬────────────┘ └──────────────────┘ │
|
|
│ │ RT-path API │
|
|
└────────────────────────────┼───────────────────────────────────────┘
|
|
│
|
|
┌────────▼────────┐
|
|
│ DebugServiceI │ ← Abstract interface (singleton)
|
|
└────────┬────────┘
|
|
┌─────────────┴──────────────┐
|
|
│ │
|
|
┌────────▼────────┐ ┌─────────▼────────┐
|
|
│ DebugService │ │ WebDebugService │
|
|
│ TCP/UDP │ │ HTTP/SSE │
|
|
└────────┬────────┘ └─────────┬─────────┘
|
|
│ │
|
|
┌──────────┼──────────┐ ┌──────────┼──────────┐
|
|
│TCP 8080 │UDP 8081 │ │HTTP 8090 │SSE stream │
|
|
│(commands)│(telemetry│ │(commands)│(telemetry)│
|
|
└────┬─────┴────┬─────┘ └─────┬────┴────┬──────┘
|
|
│ │ │ │
|
|
┌────▼──────────▼─────┐ ┌─────────▼─────────▼──────┐
|
|
│ Rust/egui GUI │ │ Web Browser (built-in UI)│
|
|
│ (Tools/gui_client) │ │ http://localhost:8090 │
|
|
└─────────────────────┘ └───────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Core Mechanism: Registry Patching
|
|
|
|
The "Zero-Code-Change" requirement is met by intercepting the MARTe2 `ClassRegistryDatabase`.
|
|
When `DebugService` or `WebDebugService` initialises, `PatchRegistry()` replaces the
|
|
`ObjectBuilder` for every standard `MemoryMap*Broker` class in the registry. When
|
|
`RealTimeApplication::ConfigureApplication()` runs afterward it instantiates wrapped broker
|
|
objects instead of the originals — the application sees no difference.
|
|
|
|
Patched broker types:
|
|
|
|
| Class | Wrapper |
|
|
|---|---|
|
|
| `MemoryMapInputBroker` | `DebugBrokerWrapper<MemoryMapInputBroker>` |
|
|
| `MemoryMapOutputBroker` | `DebugBrokerWrapper<MemoryMapOutputBroker>` |
|
|
| `MemoryMapSynchronisedInputBroker` | `DebugBrokerWrapper<MemoryMapSynchronisedInputBroker>` |
|
|
| `MemoryMapSynchronisedOutputBroker` | `DebugBrokerWrapper<MemoryMapSynchronisedOutputBroker>` |
|
|
| `MemoryMapMultiBufferBroker` | `DebugBrokerWrapper<MemoryMapMultiBufferBroker>` |
|
|
| `MemoryMapMultiBufferOutputBroker` | `DebugBrokerWrapper<MemoryMapMultiBufferOutputBroker>` |
|
|
| `MemoryMapAsynchronousInputBroker` | `DebugBrokerWrapperAsync<MemoryMapAsynchronousInputBroker>` |
|
|
| `MemoryMapAsynchronousOutputBroker` | `DebugBrokerWrapperAsync<MemoryMapAsynchronousOutputBroker>` |
|
|
| `MemoryMapInterpolatedInputBroker` | `DebugBrokerWrapper<MemoryMapInterpolatedInputBroker>` |
|
|
| `MemoryMapStatefulOutputBroker` | `DebugBrokerWrapper<MemoryMapStatefulOutputBroker>` |
|
|
| `MemoryMapStatefulInputBroker` | `DebugBrokerWrapper<MemoryMapStatefulInputBroker>` |
|
|
|
|
---
|
|
|
|
## 3. DebugServiceI Abstraction Layer
|
|
|
|
`Source/Components/Interfaces/DebugService/DebugServiceI.h` defines the abstract singleton
|
|
interface. It has two logical groups:
|
|
|
|
### 3.1 RT-path API
|
|
Called from broker threads on every RT cycle. Must be as cheap as possible; the only
|
|
permissible synchronisation primitive is `FastPollingMutexSem`.
|
|
|
|
| Method | Purpose |
|
|
|---|---|
|
|
| `RegisterSignal(addr, type, name, …)` | Called once per signal during broker `Init()` |
|
|
| `ProcessSignal(info, size, timestamp)` | Apply forcing, check break, push trace sample |
|
|
| `RegisterBroker(…)` | Cache broker metadata for active/break index updates |
|
|
| `IsPaused() / SetPaused(bool)` | Query / set global pause state |
|
|
| `ConsumeStepIfNeeded(gamName, thread)` | Decrement step counter on output-broker Execute |
|
|
|
|
### 3.2 Control-path API
|
|
Called from server/command-handler threads; latency acceptable.
|
|
|
|
| Method | Purpose |
|
|
|---|---|
|
|
| `ForceSignal / UnforceSignal` | Persistent value injection |
|
|
| `TraceSignal(name, enable, decimation)` | Enable/disable ring-buffer tracing |
|
|
| `SetBreak / ClearBreak` | Conditional breakpoints |
|
|
| `IsInstrumented(path, …)` | Query whether a signal is reachable |
|
|
| `RegisterMonitorSignal / UnmonitorSignal` | Slow-rate signal polling |
|
|
|
|
### 3.3 Static singleton pattern
|
|
|
|
```cpp
|
|
// Concrete implementation registers itself:
|
|
DebugServiceI::SetInstance(this); // in Initialise()
|
|
DebugServiceI::SetInstance(NULL_PTR(…)); // in destructor
|
|
|
|
// Broker wrappers retrieve it (no ORD search on the hot path):
|
|
DebugServiceI *svc = DebugServiceI::GetInstance();
|
|
if (svc != NULL_PTR(DebugServiceI*)) { ... }
|
|
```
|
|
|
|
Shared data structures (`SignalAlias`, `BrokerInfo`) are also defined in `DebugServiceI.h`
|
|
so both transport implementations can use them without a circular dependency.
|
|
|
|
---
|
|
|
|
## 4. Transport Implementations
|
|
|
|
### 4.1 DebugService (TCP/UDP)
|
|
|
|
`Source/Components/Interfaces/DebugService/DebugService.h/.cpp`
|
|
|
|
Runs two `SingleThreadService` threads:
|
|
|
|
| Thread | Role |
|
|
|---|---|
|
|
| `Server()` | Accepts one TCP client at a time; reads text commands; writes JSON/text responses |
|
|
| `Streamer()` | Drains `TraceRingBuffer`; assembles and sends UDP datagrams; polls monitored signals |
|
|
|
|
Communication ports (all configurable):
|
|
|
|
| Port | Protocol | Purpose |
|
|
|---|---|---|
|
|
| `ControlPort` (default 8080) | TCP text | Commands and responses |
|
|
| `UdpPort` (default 8081) | UDP binary | High-speed telemetry (`TraceHeader` + samples) |
|
|
| `LogPort` (default 8082) | TCP stream | Log forwarding via `TcpLogger` sidecar |
|
|
|
|
### 4.2 WebDebugService (HTTP/SSE)
|
|
|
|
`Source/Components/Interfaces/WebDebugService/WebDebugService.h/.cpp`
|
|
|
|
Serves a single HTTP port (default 8090) with an embedded browser UI. Runs two
|
|
`SingleThreadService` threads:
|
|
|
|
| Thread | Role |
|
|
|---|---|
|
|
| `HttpServer()` | Accepts TCP connections; routes GET/POST/OPTIONS; upgrades SSE clients |
|
|
| `Streamer()` | Drains `TraceRingBuffer` → SSE events; polls monitored signals; heartbeat |
|
|
|
|
HTTP endpoints:
|
|
|
|
| Endpoint | Method | Purpose |
|
|
|---|---|---|
|
|
| `/` | GET | Serves embedded single-page application (HTML/CSS/JS) |
|
|
| `/api/command` | POST | Execute a debug command; returns text response |
|
|
| `/api/events` | GET | Server-Sent Events stream for real-time telemetry (up to 8 clients) |
|
|
|
|
The embedded web UI (`Source/Components/Interfaces/WebDebugService/WebUI.h`) is a
|
|
self-contained single-page application using Chart.js for real-time signal plots and a
|
|
Catppuccin Mocha dark theme. No install or server-side tooling required.
|
|
|
|
---
|
|
|
|
## 5. Signal Flow
|
|
|
|
### 5.1 Registration (startup, single-threaded)
|
|
|
|
```
|
|
ConfigureApplication()
|
|
└─► DebugBrokerWrapper::Init()
|
|
└─► DebugBrokerHelper::InitSignals()
|
|
├─► DebugServiceI::RegisterSignal() (canonical name + GAM alias)
|
|
└─► DebugServiceI::RegisterBroker()
|
|
```
|
|
|
|
### 5.2 RT loop (hot path, multi-threaded)
|
|
|
|
```
|
|
RealTimeThread::Execute()
|
|
└─► DebugBrokerWrapper::Execute()
|
|
├─► Base::Execute() (data moved in/out of GAM memory)
|
|
└─► DebugBrokerHelper::Process()
|
|
├─► For each active signal:
|
|
│ └─► DebugServiceI::ProcessSignal()
|
|
│ ├─► if isForcing: memcpy forcedValue → signal memory
|
|
│ ├─► if isTracing & decimation fires:
|
|
│ │ └─► tracePushMutex → TraceRingBuffer::Push()
|
|
│ └─► if breakOp set: evaluate condition → SetPaused(true)
|
|
└─► (output brokers only) DebugServiceI::ConsumeStepIfNeeded()
|
|
```
|
|
|
|
### 5.3 Telemetry delivery
|
|
|
|
**DebugService (UDP)**
|
|
```
|
|
Streamer thread
|
|
└─► TraceRingBuffer::Pop()
|
|
└─► assemble into streamerPacketBuffer (MTU=1400)
|
|
└─► BasicUDPSocket::Write() → client
|
|
```
|
|
|
|
**WebDebugService (SSE)**
|
|
```
|
|
Streamer thread
|
|
└─► TraceRingBuffer::Pop()
|
|
└─► BroadcastSse("message", {"type":"trace","name":…,"value":…})
|
|
└─► write "event: message\ndata: {…}\n\n" to each SSE socket
|
|
```
|
|
|
|
---
|
|
|
|
## 6. TraceRingBuffer
|
|
|
|
Single-producer / single-consumer circular byte buffer defined in `DebugCore.h`.
|
|
|
|
```
|
|
Buffer layout per entry: [ID:4][Timestamp:8][Size:4][Data:N]
|
|
```
|
|
|
|
- `Push()` is serialised by `tracePushMutex` (multiple RT threads write; one Streamer reads).
|
|
- `Pop()` is called exclusively by the Streamer thread.
|
|
- Ring is initialised to 4 MB in both transport implementations.
|
|
- On corrupt `size` field (`size >= bufferSize`): discards all entries to prevent cascading
|
|
bad reads (Fix #5).
|
|
|
|
---
|
|
|
|
## 7. Performance Hardening
|
|
|
|
The following fixes were applied after initial implementation to improve correctness and
|
|
real-time safety.
|
|
|
|
| Fix | Area | Change |
|
|
|---|---|---|
|
|
| **#1** | UDP streamer | Named constants `STREAMER_MTU = 1400` and `STREAMER_BUFFER_SIZE = 4096`; removed magic numbers |
|
|
| **#2** | Multi-producer ring | `tracePushMutex` serialises `TraceRingBuffer::Push()` from concurrent RT threads |
|
|
| **#3** | Break evaluation | Break indices copied to stack-local array while holding `activeMutex`; condition evaluated outside the lock |
|
|
| **#4** | Decimation counter | `Atomic::Add` used for lock-free per-signal decimation in multi-producer `ProcessSignal` |
|
|
| **#5** | Ring corruption | `Pop()` skips only the current corrupt entry; falls back to full discard only when `size >= bufferSize` |
|
|
| **#6** | TCP rate limit | Server disconnects clients sending > 100 commands/second (`CMD_RATE_LIMIT = 100`) |
|
|
| **#7** | Broker index swap | `Vec::Swap` inside spinlock followed by free outside; avoids holding the lock during deallocation |
|
|
| **#8** | Idle timeout | Active TCP client disconnected after 30 s of silence (`CLIENT_IDLE_TIMEOUT_MS = 30000`) |
|
|
| **#9** | VALUE output size | `GetSignalValue` caps output at 256 elements (`GET_VALUE_MAX_ELEMENTS`); includes `"Truncated"` flag |
|
|
| **#10** | TCP framing | `inputBuffer` accumulates partial commands across `Read()` calls; bounded at 8 KiB (`INPUT_BUFFER_MAX`) |
|
|
| **#11** | CDB cursor | `Initialise()` never calls `MoveToRoot()` on the CDB passed by the framework; avoids corrupting `ReferenceContainer` traversal |
|
|
|
|
---
|
|
|
|
## 8. Repository Layout
|
|
|
|
```
|
|
Source/
|
|
Core/Types/
|
|
Vec/ — RT-safe dynamic array (no STL)
|
|
Result/ — Error-carrying return type
|
|
Components/Interfaces/
|
|
DebugService/
|
|
DebugCore.h — DebugSignalInfo, TraceRingBuffer, BreakOp
|
|
DebugServiceI.h — Abstract interface + SignalAlias, BrokerInfo
|
|
DebugBrokerWrapper.h — DebugBrokerHelper, wrapper templates, registry builder
|
|
DebugService.h/.cpp — TCP/UDP transport implementation
|
|
WebDebugService/
|
|
WebUI.h — Embedded single-page HTML/CSS/JS application
|
|
WebDebugService.h/.cpp — HTTP/SSE transport implementation
|
|
TCPLogger/
|
|
TcpLogger.h/.cpp — LoggerConsumerI forwarding logs to TCP clients
|
|
Test/
|
|
Integration/ — Integration test suite (180 s SIGALRM guard)
|
|
UnitTests/ — Unit test suite
|
|
Tools/
|
|
gui_client/ — Rust/egui native GUI (DebugService TCP/UDP transport)
|
|
```
|