739 lines
15 KiB
Markdown
739 lines
15 KiB
Markdown
# MARTe Benchmark Suite
|
|
|
|
## Overview
|
|
|
|
`marte-benchmark-suite` is a validation and benchmarking infrastructure
|
|
developed around the MARTe2 real-time framework.
|
|
|
|
The project is designed to build a sequence of progressively more advanced
|
|
test scenarios while preserving known-good baselines from the previous stages.
|
|
|
|
The suite does not modify MARTe2 itself.
|
|
|
|
Instead, it provides:
|
|
|
|
- MARTe configuration files;
|
|
- launchers;
|
|
- configuration patchers;
|
|
- runtime orchestration;
|
|
- output parsers;
|
|
- structural validators;
|
|
- semantic validators;
|
|
- execution evidence.
|
|
|
|
The development philosophy is incremental:
|
|
|
|
> each new stage introduces one new capability while preserving the validated
|
|
> behaviour of the previous stages.
|
|
|
|
---
|
|
|
|
# Current project status
|
|
|
|
Current validated development baseline:
|
|
|
|
```text
|
|
S03-H3
|
|
STATUS=CLOSED
|
|
BASELINE=KNOWN-GOOD
|
|
```
|
|
|
|
S03-H3 is currently the authoritative baseline on which the next stages of the
|
|
benchmark suite are intended to be built.
|
|
|
|
The next planned stages are:
|
|
|
|
```text
|
|
S04 → Repeatability / Multi-Run
|
|
S05 → Deterministic Parametric Generator
|
|
S06 → MARTe2 Core Scaling
|
|
```
|
|
|
|
These stages are not part of the current S03-H3 baseline unless explicitly
|
|
implemented and accepted later.
|
|
|
|
---
|
|
|
|
# Project purpose
|
|
|
|
The suite is intended to answer increasingly complex questions about a MARTe2
|
|
application.
|
|
|
|
Conceptually:
|
|
|
|
```text
|
|
S00
|
|
Can MARTe2 execute the basic reference application?
|
|
↓
|
|
S01
|
|
Can execution timing information be observed?
|
|
↓
|
|
S02
|
|
Can deterministic binary data be recorded?
|
|
↓
|
|
S03
|
|
Can the recorded data be checked semantically?
|
|
↓
|
|
S03-H1
|
|
Can different execution/debug profiles be supported?
|
|
↓
|
|
S03-H3
|
|
Can those profiles be formalized and validated
|
|
against an authoritative runtime contract?
|
|
↓
|
|
S04
|
|
Does the same valid workload remain correct
|
|
when executed repeatedly?
|
|
↓
|
|
S05
|
|
Can deterministic test workloads be generated
|
|
from parameters?
|
|
↓
|
|
S06
|
|
How does MARTe2 Core behave as workload complexity grows?
|
|
```
|
|
|
|
---
|
|
|
|
# High-level architecture
|
|
|
|
The benchmark suite is layered around the installed MARTe2 runtime.
|
|
|
|
```text
|
|
+------------------+
|
|
| MARTe2 |
|
|
| MARTeApp.ex |
|
|
+---------+--------+
|
|
^
|
|
|
|
|
common/marte_runner.sh
|
|
^
|
|
|
|
|
launcher/
|
|
^
|
|
|
|
|
scenario / configuration
|
|
|
|
|
v
|
|
MARTe execution
|
|
|
|
|
v
|
|
RUN DIRECTORY
|
|
|
|
|
+------------+-------------+
|
|
| |
|
|
v v
|
|
runtime logs output files
|
|
| |
|
|
+------------+-------------+
|
|
|
|
|
v
|
|
parser / validator
|
|
|
|
|
v
|
|
PASS / FAIL result
|
|
```
|
|
|
|
MARTe2 and MARTe2-components remain external dependencies.
|
|
|
|
The benchmark suite orchestrates and validates them but is intentionally kept
|
|
separate from their source trees.
|
|
|
|
---
|
|
|
|
# Repository structure
|
|
|
|
Current source structure:
|
|
|
|
```text
|
|
marte-benchmark-suite/
|
|
│
|
|
├── common/
|
|
│ ├── environment.sh
|
|
│ ├── marte_runner.sh
|
|
│ └── paths.sh
|
|
│
|
|
├── configurations/
|
|
│ └── generated/
|
|
│ ├── S00/
|
|
│ ├── S01/
|
|
│ ├── S02/
|
|
│ ├── S03/
|
|
│ ├── S03H1/
|
|
│ └── S03H3/
|
|
│
|
|
├── launcher/
|
|
│ ├── marte_benchmark_launcher.sh
|
|
│ ├── marte_benchmark_launcher_s03h1.sh
|
|
│ ├── marte_benchmark_launcher_s03h3.sh
|
|
│ └── marte_benchmark_menu.sh
|
|
│
|
|
├── patchers/
|
|
│ └── configuration/
|
|
│
|
|
├── scenarios/
|
|
│ ├── S01_timing_display/
|
|
│ ├── S02_binary_recording/
|
|
│ ├── S03_semantic_sentinels/
|
|
│ ├── S03H1_execution_profiles/
|
|
│ └── S03H3_execution_profiles/
|
|
│
|
|
└── runs/
|
|
```
|
|
|
|
`runs/` contains execution outputs and is intentionally excluded from normal
|
|
Git version control.
|
|
|
|
The configurations under `configurations/generated/` are NOT generic build
|
|
artifacts. They represent configuration baselines produced by controlled
|
|
project stages and are therefore versioned.
|
|
|
|
---
|
|
|
|
# Common runtime layer
|
|
|
|
## `common/paths.sh`
|
|
|
|
Centralizes the paths used by the benchmark infrastructure.
|
|
|
|
Its purpose is to avoid hard-coding the same MARTe and project locations in
|
|
multiple scripts.
|
|
|
|
---
|
|
|
|
## `common/environment.sh`
|
|
|
|
Defines the common execution environment used by the benchmark scripts.
|
|
|
|
Keeping environment handling in one place improves reproducibility and avoids
|
|
different scenarios silently using different runtime assumptions.
|
|
|
|
---
|
|
|
|
## `common/marte_runner.sh`
|
|
|
|
Provides the common MARTe execution mechanism shared by benchmark stages.
|
|
|
|
Conceptually:
|
|
|
|
```text
|
|
launcher
|
|
↓
|
|
common runner
|
|
↓
|
|
MARTeApp.ex
|
|
↓
|
|
run directory
|
|
↓
|
|
logs / outputs
|
|
```
|
|
|
|
Higher-level stages should reuse the authoritative runner rather than create
|
|
alternative MARTe execution paths unless a future specification explicitly
|
|
authorizes such a change.
|
|
|
|
---
|
|
|
|
# Launchers
|
|
|
|
The `launcher/` directory contains the user-facing execution layer.
|
|
|
|
The project currently includes launchers from different development stages,
|
|
including the S03-H1 and S03-H3 generations.
|
|
|
|
The current S03-H3 architecture supports execution profiles such as:
|
|
|
|
```text
|
|
MINIMAL
|
|
TEST
|
|
DIAGNOSTIC
|
|
```
|
|
|
|
The profiles allow the same benchmark infrastructure to expose different
|
|
levels of runtime/debug functionality without changing the fundamental
|
|
correctness contract.
|
|
|
|
---
|
|
|
|
# Patchers
|
|
|
|
Configuration patchers live under:
|
|
|
|
```text
|
|
patchers/configuration/
|
|
```
|
|
|
|
They transform an already-known configuration into the configuration required
|
|
by the next development stage.
|
|
|
|
The dependency chain currently visible in the repository is:
|
|
|
|
```text
|
|
S00
|
|
↓
|
|
build_s01_from_s00.sh
|
|
↓
|
|
S01
|
|
↓
|
|
build_s02_from_s01.sh
|
|
↓
|
|
S02
|
|
↓
|
|
build_s03_from_s02.sh
|
|
↓
|
|
S03
|
|
```
|
|
|
|
Additional controlled transformations support S03-H1 and S03-H3.
|
|
|
|
This model helps preserve provenance:
|
|
|
|
```text
|
|
new configuration
|
|
=
|
|
known parent
|
|
+
|
|
explicit transformation
|
|
```
|
|
|
|
rather than independently rewriting every configuration from scratch.
|
|
|
|
---
|
|
|
|
# Validation layer
|
|
|
|
The suite does not consider a MARTe process exit by itself sufficient proof
|
|
that a test succeeded.
|
|
|
|
Validation is layered.
|
|
|
|
Depending on the stage, the suite can inspect:
|
|
|
|
```text
|
|
configuration structure
|
|
+
|
|
runtime execution
|
|
+
|
|
generated files
|
|
+
|
|
semantic content
|
|
+
|
|
expected sentinels
|
|
```
|
|
|
|
and only then derive the corresponding result.
|
|
|
|
---
|
|
|
|
# S00 — Core reference
|
|
|
|
S00 provides the basic MARTe2 reference configuration:
|
|
|
|
```text
|
|
configurations/generated/S00/smoke_s0_core.marte
|
|
```
|
|
|
|
It acts as the root from which later benchmark configurations are progressively
|
|
derived.
|
|
|
|
---
|
|
|
|
# S01 — Timing
|
|
|
|
S01 introduces timing-related observation on top of the S00 reference
|
|
configuration.
|
|
|
|
Relevant current files include:
|
|
|
|
```text
|
|
patchers/configuration/build_s01_from_s00.sh
|
|
configurations/generated/S01/from_patcher/
|
|
scenarios/S01_timing_display/validate.sh
|
|
```
|
|
|
|
S01 established infrastructure that later stages could reuse rather than
|
|
reimplement.
|
|
|
|
---
|
|
|
|
# S02 — Binary recording
|
|
|
|
S02 extends the previous stage with binary data recording.
|
|
|
|
Relevant components include:
|
|
|
|
```text
|
|
patchers/configuration/build_s02_from_s01.sh
|
|
|
|
configurations/generated/S02/from_patcher/
|
|
smoke_s2_binary_recording_from_s01.marte
|
|
|
|
scenarios/S02_binary_recording/validate.sh
|
|
```
|
|
|
|
Binary recording provides machine-readable data that later stages can parse
|
|
and validate.
|
|
|
|
---
|
|
|
|
# S03 — Semantic validation
|
|
|
|
S03 adds semantic verification of the generated data.
|
|
|
|
Relevant components include:
|
|
|
|
```text
|
|
patchers/configuration/build_s03_from_s02.sh
|
|
|
|
configurations/generated/S03/from_patcher/
|
|
smoke_s3_semantic_sentinels_from_s02.marte
|
|
|
|
scenarios/S03_semantic_sentinels/
|
|
parse_s03_binary.py
|
|
validate.sh
|
|
```
|
|
|
|
The key conceptual change is:
|
|
|
|
```text
|
|
"data file exists"
|
|
```
|
|
|
|
is not enough.
|
|
|
|
The project also checks whether the data means what the test expected it to
|
|
mean.
|
|
|
|
---
|
|
|
|
# S03-H1 — Execution profiles
|
|
|
|
S03-H1 introduced explicit execution profiles and supporting validation
|
|
infrastructure.
|
|
|
|
Current repository elements include:
|
|
|
|
```text
|
|
launcher/marte_benchmark_launcher_s03h1.sh
|
|
|
|
patchers/configuration/
|
|
build_s03h1_from_s02.sh
|
|
|
|
scenarios/S03H1_execution_profiles/
|
|
export_s03_csv.py
|
|
validate.sh
|
|
verify_s03h1_config.py
|
|
```
|
|
|
|
The historical S03-H1 configurations currently retained are:
|
|
|
|
```text
|
|
TEST
|
|
DIAGNOSTIC
|
|
```
|
|
|
|
They remain important because S03-H3 intentionally reuses validated historical
|
|
configurations where appropriate instead of duplicating them unnecessarily.
|
|
|
|
---
|
|
|
|
# S03-H3 — Current known-good baseline
|
|
|
|
S03-H3 is the current closed benchmark stage.
|
|
|
|
Its role is to formalize the execution-profile architecture and protect the
|
|
runtime contract used by future development.
|
|
|
|
Relevant repository elements include:
|
|
|
|
```text
|
|
launcher/
|
|
marte_benchmark_launcher_s03h3.sh
|
|
marte_benchmark_menu.sh
|
|
|
|
patchers/configuration/
|
|
build_s03h3_minimal_from_s03h1_test.sh
|
|
|
|
configurations/generated/S03H3/from_patcher/
|
|
smoke_s03h3_execution_profiles_minimal_from_s03h1_test.marte
|
|
|
|
scenarios/S03H3_execution_profiles/
|
|
validate.sh
|
|
verify_s03h3_minimal_structure.py
|
|
```
|
|
|
|
S03-H3 defines three go-forward execution profiles:
|
|
|
|
```text
|
|
MINIMAL
|
|
TEST
|
|
DIAGNOSTIC
|
|
```
|
|
|
|
The MINIMAL configuration is derived from the validated S03-H1 TEST
|
|
configuration by explicitly removing unnecessary display functionality while
|
|
preserving the required execution behaviour.
|
|
|
|
TEST and DIAGNOSTIC reuse their validated historical configurations.
|
|
|
|
This avoids unnecessary duplication and reduces the number of independent
|
|
configuration variants that must be trusted.
|
|
|
|
---
|
|
|
|
# Why S03-H3 matters
|
|
|
|
Before advancing toward larger benchmark campaigns, the project needs one
|
|
well-understood baseline.
|
|
|
|
S03-H3 provides that reference point.
|
|
|
|
Conceptually:
|
|
|
|
```text
|
|
KNOWN INPUT
|
|
↓
|
|
KNOWN CONFIGURATION
|
|
↓
|
|
KNOWN LAUNCHER
|
|
↓
|
|
KNOWN RUNTIME DEPENDENCIES
|
|
↓
|
|
KNOWN VALIDATION
|
|
↓
|
|
KNOWN-GOOD RESULT
|
|
```
|
|
|
|
Future stages should extend this baseline rather than silently redefine it.
|
|
|
|
---
|
|
|
|
# Runtime dependencies
|
|
|
|
The benchmark suite depends on an external MARTe installation.
|
|
|
|
The current runtime environment includes:
|
|
|
|
```text
|
|
MARTe2
|
|
MARTe2-components
|
|
```
|
|
|
|
Important runtime objects include MARTe executables/libraries and component
|
|
libraries required by the active configurations.
|
|
|
|
These external projects are not copied into this repository.
|
|
|
|
This repository therefore acts as:
|
|
|
|
```text
|
|
benchmark source + configuration + validation infrastructure
|
|
```
|
|
|
|
and not as a complete binary backup of MARTe.
|
|
|
|
A separate verified restore mechanism is required for complete disaster
|
|
recovery of the known-good runtime environment.
|
|
|
|
---
|
|
|
|
# Run directories
|
|
|
|
Runtime execution artifacts are written under:
|
|
|
|
```text
|
|
runs/
|
|
```
|
|
|
|
A run directory may contain logs, generated data and validation evidence.
|
|
|
|
These files can be large and are produced repeatedly.
|
|
|
|
For this reason:
|
|
|
|
```text
|
|
runs/
|
|
```
|
|
|
|
is intentionally excluded through `.gitignore`.
|
|
|
|
Git is used for source/configuration history.
|
|
|
|
Runtime evidence and full environment recovery are handled separately.
|
|
|
|
---
|
|
|
|
# Git versus full restore backup
|
|
|
|
The Git repository and the full restore point serve different purposes.
|
|
|
|
```text
|
|
Git
|
|
│
|
|
├── scripts
|
|
├── configurations
|
|
├── validators
|
|
├── parsers
|
|
├── documentation
|
|
└── development history
|
|
```
|
|
|
|
while a complete restore point additionally needs the external runtime
|
|
environment required to reproduce the validated MARTe execution.
|
|
|
|
Therefore:
|
|
|
|
```text
|
|
Git backup ≠ complete runtime restore point
|
|
```
|
|
|
|
Both mechanisms are complementary.
|
|
|
|
---
|
|
|
|
# Development governance
|
|
|
|
The project follows a controlled development process.
|
|
|
|
A typical new stage follows approximately:
|
|
|
|
```text
|
|
DESIGN / SPECIFICATION
|
|
↓
|
|
independent technical review
|
|
↓
|
|
technical arbitration
|
|
↓
|
|
human freeze
|
|
↓
|
|
implementation
|
|
↓
|
|
tests
|
|
↓
|
|
independent implementation review
|
|
↓
|
|
technical arbitration
|
|
↓
|
|
human acceptance
|
|
↓
|
|
new known-good baseline
|
|
```
|
|
|
|
A development-stage implementation is therefore not automatically considered
|
|
part of the authoritative baseline simply because code exists.
|
|
|
|
---
|
|
|
|
# Planned roadmap
|
|
|
|
## S04 — Repeatability / Multi-Run
|
|
|
|
Main question:
|
|
|
|
> Does the same already-valid S03-H3 workload remain correct when executed
|
|
> repeatedly?
|
|
|
|
Conceptually:
|
|
|
|
```text
|
|
S03-H3 run #1 → result
|
|
S03-H3 run #2 → result
|
|
S03-H3 run #3 → result
|
|
...
|
|
S03-H3 run #N → result
|
|
↓
|
|
campaign-level repeatability result
|
|
```
|
|
|
|
S04 is primarily a correctness/repeatability stage.
|
|
|
|
It is not intended to turn repeatability percentages or CycleTime observations
|
|
into performance claims.
|
|
|
|
---
|
|
|
|
## S05 — Deterministic Parametric Generator
|
|
|
|
Main question:
|
|
|
|
> Can test workloads be generated automatically from controlled parameters
|
|
> while remaining reproducible?
|
|
|
|
Conceptually:
|
|
|
|
```text
|
|
parameters
|
|
+
|
|
seed
|
|
↓
|
|
generator
|
|
↓
|
|
deterministic MARTe scenario
|
|
```
|
|
|
|
The same inputs should regenerate the same intended scenario.
|
|
|
|
---
|
|
|
|
## S06 — MARTe2 Core Scaling
|
|
|
|
Main question:
|
|
|
|
> How does the MARTe2 Core workload behave as controlled complexity increases?
|
|
|
|
S06 is expected to reuse the capabilities established by the preceding stages:
|
|
|
|
```text
|
|
S03-H3 → correctness baseline
|
|
S04 → repeatable execution
|
|
S05 → deterministic workload generation
|
|
S06 → controlled scaling study
|
|
```
|
|
|
|
---
|
|
|
|
# Current boundary
|
|
|
|
At the time of this README baseline:
|
|
|
|
```text
|
|
S03-H3 = CLOSED / KNOWN-GOOD
|
|
|
|
S04 = next development target
|
|
S05 = planned
|
|
S06 = planned
|
|
```
|
|
|
|
The existence of this roadmap must not be interpreted as evidence that S04,
|
|
S05 or S06 have already been implemented or validated.
|
|
|
|
---
|
|
|
|
# Repository philosophy
|
|
|
|
The repository should remain:
|
|
|
|
```text
|
|
small
|
|
reviewable
|
|
deterministic
|
|
traceable
|
|
reproducible
|
|
```
|
|
|
|
Generated runtime outputs should not be mixed with source history.
|
|
|
|
Known-good configurations, however, may intentionally be versioned even when
|
|
they were produced by controlled patchers, because they are part of the
|
|
validated benchmark baseline.
|
|
|
|
---
|
|
|
|
# License and distribution
|
|
|
|
The licensing and distribution policy for this benchmark suite must be defined
|
|
by the project owners before public redistribution.
|
|
|
|
MARTe2 and MARTe2-components retain their respective upstream licensing and
|
|
ownership terms.
|