# MARTe Benchmark Suite ## Overview `marte-benchmark-suite` is a validation and benchmarking infrastructure developed around the MARTe2 real-time framework. The project is designed to build a sequence of progressively more advanced test scenarios while preserving known-good baselines from the previous stages. The suite does not modify MARTe2 itself. Instead, it provides: - MARTe configuration files; - launchers; - configuration patchers; - runtime orchestration; - output parsers; - structural validators; - semantic validators; - execution evidence. The development philosophy is incremental: > each new stage introduces one new capability while preserving the validated > behaviour of the previous stages. --- # Current project status Current validated development baseline: ```text S03-H3 STATUS=CLOSED BASELINE=KNOWN-GOOD ``` S03-H3 is currently the authoritative baseline on which the next stages of the benchmark suite are intended to be built. The next planned stages are: ```text S04 → Repeatability / Multi-Run S05 → Deterministic Parametric Generator S06 → MARTe2 Core Scaling ``` These stages are not part of the current S03-H3 baseline unless explicitly implemented and accepted later. --- # Project purpose The suite is intended to answer increasingly complex questions about a MARTe2 application. Conceptually: ```text S00 Can MARTe2 execute the basic reference application? ↓ S01 Can execution timing information be observed? ↓ S02 Can deterministic binary data be recorded? ↓ S03 Can the recorded data be checked semantically? ↓ S03-H1 Can different execution/debug profiles be supported? ↓ S03-H3 Can those profiles be formalized and validated against an authoritative runtime contract? ↓ S04 Does the same valid workload remain correct when executed repeatedly? ↓ S05 Can deterministic test workloads be generated from parameters? ↓ S06 How does MARTe2 Core behave as workload complexity grows? ``` --- # High-level architecture The benchmark suite is layered around the installed MARTe2 runtime. ```text +------------------+ | MARTe2 | | MARTeApp.ex | +---------+--------+ ^ | common/marte_runner.sh ^ | launcher/ ^ | scenario / configuration | v MARTe execution | v RUN DIRECTORY | +------------+-------------+ | | v v runtime logs output files | | +------------+-------------+ | v parser / validator | v PASS / FAIL result ``` MARTe2 and MARTe2-components remain external dependencies. The benchmark suite orchestrates and validates them but is intentionally kept separate from their source trees. --- # Repository structure Current source structure: ```text marte-benchmark-suite/ │ ├── common/ │ ├── environment.sh │ ├── marte_runner.sh │ └── paths.sh │ ├── configurations/ │ └── generated/ │ ├── S00/ │ ├── S01/ │ ├── S02/ │ ├── S03/ │ ├── S03H1/ │ └── S03H3/ │ ├── launcher/ │ ├── marte_benchmark_launcher.sh │ ├── marte_benchmark_launcher_s03h1.sh │ ├── marte_benchmark_launcher_s03h3.sh │ └── marte_benchmark_menu.sh │ ├── patchers/ │ └── configuration/ │ ├── scenarios/ │ ├── S01_timing_display/ │ ├── S02_binary_recording/ │ ├── S03_semantic_sentinels/ │ ├── S03H1_execution_profiles/ │ └── S03H3_execution_profiles/ │ └── runs/ ``` `runs/` contains execution outputs and is intentionally excluded from normal Git version control. The configurations under `configurations/generated/` are NOT generic build artifacts. They represent configuration baselines produced by controlled project stages and are therefore versioned. --- # Common runtime layer ## `common/paths.sh` Centralizes the paths used by the benchmark infrastructure. Its purpose is to avoid hard-coding the same MARTe and project locations in multiple scripts. --- ## `common/environment.sh` Defines the common execution environment used by the benchmark scripts. Keeping environment handling in one place improves reproducibility and avoids different scenarios silently using different runtime assumptions. --- ## `common/marte_runner.sh` Provides the common MARTe execution mechanism shared by benchmark stages. Conceptually: ```text launcher ↓ common runner ↓ MARTeApp.ex ↓ run directory ↓ logs / outputs ``` Higher-level stages should reuse the authoritative runner rather than create alternative MARTe execution paths unless a future specification explicitly authorizes such a change. --- # Launchers The `launcher/` directory contains the user-facing execution layer. The project currently includes launchers from different development stages, including the S03-H1 and S03-H3 generations. The current S03-H3 architecture supports execution profiles such as: ```text MINIMAL TEST DIAGNOSTIC ``` The profiles allow the same benchmark infrastructure to expose different levels of runtime/debug functionality without changing the fundamental correctness contract. --- # Patchers Configuration patchers live under: ```text patchers/configuration/ ``` They transform an already-known configuration into the configuration required by the next development stage. The dependency chain currently visible in the repository is: ```text S00 ↓ build_s01_from_s00.sh ↓ S01 ↓ build_s02_from_s01.sh ↓ S02 ↓ build_s03_from_s02.sh ↓ S03 ``` Additional controlled transformations support S03-H1 and S03-H3. This model helps preserve provenance: ```text new configuration = known parent + explicit transformation ``` rather than independently rewriting every configuration from scratch. --- # Validation layer The suite does not consider a MARTe process exit by itself sufficient proof that a test succeeded. Validation is layered. Depending on the stage, the suite can inspect: ```text configuration structure + runtime execution + generated files + semantic content + expected sentinels ``` and only then derive the corresponding result. --- # S00 — Core reference S00 provides the basic MARTe2 reference configuration: ```text configurations/generated/S00/smoke_s0_core.marte ``` It acts as the root from which later benchmark configurations are progressively derived. --- # S01 — Timing S01 introduces timing-related observation on top of the S00 reference configuration. Relevant current files include: ```text patchers/configuration/build_s01_from_s00.sh configurations/generated/S01/from_patcher/ scenarios/S01_timing_display/validate.sh ``` S01 established infrastructure that later stages could reuse rather than reimplement. --- # S02 — Binary recording S02 extends the previous stage with binary data recording. Relevant components include: ```text patchers/configuration/build_s02_from_s01.sh configurations/generated/S02/from_patcher/ smoke_s2_binary_recording_from_s01.marte scenarios/S02_binary_recording/validate.sh ``` Binary recording provides machine-readable data that later stages can parse and validate. --- # S03 — Semantic validation S03 adds semantic verification of the generated data. Relevant components include: ```text patchers/configuration/build_s03_from_s02.sh configurations/generated/S03/from_patcher/ smoke_s3_semantic_sentinels_from_s02.marte scenarios/S03_semantic_sentinels/ parse_s03_binary.py validate.sh ``` The key conceptual change is: ```text "data file exists" ``` is not enough. The project also checks whether the data means what the test expected it to mean. --- # S03-H1 — Execution profiles S03-H1 introduced explicit execution profiles and supporting validation infrastructure. Current repository elements include: ```text launcher/marte_benchmark_launcher_s03h1.sh patchers/configuration/ build_s03h1_from_s02.sh scenarios/S03H1_execution_profiles/ export_s03_csv.py validate.sh verify_s03h1_config.py ``` The historical S03-H1 configurations currently retained are: ```text TEST DIAGNOSTIC ``` They remain important because S03-H3 intentionally reuses validated historical configurations where appropriate instead of duplicating them unnecessarily. --- # S03-H3 — Current known-good baseline S03-H3 is the current closed benchmark stage. Its role is to formalize the execution-profile architecture and protect the runtime contract used by future development. Relevant repository elements include: ```text launcher/ marte_benchmark_launcher_s03h3.sh marte_benchmark_menu.sh patchers/configuration/ build_s03h3_minimal_from_s03h1_test.sh configurations/generated/S03H3/from_patcher/ smoke_s03h3_execution_profiles_minimal_from_s03h1_test.marte scenarios/S03H3_execution_profiles/ validate.sh verify_s03h3_minimal_structure.py ``` S03-H3 defines three go-forward execution profiles: ```text MINIMAL TEST DIAGNOSTIC ``` The MINIMAL configuration is derived from the validated S03-H1 TEST configuration by explicitly removing unnecessary display functionality while preserving the required execution behaviour. TEST and DIAGNOSTIC reuse their validated historical configurations. This avoids unnecessary duplication and reduces the number of independent configuration variants that must be trusted. --- # Why S03-H3 matters Before advancing toward larger benchmark campaigns, the project needs one well-understood baseline. S03-H3 provides that reference point. Conceptually: ```text KNOWN INPUT ↓ KNOWN CONFIGURATION ↓ KNOWN LAUNCHER ↓ KNOWN RUNTIME DEPENDENCIES ↓ KNOWN VALIDATION ↓ KNOWN-GOOD RESULT ``` Future stages should extend this baseline rather than silently redefine it. --- # Runtime dependencies The benchmark suite depends on an external MARTe installation. The current runtime environment includes: ```text MARTe2 MARTe2-components ``` Important runtime objects include MARTe executables/libraries and component libraries required by the active configurations. These external projects are not copied into this repository. This repository therefore acts as: ```text benchmark source + configuration + validation infrastructure ``` and not as a complete binary backup of MARTe. A separate verified restore mechanism is required for complete disaster recovery of the known-good runtime environment. --- # Run directories Runtime execution artifacts are written under: ```text runs/ ``` A run directory may contain logs, generated data and validation evidence. These files can be large and are produced repeatedly. For this reason: ```text runs/ ``` is intentionally excluded through `.gitignore`. Git is used for source/configuration history. Runtime evidence and full environment recovery are handled separately. --- # Git versus full restore backup The Git repository and the full restore point serve different purposes. ```text Git │ ├── scripts ├── configurations ├── validators ├── parsers ├── documentation └── development history ``` while a complete restore point additionally needs the external runtime environment required to reproduce the validated MARTe execution. Therefore: ```text Git backup ≠ complete runtime restore point ``` Both mechanisms are complementary. --- # Development governance The project follows a controlled development process. A typical new stage follows approximately: ```text DESIGN / SPECIFICATION ↓ independent technical review ↓ technical arbitration ↓ human freeze ↓ implementation ↓ tests ↓ independent implementation review ↓ technical arbitration ↓ human acceptance ↓ new known-good baseline ``` A development-stage implementation is therefore not automatically considered part of the authoritative baseline simply because code exists. --- # Planned roadmap ## S04 — Repeatability / Multi-Run Main question: > Does the same already-valid S03-H3 workload remain correct when executed > repeatedly? Conceptually: ```text S03-H3 run #1 → result S03-H3 run #2 → result S03-H3 run #3 → result ... S03-H3 run #N → result ↓ campaign-level repeatability result ``` S04 is primarily a correctness/repeatability stage. It is not intended to turn repeatability percentages or CycleTime observations into performance claims. --- ## S05 — Deterministic Parametric Generator Main question: > Can test workloads be generated automatically from controlled parameters > while remaining reproducible? Conceptually: ```text parameters + seed ↓ generator ↓ deterministic MARTe scenario ``` The same inputs should regenerate the same intended scenario. --- ## S06 — MARTe2 Core Scaling Main question: > How does the MARTe2 Core workload behave as controlled complexity increases? S06 is expected to reuse the capabilities established by the preceding stages: ```text S03-H3 → correctness baseline S04 → repeatable execution S05 → deterministic workload generation S06 → controlled scaling study ``` --- # Current boundary At the time of this README baseline: ```text S03-H3 = CLOSED / KNOWN-GOOD S04 = next development target S05 = planned S06 = planned ``` The existence of this roadmap must not be interpreted as evidence that S04, S05 or S06 have already been implemented or validated. --- # Repository philosophy The repository should remain: ```text small reviewable deterministic traceable reproducible ``` Generated runtime outputs should not be mixed with source history. Known-good configurations, however, may intentionally be versioned even when they were produced by controlled patchers, because they are part of the validated benchmark baseline. --- # License and distribution The licensing and distribution policy for this benchmark suite must be defined by the project owners before public redistribution. MARTe2 and MARTe2-components retain their respective upstream licensing and ownership terms.