S03-H3 CLOSED baseline

2026-08-27 11:12:22 +02:00
commit 4b5fbb6c23
30 changed files with 10130 additions and 0 deletions
@@ -0,0 +1,738 @@
+# MARTe Benchmark Suite
+
+## Overview
+
+`marte-benchmark-suite` is a validation and benchmarking infrastructure
+developed around the MARTe2 real-time framework.
+
+The project is designed to build a sequence of progressively more advanced
+test scenarios while preserving known-good baselines from the previous stages.
+
+The suite does not modify MARTe2 itself.
+
+Instead, it provides:
+
+- MARTe configuration files;
+- launchers;
+- configuration patchers;
+- runtime orchestration;
+- output parsers;
+- structural validators;
+- semantic validators;
+- execution evidence.
+
+The development philosophy is incremental:
+
+> each new stage introduces one new capability while preserving the validated
+> behaviour of the previous stages.
+
+---
+
+# Current project status
+
+Current validated development baseline:
+
+```text
+S03-H3
+STATUS=CLOSED
+BASELINE=KNOWN-GOOD
+```
+
+S03-H3 is currently the authoritative baseline on which the next stages of the
+benchmark suite are intended to be built.
+
+The next planned stages are:
+
+```text
+S04 → Repeatability / Multi-Run
+S05 → Deterministic Parametric Generator
+S06 → MARTe2 Core Scaling
+```
+
+These stages are not part of the current S03-H3 baseline unless explicitly
+implemented and accepted later.
+
+---
+
+# Project purpose
+
+The suite is intended to answer increasingly complex questions about a MARTe2
+application.
+
+Conceptually:
+
+```text
+S00
+Can MARTe2 execute the basic reference application?
+        ↓
+S01
+Can execution timing information be observed?
+        ↓
+S02
+Can deterministic binary data be recorded?
+        ↓
+S03
+Can the recorded data be checked semantically?
+        ↓
+S03-H1
+Can different execution/debug profiles be supported?
+        ↓
+S03-H3
+Can those profiles be formalized and validated
+against an authoritative runtime contract?
+        ↓
+S04
+Does the same valid workload remain correct
+when executed repeatedly?
+        ↓
+S05
+Can deterministic test workloads be generated
+from parameters?
+        ↓
+S06
+How does MARTe2 Core behave as workload complexity grows?
+```
+
+---
+
+# High-level architecture
+
+The benchmark suite is layered around the installed MARTe2 runtime.
+
+```text
+                    +------------------+
+                    |      MARTe2      |
+                    |   MARTeApp.ex    |
+                    +---------+--------+
+                              ^
+                              |
+                     common/marte_runner.sh
+                              ^
+                              |
+                         launcher/
+                              ^
+                              |
+                 scenario / configuration
+                              |
+                              v
+                      MARTe execution
+                              |
+                              v
+                         RUN DIRECTORY
+                              |
+                 +------------+-------------+
+                 |                          |
+                 v                          v
+             runtime logs              output files
+                 |                          |
+                 +------------+-------------+
+                              |
+                              v
+                       parser / validator
+                              |
+                              v
+                       PASS / FAIL result
+```
+
+MARTe2 and MARTe2-components remain external dependencies.
+
+The benchmark suite orchestrates and validates them but is intentionally kept
+separate from their source trees.
+
+---
+
+# Repository structure
+
+Current source structure:
+
+```text
+marte-benchmark-suite/
+│
+├── common/
+│   ├── environment.sh
+│   ├── marte_runner.sh
+│   └── paths.sh
+│
+├── configurations/
+│   └── generated/
+│       ├── S00/
+│       ├── S01/
+│       ├── S02/
+│       ├── S03/
+│       ├── S03H1/
+│       └── S03H3/
+│
+├── launcher/
+│   ├── marte_benchmark_launcher.sh
+│   ├── marte_benchmark_launcher_s03h1.sh
+│   ├── marte_benchmark_launcher_s03h3.sh
+│   └── marte_benchmark_menu.sh
+│
+├── patchers/
+│   └── configuration/
+│
+├── scenarios/
+│   ├── S01_timing_display/
+│   ├── S02_binary_recording/
+│   ├── S03_semantic_sentinels/
+│   ├── S03H1_execution_profiles/
+│   └── S03H3_execution_profiles/
+│
+└── runs/
+```
+
+`runs/` contains execution outputs and is intentionally excluded from normal
+Git version control.
+
+The configurations under `configurations/generated/` are NOT generic build
+artifacts. They represent configuration baselines produced by controlled
+project stages and are therefore versioned.
+
+---
+
+# Common runtime layer
+
+## `common/paths.sh`
+
+Centralizes the paths used by the benchmark infrastructure.
+
+Its purpose is to avoid hard-coding the same MARTe and project locations in
+multiple scripts.
+
+---
+
+## `common/environment.sh`
+
+Defines the common execution environment used by the benchmark scripts.
+
+Keeping environment handling in one place improves reproducibility and avoids
+different scenarios silently using different runtime assumptions.
+
+---
+
+## `common/marte_runner.sh`
+
+Provides the common MARTe execution mechanism shared by benchmark stages.
+
+Conceptually:
+
+```text
+launcher
+   ↓
+common runner
+   ↓
+MARTeApp.ex
+   ↓
+run directory
+   ↓
+logs / outputs
+```
+
+Higher-level stages should reuse the authoritative runner rather than create
+alternative MARTe execution paths unless a future specification explicitly
+authorizes such a change.
+
+---
+
+# Launchers
+
+The `launcher/` directory contains the user-facing execution layer.
+
+The project currently includes launchers from different development stages,
+including the S03-H1 and S03-H3 generations.
+
+The current S03-H3 architecture supports execution profiles such as:
+
+```text
+MINIMAL
+TEST
+DIAGNOSTIC
+```
+
+The profiles allow the same benchmark infrastructure to expose different
+levels of runtime/debug functionality without changing the fundamental
+correctness contract.
+
+---
+
+# Patchers
+
+Configuration patchers live under:
+
+```text
+patchers/configuration/
+```
+
+They transform an already-known configuration into the configuration required
+by the next development stage.
+
+The dependency chain currently visible in the repository is:
+
+```text
+S00
+ ↓
+build_s01_from_s00.sh
+ ↓
+S01
+ ↓
+build_s02_from_s01.sh
+ ↓
+S02
+ ↓
+build_s03_from_s02.sh
+ ↓
+S03
+```
+
+Additional controlled transformations support S03-H1 and S03-H3.
+
+This model helps preserve provenance:
+
+```text
+new configuration
+=
+known parent
+
+explicit transformation
+```
+
+rather than independently rewriting every configuration from scratch.
+
+---
+
+# Validation layer
+
+The suite does not consider a MARTe process exit by itself sufficient proof
+that a test succeeded.
+
+Validation is layered.
+
+Depending on the stage, the suite can inspect:
+
+```text
+configuration structure
+        +
+runtime execution
+        +
+generated files
+        +
+semantic content
+        +
+expected sentinels
+```
+
+and only then derive the corresponding result.
+
+---
+
+# S00 — Core reference
+
+S00 provides the basic MARTe2 reference configuration:
+
+```text
+configurations/generated/S00/smoke_s0_core.marte
+```
+
+It acts as the root from which later benchmark configurations are progressively
+derived.
+
+---
+
+# S01 — Timing
+
+S01 introduces timing-related observation on top of the S00 reference
+configuration.
+
+Relevant current files include:
+
+```text
+patchers/configuration/build_s01_from_s00.sh
+configurations/generated/S01/from_patcher/
+scenarios/S01_timing_display/validate.sh
+```
+
+S01 established infrastructure that later stages could reuse rather than
+reimplement.
+
+---
+
+# S02 — Binary recording
+
+S02 extends the previous stage with binary data recording.
+
+Relevant components include:
+
+```text
+patchers/configuration/build_s02_from_s01.sh
+
+configurations/generated/S02/from_patcher/
+    smoke_s2_binary_recording_from_s01.marte
+
+scenarios/S02_binary_recording/validate.sh
+```
+
+Binary recording provides machine-readable data that later stages can parse
+and validate.
+
+---
+
+# S03 — Semantic validation
+
+S03 adds semantic verification of the generated data.
+
+Relevant components include:
+
+```text
+patchers/configuration/build_s03_from_s02.sh
+
+configurations/generated/S03/from_patcher/
+    smoke_s3_semantic_sentinels_from_s02.marte
+
+scenarios/S03_semantic_sentinels/
+    parse_s03_binary.py
+    validate.sh
+```
+
+The key conceptual change is:
+
+```text
+"data file exists"
+```
+
+is not enough.
+
+The project also checks whether the data means what the test expected it to
+mean.
+
+---
+
+# S03-H1 — Execution profiles
+
+S03-H1 introduced explicit execution profiles and supporting validation
+infrastructure.
+
+Current repository elements include:
+
+```text
+launcher/marte_benchmark_launcher_s03h1.sh
+
+patchers/configuration/
+    build_s03h1_from_s02.sh
+
+scenarios/S03H1_execution_profiles/
+    export_s03_csv.py
+    validate.sh
+    verify_s03h1_config.py
+```
+
+The historical S03-H1 configurations currently retained are:
+
+```text
+TEST
+DIAGNOSTIC
+```
+
+They remain important because S03-H3 intentionally reuses validated historical
+configurations where appropriate instead of duplicating them unnecessarily.
+
+---
+
+# S03-H3 — Current known-good baseline
+
+S03-H3 is the current closed benchmark stage.
+
+Its role is to formalize the execution-profile architecture and protect the
+runtime contract used by future development.
+
+Relevant repository elements include:
+
+```text
+launcher/
+    marte_benchmark_launcher_s03h3.sh
+    marte_benchmark_menu.sh
+
+patchers/configuration/
+    build_s03h3_minimal_from_s03h1_test.sh
+
+configurations/generated/S03H3/from_patcher/
+    smoke_s03h3_execution_profiles_minimal_from_s03h1_test.marte
+
+scenarios/S03H3_execution_profiles/
+    validate.sh
+    verify_s03h3_minimal_structure.py
+```
+
+S03-H3 defines three go-forward execution profiles:
+
+```text
+MINIMAL
+TEST
+DIAGNOSTIC
+```
+
+The MINIMAL configuration is derived from the validated S03-H1 TEST
+configuration by explicitly removing unnecessary display functionality while
+preserving the required execution behaviour.
+
+TEST and DIAGNOSTIC reuse their validated historical configurations.
+
+This avoids unnecessary duplication and reduces the number of independent
+configuration variants that must be trusted.
+
+---
+
+# Why S03-H3 matters
+
+Before advancing toward larger benchmark campaigns, the project needs one
+well-understood baseline.
+
+S03-H3 provides that reference point.
+
+Conceptually:
+
+```text
+KNOWN INPUT
+    ↓
+KNOWN CONFIGURATION
+    ↓
+KNOWN LAUNCHER
+    ↓
+KNOWN RUNTIME DEPENDENCIES
+    ↓
+KNOWN VALIDATION
+    ↓
+KNOWN-GOOD RESULT
+```
+
+Future stages should extend this baseline rather than silently redefine it.
+
+---
+
+# Runtime dependencies
+
+The benchmark suite depends on an external MARTe installation.
+
+The current runtime environment includes:
+
+```text
+MARTe2
+MARTe2-components
+```
+
+Important runtime objects include MARTe executables/libraries and component
+libraries required by the active configurations.
+
+These external projects are not copied into this repository.
+
+This repository therefore acts as:
+
+```text
+benchmark source + configuration + validation infrastructure
+```
+
+and not as a complete binary backup of MARTe.
+
+A separate verified restore mechanism is required for complete disaster
+recovery of the known-good runtime environment.
+
+---
+
+# Run directories
+
+Runtime execution artifacts are written under:
+
+```text
+runs/
+```
+
+A run directory may contain logs, generated data and validation evidence.
+
+These files can be large and are produced repeatedly.
+
+For this reason:
+
+```text
+runs/
+```
+
+is intentionally excluded through `.gitignore`.
+
+Git is used for source/configuration history.
+
+Runtime evidence and full environment recovery are handled separately.
+
+---
+
+# Git versus full restore backup
+
+The Git repository and the full restore point serve different purposes.
+
+```text
+Git
+│
+├── scripts
+├── configurations
+├── validators
+├── parsers
+├── documentation
+└── development history
+```
+
+while a complete restore point additionally needs the external runtime
+environment required to reproduce the validated MARTe execution.
+
+Therefore:
+
+```text
+Git backup ≠ complete runtime restore point
+```
+
+Both mechanisms are complementary.
+
+---
+
+# Development governance
+
+The project follows a controlled development process.
+
+A typical new stage follows approximately:
+
+```text
+DESIGN / SPECIFICATION
+        ↓
+independent technical review
+        ↓
+technical arbitration
+        ↓
+human freeze
+        ↓
+implementation
+        ↓
+tests
+        ↓
+independent implementation review
+        ↓
+technical arbitration
+        ↓
+human acceptance
+        ↓
+new known-good baseline
+```
+
+A development-stage implementation is therefore not automatically considered
+part of the authoritative baseline simply because code exists.
+
+---
+
+# Planned roadmap
+
+## S04 — Repeatability / Multi-Run
+
+Main question:
+
+> Does the same already-valid S03-H3 workload remain correct when executed
+> repeatedly?
+
+Conceptually:
+
+```text
+S03-H3 run #1 → result
+S03-H3 run #2 → result
+S03-H3 run #3 → result
+...
+S03-H3 run #N → result
+        ↓
+campaign-level repeatability result
+```
+
+S04 is primarily a correctness/repeatability stage.
+
+It is not intended to turn repeatability percentages or CycleTime observations
+into performance claims.
+
+---
+
+## S05 — Deterministic Parametric Generator
+
+Main question:
+
+> Can test workloads be generated automatically from controlled parameters
+> while remaining reproducible?
+
+Conceptually:
+
+```text
+parameters
+
+seed
+   ↓
+generator
+   ↓
+deterministic MARTe scenario
+```
+
+The same inputs should regenerate the same intended scenario.
+
+---
+
+## S06 — MARTe2 Core Scaling
+
+Main question:
+
+> How does the MARTe2 Core workload behave as controlled complexity increases?
+
+S06 is expected to reuse the capabilities established by the preceding stages:
+
+```text
+S03-H3 → correctness baseline
+S04    → repeatable execution
+S05    → deterministic workload generation
+S06    → controlled scaling study
+```
+
+---
+
+# Current boundary
+
+At the time of this README baseline:
+
+```text
+S03-H3 = CLOSED / KNOWN-GOOD
+
+S04 = next development target
+S05 = planned
+S06 = planned
+```
+
+The existence of this roadmap must not be interpreted as evidence that S04,
+S05 or S06 have already been implemented or validated.
+
+---
+
+# Repository philosophy
+
+The repository should remain:
+
+```text
+small
+reviewable
+deterministic
+traceable
+reproducible
+```
+
+Generated runtime outputs should not be mixed with source history.
+
+Known-good configurations, however, may intentionally be versioned even when
+they were produced by controlled patchers, because they are part of the
+validated benchmark baseline.
+
+---
+
+# License and distribution
+
+The licensing and distribution policy for this benchmark suite must be defined
+by the project owners before public redistribution.
+
+MARTe2 and MARTe2-components retain their respective upstream licensing and
+ownership terms.