Improving CLI tool and improving documentation

docs/CODE_DOCUMENTATION.md

@@ -0,0 +1,106 @@
+# mdt Internal Code Documentation
+
+This document provides a detailed overview of the `mdt` codebase architecture and internal components.
+
+## Architecture Overview
+
+`mdt` is built as a modular system where core functionalities are separated into internal packages. The data flow typically follows this pattern:
+
+1.  **Parsing**: Source code is parsed into an Abstract Syntax Tree (AST).
+2.  **Indexing**: ASTs from multiple files are aggregated into a unified `ProjectTree`.
+3.  **Processing**: The `ProjectTree` is used by the Validator, Builder, and LSP server to perform their respective tasks.
+
+## Package Structure
+
+```
+cmd/
+  mdt/              # Application entry point (CLI)
+internal/
+  builder/          # Logic for merging and building configurations
+  formatter/        # Code formatting engine
+  index/            # Symbol table and project structure management
+  logger/           # Centralized logging
+  lsp/              # Language Server Protocol implementation
+  parser/           # Lexer, Parser, and AST definitions
+  schema/           # CUE schema loading and integration
+  validator/        # Semantic analysis and validation logic
+```
+
+## Core Packages
+
+### 1. `internal/parser`
+
+Responsible for converting MARTe configuration text into structured data.
+
+*   **Lexer (`lexer.go`)**: Tokenizes the input stream. Handles MARTe specific syntax like `#package`, `//!` pragmas, and `//#` docstrings. Supports standard identifiers and `#`-prefixed identifiers.
+*   **Parser (`parser.go`)**: Recursive descent parser. Converts tokens into a `Configuration` object containing definitions, comments, and pragmas.
+*   **AST (`ast.go`)**: Defines the node types (`ObjectNode`, `Field`, `Value`, etc.). All nodes implement the `Node` interface providing position information.
+
+### 2. `internal/index`
+
+The brain of the system. It maintains a holistic view of the project.
+
+*   **ProjectTree**: The central data structure. It holds the root of the configuration hierarchy (`Root`), references, and isolated files.
+*   **ProjectNode**: Represents a logical node in the configuration. Since a node can be defined across multiple files (fragments), `ProjectNode` aggregates these fragments.
+*   **NodeMap**: A hash map index (`map[string][]*ProjectNode`) for $O(1)$ symbol lookups, optimizing `FindNode` operations.
+*   **Reference Resolution**: The `ResolveReferences` method links `Reference` objects to their target `ProjectNode` using the `NodeMap`.
+
+### 3. `internal/validator`
+
+Ensures configuration correctness.
+
+*   **Validator**: Iterates over the `ProjectTree` to check rules.
+*   **Checks**:
+    *   **Structure**: Duplicate fields, invalid content.
+    *   **Schema**: Unifies nodes with CUE schemas (loaded via `internal/schema`) to validate types and mandatory fields.
+    *   **Signals**: Verifies that signals referenced in GAMs exist in DataSources and match types.
+    *   **Threading**: Checks `checkDataSourceThreading` to ensure non-multithreaded DataSources are not shared across threads in the same state.
+    *   **Unused**: Detects unused GAMs and Signals (suppressible via pragmas).
+
+### 4. `internal/lsp`
+
+Implements the Language Server Protocol.
+
+*   **Server (`server.go`)**: Handles JSON-RPC messages over stdio.
+*   **Incremental Sync**: Supports `textDocumentSync: 2`. `HandleDidChange` applies patches to the in-memory document buffers using `offsetAt` logic.
+*   **Features**:
+    *   `HandleCompletion`: Context-aware suggestions (Schema fields, Signal references, Class names).
+    *   `HandleHover`: Shows documentation, signal types, and usage analysis (e.g., "Used in GAMs: Controller (Input)").
+    *   `HandleDefinition` / `HandleReferences`: specific lookup using the `index`.
+
+### 5. `internal/builder`
+
+Merges multiple MARTe files into a single output.
+
+*   **Logic**: It parses all input files, builds a temporary `ProjectTree`, and then reconstructs the source code.
+*   **Merging**: It interleaves fields and subnodes from different file fragments to produce a coherent single-file configuration, respecting the `#package` hierarchy.
+
+### 6. `internal/schema`
+
+Manages CUE schemas.
+
+*   **Loading**: Loads the embedded default schema (`marte.cue`) and merges it with any user-provided `.marte_schema.cue`.
+*   **Metadata**: Handles the `#meta` field in schemas to extract properties like `direction` and `multithreaded` support for the validator.
+
+## Key Data Flows
+
+### Reference Resolution
+1.  **Scan**: Files are parsed and added to the `ProjectTree`.
+2.  **Index**: `RebuildIndex` populates `NodeMap`.
+3.  **Resolve**: `ResolveReferences` iterates all recorded references (values) and calls `FindNode`.
+4.  **Link**: If found, `ref.Target` is set to the `ProjectNode`.
+
+### Validation Lifecycle
+1.  `mdt check` or LSP `didChange` triggers validation.
+2.  A new `Validator` is created with the current `Tree`.
+3.  `ValidateProject` is called.
+4.  It walks the tree, runs checks, and populates `Diagnostics`.
+5.  Diagnostics are printed (CLI) or published via `textDocument/publishDiagnostics` (LSP).
+
+### Threading Check Logic
+1.  Finds the `RealTimeApplication` node.
+2.  Iterates through `States` and `Threads`.
+3.  For each Thread, resolves the `Functions` (GAMs).
+4.  For each GAM, resolves connected `DataSources` via Input/Output signals.
+5.  Maps `DataSource -> Thread` within the context of a State.
+6.  If a DataSource is seen in >1 Thread, it checks the `#meta.multithreaded` property. If false (default), an error is raised.