diff --git a/docs/superpowers/specs/2026-06-24-local-arrays-design.md b/docs/superpowers/specs/2026-06-24-local-arrays-design.md new file mode 100644 index 0000000..b36ea9a --- /dev/null +++ b/docs/superpowers/specs/2026-06-24-local-arrays-design.md @@ -0,0 +1,266 @@ +# Design: Local array values for the node-editor flow engines + +**Date:** 2026-06-24 +**Status:** Approved (design phase) +**TODO refs:** "Logic editor → add full support to local array values: dynamic, +dynamic but capped max, fixed size etc; array functions should work with new +local array" and "Control loop → add full support to server side array values". + +## 1. Goal & scope + +Add first-class **array-valued local variables** to both flow engines: + +- **Panel logic** (client TS): `web/src/lib/logic.ts`, `LogicEditor.tsx`, + `web/src/lib/types.ts`, `web/src/lib/expr.ts`, `web/src/lib/localstate.ts`, + `web/src/lib/xml.ts`. +- **Control logic** (server Go): `internal/controllogic/` (`model.go`, + `engine.go`, `expr.go`), editor `web/src/ControlLogicEditor.tsx`. + +**Build order:** design both together; implement **panel logic (TS) first**, +then port the same design to control logic (Go). The panel `StateVar` schema is +the richer reference and doing TS first de-risks the expression-language change +before the Go port. + +**In scope for v1:** declaration + sizing policies, an array-aware expression +language (shared by both engines), array mutation nodes, persistence, **and +widget binding** (multi-LED/bitset, table, plot read array locals). + +## 2. Data model — array local declaration + +A new array kind on the existing `StateVar` (TS `types.ts`; mirrored as a Go +struct in control logic, which gains state-var declarations for the first time): + +``` +StateVar { + name: string + type: 'number' | 'bool' | 'string' | 'array' // 'array' is new + initial: string // arrays: JSON literal e.g. "[0,0,0]" / "[[1,2],[3,4]]", or "" = empty/zero-fill + unit?, low?, high? // existing scalar fields + + // present only when type === 'array': + elem?: 'number' | 'bool' | 'array' // element type; 'array' ⇒ nested (recursive, arbitrary depth, jagged allowed) + sizing?: 'dynamic' | 'capped' | 'fixed' // default 'dynamic' + capacity?: number // required for capped/fixed +} +``` + +Semantics: + +- **dynamic** — unbounded, guarded by a global safety cap (1e6 elements) to + prevent runaway growth. +- **capped** — `capacity` max; pushing past full **drops the oldest** element + (ring / FIFO). +- **fixed** — exactly `capacity` slots. Initialised from the `initial` literal + (truncated / zero-padded to `capacity`), else **zero-filled**. `push` is a + no-op (write via index/set); `clear` zero-refills rather than emptying. +- **initial** — non-empty JSON literal is parsed and used; empty ⇒ dynamic and + capped start `[]`, fixed starts zero-filled. +- `elem: 'bool'` is **display metadata only**. Runtime leaves are numeric + `1`/`0` (consistent with the engine's existing "booleans are 1/0" rule); + widgets use the declaration to render on/off. + +**Declaration-time validation:** capped/fixed require `capacity >= 1`; +`initial`, if non-empty, must parse as JSON and match the declared element +type / nesting. Errors surface inline in the editor. + +## 3. Expression engine (`expr.ts` + `expr.go`) + +**Value model (tagged union — "Approach 1"):** + +- TS: `type EvalValue = number | EvalValue[]`. The evaluator returns + `EvalValue`; functions/indexing type-check at runtime and throw on misuse + (caught → node error badge). +- Go: a boxed `value{ num float64; arr []value; isArr bool }`. `Resolver` + returns `value`; every `*Node.eval` returns `value` (a contained, mechanical + refactor of `expr.go`, which today returns `float64`). The `float64` leaf + stays the fast path. + +**New syntax (both parsers):** + +- **Array literal:** `[a, b, c]`, nested `[[1,2],[3,4]]`. +- **Indexing:** postfix `expr[expr]`, chainable `a[i][j]`. Index rounds to int; + **negative index counts from the end** (`a[-1]` = last); out-of-range → node + error. + +**Functions** — added to the existing `abs/min/max/sqrt/...` table. All the +read/transform functions are **pure** (return new values; never mutate a local): + +| Function | Result | Meaning / notes | +|---|---|---| +| `len(a)` | number | element count (top level) | +| `sum(a)`, `mean(a)`, `min(a)`, `max(a)` | number | over a 1-D numeric array; error if elements are arrays | +| `slice(a, s, e)` | array | subrange, `e` exclusive, negative indices allowed | +| `concat(a, b)` | array | join | +| `reverse(a)` | array | | +| `sort(a)` | array | ascending numeric | +| `scale(a, k)` | array | element-wise `a[i]*k` | +| `add(a, b)`, `sub(a, b)` | array | element-wise pairwise; length = min(len a, len b) | +| `push(a, v)` | array | copy with `v` appended | +| `set(a, i, v)` | array | copy with element `i` replaced (negative `i` ok) | +| `insert(a, i, v)` | array | copy with `v` inserted at `i` | +| `remove(a, i)` | array | copy without element `i` | +| `pop(a)` | array | copy without the last element (read it with `a[-1]`) | +| `shift(a)` | array | copy without the first element (read with `a[0]`) | +| `indexOf(a, v)` | number | first index of `v`, else `-1` | +| `contains(a, v)` | number | `1`/`0` | +| `fill(n, v)` | array | new length-`n` array of `v` | + +- `min`/`max` keep their existing **scalar variadic** form (`min(x,y,z)`) and + gain a 1-arg **array** form (`min(a)`) — dispatch on arg count + type. +- **Resolution:** a bare identifier naming an array local returns the whole + array value; `{ds:sig}` waveform signals (EPICS `float64[]`) become first-class + array values usable by every function above. + +**Purity & persistence interplay:** the mutator-named functions +(`push/set/insert/remove/pop/shift`) are **immutable transforms** — they return a +new array and do not touch the local. You persist a result by writing it back; +the **sizing policy is enforced at store time** in `writeLocalState` (TS) / +`setLocal` (Go) whenever the write target is a typed array local (ring-drop for +capped, clamp / no-op for fixed). The mutation nodes (§4) are convenient, visible +sugar for "store with policy". + +**Errors** (e.g. `sum` of nested array, indexing a scalar, `add` of non-arrays) +throw in the evaluator and surface as the node's error reason via the existing +`checkExpr` validation + runtime-catch / badge path. + +**Ref-collection** (`collectRefs` / `CollectRefs`) walks the new literal/index +AST so subscriptions still discover every `{ds:sig}` inside array expressions. + +## 4. Mutation nodes + accumulate/export unification + +**New action nodes** (panel `LogicNodeKind`, mirrored in Go control-logic kinds): + +- `action.array.push{array, expr}` — append `eval(expr)`; sizing-policy aware + (ring-drop if capped; no-op if fixed). +- `action.array.set{array, index, expr}` — store at `eval(index)`. Supports + **nested targets via an index path**: `index = "i, j"` ⇒ `a[i][j]`. Negative + indices allowed; out-of-range → node error. (This is the imperative + path-assignment style.) +- `action.array.remove{array, index}` — remove element at index (in place). +- `action.array.pop{array}` — remove last element (in place). +- `action.array.clear{array}` — empty (dynamic/capped) or zero-refill (fixed). + +These are the sizing-policy-aware, in-place counterparts to the pure expression +functions. + +**Unification of the existing `{t,v}` array system (decision: unify):** + +- `action.accumulate{array, expr}` → reframed as `action.array.push` (append, + policy-enforced). Old kind **kept as a compile alias** so saved panels run. +- `action.clear{array}` → `action.array.clear` (alias retained). +- `action.export{columns, align, filename}` → serializes **array locals** by + column. Array locals are plain numeric (no per-sample `t`), so **time-based + alignment (`common`/`any`/`interpolate`) is dropped**; columns are emitted + **side-by-side by index** (ragged columns padded blank). The `align` param is + ignored and hidden in the inspector. To keep a timestamp column, push + `{sys:time}` into a parallel array local and add it as a column. +- **Custom column names:** each export column keeps its `label`, surfaced as an + **editable header name** in the inspector (default = array-local name); the CSV + header row uses the chosen names. + +**Migration (non-destructive, at engine `load()`):** any +`accumulate`/`clear`/`export` node referencing an array name with **no** matching +`StateVar` declaration triggers an **auto-declared dynamic numeric array local** +of that name. No file rewrite. + +## 5. Persistence + +**Panel logic (XML, `xml.ts`):** `` gains optional array attributes, +written only for arrays: + +```xml + + +``` + +Round-trips through the existing verbatim-body store (no Go change for panels). +New nodes serialize via the existing `` mechanism. + +**Control logic (Go, `model.go`):** control logic has **no** state-var +declarations today (`locals` is an untyped `map[string]float64`). Additions: + +- `Graph` gains `StateVars []StateVar` (Go struct mirroring the TS shape: + `Name, Type, Elem, Sizing string; Capacity int; Initial, Unit string; Low, + High float64`), serialized in `controllogic.json`. +- `compiledGraph.locals` changes from `map[string]float64` to + `map[string]value`, initialised from `StateVars` (applying sizing/initial) at + compile time. +- `getLocal`/`setLocal` operate on `value`; `setLocal` enforces sizing policy. + Config-apply / snapshot paths that read/write locals as `float64` box/unbox. + +**Versioning:** control-logic graphs are already git-style versioned; the new +`StateVars` field rides along in each revision (no diff-engine change — just more +JSON). + +## 6. Editor UI + debug/live badges + +**Panel `LogicEditor.tsx`:** the `LocalVars` palette subcomponent gains an array +declaration form (type=array reveals element-type / sizing / capacity / `initial` +JSON with inline validation). New array action nodes added to the Actions palette +group with inspectors (array name + expr/index fields, reusing `ExprField` with +array-aware `checkExpr`). + +**Control `ControlLogicEditor.tsx`:** control logic has **no** local-var +declaration UI today. Add a `LocalVars` panel mirroring the panel editor (same +component, driven by `Graph.StateVars`) plus the array nodes in its palette. This +brings control-logic locals to parity — scalars *and* arrays become declarable +there for the first time. + +**Debug/live badges (`flowDebug.ts` + Go `DebugObserver` / synthetic trace):** +array node values render as a truncated literal, e.g. `[1, 2, 3, …](n=100)`. The +Go debug event payload (`debugNode`) and the synthetic trace already serialize a +value — extended to carry array JSON; the badge formatter stringifies arrays +compactly. + +## 7. Widgets reading array locals + +The `ds:'local'` plumbing already routes through `stores.ts`/`ws.ts`; the change +is that a local's `SignalValue.value` can be an array (number / nested), held and +initialised per panel instance by `localstate.ts`. No new widget types — new +source modes on three existing widgets: + +- **Plot** — a 1-D numeric array local binds as a **waveform sample** (same path + EPICS `float64[]` waveforms already use for multidimensional/FFT/waterfall). + Each engine tick that rewrites the array updates the trace. Nested arrays show + "unsupported shape". +- **Table widget** — gains an **array source mode**: bound to one array local, + renders **one row per element** (index + value, per-signal value-format applied). + For an `elem:'array'` (2-D) local, rows are indices and the configured columns + map to inner-array positions. Falls back to multi-signal mode for scalars. +- **Multi-LED / bitset** — gains an **array source mode**: one LED per element, + lit per element truthiness; when `elem:'bool'`, on/off labels come from the + declaration. Existing integer-bitset mode unchanged. + +`getLocalMetaStore` carries `elem`/`sizing`/`capacity` so widgets self-configure +(e.g. multi-LED LED count = array length, growing/shrinking live for dynamic +arrays). + +## 8. Testing + +- **TS unit (expr):** literals; indexing (negative, nested, out-of-range error); + every new function incl. type-error cases; sizing-policy enforcement on store + (ring drop-oldest, fixed no-op / zero-refill); accumulate→push migration; CSV + export by-index with custom headers. +- **Go unit (`internal/controllogic`):** port of the expr suite (boxed `value`, + all functions/indexing/errors); `StateVars` init from declarations; `setLocal` + policy enforcement; JSON round-trip of `StateVars`; config-apply/snapshot with + boxed locals. +- **Cross-engine parity:** a `(expr, expected)` table asserted identical in both + `expr.ts` and `expr.go` to keep them in lockstep. +- **Widgets:** light component tests for the new array source modes if widget + tests exist; else manual verification. +- **Gates:** `gofmt`, `go vet`, `go test ./... -race`, frontend typecheck/build. + +## 9. Risks + +- **Go `expr.go` refactor (`float64` → boxed `value`):** touches every node's + `eval` and the `Resolver`. Mechanical but broad; the parity test guards + behavioral drift from `expr.ts`. +- **Backward compatibility of `accumulate`/`export`:** aliasing + auto-declared + locals keep old panels running, but the **dropped time-alignment in export** is + a behavioral change for any panel relying on `interpolate`/`common` align. Call + this out in release notes. +- **Two engines staying in lockstep:** the function set and semantics must match + exactly across TS and Go; the cross-engine parity fixture is the safeguard. +- **Hot-path purity:** array expression functions are evaluated repeatedly; they + must remain allocation-light and side-effect-free (mutation only at store time).