uopi/docs/superpowers/specs/2026-06-24-local-arrays-design.md

Design: Local array values for the node-editor flow engines

Date: 2026-06-24 Status: Approved (design phase) TODO refs: "Logic editor → add full support to local array values: dynamic, dynamic but capped max, fixed size etc; array functions should work with new local array" and "Control loop → add full support to server side array values".

1. Goal & scope

Add first-class array-valued local variables to both flow engines:

• Panel logic (client TS): web/src/lib/logic.ts, LogicEditor.tsx, web/src/lib/types.ts, web/src/lib/expr.ts, web/src/lib/localstate.ts, web/src/lib/xml.ts.
• Control logic (server Go): internal/controllogic/ (model.go, engine.go, expr.go), editor web/src/ControlLogicEditor.tsx.

Build order: design both together; implement panel logic (TS) first, then port the same design to control logic (Go). The panel StateVar schema is the richer reference and doing TS first de-risks the expression-language change before the Go port.

In scope for v1: declaration + sizing policies, an array-aware expression language (shared by both engines), array mutation nodes, persistence, and widget binding (multi-LED/bitset, table, plot read array locals).

2. Data model — array local declaration

A new array kind on the existing StateVar (TS types.ts; mirrored as a Go struct in control logic, which gains state-var declarations for the first time):

StateVar {
  name: string
  type: 'number' | 'bool' | 'string' | 'array'   // 'array' is new
  initial: string            // arrays: JSON literal e.g. "[0,0,0]" / "[[1,2],[3,4]]", or "" = empty/zero-fill
  unit?, low?, high?         // existing scalar fields

  // present only when type === 'array':
  elem?: 'number' | 'bool' | 'array'  // element type; 'array' ⇒ nested (recursive, arbitrary depth, jagged allowed)
  sizing?: 'dynamic' | 'capped' | 'fixed'   // default 'dynamic'
  capacity?: number          // required for capped/fixed
}

Semantics:

• dynamic — unbounded, guarded by a global safety cap (1e6 elements) to prevent runaway growth.
• capped — capacity max; pushing past full drops the oldest element (ring / FIFO).
• fixed — exactly capacity slots. Initialised from the initial literal (truncated / zero-padded to capacity), else zero-filled. push is a no-op (write via index/set); clear zero-refills rather than emptying.
• initial — non-empty JSON literal is parsed and used; empty ⇒ dynamic and capped start [], fixed starts zero-filled.
• elem: 'bool' is display metadata only. Runtime leaves are numeric 1/0 (consistent with the engine's existing "booleans are 1/0" rule); widgets use the declaration to render on/off.

Declaration-time validation: capped/fixed require capacity >= 1; initial, if non-empty, must parse as JSON and match the declared element type / nesting. Errors surface inline in the editor.

3. Expression engine (expr.ts + expr.go)

Value model (tagged union — "Approach 1"):

• TS: type EvalValue = number | EvalValue[]. The evaluator returns EvalValue; functions/indexing type-check at runtime and throw on misuse (caught → node error badge).
• Go: a boxed value{ num float64; arr []value; isArr bool }. Resolver returns value; every *Node.eval returns value (a contained, mechanical refactor of expr.go, which today returns float64). The float64 leaf stays the fast path.

New syntax (both parsers):

• Array literal: [a, b, c], nested [[1,2],[3,4]].
• Indexing: postfix expr[expr], chainable a[i][j]. Index rounds to int; negative index counts from the end (a[-1] = last); out-of-range → node error.

Functions — added to the existing abs/min/max/sqrt/... table. All the read/transform functions are pure (return new values; never mutate a local):

Function	Result	Meaning / notes
len(a)	number	element count (top level)
sum(a), mean(a), min(a), max(a)	number	over a 1-D numeric array; error if elements are arrays
slice(a, s, e)	array	subrange, e exclusive, negative indices allowed
concat(a, b)	array	join
reverse(a)	array
sort(a)	array	ascending numeric
scale(a, k)	array	element-wise a[i]*k
add(a, b), sub(a, b)	array	element-wise pairwise; length = min(len a, len b)
push(a, v)	array	copy with v appended
set(a, i, v)	array	copy with element i replaced (negative i ok)
insert(a, i, v)	array	copy with v inserted at i
remove(a, i)	array	copy without element i
pop(a)	array	copy without the last element (read it with a[-1])
shift(a)	array	copy without the first element (read with a[0])
indexOf(a, v)	number	first index of v, else -1
contains(a, v)	number	1/0
fill(n, v)	array	new length-n array of v

• min/max keep their existing scalar variadic form (min(x,y,z)) and gain a 1-arg array form (min(a)) — dispatch on arg count + type.
• Resolution: a bare identifier naming an array local returns the whole array value; {ds:sig} waveform signals (EPICS float64[]) become first-class array values usable by every function above.

Purity & persistence interplay: the mutator-named functions (push/set/insert/remove/pop/shift) are immutable transforms — they return a new array and do not touch the local. You persist a result by writing it back; the sizing policy is enforced at store time in writeLocalState (TS) / setLocal (Go) whenever the write target is a typed array local (ring-drop for capped, clamp / no-op for fixed). The mutation nodes (§4) are convenient, visible sugar for "store with policy".

Errors (e.g. sum of nested array, indexing a scalar, add of non-arrays) throw in the evaluator and surface as the node's error reason via the existing checkExpr validation + runtime-catch / badge path.

Ref-collection (collectRefs / CollectRefs) walks the new literal/index AST so subscriptions still discover every {ds:sig} inside array expressions.

4. Mutation nodes + accumulate/export unification

New action nodes (panel LogicNodeKind, mirrored in Go control-logic kinds):

• action.array.push{array, expr} — append eval(expr); sizing-policy aware (ring-drop if capped; no-op if fixed).
• action.array.set{array, index, expr} — store at eval(index). Supports nested targets via an index path: index = "i, j" ⇒ a[i][j]. Negative indices allowed; out-of-range → node error. (This is the imperative path-assignment style.)
• action.array.remove{array, index} — remove element at index (in place).
• action.array.pop{array} — remove last element (in place).
• action.array.clear{array} — empty (dynamic/capped) or zero-refill (fixed).

These are the sizing-policy-aware, in-place counterparts to the pure expression functions.

Unification of the existing {t,v} array system (decision: unify):

• action.accumulate{array, expr} → reframed as action.array.push (append, policy-enforced). Old kind kept as a compile alias so saved panels run.
• action.clear{array} → action.array.clear (alias retained).
• action.export{columns, align, filename} → serializes array locals by column. Array locals are plain numeric (no per-sample t), so time-based alignment (common/any/interpolate) is dropped; columns are emitted side-by-side by index (ragged columns padded blank). The align param is ignored and hidden in the inspector. To keep a timestamp column, push {sys:time} into a parallel array local and add it as a column.
• Custom column names: each export column keeps its label, surfaced as an editable header name in the inspector (default = array-local name); the CSV header row uses the chosen names.

Migration (non-destructive, at engine load()): any accumulate/clear/export node referencing an array name with no matching StateVar declaration triggers an auto-declared dynamic numeric array local of that name. No file rewrite.

5. Persistence

Panel logic (XML, xml.ts): <statevar> gains optional array attributes, written only for arrays:

<statevar name="hist" type="array" elem="number" sizing="capped" capacity="100" initial=""/>
<statevar name="grid" type="array" elem="array" sizing="fixed" capacity="4" initial="[[0,0],[0,0]]"/>

Round-trips through the existing verbatim-body store (no Go change for panels). New nodes serialize via the existing <node><param/></node> mechanism.

Control logic (Go, model.go): control logic has no state-var declarations today (locals is an untyped map[string]float64). Additions:

• Graph gains StateVars []StateVar (Go struct mirroring the TS shape: Name, Type, Elem, Sizing string; Capacity int; Initial, Unit string; Low, High float64), serialized in controllogic.json.
• compiledGraph.locals changes from map[string]float64 to map[string]value, initialised from StateVars (applying sizing/initial) at compile time.
• getLocal/setLocal operate on value; setLocal enforces sizing policy. Config-apply / snapshot paths that read/write locals as float64 box/unbox.

Versioning: control-logic graphs are already git-style versioned; the new StateVars field rides along in each revision (no diff-engine change — just more JSON).

6. Editor UI + debug/live badges

Panel LogicEditor.tsx: the LocalVars palette subcomponent gains an array declaration form (type=array reveals element-type / sizing / capacity / initial JSON with inline validation). New array action nodes added to the Actions palette group with inspectors (array name + expr/index fields, reusing ExprField with array-aware checkExpr).

Control ControlLogicEditor.tsx: control logic has no local-var declaration UI today. Add a LocalVars panel mirroring the panel editor (same component, driven by Graph.StateVars) plus the array nodes in its palette. This brings control-logic locals to parity — scalars and arrays become declarable there for the first time.

Debug/live badges (flowDebug.ts + Go DebugObserver / synthetic trace): array node values render as a truncated literal, e.g. [1, 2, 3, …](n=100). The Go debug event payload (debugNode) and the synthetic trace already serialize a value — extended to carry array JSON; the badge formatter stringifies arrays compactly.

7. Widgets reading array locals

The ds:'local' plumbing already routes through stores.ts/ws.ts; the change is that a local's SignalValue.value can be an array (number / nested), held and initialised per panel instance by localstate.ts. No new widget types — new source modes on three existing widgets:

• Plot — a 1-D numeric array local binds as a waveform sample (same path EPICS float64[] waveforms already use for multidimensional/FFT/waterfall). Each engine tick that rewrites the array updates the trace. Nested arrays show "unsupported shape".
• Table widget — gains an array source mode: bound to one array local, renders one row per element (index + value, per-signal value-format applied). For an elem:'array' (2-D) local, rows are indices and the configured columns map to inner-array positions. Falls back to multi-signal mode for scalars.
• Multi-LED / bitset — gains an array source mode: one LED per element, lit per element truthiness; when elem:'bool', on/off labels come from the declaration. Existing integer-bitset mode unchanged.

getLocalMetaStore carries elem/sizing/capacity so widgets self-configure (e.g. multi-LED LED count = array length, growing/shrinking live for dynamic arrays).

8. Testing

• TS unit (expr): literals; indexing (negative, nested, out-of-range error); every new function incl. type-error cases; sizing-policy enforcement on store (ring drop-oldest, fixed no-op / zero-refill); accumulate→push migration; CSV export by-index with custom headers.
• Go unit (internal/controllogic): port of the expr suite (boxed value, all functions/indexing/errors); StateVars init from declarations; setLocal policy enforcement; JSON round-trip of StateVars; config-apply/snapshot with boxed locals.
• Cross-engine parity: a (expr, expected) table asserted identical in both expr.ts and expr.go to keep them in lockstep.
• Widgets: light component tests for the new array source modes if widget tests exist; else manual verification.
• Gates: gofmt, go vet, go test ./... -race, frontend typecheck/build.

9. Risks

• Go expr.go refactor (float64 → boxed value): touches every node's eval and the Resolver. Mechanical but broad; the parity test guards behavioral drift from expr.ts.
• Backward compatibility of accumulate/export: aliasing + auto-declared locals keep old panels running, but the dropped time-alignment in export is a behavioral change for any panel relying on interpolate/common align. Call this out in release notes.
• Two engines staying in lockstep: the function set and semantics must match exactly across TS and Go; the cross-engine parity fixture is the safeguard.
• Hot-path purity: array expression functions are evaluated repeatedly; they must remain allocation-light and side-effect-free (mutation only at store time).