feat: the export writes an MDF 3.30 file, so a window loads in ScryLab

A third format beside `.npy` and `.csv`, written by hand in `src/export.cpp` --
MF4 is HDF5 and would mean a library, MF3 is a handful of fixed blocks. The
layout is the asammdf 8.8.27 reader's own field list, byte-diffed against a
known-good 3.30 file it wrote; the shape is the one ScryLab's data-source
contract asks for: one channel group, record-interleaved, the master as the x
axis, and the unit in the channel's conversion block.

So in MDF the master is `abs_seconds`, named `time` -- the clock that is the
same in every display mode -- and the plotted axis goes out as an ordinary
channel, `time_relative`. `ExportTable::Column` gained the CONFIG's unit, which
only this writer spells out; the `.npy`/`.csv` bytes are unchanged, and
`write_export()` now picks the writer for both the dialog and the probe.
This commit is contained in:
Martino Ferrari committed 2026-10-14 15:56:27 +02:00
1 parent 761615c2fe
commit 908286ee89
10 files changed
+505 -28

No files matched your search

+28 -2
View File
@@ -89,7 +89,7 @@ src/channel.* ChannelBuffer (a ring pair as a plot::data::Series) and SlotS
src/acquisition.* the UDPStreamer client, the producer thread, frame decoding src/acquisition.* the UDPStreamer client, the producer thread, frame decoding
src/trigger.* the trigger configuration and the producer-side state machine src/trigger.* the trigger configuration and the producer-side state machine
src/plot_host.* the one plot::Figure, its rows' traces and cursors, the x link group src/plot_host.* the one plot::Figure, its rows' traces and cursors, the x link group
src/export.* the visible range as a table, and the .npy / .csv writers src/export.* the visible range as a table, and the .npy / .csv / .mf3 writers
src/layout_store.* the per-source session layout file (TOML), written and read back src/layout_store.* the per-source session layout file (TOML), written and read back
src/event_series.hpp an immutable captured event as a plot::data::Series src/event_series.hpp an immutable captured event as a plot::data::Series
src/ui.* the GTK4 window: toolbar, trigger panel, signal panel, plot box src/ui.* the GTK4 window: toolbar, trigger panel, signal panel, plot box
@@ -2261,16 +2261,42 @@ magnitudes, so consecutive differences wobble by a few ULP of the stream's uptim
(femtoseconds at any plausible uptime — note that the `f32` defect this used to guard against (femtoseconds at any plausible uptime — note that the `f32` defect this used to guard against
was *microseconds*, four orders of magnitude up). was *microseconds*, four orders of magnitude up).
Two formats, picked by the extension the save dialog ends up with: Three formats, picked by the extension the save dialog ends up with:
| file | what it is | | file | what it is |
|---|---| |---|---|
| `.npy` | one NumPy structured array, `time`/`abs_seconds` as `<f8` and every signal as `<f4` — the width the ring stores, so the file invents no precision and is about half an all-`f8` one. `pd.DataFrame(np.load(path))` is the whole reader. | | `.npy` | one NumPy structured array, `time`/`abs_seconds` as `<f8` and every signal as `<f4` — the width the ring stores, so the file invents no precision and is about half an all-`f8` one. `pd.DataFrame(np.load(path))` is the whole reader. |
| `.csv` | the same table as text, `%.17g` for the times and `%.9g` for the values: the shortest widths that round-trip each type exactly. | | `.csv` | the same table as text, `%.17g` for the times and `%.9g` for the values: the shortest widths that round-trip each type exactly. |
| `.mf3`, `.mdf` | **MDF 3.30**, one channel group with every channel record-interleaved, written by hand — see *The MDF export* below. |
Parquet (the original ask) would have meant linking Arrow — tens of megabytes of Parquet (the original ask) would have meant linking Arrow — tens of megabytes of
dependency for a feature whose output a three-line reader already handles. dependency for a feature whose output a three-line reader already handles.
**The MDF export** is the format for handing a window to another tool, and specifically to
[ScryLab](https://docs.scrylab.de/docs/data-sources/), whose data-source contract is three
rules this file is shaped by: *every data channel becomes a signal* — the channel name is the
signal name — *the master channel of each channel group is the x axis*, and *the unit is taken
from the channel metadata*, which in MDF 3 means the channel's conversion block (`CCBLOCK`) and
nowhere else. A second occurrence of a name is dropped by that reader, so the channel names are
de-duplicated against each other and the master's.
Two things follow from that, and both differ from what the `.npy`/`.csv` table holds:
- **The master is `abs_seconds`, named `time`**, not the axis the plot showed. The relative
column is 0 at a different sample in each display mode, so a master off it would put a live
export and a held event's export on two different timelines — the one thing that column
exists to avoid. The relative axis goes out as an ordinary channel, **`time_relative`**,
right after the master, and both carry the unit `s`. The signal columns follow in the same
order the `.npy`/`.csv` file writes them.
- **The unit travels with a column**, taken from the stream's CONFIG (or a math channel's own),
and is written into the conversion block. The `.npy`/`.csv` writers are unchanged: their
readers get the names and the numbers, and their bytes are what they always were.
There is **no MDF writer library here and no new dependency**: the blocks are spelled out in
`src/export.cpp` against the field list the `asammdf` 8.8.27 reader walks the format with, and
`docs/claude/invariants-session-and-export.md` records the layout, the version decision (MDF
3.30, not MF4 — that is HDF5) and what was verified against a known-good reference file.
The table is collected when the menu item is clicked, not when the dialog is The table is collected when the menu item is clicked, not when the dialog is
answered: a live view keeps scrolling while the file chooser is up. The live cut goes answered: a live view keeps scrolling while the file chooser is up. The live cut goes
through `ScopeModel::live_window_locked()`, the same index range `publish_event()` through `ScopeModel::live_window_locked()`, the same index range `publish_event()`
@@ -52,6 +52,40 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear
- **The export is collected at click time, not in the file-dialog callback.** The live - **The export is collected at click time, not in the file-dialog callback.** The live
view keeps scrolling while the chooser is up, so a late collection would write a view keeps scrolling while the chooser is up, so a late collection would write a
window the user never saw. The writer is picked by the chosen path's extension. window the user never saw. The writer is picked by the chosen path's extension.
- **The MDF export is MDF 3.30 written by hand, and its layout is the reader's, not the
standard's prose.** No dependency is added: MF4 is HDF5 and would need a library, while MF3 is a
handful of fixed-size blocks, so `write_mdf()` spells them out. The field list comes from
`asammdf` 8.8.27 (`blocks/v2_v3_blocks.py`, `blocks/mdf_v3.py`, which parses MF3 by explicit
offsets) and was confirmed byte for byte against a known-good 3.30 file asammdf wrote (one
group, a `time` master and two `V` channels). The shape, in file order: the **ID block** is 64
bytes with *no id/length header* (`"MDF 3.30"`, byte order, version `330`) — a writer that
emits `ID` + size + links at offset 0 produces something no reader accepts, since the first real
block starts at 64; the **HD block** is 208 (164 common + 44 of the 3.2-era time fields) and
points at the data group; a **TX** comment follows it; then the **samples**; the **DG** is 28; a
**CC** (conversion, 46 — a `NONE` conversion is the common fields alone) then a **CN** (channel,
228) precede each channel; and the **CG** (30) is last. Address fields are absolute file offsets
written *forward*, so the whole address map is computed before a byte is written, and every
block length is checked after it is built: a field at the wrong width does not fail the write, it
leaves the reader in the middle of the next block.
- **The data group's data address holds the raw records, with no `DT` header** — which is what
asammdf's *writer* does and what its *reader* assumes (`DataBlock` is documented as "a pseudo
block not defined by the MDF 3 standard"). This is the one place the file follows the reader
rather than a narrow reading of the standard, and it is the assumption ScryLab compatibility
rests on: a file with a `DT` block there would have its header read as sample bytes.
- **In MDF the master is `abs_seconds`, named `time`, and the relative axis is a channel
`time_relative`** — the same two numbers the `.npy`/`.csv` table holds, in the other roles. A
master off the relative column is 0 at a different sample in each display mode, so it would put a
live export and a held event's export on two timelines, which is the one thing the second column
exists to avoid; both carry the unit `s`, and the signal columns follow in the `.npy`/`.csv`
order. **The unit** is the client's CONFIG unit (a math channel's own), and it is written into
the CC block because that is the only place MDF 3 keeps one — and the place ScryLab reads a
signal's unit from. `ExportTable::Column` gained the field for it; the two older writers ignore
it, and their bytes are unchanged. **The names** are sanitised and de-duplicated against each
other *and* against the two reserved names before being cut to 31 bytes (the CN name field is 32
with its NUL): ScryLab imports only the **first** occurrence of a channel name, so a cut made
before de-duplication could silently drop a signal. `write_export()` is the one place the
extension picks the writer (`.csv`, `.mf3`/`.mdf`, anything else `.npy`), and both the dialog and
the self-check's export probe come through it so the two cannot offer different formats.
- **The chooser runs before `ScopeModel` exists, and `--server` is what decides.** `Config` carries - **The chooser runs before `ScopeModel` exists, and `--server` is what decides.** `Config` carries
`server_specified`, set only by the `--server` branch, and `main()` asks only when it is false — `server_specified`, set only by the `--server` branch, and `main()` asks only when it is false —
so declining costs no buffer and no socket, and `ScopeModel`, `Acquisition`, `RowSet` and so declining costs no buffer and no socket, and `ScopeModel`, `Acquisition`, `RowSet` and
+18 -1
View File
@@ -126,10 +126,27 @@ Two lines of the self-check output are assertions in disguise:
An export taken **mid-capture** reports `mode=live` with `time=[-pre..e]` where `e` is how far An export taken **mid-capture** reports `mode=live` with `time=[-pre..e]` where `e` is how far
the post window has filled — the export is what is on screen, and during a capture that is a the post window has filled — the export is what is on screen, and during a capture that is a
partly filled window. Pin the probe outside a capture when checking the `rows=` count. partly filled window. Pin the probe outside a capture when checking the `rows=` count.
The two `bytes=` lines are the writers: the `.npy` is The three `bytes=` lines are the writers: the `.npy` is
`64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)`, so a size that is not that is a `64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)`, so a size that is not that is a
mispacked record. Read it back with `pd.DataFrame(np.load(path))` after changing mispacked record. Read it back with `pd.DataFrame(np.load(path))` after changing
either writer. either writer.
The `.mf3` line's size is the MDF 3.30 layout the writer spells out, and every term of it is
observable: `272 + comment + rows·record + 28 + channels·274 + 30`, where the ID block is 64,
the HD block 208, the comment is a `TX` block of its text plus 5 bytes, `record` is
`16 + 4·cols` (two `f64` time columns then one `f32` per signal), the data group is 28, each of
`channels = cols + 2` gets a 46-byte conversion block and a 228-byte channel block, and the
channel group is 30. Read the file back with asammdf: `MDF(path).info()` must show one group
whose `channel 0` is `name="time" type=master`, and `get(name, group=0).unit` must be the unit
the next line reports.
`[export-probe] mdf master=time master_unit=s channels=N record=B units=['s' 's' 'V' …]`
is the other half, and it is the assertion on the units: `channels` and `record` are the two
numbers the `.mf3` size above is built from (so `record` must be `16 + 4·cols` and `channels`
`cols + 2`), and `units` must be **the CONFIG's**, in the table's column order, behind the
master's and the relative axis's two `s` — one `'s' 's'` pair plus one entry per exported
column, each exactly the unit the stream sent (an empty one reads `''`). A `V` where the mock
sent `s`, or a missing entry, is the unit not reaching the conversion block, which is the only
thing ScryLab reads a signal's unit from; `write_npy`/`write_csv` must be unchanged either way
(their bytes are pinned by the two lines above).
- `[window-probe] <action> label=.. entry=.. unit=U window=W event_span=E cap=C decim=M - `[window-probe] <action> label=.. entry=.. unit=U window=W event_span=E cap=C decim=M
stored=R aa=F buffered=B` (stderr, `UDPSCOPE_PROBE_WINDOW_MS`) — **`window` must equal stored=R aa=F buffered=B` (stderr, `UDPSCOPE_PROBE_WINDOW_MS`) — **`window` must equal
`event_span`** on every line (the trigger's event span is the display window; a capture `event_span`** on every line (the trigger's event span is the display window; a capture
+3 -3
View File
@@ -15,7 +15,7 @@ xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=1000 \
Two baselines every run is read against. A clean build emits **four** warnings and no more: two Two baselines every run is read against. A clean build emits **four** warnings and no more: two
`-Wmissing-field-initializers` on `MathNode::source` and `-Woverlength-strings` on **both** `-Wmissing-field-initializers` on `MathNode::source` and `-Woverlength-strings` on **both**
generated GResource `.c` files — the help pages' (**514660** bytes) and the icons', whose blob is generated GResource `.c` files — the help pages' (**515700** bytes) and the icons', whose blob is
**12352** against C99's 4095-byte string-literal limit with the seven action icons and the two app icons **12352** against C99's 4095-byte string-literal limit with the seven action icons and the two app icons
that remain (a note about a generated file, which GCC itself supports) -- re-measured when that remain (a note about a generated file, which GCC itself supports) -- re-measured when
the tab work changed the set (that icon left, the two tab glyphs arrived, and this file's the tab work changed the set (that icon left, the two tab glyphs arrived, and this file's
@@ -194,8 +194,8 @@ precedent, not an exception.
nothing. Its four lines are in nothing. Its four lines are in
[probe-readings.md](probe-readings.md). [probe-readings.md](probe-readings.md).
- `UDPSCOPE_EXPORT_MS` (+ optional `UDPSCOPE_EXPORT_PATH`, default - `UDPSCOPE_EXPORT_MS` (+ optional `UDPSCOPE_EXPORT_PATH`, default
`/tmp/udpscope-export`) — collect the export table and write it both ways, skipping `/tmp/udpscope-export`) — collect the export table and write it all three ways (`.npy`,
only the file chooser. `.csv`, `.mf3`), skipping only the file chooser.
- `UDPSCOPE_PROBE_CONDITION_MS` — arms a qualified trigger against the mock and reports the latched - `UDPSCOPE_PROBE_CONDITION_MS` — arms a qualified trigger against the mock and reports the latched
`t0`, the measured pulse or band width, and sample values for the window condition. Schedules its `t0`, the measured pulse or band width, and sample values for the window condition. Schedules its
+354 -12
View File
@@ -6,8 +6,11 @@
#include <algorithm> #include <algorithm>
#include <bit> #include <bit>
#include <cerrno> #include <cerrno>
#include <cstdint>
#include <cstdio> #include <cstdio>
#include <cstring> #include <cstring>
#include <ctime>
#include <utility>
namespace udpscope { namespace udpscope {
namespace { namespace {
@@ -34,18 +37,29 @@ std::string sanitise(const std::string& name) {
return out; return out;
} }
/// The final labels: sanitised, and de-duplicated against each other *and* against the /// `base` carrying a `_2`, `_3` … suffix for the `n`-th copy (`n == 1` is the name itself),
/// two time columns, with a `_2`, `_3` suffix. Two elements of one vector signal can /// cut to `max_len` bytes when one is given -- MDF's name field is 32 bytes wide and the file
/// easily sanitise to the same string. /// is re-read by a reader that keeps only the *first* occurrence of a name, so a cut that made
std::vector<std::string> column_labels(const ExportTable& table) { /// two channels equal would silently lose one.
std::vector<std::string> taken{"time", "abs_seconds"}; std::string numbered(const std::string& base, int n, std::size_t max_len) {
const std::string suffix = n == 1 ? std::string() : "_" + std::to_string(n);
std::size_t room = base.size();
if (max_len != 0) room = max_len > suffix.size() ? max_len - suffix.size() : 0;
return base.substr(0, room) + suffix;
}
/// The final labels: sanitised, and de-duplicated against each other *and* against
/// `taken` -- the names the writer already spends -- with a `_2`, `_3` suffix. Two
/// elements of one vector signal can easily sanitise to the same string.
std::vector<std::string> column_labels(const ExportTable& table, std::vector<std::string> taken,
std::size_t max_len = 0) {
std::vector<std::string> out; std::vector<std::string> out;
out.reserve(table.columns.size()); out.reserve(table.columns.size());
for (const ExportTable::Column& col : table.columns) { for (const ExportTable::Column& col : table.columns) {
const std::string base = sanitise(col.name); const std::string base = sanitise(col.name);
std::string label = base; std::string label = numbered(base, 1, max_len);
for (int n = 2; std::find(taken.begin(), taken.end(), label) != taken.end(); ++n) { for (int n = 2; std::find(taken.begin(), taken.end(), label) != taken.end(); ++n) {
label = base + "_" + std::to_string(n); label = numbered(base, n, max_len);
} }
taken.push_back(label); taken.push_back(label);
out.push_back(std::move(label)); out.push_back(std::move(label));
@@ -69,6 +83,74 @@ bool opened(const File& out, const std::string& path, std::string& error) {
return false; return false;
} }
/// The block lengths of MDF 3.30, from the **reader's** own field lists
/// (`asammdf/blocks/v2_v3_constants.py`, 8.8.27): every block is a two-letter id, a 16-bit
/// length and then its own fields, and the reader walks the file by those lengths alone, so a
/// block written one field too wide is a file whose reader lands in the middle of the next one.
/// The widths here are the ones a `bytes(block)` of the reference file measures (the known-good
/// 3.30 file `/tmp/ref.mdf` this writer was byte-diffed against).
constexpr std::size_t kMdfIdBlock = 64; // "MDF 3.30" and the flags, no id/length
constexpr std::size_t kMdfHeader = 208; // 164 common + 44 of the 3.2-era time fields
constexpr std::size_t kMdfTxOverhead = 5; // "TX", the u16 length and the terminating NUL
constexpr std::size_t kMdfDataGroup = 28; // 4 addresses, the group count, 4 reserved
constexpr std::size_t kMdfConversion = 46; // a NONE conversion is the common fields alone
constexpr std::size_t kMdfChannel = 228; // the display-name-length CNBLOCK
constexpr std::size_t kMdfChannelGroup = 30; // 26 + the sample-reduction address of 3.30
constexpr std::size_t kMdfNameBytes = 31; // a CNBLOCK's 32-byte name field, less its NUL
/// A byte sink for the MDF header blocks, which are a few hundred bytes each and have to be
/// laid out field by field. The little-endian `raw()` writes are the same assumption the `.npy`
/// writer states at the top of this file, and `static_assert`s there.
struct Bytes {
std::vector<unsigned char> v;
void raw(const void* p, std::size_t n) {
const auto* b = static_cast<const unsigned char*>(p);
v.insert(v.end(), b, b + n);
}
void u16(std::uint16_t x) {
const unsigned char b[2] = {static_cast<unsigned char>(x & 0xFFu),
static_cast<unsigned char>((x >> 8) & 0xFFu)};
raw(b, sizeof(b));
}
void i16(std::int16_t x) { u16(static_cast<std::uint16_t>(x)); }
void u32(std::uint32_t x) {
const unsigned char b[4] = {static_cast<unsigned char>(x & 0xFFu),
static_cast<unsigned char>((x >> 8) & 0xFFu),
static_cast<unsigned char>((x >> 16) & 0xFFu),
static_cast<unsigned char>((x >> 24) & 0xFFu)};
raw(b, sizeof(b));
}
void u64(std::uint64_t x) {
u32(static_cast<std::uint32_t>(x & 0xFFFFFFFFu));
u32(static_cast<std::uint32_t>(x >> 32));
}
void f64(double x) { raw(&x, sizeof(x)); }
void f32(float x) { raw(&x, sizeof(x)); }
void zeros(std::size_t n) { v.insert(v.end(), n, 0); }
/// A fixed-width text field: the bytes, then NUL padding. MDF's names and units are
/// NUL-terminated inside a field of a width the reader is told by the block size, so this is
/// how every string in the format is spelled.
void text(const std::string& s, std::size_t width) {
const std::size_t n = std::min(s.size(), width);
raw(s.data(), n);
zeros(width - n);
}
std::size_t size() const noexcept { return v.size(); }
};
/// Checked rather than trusted: a field written at the wrong width would not fail here, it
/// would produce a file whose reader walks into the middle of the next block. The widths are
/// the asammdf 8.8.27 reader's own field lists (`blocks/v2_v3_constants.py`).
bool mdf_block_size(const Bytes& block, std::size_t want, const char* what, std::string& error) {
if (block.size() == want) return true;
error = std::string("internal: the MDF ") + what + " block is " + std::to_string(block.size()) +
" bytes, the layout wants " + std::to_string(want);
return false;
}
} // namespace } // namespace
ExportTable collect_export(ScopeModel& model, RowSet& rows) { ExportTable collect_export(ScopeModel& model, RowSet& rows) {
@@ -97,6 +179,12 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].name return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].name
: std::string("ch") + std::to_string(c); : std::string("ch") + std::to_string(c);
}; };
// The unit travels with the column so the MDF writer can put it on the channel; the other
// writers ignore it, as they always have.
const auto unit_of = [&](int c) {
return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].unit
: std::string();
};
// What the rows bind to is what gets exported, so the split is the same `DisplayMode` // What the rows bind to is what gets exported, so the split is the same `DisplayMode`
// `RowSet::bind()` switches on: the frozen event only while one is held, the live ring // `RowSet::bind()` switches on: the frozen event only while one is held, the live ring
@@ -136,8 +224,9 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
if (ch >= snap->values.size()) continue; if (ch >= snap->values.size()) continue;
const std::vector<f32>& src = snap->values[ch]; const std::vector<f32>& src = snap->values[ch];
if (src.size() < first + n) continue; if (src.size() < first + n) continue;
table.columns.push_back({name_of(c), std::vector<f32>(src.begin() + static_cast<std::ptrdiff_t>(first), table.columns.push_back({name_of(c), unit_of(c),
src.begin() + static_cast<std::ptrdiff_t>(first + n))}); std::vector<f32>(src.begin() + static_cast<std::ptrdiff_t>(first),
src.begin() + static_cast<std::ptrdiff_t>(first + n))});
} }
return table; return table;
} }
@@ -177,7 +266,7 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
if (ch >= model.channel_count()) continue; if (ch >= model.channel_count()) continue;
std::vector<f32> values(static_cast<std::size_t>(n)); std::vector<f32> values(static_cast<std::size_t>(n));
store.values(ch).copy_from(first, n, values.data()); store.values(ch).copy_from(first, n, values.data());
table.columns.push_back({name_of(c), std::move(values)}); table.columns.push_back({name_of(c), unit_of(c), std::move(values)});
} }
return table; return table;
} }
@@ -191,6 +280,7 @@ ExportTable event_table(ScopeModel& model, const EventSnapshot& snapshot) {
for (std::size_t c = 0; c < snapshot.values.size() && c < channels.size(); ++c) { for (std::size_t c = 0; c < snapshot.values.size() && c < channels.size(); ++c) {
ExportTable::Column column; ExportTable::Column column;
column.name = channels[c].name; column.name = channels[c].name;
column.unit = channels[c].unit;
column.values = snapshot.values[c]; column.values = snapshot.values[c];
table.columns.push_back(std::move(column)); table.columns.push_back(std::move(column));
} }
@@ -202,7 +292,7 @@ bool write_npy(const std::string& path, const ExportTable& table, std::string& e
error = "nothing to export: no signal is plotted in the visible range"; error = "nothing to export: no signal is plotted in the visible range";
return false; return false;
} }
const std::vector<std::string> labels = column_labels(table); const std::vector<std::string> labels = column_labels(table, {"time", "abs_seconds"});
// `descr` as a list of (name, format) pairs is a *packed* structured dtype -- NumPy // `descr` as a list of (name, format) pairs is a *packed* structured dtype -- NumPy
// only inserts alignment padding when asked to (`align=True`) -- so the records below // only inserts alignment padding when asked to (`align=True`) -- so the records below
@@ -266,7 +356,7 @@ bool write_csv(const std::string& path, const ExportTable& table, std::string& e
File out(path); File out(path);
if (!opened(out, path, error)) return false; if (!opened(out, path, error)) return false;
const std::vector<std::string> labels = column_labels(table); const std::vector<std::string> labels = column_labels(table, {"time", "abs_seconds"});
std::string line = "time,abs_seconds"; std::string line = "time,abs_seconds";
for (const std::string& label : labels) line += "," + label; for (const std::string& label : labels) line += "," + label;
line += "\n"; line += "\n";
@@ -302,4 +392,256 @@ bool write_csv(const std::string& path, const ExportTable& table, std::string& e
return true; return true;
} }
bool write_mdf(const std::string& path, const ExportTable& table, std::string& error) {
if (table.empty()) {
error = "nothing to export: no signal is plotted in the visible range";
return false;
}
const std::size_t rows = table.rows();
const std::size_t cols = table.columns.size();
// **The master is the absolute clock, named `time`**, and the axis the plot showed goes out as
// an ordinary channel, `time_relative`. The relative column is 0 at a different sample in each
// display mode, so a master off it would put a live export and a held event's export on two
// timelines -- the one thing the second column exists to avoid -- while `abs_seconds` is the
// number the stream itself stamped the samples with. Both carry the unit `s`.
const std::vector<std::string> labels =
column_labels(table, {"time", "time_relative"}, kMdfNameBytes);
const std::size_t channels = cols + 2;
const std::size_t record = 2 * sizeof(f64) + cols * sizeof(f32);
if (rows > 0xFFFFFFFFull) {
error = "too many samples for an MDF 3 file: the channel group counts cycles in 32 bits";
return false;
}
if (128 + (cols - 1) * 32 > 0xFFFFu) {
error = "too many signals for an MDF 3 record: a channel's start offset is 16 bits";
return false;
}
// One channel per column, described once: what the CNBLOCK says about it (where in the record
// its samples start, how wide they are, whether it is the master) and what the CCBLOCK says
// (its unit). The range is measured from the samples rather than left at zero, so
// `range_flag = 1` means what it says.
struct Channel {
std::string name;
std::string unit;
std::uint16_t bit_offset;
std::uint16_t bit_count;
std::uint16_t data_type; // MDF 3: 2 = float, 3 = double, both in the ID block's byte order
std::uint16_t type; // 0 = value, 1 = master
f64 lo, hi;
};
const auto range_of = [](const auto& values) {
if (values.empty()) return std::pair<f64, f64>{0.0, 0.0};
const auto mm = std::minmax_element(values.begin(), values.end());
return std::pair<f64, f64>{static_cast<f64>(*mm.first), static_cast<f64>(*mm.second)};
};
std::vector<Channel> chans;
chans.reserve(channels);
const auto abs_range = range_of(table.abs_seconds);
const auto rel_range = range_of(table.time);
chans.push_back({"time", "s", 0, 64, 3, 1, abs_range.first, abs_range.second});
chans.push_back({"time_relative", "s", 64, 64, 3, 0, rel_range.first, rel_range.second});
for (std::size_t i = 0; i < cols; ++i) {
const auto range = range_of(table.columns[i].values);
chans.push_back({labels[i], table.columns[i].unit,
static_cast<std::uint16_t>(128 + i * 32), 32, 2, 0, range.first, range.second});
}
// Every address in the file, from the fixed block lengths above: the writer therefore knows
// where it is going before it writes a byte, which is what lets each block carry its links.
const std::string comment = "udpscope export: the visible window, " + std::to_string(rows) +
" samples x " + std::to_string(cols) + " signals";
const std::size_t hd_end = kMdfIdBlock + kMdfHeader;
const std::size_t comment_addr = hd_end;
const std::size_t data_addr = comment_addr + kMdfTxOverhead + comment.size();
const std::size_t dg_addr = data_addr + rows * record;
const std::size_t ch_addr = dg_addr + kMdfDataGroup;
const std::size_t cg_addr = ch_addr + channels * (kMdfConversion + kMdfChannel);
const std::uint64_t total = cg_addr + kMdfChannelGroup;
if (total > 0xFFFFFFFFull) {
error = "the export is larger than an MDF 3 file can address";
return false;
}
const auto cc_addr = [&](std::size_t i) {
return ch_addr + i * (kMdfConversion + kMdfChannel);
};
// The wall clock the measurement file was written at. `abs_time` is the same instant in
// nanoseconds since the epoch, which is what the 3.2-era header field asks for.
const std::time_t now = std::time(nullptr);
std::tm tm{};
localtime_r(&now, &tm);
char date[11] = {};
char clock[9] = {};
std::strftime(date, sizeof(date), "%d:%m:%Y", &tm);
std::strftime(clock, sizeof(clock), "%H:%M:%S", &tm);
Bytes head;
// Identification block: the version string, who wrote the file, and the byte order every
// sample in it is in.
head.text("MDF ", 8);
head.text("3.30", 8);
head.text("udpscope", 8);
head.u16(0); // byte order: Intel, ahead of every value and the float format below
head.u16(0); // float format: IEEE
head.u16(330); // MDF version, digits without the separator
head.u16(0); // code page: latin-1
head.zeros(2); // reserved
head.zeros(26); // reserved
head.u16(0); // unfinalised standard flags
head.u16(0); // unfinalised custom flags
if (!mdf_block_size(head, kMdfIdBlock, "identification", error)) return false;
// Header block, which points at the data group and at the comment that follows it.
head.raw("HD", 2);
head.u16(static_cast<std::uint16_t>(kMdfHeader));
head.u32(static_cast<std::uint32_t>(dg_addr)); // first data group
head.u32(static_cast<std::uint32_t>(comment_addr));
head.u32(0); // program block: nothing wrote this but us
head.u16(1); // data groups
head.text(date, 10);
head.text(clock, 8);
head.text("", 32); // author
head.text("", 32); // department
head.text("", 32); // project
head.text("", 32); // subject
head.u64(static_cast<std::uint64_t>(now) * 1000000000ull);
head.i16(static_cast<std::int16_t>(tm.tm_gmtoff / 3600));
head.u16(0); // time quality
head.text("Local PC Reference Time", 32);
if (!mdf_block_size(head, hd_end, "header", error)) return false;
// The comment, in the one text block the file carries: plain words rather than the XML a
// writer that also *edits* files keeps its structured properties in.
head.raw("TX", 2);
head.u16(static_cast<std::uint16_t>(comment.size() + kMdfTxOverhead));
head.raw(comment.data(), comment.size());
head.zeros(1);
Bytes tail;
// Data group: one channel group, whose channels are chained, and the address of the samples.
tail.raw("DG", 2);
tail.u16(static_cast<std::uint16_t>(kMdfDataGroup));
tail.u32(0); // next data group
tail.u32(static_cast<std::uint32_t>(cg_addr));
tail.u32(0); // trigger block
tail.u32(static_cast<std::uint32_t>(data_addr));
tail.u16(1); // channel groups
tail.u16(0); // record IDs: the group is sorted
tail.zeros(4); // reserved
if (!mdf_block_size(tail, kMdfDataGroup, "data group", error)) return false;
for (std::size_t i = 0; i < channels; ++i) {
const Channel& c = chans[i];
// Conversion block: the unit lives here and nowhere else in MDF 3, which is where ScryLab
// reads a signal's unit from. NONE because the samples are already the physical values --
// the table was collected as `f32`, so a conversion would only be a second copy of it.
Bytes cc;
cc.raw("CC", 2);
cc.u16(static_cast<std::uint16_t>(kMdfConversion));
cc.u16(1); // range_flag: the range below is measured, so it is valid
cc.f64(c.lo);
cc.f64(c.hi);
cc.text(c.unit, 20);
cc.u16(0xFFFFu);
cc.u16(0); // referenced parameters
if (!mdf_block_size(cc, kMdfConversion, "conversion", error)) return false;
// Channel block: the name ScryLab turns into the signal name, where this channel's samples
// sit in a record, and the link to the next channel.
Bytes cn;
cn.raw("CN", 2);
cn.u16(static_cast<std::uint16_t>(kMdfChannel));
cn.u32(i + 1 < channels ? static_cast<std::uint32_t>(cc_addr(i + 1) + kMdfConversion) : 0);
cn.u32(static_cast<std::uint32_t>(cc_addr(i)));
cn.u32(0); // source: the stream does not say what produced a sample
cn.u32(0); // dependency block
cn.u32(0); // comment
cn.u16(c.type);
cn.text(c.name, 32);
cn.zeros(128); // description
cn.u16(c.bit_offset);
cn.u16(c.bit_count);
cn.u16(c.data_type);
cn.u16(1); // range_flag
cn.f64(c.lo);
cn.f64(c.hi);
cn.f64(0.0); // sampling rate: only a virtual time channel carries one
cn.u32(0); // long name
cn.u32(0); // display name
cn.u16(0); // additional byte offset
if (!mdf_block_size(cn, kMdfChannel, "channel", error)) return false;
tail.v.insert(tail.v.end(), cc.v.begin(), cc.v.end());
tail.v.insert(tail.v.end(), cn.v.begin(), cn.v.end());
}
// Channel group, last: it names the first channel, the record size and the cycle count from
// which a reader derives the size of the sample block at `data_addr`.
tail.raw("CG", 2);
tail.u16(static_cast<std::uint16_t>(kMdfChannelGroup));
tail.u32(0); // next channel group
tail.u32(static_cast<std::uint32_t>(ch_addr + kMdfConversion));
tail.u32(0); // comment
tail.u16(1); // record ID
tail.u16(static_cast<std::uint16_t>(channels));
tail.u16(static_cast<std::uint16_t>(record));
tail.u32(static_cast<std::uint32_t>(rows));
tail.zeros(4); // sample reduction address
if (!mdf_block_size(tail, kMdfDataGroup + channels * (kMdfConversion + kMdfChannel) +
kMdfChannelGroup,
"data group and channels", error)) {
return false;
}
File out(path);
if (!opened(out, path, error)) return false;
const auto put = [&](const Bytes& bytes) {
return bytes.v.empty() || std::fwrite(bytes.v.data(), 1, bytes.v.size(), out.f) == bytes.v.size();
};
if (!put(head)) {
error = "short write to '" + path + "': " + std::strerror(errno);
return false;
}
// One record at a time, in the channel order the blocks above declare, so a full ring is not
// held twice -- the same shape the `.npy` writer streams its records in.
std::vector<unsigned char> rec(record);
for (std::size_t i = 0; i < rows; ++i) {
std::size_t at = 0;
std::memcpy(rec.data() + at, &table.abs_seconds[i], sizeof(f64));
at += sizeof(f64);
std::memcpy(rec.data() + at, &table.time[i], sizeof(f64));
at += sizeof(f64);
for (const ExportTable::Column& col : table.columns) {
const f32 v = i < col.values.size() ? col.values[i] : 0.0f;
std::memcpy(rec.data() + at, &v, sizeof(f32));
at += sizeof(f32);
}
if (std::fwrite(rec.data(), 1, rec.size(), out.f) != rec.size()) {
error = "short write to '" + path + "': " + std::strerror(errno);
return false;
}
}
if (!put(tail)) {
error = "short write to '" + path + "': " + std::strerror(errno);
return false;
}
if (std::fflush(out.f) != 0) {
error = "cannot flush '" + path + "': " + std::strerror(errno);
return false;
}
return true;
}
bool write_export(const std::string& path, const ExportTable& table, std::string& error) {
const auto ends_with = [&path](const char* ext) {
const std::size_t n = std::strlen(ext);
return path.size() >= n && path.compare(path.size() - n, n, ext) == 0;
};
if (ends_with(".csv")) return write_csv(path, table, error);
if (ends_with(".mf3") || ends_with(".mdf")) return write_mdf(path, table, error);
return write_npy(path, table, error);
}
} // namespace udpscope } // namespace udpscope
+28
View File
@@ -21,6 +21,11 @@ namespace udpscope {
struct ExportTable { struct ExportTable {
struct Column { struct Column {
std::string name; ///< the legend name, sanitised for a column label std::string name; ///< the legend name, sanitised for a column label
/// The signal's unit exactly as the CONFIG carried it, the empty string when it
/// carried none. The `.npy`/`.csv` writers do not spell it out (their readers get the
/// names and the numbers); MDF does, because the unit *is* part of what a channel is
/// there, and it is the only thing ScryLab reads a signal's unit from.
std::string unit;
std::vector<f32> values; ///< `rows()` long, the ring's own `f32` std::vector<f32> values; ///< `rows()` long, the ring's own `f32`
}; };
@@ -71,4 +76,27 @@ bool write_npy(const std::string& path, const ExportTable& table, std::string& e
/// enough digits to round-trip each column's own type and no more. /// enough digits to round-trip each column's own type and no more.
bool write_csv(const std::string& path, const ExportTable& table, std::string& error); bool write_csv(const std::string& path, const ExportTable& table, std::string& error);
/// Writes the same table as **MDF 3.30** (`.mf3`/`.mdf`), byte for byte by hand -- there is no
/// writer library here and MF4 (HDF5) is the one format this app will not link.
///
/// One channel group, all channels record-interleaved, master first, which is the shape the
/// [ScryLab data sources](https://docs.scrylab.de/docs/data-sources/) contract asks for: *every
/// data channel becomes a signal* (the CNBLOCK's `short_name`), *the master channel of each
/// group is the x axis* (our master is `abs_seconds`, named `time`), and *the unit is taken from
/// the channel metadata* (MDF 3 keeps it in the channel's conversion block, the CCBLOCK). A
/// second occurrence of a name is dropped by that reader, so the labels are de-duplicated and cut
/// to the 31 bytes a CNBLOCK's name field holds.
///
/// The master is the **absolute** clock and the axis the plot showed is an ordinary channel,
/// `time_relative`: the relative column is `0` at a different sample in every mode, so a master
/// off it would put two exports from different modes on two timelines, while `abs_seconds` is the
/// number the stream itself stamped the samples with. Both carry the unit `s`.
bool write_mdf(const std::string& path, const ExportTable& table, std::string& error);
/// Writes `table` with the writer **`path`'s extension** names: `.csv` -> `write_csv`,
/// `.mf3`/`.mdf` -> `write_mdf`, anything else -> `write_npy` (the default, as the save dialog's
/// default filter is). The export dialog and the self-check's export probe both come through
/// here, so the two cannot offer different formats.
bool write_export(const std::string& path, const ExportTable& table, std::string& error);
} // namespace udpscope } // namespace udpscope
+1 -1
View File
@@ -64,7 +64,7 @@ panel is working correctly, not failing to connect.
7. <b>Measure it</b> -- the measurements pane computes a statistic of the visible window, and 7. <b>Measure it</b> -- the measurements pane computes a statistic of the visible window, and
<b>show on plot</b> marks it as a ruler or a pair of markers. <b>show on plot</b> marks it as a ruler or a pair of markers.
8. <b>Save the data</b> with Export visible range, which writes the plotted signals over the 8. <b>Save the data</b> with Export visible range, which writes the plotted signals over the
window at the stored rate to <tt>.npy</tt> or <tt>.csv</tt>. window at the stored rate to <tt>.npy</tt>, <tt>.csv</tt> or <tt>.mf3</tt> (MDF 3.30).
<b>Worth knowing</b> <b>Worth knowing</b>
+14 -3
View File
@@ -1,7 +1,8 @@
<b>Saving and export</b> <b>Saving and export</b>
Export writes the <i>visible range</i> of every plotted trace to a file: NumPy <tt>.npy</tt> or Export writes the <i>visible range</i> of every plotted trace to a file: NumPy <tt>.npy</tt>,
<tt>.csv</tt>, chosen by the extension you type. <tt>.csv</tt>, or <b>MDF 3.30</b> (<tt>.mf3</tt> or <tt>.mdf</tt>), chosen by the extension you
type.
<b>How to reach it</b> <b>How to reach it</b>
@@ -24,10 +25,20 @@ the format follows the path's extension.
evenly spaced; duplicate or backwards stamps would mean the file held two clocks. evenly spaced; duplicate or backwards stamps would mean the file held two clocks.
• An export taken <i>mid-capture</i> writes the partly filled window that is on screen, which is • An export taken <i>mid-capture</i> writes the partly filled window that is on screen, which is
shorter than the event span. Export outside a capture if you want the whole event. shorter than the event span. Export outside a capture if you want the whole event.
• <b>MDF</b> exists for handing a window to another tool, and its shape follows the contract that
reader uses: one channel group, every channel record-interleaved, the master channel as the x
axis and the unit in the channel metadata. So there the <b>master is <tt>abs_seconds</tt></b>,
named <tt>time</tt> — the clock that is the same in every display mode — and the display axis
goes out beside it as an ordinary channel, <tt>time_relative</tt>. Both are seconds; each signal
carries the source's own unit. That is also what <b>ScryLab</b> reads: channel name → signal
name, master → x axis, unit → the signal's unit.
<b>Example</b> <b>Example</b>
A 2 s window at 10 kS/s over 3 traces writes 20001 rows and 5 columns A 2 s window at 10 kS/s over 3 traces writes 20001 rows and 5 columns
(<tt>time</tt>, <tt>abs_seconds</tt>, three signals). The <tt>.npy</tt> is (<tt>time</tt>, <tt>abs_seconds</tt>, three signals). The <tt>.npy</tt> is
<tt>64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)</tt> bytes. Read it back with <tt>64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)</tt> bytes. Read it back with
<tt>pd.DataFrame(np.load(path))</tt>. <tt>pd.DataFrame(np.load(path))</tt>. The <tt>.mf3</tt> is bigger — a block per channel plus a
comment — <tt>272 + comment + rows·record + 28 + channels·274 + 30</tt> bytes, with
<tt>record = 16 + 4·cols</tt> and <tt>channels = cols + 2</tt> for the master and the relative
axis. Read that one back with <tt>asammdf.MDF(path)</tt> or in ScryLab.
+11 -3
View File
@@ -2371,10 +2371,10 @@ void on_export_response(GObject* source, GAsyncResult* result, gpointer user) {
} }
// The extension picks the writer, so the filter the user chose is honoured by way of // The extension picks the writer, so the filter the user chose is honoured by way of
// the name it put in the entry rather than by a second piece of state. // the name it put in the entry rather than by a second piece of state. `.npy` is what an
const bool csv = path.size() >= 4 && path.compare(path.size() - 4, 4, ".csv") == 0; // unknown extension gets, which is the dialog's default filter too.
std::string error; std::string error;
const bool ok = csv ? write_csv(path, ctx->table, error) : write_npy(path, ctx->table, error); const bool ok = write_export(path, ctx->table, error);
if (!ok) { if (!ok) {
set_notice(*ctx->app, "export failed: " + error); set_notice(*ctx->app, "export failed: " + error);
return; return;
@@ -2411,10 +2411,18 @@ void on_export_clicked(GtkButton*, gpointer user) {
gtk_file_filter_set_name(csv, "CSV (*.csv)"); gtk_file_filter_set_name(csv, "CSV (*.csv)");
gtk_file_filter_add_pattern(csv, "*.csv"); gtk_file_filter_add_pattern(csv, "*.csv");
g_list_store_append(filters, csv); g_list_store_append(filters, csv);
// MDF 3.30, which is what ScryLab (and every MDF reader) opens: offered by both of the
// extensions the format is stored under.
GtkFileFilter* mdf = gtk_file_filter_new();
gtk_file_filter_set_name(mdf, "MDF 3.30 (*.mf3, *.mdf)");
gtk_file_filter_add_pattern(mdf, "*.mf3");
gtk_file_filter_add_pattern(mdf, "*.mdf");
g_list_store_append(filters, mdf);
gtk_file_dialog_set_filters(dialog, G_LIST_MODEL(filters)); gtk_file_dialog_set_filters(dialog, G_LIST_MODEL(filters));
gtk_file_dialog_set_default_filter(dialog, npy); gtk_file_dialog_set_default_filter(dialog, npy);
g_object_unref(npy); g_object_unref(npy);
g_object_unref(csv); g_object_unref(csv);
g_object_unref(mdf);
g_object_unref(filters); g_object_unref(filters);
gtk_file_dialog_save(dialog, GTK_WINDOW(app->window), nullptr, on_export_response, gtk_file_dialog_save(dialog, GTK_WINDOW(app->window), nullptr, on_export_response,
+14 -3
View File
@@ -2345,11 +2345,10 @@ gboolean export_probe(gpointer user) {
std::fprintf(stderr, "[export-probe] column '%s' n=%zu\n", col.name.c_str(), col.values.size()); std::fprintf(stderr, "[export-probe] column '%s' n=%zu\n", col.name.c_str(), col.values.size());
} }
for (const char* ext : {".npy", ".csv"}) { for (const char* ext : {".npy", ".csv", ".mf3"}) {
const std::string path = prefix + ext; const std::string path = prefix + ext;
std::string error; std::string error;
const bool ok = std::strcmp(ext, ".csv") == 0 ? write_csv(path, table, error) const bool ok = write_export(path, table, error);
: write_npy(path, table, error);
long bytes = -1; long bytes = -1;
if (ok) { if (ok) {
if (std::FILE* f = std::fopen(path.c_str(), "rb"); f != nullptr) { if (std::FILE* f = std::fopen(path.c_str(), "rb"); f != nullptr) {
@@ -2361,6 +2360,18 @@ gboolean export_probe(gpointer user) {
std::fprintf(stderr, "[export-probe] %s %s bytes=%ld%s%s\n", path.c_str(), std::fprintf(stderr, "[export-probe] %s %s bytes=%ld%s%s\n", path.c_str(),
ok ? "written" : "FAILED", bytes, ok ? "" : " error=", ok ? "" : error.c_str()); ok ? "written" : "FAILED", bytes, ok ? "" : " error=", ok ? "" : error.c_str());
} }
// What the `.mf3` was handed, in the shape the file stores it: the master and the relative
// axis are channels too, and `record` is the bytes of one interleaved record. So the size the
// line above printed is `272 + comment + rows*record + 28 + channels*274 + 30`, and the units
// are the ones ScryLab reads each signal's unit from -- the CONFIG's, per column.
std::string units;
for (const char* u : {"s", "s"}) units += std::string(units.empty() ? "" : " ") + "'" + u + "'";
for (const ExportTable::Column& col : table.columns) {
units += std::string(units.empty() ? "" : " ") + "'" + col.unit + "'";
}
std::fprintf(stderr,
"[export-probe] mdf master=time master_unit=s channels=%zu record=%zu units=[%s]\n",
table.columns.size() + 2, 16 + 4 * table.columns.size(), units.c_str());
std::fflush(stderr); std::fflush(stderr);
return G_SOURCE_REMOVE; return G_SOURCE_REMOVE;
} }