feat: the export writes an MDF 3.30 file, so a window loads in ScryLab
A third format beside `.npy` and `.csv`, written by hand in `src/export.cpp` -- MF4 is HDF5 and would mean a library, MF3 is a handful of fixed blocks. The layout is the asammdf 8.8.27 reader's own field list, byte-diffed against a known-good 3.30 file it wrote; the shape is the one ScryLab's data-source contract asks for: one channel group, record-interleaved, the master as the x axis, and the unit in the channel's conversion block. So in MDF the master is `abs_seconds`, named `time` -- the clock that is the same in every display mode -- and the plotted axis goes out as an ordinary channel, `time_relative`. `ExportTable::Column` gained the CONFIG's unit, which only this writer spells out; the `.npy`/`.csv` bytes are unchanged, and `write_export()` now picks the writer for both the dialog and the probe.
This commit is contained in:
1 parent
761615c2fe
commit
908286ee89
10 files changed
+505
-28
No files matched your search
@@ -89,7 +89,7 @@ src/channel.* ChannelBuffer (a ring pair as a plot::data::Series) and SlotS
|
|||||||
src/acquisition.* the UDPStreamer client, the producer thread, frame decoding
|
src/acquisition.* the UDPStreamer client, the producer thread, frame decoding
|
||||||
src/trigger.* the trigger configuration and the producer-side state machine
|
src/trigger.* the trigger configuration and the producer-side state machine
|
||||||
src/plot_host.* the one plot::Figure, its rows' traces and cursors, the x link group
|
src/plot_host.* the one plot::Figure, its rows' traces and cursors, the x link group
|
||||||
src/export.* the visible range as a table, and the .npy / .csv writers
|
src/export.* the visible range as a table, and the .npy / .csv / .mf3 writers
|
||||||
src/layout_store.* the per-source session layout file (TOML), written and read back
|
src/layout_store.* the per-source session layout file (TOML), written and read back
|
||||||
src/event_series.hpp an immutable captured event as a plot::data::Series
|
src/event_series.hpp an immutable captured event as a plot::data::Series
|
||||||
src/ui.* the GTK4 window: toolbar, trigger panel, signal panel, plot box
|
src/ui.* the GTK4 window: toolbar, trigger panel, signal panel, plot box
|
||||||
@@ -2261,16 +2261,42 @@ magnitudes, so consecutive differences wobble by a few ULP of the stream's uptim
|
|||||||
(femtoseconds at any plausible uptime — note that the `f32` defect this used to guard against
|
(femtoseconds at any plausible uptime — note that the `f32` defect this used to guard against
|
||||||
was *microseconds*, four orders of magnitude up).
|
was *microseconds*, four orders of magnitude up).
|
||||||
|
|
||||||
Two formats, picked by the extension the save dialog ends up with:
|
Three formats, picked by the extension the save dialog ends up with:
|
||||||
|
|
||||||
| file | what it is |
|
| file | what it is |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `.npy` | one NumPy structured array, `time`/`abs_seconds` as `<f8` and every signal as `<f4` — the width the ring stores, so the file invents no precision and is about half an all-`f8` one. `pd.DataFrame(np.load(path))` is the whole reader. |
|
| `.npy` | one NumPy structured array, `time`/`abs_seconds` as `<f8` and every signal as `<f4` — the width the ring stores, so the file invents no precision and is about half an all-`f8` one. `pd.DataFrame(np.load(path))` is the whole reader. |
|
||||||
| `.csv` | the same table as text, `%.17g` for the times and `%.9g` for the values: the shortest widths that round-trip each type exactly. |
|
| `.csv` | the same table as text, `%.17g` for the times and `%.9g` for the values: the shortest widths that round-trip each type exactly. |
|
||||||
|
| `.mf3`, `.mdf` | **MDF 3.30**, one channel group with every channel record-interleaved, written by hand — see *The MDF export* below. |
|
||||||
|
|
||||||
Parquet (the original ask) would have meant linking Arrow — tens of megabytes of
|
Parquet (the original ask) would have meant linking Arrow — tens of megabytes of
|
||||||
dependency for a feature whose output a three-line reader already handles.
|
dependency for a feature whose output a three-line reader already handles.
|
||||||
|
|
||||||
|
**The MDF export** is the format for handing a window to another tool, and specifically to
|
||||||
|
[ScryLab](https://docs.scrylab.de/docs/data-sources/), whose data-source contract is three
|
||||||
|
rules this file is shaped by: *every data channel becomes a signal* — the channel name is the
|
||||||
|
signal name — *the master channel of each channel group is the x axis*, and *the unit is taken
|
||||||
|
from the channel metadata*, which in MDF 3 means the channel's conversion block (`CCBLOCK`) and
|
||||||
|
nowhere else. A second occurrence of a name is dropped by that reader, so the channel names are
|
||||||
|
de-duplicated against each other and the master's.
|
||||||
|
|
||||||
|
Two things follow from that, and both differ from what the `.npy`/`.csv` table holds:
|
||||||
|
|
||||||
|
- **The master is `abs_seconds`, named `time`**, not the axis the plot showed. The relative
|
||||||
|
column is 0 at a different sample in each display mode, so a master off it would put a live
|
||||||
|
export and a held event's export on two different timelines — the one thing that column
|
||||||
|
exists to avoid. The relative axis goes out as an ordinary channel, **`time_relative`**,
|
||||||
|
right after the master, and both carry the unit `s`. The signal columns follow in the same
|
||||||
|
order the `.npy`/`.csv` file writes them.
|
||||||
|
- **The unit travels with a column**, taken from the stream's CONFIG (or a math channel's own),
|
||||||
|
and is written into the conversion block. The `.npy`/`.csv` writers are unchanged: their
|
||||||
|
readers get the names and the numbers, and their bytes are what they always were.
|
||||||
|
|
||||||
|
There is **no MDF writer library here and no new dependency**: the blocks are spelled out in
|
||||||
|
`src/export.cpp` against the field list the `asammdf` 8.8.27 reader walks the format with, and
|
||||||
|
`docs/claude/invariants-session-and-export.md` records the layout, the version decision (MDF
|
||||||
|
3.30, not MF4 — that is HDF5) and what was verified against a known-good reference file.
|
||||||
|
|
||||||
The table is collected when the menu item is clicked, not when the dialog is
|
The table is collected when the menu item is clicked, not when the dialog is
|
||||||
answered: a live view keeps scrolling while the file chooser is up. The live cut goes
|
answered: a live view keeps scrolling while the file chooser is up. The live cut goes
|
||||||
through `ScopeModel::live_window_locked()`, the same index range `publish_event()`
|
through `ScopeModel::live_window_locked()`, the same index range `publish_event()`
|
||||||
|
|||||||
@@ -52,6 +52,40 @@ Part of [CLAUDE.md](../../CLAUDE.md). These are rules that cost a defect to lear
|
|||||||
- **The export is collected at click time, not in the file-dialog callback.** The live
|
- **The export is collected at click time, not in the file-dialog callback.** The live
|
||||||
view keeps scrolling while the chooser is up, so a late collection would write a
|
view keeps scrolling while the chooser is up, so a late collection would write a
|
||||||
window the user never saw. The writer is picked by the chosen path's extension.
|
window the user never saw. The writer is picked by the chosen path's extension.
|
||||||
|
- **The MDF export is MDF 3.30 written by hand, and its layout is the reader's, not the
|
||||||
|
standard's prose.** No dependency is added: MF4 is HDF5 and would need a library, while MF3 is a
|
||||||
|
handful of fixed-size blocks, so `write_mdf()` spells them out. The field list comes from
|
||||||
|
`asammdf` 8.8.27 (`blocks/v2_v3_blocks.py`, `blocks/mdf_v3.py`, which parses MF3 by explicit
|
||||||
|
offsets) and was confirmed byte for byte against a known-good 3.30 file asammdf wrote (one
|
||||||
|
group, a `time` master and two `V` channels). The shape, in file order: the **ID block** is 64
|
||||||
|
bytes with *no id/length header* (`"MDF 3.30"`, byte order, version `330`) — a writer that
|
||||||
|
emits `ID` + size + links at offset 0 produces something no reader accepts, since the first real
|
||||||
|
block starts at 64; the **HD block** is 208 (164 common + 44 of the 3.2-era time fields) and
|
||||||
|
points at the data group; a **TX** comment follows it; then the **samples**; the **DG** is 28; a
|
||||||
|
**CC** (conversion, 46 — a `NONE` conversion is the common fields alone) then a **CN** (channel,
|
||||||
|
228) precede each channel; and the **CG** (30) is last. Address fields are absolute file offsets
|
||||||
|
written *forward*, so the whole address map is computed before a byte is written, and every
|
||||||
|
block length is checked after it is built: a field at the wrong width does not fail the write, it
|
||||||
|
leaves the reader in the middle of the next block.
|
||||||
|
- **The data group's data address holds the raw records, with no `DT` header** — which is what
|
||||||
|
asammdf's *writer* does and what its *reader* assumes (`DataBlock` is documented as "a pseudo
|
||||||
|
block not defined by the MDF 3 standard"). This is the one place the file follows the reader
|
||||||
|
rather than a narrow reading of the standard, and it is the assumption ScryLab compatibility
|
||||||
|
rests on: a file with a `DT` block there would have its header read as sample bytes.
|
||||||
|
- **In MDF the master is `abs_seconds`, named `time`, and the relative axis is a channel
|
||||||
|
`time_relative`** — the same two numbers the `.npy`/`.csv` table holds, in the other roles. A
|
||||||
|
master off the relative column is 0 at a different sample in each display mode, so it would put a
|
||||||
|
live export and a held event's export on two timelines, which is the one thing the second column
|
||||||
|
exists to avoid; both carry the unit `s`, and the signal columns follow in the `.npy`/`.csv`
|
||||||
|
order. **The unit** is the client's CONFIG unit (a math channel's own), and it is written into
|
||||||
|
the CC block because that is the only place MDF 3 keeps one — and the place ScryLab reads a
|
||||||
|
signal's unit from. `ExportTable::Column` gained the field for it; the two older writers ignore
|
||||||
|
it, and their bytes are unchanged. **The names** are sanitised and de-duplicated against each
|
||||||
|
other *and* against the two reserved names before being cut to 31 bytes (the CN name field is 32
|
||||||
|
with its NUL): ScryLab imports only the **first** occurrence of a channel name, so a cut made
|
||||||
|
before de-duplication could silently drop a signal. `write_export()` is the one place the
|
||||||
|
extension picks the writer (`.csv`, `.mf3`/`.mdf`, anything else `.npy`), and both the dialog and
|
||||||
|
the self-check's export probe come through it so the two cannot offer different formats.
|
||||||
- **The chooser runs before `ScopeModel` exists, and `--server` is what decides.** `Config` carries
|
- **The chooser runs before `ScopeModel` exists, and `--server` is what decides.** `Config` carries
|
||||||
`server_specified`, set only by the `--server` branch, and `main()` asks only when it is false —
|
`server_specified`, set only by the `--server` branch, and `main()` asks only when it is false —
|
||||||
so declining costs no buffer and no socket, and `ScopeModel`, `Acquisition`, `RowSet` and
|
so declining costs no buffer and no socket, and `ScopeModel`, `Acquisition`, `RowSet` and
|
||||||
|
|||||||
@@ -126,10 +126,27 @@ Two lines of the self-check output are assertions in disguise:
|
|||||||
An export taken **mid-capture** reports `mode=live` with `time=[-pre..e]` where `e` is how far
|
An export taken **mid-capture** reports `mode=live` with `time=[-pre..e]` where `e` is how far
|
||||||
the post window has filled — the export is what is on screen, and during a capture that is a
|
the post window has filled — the export is what is on screen, and during a capture that is a
|
||||||
partly filled window. Pin the probe outside a capture when checking the `rows=` count.
|
partly filled window. Pin the probe outside a capture when checking the `rows=` count.
|
||||||
The two `bytes=` lines are the writers: the `.npy` is
|
The three `bytes=` lines are the writers: the `.npy` is
|
||||||
`64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)`, so a size that is not that is a
|
`64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)`, so a size that is not that is a
|
||||||
mispacked record. Read it back with `pd.DataFrame(np.load(path))` after changing
|
mispacked record. Read it back with `pd.DataFrame(np.load(path))` after changing
|
||||||
either writer.
|
either writer.
|
||||||
|
The `.mf3` line's size is the MDF 3.30 layout the writer spells out, and every term of it is
|
||||||
|
observable: `272 + comment + rows·record + 28 + channels·274 + 30`, where the ID block is 64,
|
||||||
|
the HD block 208, the comment is a `TX` block of its text plus 5 bytes, `record` is
|
||||||
|
`16 + 4·cols` (two `f64` time columns then one `f32` per signal), the data group is 28, each of
|
||||||
|
`channels = cols + 2` gets a 46-byte conversion block and a 228-byte channel block, and the
|
||||||
|
channel group is 30. Read the file back with asammdf: `MDF(path).info()` must show one group
|
||||||
|
whose `channel 0` is `name="time" type=master`, and `get(name, group=0).unit` must be the unit
|
||||||
|
the next line reports.
|
||||||
|
`[export-probe] mdf master=time master_unit=s channels=N record=B units=['s' 's' 'V' …]`
|
||||||
|
is the other half, and it is the assertion on the units: `channels` and `record` are the two
|
||||||
|
numbers the `.mf3` size above is built from (so `record` must be `16 + 4·cols` and `channels`
|
||||||
|
`cols + 2`), and `units` must be **the CONFIG's**, in the table's column order, behind the
|
||||||
|
master's and the relative axis's two `s` — one `'s' 's'` pair plus one entry per exported
|
||||||
|
column, each exactly the unit the stream sent (an empty one reads `''`). A `V` where the mock
|
||||||
|
sent `s`, or a missing entry, is the unit not reaching the conversion block, which is the only
|
||||||
|
thing ScryLab reads a signal's unit from; `write_npy`/`write_csv` must be unchanged either way
|
||||||
|
(their bytes are pinned by the two lines above).
|
||||||
- `[window-probe] <action> label=.. entry=.. unit=U window=W event_span=E cap=C decim=M
|
- `[window-probe] <action> label=.. entry=.. unit=U window=W event_span=E cap=C decim=M
|
||||||
stored=R aa=F buffered=B` (stderr, `UDPSCOPE_PROBE_WINDOW_MS`) — **`window` must equal
|
stored=R aa=F buffered=B` (stderr, `UDPSCOPE_PROBE_WINDOW_MS`) — **`window` must equal
|
||||||
`event_span`** on every line (the trigger's event span is the display window; a capture
|
`event_span`** on every line (the trigger's event span is the display window; a capture
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ xvfb-run -a env UDPSCOPE_EXIT_AFTER_MS=8000 UDPSCOPE_ENABLE_ALL_MS=1000 \
|
|||||||
|
|
||||||
Two baselines every run is read against. A clean build emits **four** warnings and no more: two
|
Two baselines every run is read against. A clean build emits **four** warnings and no more: two
|
||||||
`-Wmissing-field-initializers` on `MathNode::source` and `-Woverlength-strings` on **both**
|
`-Wmissing-field-initializers` on `MathNode::source` and `-Woverlength-strings` on **both**
|
||||||
generated GResource `.c` files — the help pages' (**514660** bytes) and the icons', whose blob is
|
generated GResource `.c` files — the help pages' (**515700** bytes) and the icons', whose blob is
|
||||||
**12352** against C99's 4095-byte string-literal limit with the seven action icons and the two app icons
|
**12352** against C99's 4095-byte string-literal limit with the seven action icons and the two app icons
|
||||||
that remain (a note about a generated file, which GCC itself supports) -- re-measured when
|
that remain (a note about a generated file, which GCC itself supports) -- re-measured when
|
||||||
the tab work changed the set (that icon left, the two tab glyphs arrived, and this file's
|
the tab work changed the set (that icon left, the two tab glyphs arrived, and this file's
|
||||||
@@ -194,8 +194,8 @@ precedent, not an exception.
|
|||||||
nothing. Its four lines are in
|
nothing. Its four lines are in
|
||||||
[probe-readings.md](probe-readings.md).
|
[probe-readings.md](probe-readings.md).
|
||||||
- `UDPSCOPE_EXPORT_MS` (+ optional `UDPSCOPE_EXPORT_PATH`, default
|
- `UDPSCOPE_EXPORT_MS` (+ optional `UDPSCOPE_EXPORT_PATH`, default
|
||||||
`/tmp/udpscope-export`) — collect the export table and write it both ways, skipping
|
`/tmp/udpscope-export`) — collect the export table and write it all three ways (`.npy`,
|
||||||
only the file chooser.
|
`.csv`, `.mf3`), skipping only the file chooser.
|
||||||
|
|
||||||
- `UDPSCOPE_PROBE_CONDITION_MS` — arms a qualified trigger against the mock and reports the latched
|
- `UDPSCOPE_PROBE_CONDITION_MS` — arms a qualified trigger against the mock and reports the latched
|
||||||
`t0`, the measured pulse or band width, and sample values for the window condition. Schedules its
|
`t0`, the measured pulse or band width, and sample values for the window condition. Schedules its
|
||||||
|
|||||||
+354
-12
@@ -6,8 +6,11 @@
|
|||||||
#include <algorithm>
|
#include <algorithm>
|
||||||
#include <bit>
|
#include <bit>
|
||||||
#include <cerrno>
|
#include <cerrno>
|
||||||
|
#include <cstdint>
|
||||||
#include <cstdio>
|
#include <cstdio>
|
||||||
#include <cstring>
|
#include <cstring>
|
||||||
|
#include <ctime>
|
||||||
|
#include <utility>
|
||||||
|
|
||||||
namespace udpscope {
|
namespace udpscope {
|
||||||
namespace {
|
namespace {
|
||||||
@@ -34,18 +37,29 @@ std::string sanitise(const std::string& name) {
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The final labels: sanitised, and de-duplicated against each other *and* against the
|
/// `base` carrying a `_2`, `_3` … suffix for the `n`-th copy (`n == 1` is the name itself),
|
||||||
/// two time columns, with a `_2`, `_3` suffix. Two elements of one vector signal can
|
/// cut to `max_len` bytes when one is given -- MDF's name field is 32 bytes wide and the file
|
||||||
/// easily sanitise to the same string.
|
/// is re-read by a reader that keeps only the *first* occurrence of a name, so a cut that made
|
||||||
std::vector<std::string> column_labels(const ExportTable& table) {
|
/// two channels equal would silently lose one.
|
||||||
std::vector<std::string> taken{"time", "abs_seconds"};
|
std::string numbered(const std::string& base, int n, std::size_t max_len) {
|
||||||
|
const std::string suffix = n == 1 ? std::string() : "_" + std::to_string(n);
|
||||||
|
std::size_t room = base.size();
|
||||||
|
if (max_len != 0) room = max_len > suffix.size() ? max_len - suffix.size() : 0;
|
||||||
|
return base.substr(0, room) + suffix;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The final labels: sanitised, and de-duplicated against each other *and* against
|
||||||
|
/// `taken` -- the names the writer already spends -- with a `_2`, `_3` suffix. Two
|
||||||
|
/// elements of one vector signal can easily sanitise to the same string.
|
||||||
|
std::vector<std::string> column_labels(const ExportTable& table, std::vector<std::string> taken,
|
||||||
|
std::size_t max_len = 0) {
|
||||||
std::vector<std::string> out;
|
std::vector<std::string> out;
|
||||||
out.reserve(table.columns.size());
|
out.reserve(table.columns.size());
|
||||||
for (const ExportTable::Column& col : table.columns) {
|
for (const ExportTable::Column& col : table.columns) {
|
||||||
const std::string base = sanitise(col.name);
|
const std::string base = sanitise(col.name);
|
||||||
std::string label = base;
|
std::string label = numbered(base, 1, max_len);
|
||||||
for (int n = 2; std::find(taken.begin(), taken.end(), label) != taken.end(); ++n) {
|
for (int n = 2; std::find(taken.begin(), taken.end(), label) != taken.end(); ++n) {
|
||||||
label = base + "_" + std::to_string(n);
|
label = numbered(base, n, max_len);
|
||||||
}
|
}
|
||||||
taken.push_back(label);
|
taken.push_back(label);
|
||||||
out.push_back(std::move(label));
|
out.push_back(std::move(label));
|
||||||
@@ -69,6 +83,74 @@ bool opened(const File& out, const std::string& path, std::string& error) {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The block lengths of MDF 3.30, from the **reader's** own field lists
|
||||||
|
/// (`asammdf/blocks/v2_v3_constants.py`, 8.8.27): every block is a two-letter id, a 16-bit
|
||||||
|
/// length and then its own fields, and the reader walks the file by those lengths alone, so a
|
||||||
|
/// block written one field too wide is a file whose reader lands in the middle of the next one.
|
||||||
|
/// The widths here are the ones a `bytes(block)` of the reference file measures (the known-good
|
||||||
|
/// 3.30 file `/tmp/ref.mdf` this writer was byte-diffed against).
|
||||||
|
constexpr std::size_t kMdfIdBlock = 64; // "MDF 3.30" and the flags, no id/length
|
||||||
|
constexpr std::size_t kMdfHeader = 208; // 164 common + 44 of the 3.2-era time fields
|
||||||
|
constexpr std::size_t kMdfTxOverhead = 5; // "TX", the u16 length and the terminating NUL
|
||||||
|
constexpr std::size_t kMdfDataGroup = 28; // 4 addresses, the group count, 4 reserved
|
||||||
|
constexpr std::size_t kMdfConversion = 46; // a NONE conversion is the common fields alone
|
||||||
|
constexpr std::size_t kMdfChannel = 228; // the display-name-length CNBLOCK
|
||||||
|
constexpr std::size_t kMdfChannelGroup = 30; // 26 + the sample-reduction address of 3.30
|
||||||
|
constexpr std::size_t kMdfNameBytes = 31; // a CNBLOCK's 32-byte name field, less its NUL
|
||||||
|
|
||||||
|
/// A byte sink for the MDF header blocks, which are a few hundred bytes each and have to be
|
||||||
|
/// laid out field by field. The little-endian `raw()` writes are the same assumption the `.npy`
|
||||||
|
/// writer states at the top of this file, and `static_assert`s there.
|
||||||
|
struct Bytes {
|
||||||
|
std::vector<unsigned char> v;
|
||||||
|
|
||||||
|
void raw(const void* p, std::size_t n) {
|
||||||
|
const auto* b = static_cast<const unsigned char*>(p);
|
||||||
|
v.insert(v.end(), b, b + n);
|
||||||
|
}
|
||||||
|
void u16(std::uint16_t x) {
|
||||||
|
const unsigned char b[2] = {static_cast<unsigned char>(x & 0xFFu),
|
||||||
|
static_cast<unsigned char>((x >> 8) & 0xFFu)};
|
||||||
|
raw(b, sizeof(b));
|
||||||
|
}
|
||||||
|
void i16(std::int16_t x) { u16(static_cast<std::uint16_t>(x)); }
|
||||||
|
void u32(std::uint32_t x) {
|
||||||
|
const unsigned char b[4] = {static_cast<unsigned char>(x & 0xFFu),
|
||||||
|
static_cast<unsigned char>((x >> 8) & 0xFFu),
|
||||||
|
static_cast<unsigned char>((x >> 16) & 0xFFu),
|
||||||
|
static_cast<unsigned char>((x >> 24) & 0xFFu)};
|
||||||
|
raw(b, sizeof(b));
|
||||||
|
}
|
||||||
|
void u64(std::uint64_t x) {
|
||||||
|
u32(static_cast<std::uint32_t>(x & 0xFFFFFFFFu));
|
||||||
|
u32(static_cast<std::uint32_t>(x >> 32));
|
||||||
|
}
|
||||||
|
void f64(double x) { raw(&x, sizeof(x)); }
|
||||||
|
void f32(float x) { raw(&x, sizeof(x)); }
|
||||||
|
void zeros(std::size_t n) { v.insert(v.end(), n, 0); }
|
||||||
|
|
||||||
|
/// A fixed-width text field: the bytes, then NUL padding. MDF's names and units are
|
||||||
|
/// NUL-terminated inside a field of a width the reader is told by the block size, so this is
|
||||||
|
/// how every string in the format is spelled.
|
||||||
|
void text(const std::string& s, std::size_t width) {
|
||||||
|
const std::size_t n = std::min(s.size(), width);
|
||||||
|
raw(s.data(), n);
|
||||||
|
zeros(width - n);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::size_t size() const noexcept { return v.size(); }
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Checked rather than trusted: a field written at the wrong width would not fail here, it
|
||||||
|
/// would produce a file whose reader walks into the middle of the next block. The widths are
|
||||||
|
/// the asammdf 8.8.27 reader's own field lists (`blocks/v2_v3_constants.py`).
|
||||||
|
bool mdf_block_size(const Bytes& block, std::size_t want, const char* what, std::string& error) {
|
||||||
|
if (block.size() == want) return true;
|
||||||
|
error = std::string("internal: the MDF ") + what + " block is " + std::to_string(block.size()) +
|
||||||
|
" bytes, the layout wants " + std::to_string(want);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
} // namespace
|
} // namespace
|
||||||
|
|
||||||
ExportTable collect_export(ScopeModel& model, RowSet& rows) {
|
ExportTable collect_export(ScopeModel& model, RowSet& rows) {
|
||||||
@@ -97,6 +179,12 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
|
|||||||
return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].name
|
return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].name
|
||||||
: std::string("ch") + std::to_string(c);
|
: std::string("ch") + std::to_string(c);
|
||||||
};
|
};
|
||||||
|
// The unit travels with the column so the MDF writer can put it on the channel; the other
|
||||||
|
// writers ignore it, as they always have.
|
||||||
|
const auto unit_of = [&](int c) {
|
||||||
|
return static_cast<std::size_t>(c) < infos.size() ? infos[static_cast<std::size_t>(c)].unit
|
||||||
|
: std::string();
|
||||||
|
};
|
||||||
|
|
||||||
// What the rows bind to is what gets exported, so the split is the same `DisplayMode`
|
// What the rows bind to is what gets exported, so the split is the same `DisplayMode`
|
||||||
// `RowSet::bind()` switches on: the frozen event only while one is held, the live ring
|
// `RowSet::bind()` switches on: the frozen event only while one is held, the live ring
|
||||||
@@ -136,8 +224,9 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
|
|||||||
if (ch >= snap->values.size()) continue;
|
if (ch >= snap->values.size()) continue;
|
||||||
const std::vector<f32>& src = snap->values[ch];
|
const std::vector<f32>& src = snap->values[ch];
|
||||||
if (src.size() < first + n) continue;
|
if (src.size() < first + n) continue;
|
||||||
table.columns.push_back({name_of(c), std::vector<f32>(src.begin() + static_cast<std::ptrdiff_t>(first),
|
table.columns.push_back({name_of(c), unit_of(c),
|
||||||
src.begin() + static_cast<std::ptrdiff_t>(first + n))});
|
std::vector<f32>(src.begin() + static_cast<std::ptrdiff_t>(first),
|
||||||
|
src.begin() + static_cast<std::ptrdiff_t>(first + n))});
|
||||||
}
|
}
|
||||||
return table;
|
return table;
|
||||||
}
|
}
|
||||||
@@ -177,7 +266,7 @@ ExportTable collect_export(ScopeModel& model, RowSet& rows) {
|
|||||||
if (ch >= model.channel_count()) continue;
|
if (ch >= model.channel_count()) continue;
|
||||||
std::vector<f32> values(static_cast<std::size_t>(n));
|
std::vector<f32> values(static_cast<std::size_t>(n));
|
||||||
store.values(ch).copy_from(first, n, values.data());
|
store.values(ch).copy_from(first, n, values.data());
|
||||||
table.columns.push_back({name_of(c), std::move(values)});
|
table.columns.push_back({name_of(c), unit_of(c), std::move(values)});
|
||||||
}
|
}
|
||||||
return table;
|
return table;
|
||||||
}
|
}
|
||||||
@@ -191,6 +280,7 @@ ExportTable event_table(ScopeModel& model, const EventSnapshot& snapshot) {
|
|||||||
for (std::size_t c = 0; c < snapshot.values.size() && c < channels.size(); ++c) {
|
for (std::size_t c = 0; c < snapshot.values.size() && c < channels.size(); ++c) {
|
||||||
ExportTable::Column column;
|
ExportTable::Column column;
|
||||||
column.name = channels[c].name;
|
column.name = channels[c].name;
|
||||||
|
column.unit = channels[c].unit;
|
||||||
column.values = snapshot.values[c];
|
column.values = snapshot.values[c];
|
||||||
table.columns.push_back(std::move(column));
|
table.columns.push_back(std::move(column));
|
||||||
}
|
}
|
||||||
@@ -202,7 +292,7 @@ bool write_npy(const std::string& path, const ExportTable& table, std::string& e
|
|||||||
error = "nothing to export: no signal is plotted in the visible range";
|
error = "nothing to export: no signal is plotted in the visible range";
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
const std::vector<std::string> labels = column_labels(table);
|
const std::vector<std::string> labels = column_labels(table, {"time", "abs_seconds"});
|
||||||
|
|
||||||
// `descr` as a list of (name, format) pairs is a *packed* structured dtype -- NumPy
|
// `descr` as a list of (name, format) pairs is a *packed* structured dtype -- NumPy
|
||||||
// only inserts alignment padding when asked to (`align=True`) -- so the records below
|
// only inserts alignment padding when asked to (`align=True`) -- so the records below
|
||||||
@@ -266,7 +356,7 @@ bool write_csv(const std::string& path, const ExportTable& table, std::string& e
|
|||||||
File out(path);
|
File out(path);
|
||||||
if (!opened(out, path, error)) return false;
|
if (!opened(out, path, error)) return false;
|
||||||
|
|
||||||
const std::vector<std::string> labels = column_labels(table);
|
const std::vector<std::string> labels = column_labels(table, {"time", "abs_seconds"});
|
||||||
std::string line = "time,abs_seconds";
|
std::string line = "time,abs_seconds";
|
||||||
for (const std::string& label : labels) line += "," + label;
|
for (const std::string& label : labels) line += "," + label;
|
||||||
line += "\n";
|
line += "\n";
|
||||||
@@ -302,4 +392,256 @@ bool write_csv(const std::string& path, const ExportTable& table, std::string& e
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool write_mdf(const std::string& path, const ExportTable& table, std::string& error) {
|
||||||
|
if (table.empty()) {
|
||||||
|
error = "nothing to export: no signal is plotted in the visible range";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const std::size_t rows = table.rows();
|
||||||
|
const std::size_t cols = table.columns.size();
|
||||||
|
|
||||||
|
// **The master is the absolute clock, named `time`**, and the axis the plot showed goes out as
|
||||||
|
// an ordinary channel, `time_relative`. The relative column is 0 at a different sample in each
|
||||||
|
// display mode, so a master off it would put a live export and a held event's export on two
|
||||||
|
// timelines -- the one thing the second column exists to avoid -- while `abs_seconds` is the
|
||||||
|
// number the stream itself stamped the samples with. Both carry the unit `s`.
|
||||||
|
const std::vector<std::string> labels =
|
||||||
|
column_labels(table, {"time", "time_relative"}, kMdfNameBytes);
|
||||||
|
const std::size_t channels = cols + 2;
|
||||||
|
const std::size_t record = 2 * sizeof(f64) + cols * sizeof(f32);
|
||||||
|
if (rows > 0xFFFFFFFFull) {
|
||||||
|
error = "too many samples for an MDF 3 file: the channel group counts cycles in 32 bits";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (128 + (cols - 1) * 32 > 0xFFFFu) {
|
||||||
|
error = "too many signals for an MDF 3 record: a channel's start offset is 16 bits";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// One channel per column, described once: what the CNBLOCK says about it (where in the record
|
||||||
|
// its samples start, how wide they are, whether it is the master) and what the CCBLOCK says
|
||||||
|
// (its unit). The range is measured from the samples rather than left at zero, so
|
||||||
|
// `range_flag = 1` means what it says.
|
||||||
|
struct Channel {
|
||||||
|
std::string name;
|
||||||
|
std::string unit;
|
||||||
|
std::uint16_t bit_offset;
|
||||||
|
std::uint16_t bit_count;
|
||||||
|
std::uint16_t data_type; // MDF 3: 2 = float, 3 = double, both in the ID block's byte order
|
||||||
|
std::uint16_t type; // 0 = value, 1 = master
|
||||||
|
f64 lo, hi;
|
||||||
|
};
|
||||||
|
const auto range_of = [](const auto& values) {
|
||||||
|
if (values.empty()) return std::pair<f64, f64>{0.0, 0.0};
|
||||||
|
const auto mm = std::minmax_element(values.begin(), values.end());
|
||||||
|
return std::pair<f64, f64>{static_cast<f64>(*mm.first), static_cast<f64>(*mm.second)};
|
||||||
|
};
|
||||||
|
std::vector<Channel> chans;
|
||||||
|
chans.reserve(channels);
|
||||||
|
const auto abs_range = range_of(table.abs_seconds);
|
||||||
|
const auto rel_range = range_of(table.time);
|
||||||
|
chans.push_back({"time", "s", 0, 64, 3, 1, abs_range.first, abs_range.second});
|
||||||
|
chans.push_back({"time_relative", "s", 64, 64, 3, 0, rel_range.first, rel_range.second});
|
||||||
|
for (std::size_t i = 0; i < cols; ++i) {
|
||||||
|
const auto range = range_of(table.columns[i].values);
|
||||||
|
chans.push_back({labels[i], table.columns[i].unit,
|
||||||
|
static_cast<std::uint16_t>(128 + i * 32), 32, 2, 0, range.first, range.second});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every address in the file, from the fixed block lengths above: the writer therefore knows
|
||||||
|
// where it is going before it writes a byte, which is what lets each block carry its links.
|
||||||
|
const std::string comment = "udpscope export: the visible window, " + std::to_string(rows) +
|
||||||
|
" samples x " + std::to_string(cols) + " signals";
|
||||||
|
const std::size_t hd_end = kMdfIdBlock + kMdfHeader;
|
||||||
|
const std::size_t comment_addr = hd_end;
|
||||||
|
const std::size_t data_addr = comment_addr + kMdfTxOverhead + comment.size();
|
||||||
|
const std::size_t dg_addr = data_addr + rows * record;
|
||||||
|
const std::size_t ch_addr = dg_addr + kMdfDataGroup;
|
||||||
|
const std::size_t cg_addr = ch_addr + channels * (kMdfConversion + kMdfChannel);
|
||||||
|
const std::uint64_t total = cg_addr + kMdfChannelGroup;
|
||||||
|
if (total > 0xFFFFFFFFull) {
|
||||||
|
error = "the export is larger than an MDF 3 file can address";
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
const auto cc_addr = [&](std::size_t i) {
|
||||||
|
return ch_addr + i * (kMdfConversion + kMdfChannel);
|
||||||
|
};
|
||||||
|
|
||||||
|
// The wall clock the measurement file was written at. `abs_time` is the same instant in
|
||||||
|
// nanoseconds since the epoch, which is what the 3.2-era header field asks for.
|
||||||
|
const std::time_t now = std::time(nullptr);
|
||||||
|
std::tm tm{};
|
||||||
|
localtime_r(&now, &tm);
|
||||||
|
char date[11] = {};
|
||||||
|
char clock[9] = {};
|
||||||
|
std::strftime(date, sizeof(date), "%d:%m:%Y", &tm);
|
||||||
|
std::strftime(clock, sizeof(clock), "%H:%M:%S", &tm);
|
||||||
|
|
||||||
|
Bytes head;
|
||||||
|
// Identification block: the version string, who wrote the file, and the byte order every
|
||||||
|
// sample in it is in.
|
||||||
|
head.text("MDF ", 8);
|
||||||
|
head.text("3.30", 8);
|
||||||
|
head.text("udpscope", 8);
|
||||||
|
head.u16(0); // byte order: Intel, ahead of every value and the float format below
|
||||||
|
head.u16(0); // float format: IEEE
|
||||||
|
head.u16(330); // MDF version, digits without the separator
|
||||||
|
head.u16(0); // code page: latin-1
|
||||||
|
head.zeros(2); // reserved
|
||||||
|
head.zeros(26); // reserved
|
||||||
|
head.u16(0); // unfinalised standard flags
|
||||||
|
head.u16(0); // unfinalised custom flags
|
||||||
|
if (!mdf_block_size(head, kMdfIdBlock, "identification", error)) return false;
|
||||||
|
|
||||||
|
// Header block, which points at the data group and at the comment that follows it.
|
||||||
|
head.raw("HD", 2);
|
||||||
|
head.u16(static_cast<std::uint16_t>(kMdfHeader));
|
||||||
|
head.u32(static_cast<std::uint32_t>(dg_addr)); // first data group
|
||||||
|
head.u32(static_cast<std::uint32_t>(comment_addr));
|
||||||
|
head.u32(0); // program block: nothing wrote this but us
|
||||||
|
head.u16(1); // data groups
|
||||||
|
head.text(date, 10);
|
||||||
|
head.text(clock, 8);
|
||||||
|
head.text("", 32); // author
|
||||||
|
head.text("", 32); // department
|
||||||
|
head.text("", 32); // project
|
||||||
|
head.text("", 32); // subject
|
||||||
|
head.u64(static_cast<std::uint64_t>(now) * 1000000000ull);
|
||||||
|
head.i16(static_cast<std::int16_t>(tm.tm_gmtoff / 3600));
|
||||||
|
head.u16(0); // time quality
|
||||||
|
head.text("Local PC Reference Time", 32);
|
||||||
|
if (!mdf_block_size(head, hd_end, "header", error)) return false;
|
||||||
|
|
||||||
|
// The comment, in the one text block the file carries: plain words rather than the XML a
|
||||||
|
// writer that also *edits* files keeps its structured properties in.
|
||||||
|
head.raw("TX", 2);
|
||||||
|
head.u16(static_cast<std::uint16_t>(comment.size() + kMdfTxOverhead));
|
||||||
|
head.raw(comment.data(), comment.size());
|
||||||
|
head.zeros(1);
|
||||||
|
|
||||||
|
Bytes tail;
|
||||||
|
// Data group: one channel group, whose channels are chained, and the address of the samples.
|
||||||
|
tail.raw("DG", 2);
|
||||||
|
tail.u16(static_cast<std::uint16_t>(kMdfDataGroup));
|
||||||
|
tail.u32(0); // next data group
|
||||||
|
tail.u32(static_cast<std::uint32_t>(cg_addr));
|
||||||
|
tail.u32(0); // trigger block
|
||||||
|
tail.u32(static_cast<std::uint32_t>(data_addr));
|
||||||
|
tail.u16(1); // channel groups
|
||||||
|
tail.u16(0); // record IDs: the group is sorted
|
||||||
|
tail.zeros(4); // reserved
|
||||||
|
if (!mdf_block_size(tail, kMdfDataGroup, "data group", error)) return false;
|
||||||
|
|
||||||
|
for (std::size_t i = 0; i < channels; ++i) {
|
||||||
|
const Channel& c = chans[i];
|
||||||
|
// Conversion block: the unit lives here and nowhere else in MDF 3, which is where ScryLab
|
||||||
|
// reads a signal's unit from. NONE because the samples are already the physical values --
|
||||||
|
// the table was collected as `f32`, so a conversion would only be a second copy of it.
|
||||||
|
Bytes cc;
|
||||||
|
cc.raw("CC", 2);
|
||||||
|
cc.u16(static_cast<std::uint16_t>(kMdfConversion));
|
||||||
|
cc.u16(1); // range_flag: the range below is measured, so it is valid
|
||||||
|
cc.f64(c.lo);
|
||||||
|
cc.f64(c.hi);
|
||||||
|
cc.text(c.unit, 20);
|
||||||
|
cc.u16(0xFFFFu);
|
||||||
|
cc.u16(0); // referenced parameters
|
||||||
|
if (!mdf_block_size(cc, kMdfConversion, "conversion", error)) return false;
|
||||||
|
|
||||||
|
// Channel block: the name ScryLab turns into the signal name, where this channel's samples
|
||||||
|
// sit in a record, and the link to the next channel.
|
||||||
|
Bytes cn;
|
||||||
|
cn.raw("CN", 2);
|
||||||
|
cn.u16(static_cast<std::uint16_t>(kMdfChannel));
|
||||||
|
cn.u32(i + 1 < channels ? static_cast<std::uint32_t>(cc_addr(i + 1) + kMdfConversion) : 0);
|
||||||
|
cn.u32(static_cast<std::uint32_t>(cc_addr(i)));
|
||||||
|
cn.u32(0); // source: the stream does not say what produced a sample
|
||||||
|
cn.u32(0); // dependency block
|
||||||
|
cn.u32(0); // comment
|
||||||
|
cn.u16(c.type);
|
||||||
|
cn.text(c.name, 32);
|
||||||
|
cn.zeros(128); // description
|
||||||
|
cn.u16(c.bit_offset);
|
||||||
|
cn.u16(c.bit_count);
|
||||||
|
cn.u16(c.data_type);
|
||||||
|
cn.u16(1); // range_flag
|
||||||
|
cn.f64(c.lo);
|
||||||
|
cn.f64(c.hi);
|
||||||
|
cn.f64(0.0); // sampling rate: only a virtual time channel carries one
|
||||||
|
cn.u32(0); // long name
|
||||||
|
cn.u32(0); // display name
|
||||||
|
cn.u16(0); // additional byte offset
|
||||||
|
if (!mdf_block_size(cn, kMdfChannel, "channel", error)) return false;
|
||||||
|
|
||||||
|
tail.v.insert(tail.v.end(), cc.v.begin(), cc.v.end());
|
||||||
|
tail.v.insert(tail.v.end(), cn.v.begin(), cn.v.end());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Channel group, last: it names the first channel, the record size and the cycle count from
|
||||||
|
// which a reader derives the size of the sample block at `data_addr`.
|
||||||
|
tail.raw("CG", 2);
|
||||||
|
tail.u16(static_cast<std::uint16_t>(kMdfChannelGroup));
|
||||||
|
tail.u32(0); // next channel group
|
||||||
|
tail.u32(static_cast<std::uint32_t>(ch_addr + kMdfConversion));
|
||||||
|
tail.u32(0); // comment
|
||||||
|
tail.u16(1); // record ID
|
||||||
|
tail.u16(static_cast<std::uint16_t>(channels));
|
||||||
|
tail.u16(static_cast<std::uint16_t>(record));
|
||||||
|
tail.u32(static_cast<std::uint32_t>(rows));
|
||||||
|
tail.zeros(4); // sample reduction address
|
||||||
|
if (!mdf_block_size(tail, kMdfDataGroup + channels * (kMdfConversion + kMdfChannel) +
|
||||||
|
kMdfChannelGroup,
|
||||||
|
"data group and channels", error)) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
File out(path);
|
||||||
|
if (!opened(out, path, error)) return false;
|
||||||
|
const auto put = [&](const Bytes& bytes) {
|
||||||
|
return bytes.v.empty() || std::fwrite(bytes.v.data(), 1, bytes.v.size(), out.f) == bytes.v.size();
|
||||||
|
};
|
||||||
|
if (!put(head)) {
|
||||||
|
error = "short write to '" + path + "': " + std::strerror(errno);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
// One record at a time, in the channel order the blocks above declare, so a full ring is not
|
||||||
|
// held twice -- the same shape the `.npy` writer streams its records in.
|
||||||
|
std::vector<unsigned char> rec(record);
|
||||||
|
for (std::size_t i = 0; i < rows; ++i) {
|
||||||
|
std::size_t at = 0;
|
||||||
|
std::memcpy(rec.data() + at, &table.abs_seconds[i], sizeof(f64));
|
||||||
|
at += sizeof(f64);
|
||||||
|
std::memcpy(rec.data() + at, &table.time[i], sizeof(f64));
|
||||||
|
at += sizeof(f64);
|
||||||
|
for (const ExportTable::Column& col : table.columns) {
|
||||||
|
const f32 v = i < col.values.size() ? col.values[i] : 0.0f;
|
||||||
|
std::memcpy(rec.data() + at, &v, sizeof(f32));
|
||||||
|
at += sizeof(f32);
|
||||||
|
}
|
||||||
|
if (std::fwrite(rec.data(), 1, rec.size(), out.f) != rec.size()) {
|
||||||
|
error = "short write to '" + path + "': " + std::strerror(errno);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!put(tail)) {
|
||||||
|
error = "short write to '" + path + "': " + std::strerror(errno);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
if (std::fflush(out.f) != 0) {
|
||||||
|
error = "cannot flush '" + path + "': " + std::strerror(errno);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
bool write_export(const std::string& path, const ExportTable& table, std::string& error) {
|
||||||
|
const auto ends_with = [&path](const char* ext) {
|
||||||
|
const std::size_t n = std::strlen(ext);
|
||||||
|
return path.size() >= n && path.compare(path.size() - n, n, ext) == 0;
|
||||||
|
};
|
||||||
|
if (ends_with(".csv")) return write_csv(path, table, error);
|
||||||
|
if (ends_with(".mf3") || ends_with(".mdf")) return write_mdf(path, table, error);
|
||||||
|
return write_npy(path, table, error);
|
||||||
|
}
|
||||||
|
|
||||||
} // namespace udpscope
|
} // namespace udpscope
|
||||||
@@ -21,6 +21,11 @@ namespace udpscope {
|
|||||||
struct ExportTable {
|
struct ExportTable {
|
||||||
struct Column {
|
struct Column {
|
||||||
std::string name; ///< the legend name, sanitised for a column label
|
std::string name; ///< the legend name, sanitised for a column label
|
||||||
|
/// The signal's unit exactly as the CONFIG carried it, the empty string when it
|
||||||
|
/// carried none. The `.npy`/`.csv` writers do not spell it out (their readers get the
|
||||||
|
/// names and the numbers); MDF does, because the unit *is* part of what a channel is
|
||||||
|
/// there, and it is the only thing ScryLab reads a signal's unit from.
|
||||||
|
std::string unit;
|
||||||
std::vector<f32> values; ///< `rows()` long, the ring's own `f32`
|
std::vector<f32> values; ///< `rows()` long, the ring's own `f32`
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -71,4 +76,27 @@ bool write_npy(const std::string& path, const ExportTable& table, std::string& e
|
|||||||
/// enough digits to round-trip each column's own type and no more.
|
/// enough digits to round-trip each column's own type and no more.
|
||||||
bool write_csv(const std::string& path, const ExportTable& table, std::string& error);
|
bool write_csv(const std::string& path, const ExportTable& table, std::string& error);
|
||||||
|
|
||||||
|
/// Writes the same table as **MDF 3.30** (`.mf3`/`.mdf`), byte for byte by hand -- there is no
|
||||||
|
/// writer library here and MF4 (HDF5) is the one format this app will not link.
|
||||||
|
///
|
||||||
|
/// One channel group, all channels record-interleaved, master first, which is the shape the
|
||||||
|
/// [ScryLab data sources](https://docs.scrylab.de/docs/data-sources/) contract asks for: *every
|
||||||
|
/// data channel becomes a signal* (the CNBLOCK's `short_name`), *the master channel of each
|
||||||
|
/// group is the x axis* (our master is `abs_seconds`, named `time`), and *the unit is taken from
|
||||||
|
/// the channel metadata* (MDF 3 keeps it in the channel's conversion block, the CCBLOCK). A
|
||||||
|
/// second occurrence of a name is dropped by that reader, so the labels are de-duplicated and cut
|
||||||
|
/// to the 31 bytes a CNBLOCK's name field holds.
|
||||||
|
///
|
||||||
|
/// The master is the **absolute** clock and the axis the plot showed is an ordinary channel,
|
||||||
|
/// `time_relative`: the relative column is `0` at a different sample in every mode, so a master
|
||||||
|
/// off it would put two exports from different modes on two timelines, while `abs_seconds` is the
|
||||||
|
/// number the stream itself stamped the samples with. Both carry the unit `s`.
|
||||||
|
bool write_mdf(const std::string& path, const ExportTable& table, std::string& error);
|
||||||
|
|
||||||
|
/// Writes `table` with the writer **`path`'s extension** names: `.csv` -> `write_csv`,
|
||||||
|
/// `.mf3`/`.mdf` -> `write_mdf`, anything else -> `write_npy` (the default, as the save dialog's
|
||||||
|
/// default filter is). The export dialog and the self-check's export probe both come through
|
||||||
|
/// here, so the two cannot offer different formats.
|
||||||
|
bool write_export(const std::string& path, const ExportTable& table, std::string& error);
|
||||||
|
|
||||||
} // namespace udpscope
|
} // namespace udpscope
|
||||||
@@ -64,7 +64,7 @@ panel is working correctly, not failing to connect.
|
|||||||
7. <b>Measure it</b> -- the measurements pane computes a statistic of the visible window, and
|
7. <b>Measure it</b> -- the measurements pane computes a statistic of the visible window, and
|
||||||
<b>show on plot</b> marks it as a ruler or a pair of markers.
|
<b>show on plot</b> marks it as a ruler or a pair of markers.
|
||||||
8. <b>Save the data</b> with Export visible range, which writes the plotted signals over the
|
8. <b>Save the data</b> with Export visible range, which writes the plotted signals over the
|
||||||
window at the stored rate to <tt>.npy</tt> or <tt>.csv</tt>.
|
window at the stored rate to <tt>.npy</tt>, <tt>.csv</tt> or <tt>.mf3</tt> (MDF 3.30).
|
||||||
|
|
||||||
<b>Worth knowing</b>
|
<b>Worth knowing</b>
|
||||||
|
|
||||||
|
|||||||
+14
-3
@@ -1,7 +1,8 @@
|
|||||||
<b>Saving and export</b>
|
<b>Saving and export</b>
|
||||||
|
|
||||||
Export writes the <i>visible range</i> of every plotted trace to a file: NumPy <tt>.npy</tt> or
|
Export writes the <i>visible range</i> of every plotted trace to a file: NumPy <tt>.npy</tt>,
|
||||||
<tt>.csv</tt>, chosen by the extension you type.
|
<tt>.csv</tt>, or <b>MDF 3.30</b> (<tt>.mf3</tt> or <tt>.mdf</tt>), chosen by the extension you
|
||||||
|
type.
|
||||||
|
|
||||||
<b>How to reach it</b>
|
<b>How to reach it</b>
|
||||||
|
|
||||||
@@ -24,10 +25,20 @@ the format follows the path's extension.
|
|||||||
evenly spaced; duplicate or backwards stamps would mean the file held two clocks.
|
evenly spaced; duplicate or backwards stamps would mean the file held two clocks.
|
||||||
• An export taken <i>mid-capture</i> writes the partly filled window that is on screen, which is
|
• An export taken <i>mid-capture</i> writes the partly filled window that is on screen, which is
|
||||||
shorter than the event span. Export outside a capture if you want the whole event.
|
shorter than the event span. Export outside a capture if you want the whole event.
|
||||||
|
• <b>MDF</b> exists for handing a window to another tool, and its shape follows the contract that
|
||||||
|
reader uses: one channel group, every channel record-interleaved, the master channel as the x
|
||||||
|
axis and the unit in the channel metadata. So there the <b>master is <tt>abs_seconds</tt></b>,
|
||||||
|
named <tt>time</tt> — the clock that is the same in every display mode — and the display axis
|
||||||
|
goes out beside it as an ordinary channel, <tt>time_relative</tt>. Both are seconds; each signal
|
||||||
|
carries the source's own unit. That is also what <b>ScryLab</b> reads: channel name → signal
|
||||||
|
name, master → x axis, unit → the signal's unit.
|
||||||
|
|
||||||
<b>Example</b>
|
<b>Example</b>
|
||||||
|
|
||||||
A 2 s window at 10 kS/s over 3 traces writes 20001 rows and 5 columns
|
A 2 s window at 10 kS/s over 3 traces writes 20001 rows and 5 columns
|
||||||
(<tt>time</tt>, <tt>abs_seconds</tt>, three signals). The <tt>.npy</tt> is
|
(<tt>time</tt>, <tt>abs_seconds</tt>, three signals). The <tt>.npy</tt> is
|
||||||
<tt>64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)</tt> bytes. Read it back with
|
<tt>64·⌈(10+len(header)+1)/64⌉ + rows·(16 + 4·cols)</tt> bytes. Read it back with
|
||||||
<tt>pd.DataFrame(np.load(path))</tt>.
|
<tt>pd.DataFrame(np.load(path))</tt>. The <tt>.mf3</tt> is bigger — a block per channel plus a
|
||||||
|
comment — <tt>272 + comment + rows·record + 28 + channels·274 + 30</tt> bytes, with
|
||||||
|
<tt>record = 16 + 4·cols</tt> and <tt>channels = cols + 2</tt> for the master and the relative
|
||||||
|
axis. Read that one back with <tt>asammdf.MDF(path)</tt> or in ScryLab.
|
||||||
+11
-3
@@ -2371,10 +2371,10 @@ void on_export_response(GObject* source, GAsyncResult* result, gpointer user) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The extension picks the writer, so the filter the user chose is honoured by way of
|
// The extension picks the writer, so the filter the user chose is honoured by way of
|
||||||
// the name it put in the entry rather than by a second piece of state.
|
// the name it put in the entry rather than by a second piece of state. `.npy` is what an
|
||||||
const bool csv = path.size() >= 4 && path.compare(path.size() - 4, 4, ".csv") == 0;
|
// unknown extension gets, which is the dialog's default filter too.
|
||||||
std::string error;
|
std::string error;
|
||||||
const bool ok = csv ? write_csv(path, ctx->table, error) : write_npy(path, ctx->table, error);
|
const bool ok = write_export(path, ctx->table, error);
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
set_notice(*ctx->app, "export failed: " + error);
|
set_notice(*ctx->app, "export failed: " + error);
|
||||||
return;
|
return;
|
||||||
@@ -2411,10 +2411,18 @@ void on_export_clicked(GtkButton*, gpointer user) {
|
|||||||
gtk_file_filter_set_name(csv, "CSV (*.csv)");
|
gtk_file_filter_set_name(csv, "CSV (*.csv)");
|
||||||
gtk_file_filter_add_pattern(csv, "*.csv");
|
gtk_file_filter_add_pattern(csv, "*.csv");
|
||||||
g_list_store_append(filters, csv);
|
g_list_store_append(filters, csv);
|
||||||
|
// MDF 3.30, which is what ScryLab (and every MDF reader) opens: offered by both of the
|
||||||
|
// extensions the format is stored under.
|
||||||
|
GtkFileFilter* mdf = gtk_file_filter_new();
|
||||||
|
gtk_file_filter_set_name(mdf, "MDF 3.30 (*.mf3, *.mdf)");
|
||||||
|
gtk_file_filter_add_pattern(mdf, "*.mf3");
|
||||||
|
gtk_file_filter_add_pattern(mdf, "*.mdf");
|
||||||
|
g_list_store_append(filters, mdf);
|
||||||
gtk_file_dialog_set_filters(dialog, G_LIST_MODEL(filters));
|
gtk_file_dialog_set_filters(dialog, G_LIST_MODEL(filters));
|
||||||
gtk_file_dialog_set_default_filter(dialog, npy);
|
gtk_file_dialog_set_default_filter(dialog, npy);
|
||||||
g_object_unref(npy);
|
g_object_unref(npy);
|
||||||
g_object_unref(csv);
|
g_object_unref(csv);
|
||||||
|
g_object_unref(mdf);
|
||||||
g_object_unref(filters);
|
g_object_unref(filters);
|
||||||
|
|
||||||
gtk_file_dialog_save(dialog, GTK_WINDOW(app->window), nullptr, on_export_response,
|
gtk_file_dialog_save(dialog, GTK_WINDOW(app->window), nullptr, on_export_response,
|
||||||
|
|||||||
+14
-3
@@ -2345,11 +2345,10 @@ gboolean export_probe(gpointer user) {
|
|||||||
std::fprintf(stderr, "[export-probe] column '%s' n=%zu\n", col.name.c_str(), col.values.size());
|
std::fprintf(stderr, "[export-probe] column '%s' n=%zu\n", col.name.c_str(), col.values.size());
|
||||||
}
|
}
|
||||||
|
|
||||||
for (const char* ext : {".npy", ".csv"}) {
|
for (const char* ext : {".npy", ".csv", ".mf3"}) {
|
||||||
const std::string path = prefix + ext;
|
const std::string path = prefix + ext;
|
||||||
std::string error;
|
std::string error;
|
||||||
const bool ok = std::strcmp(ext, ".csv") == 0 ? write_csv(path, table, error)
|
const bool ok = write_export(path, table, error);
|
||||||
: write_npy(path, table, error);
|
|
||||||
long bytes = -1;
|
long bytes = -1;
|
||||||
if (ok) {
|
if (ok) {
|
||||||
if (std::FILE* f = std::fopen(path.c_str(), "rb"); f != nullptr) {
|
if (std::FILE* f = std::fopen(path.c_str(), "rb"); f != nullptr) {
|
||||||
@@ -2361,6 +2360,18 @@ gboolean export_probe(gpointer user) {
|
|||||||
std::fprintf(stderr, "[export-probe] %s %s bytes=%ld%s%s\n", path.c_str(),
|
std::fprintf(stderr, "[export-probe] %s %s bytes=%ld%s%s\n", path.c_str(),
|
||||||
ok ? "written" : "FAILED", bytes, ok ? "" : " error=", ok ? "" : error.c_str());
|
ok ? "written" : "FAILED", bytes, ok ? "" : " error=", ok ? "" : error.c_str());
|
||||||
}
|
}
|
||||||
|
// What the `.mf3` was handed, in the shape the file stores it: the master and the relative
|
||||||
|
// axis are channels too, and `record` is the bytes of one interleaved record. So the size the
|
||||||
|
// line above printed is `272 + comment + rows*record + 28 + channels*274 + 30`, and the units
|
||||||
|
// are the ones ScryLab reads each signal's unit from -- the CONFIG's, per column.
|
||||||
|
std::string units;
|
||||||
|
for (const char* u : {"s", "s"}) units += std::string(units.empty() ? "" : " ") + "'" + u + "'";
|
||||||
|
for (const ExportTable::Column& col : table.columns) {
|
||||||
|
units += std::string(units.empty() ? "" : " ") + "'" + col.unit + "'";
|
||||||
|
}
|
||||||
|
std::fprintf(stderr,
|
||||||
|
"[export-probe] mdf master=time master_unit=s channels=%zu record=%zu units=[%s]\n",
|
||||||
|
table.columns.size() + 2, 16 + 4 * table.columns.size(), units.c_str());
|
||||||
std::fflush(stderr);
|
std::fflush(stderr);
|
||||||
return G_SOURCE_REMOVE;
|
return G_SOURCE_REMOVE;
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user