Record, mark and plot

Use record when you are about to change something on the router and want to see what the change does. It pulls the agent’s samples to your machine at full rate; mark adds notes and the router’s own log; plot draws the result.

Commands and flags

mikroscope record --for 5m --out cap                  # cap.jsonl, cap.csv, cap.markers.csv, cap.meta.json; type lines to mark
mikroscope mark --out cap "queue tree applied"        # a note stamped with the current time (see below)
mikroscope mark --out cap --log-markers               # the router's own log lines, over the API
mikroscope plot --in cap                              # cap.svg, deterministic

The three commands share one flag set: each accepts every flag below, and the table says which command acts on it.

Flag	Default	What it does
--out	capture-<UTC time>	Output prefix: <out>.jsonl, .csv, .markers.csv, .meta.json. mark needs the prefix of an existing recording.
--for	0	record: stop after this long. 0 records until Ctrl-C.
--from-start	off	record: backfill everything the agent’s ring holds before going live.
--poll	500ms	How often to pull the agent’s ring.
--batch	0	Samples per pull. 0 is twice what one --poll interval produces at the agent’s rate, at least 20; the relay caps a pull at 13.
--transport	auto	auto, direct (HTTP to the veth) or relay (/tool fetch over the RouterOS API).
--log-markers	off	record: adds the router log lines of the window once the recording stops. mark: adds the log lines of the recording’s window.
--topics	system,interface,container	Log topics kept as markers; add firewall or script when their lines are the story.
--router-tz	Local	IANA zone the router’s clock shows. RouterOS log times carry no zone.
--api	MIKROSCOPE_API_ADDR	RouterOS API host:port, for the relay and for --log-markers.
--api-user	MIKROSCOPE_API_USER	The API user. The password is read only from MIKROSCOPE_API_PASSWORD; there is no flag for it.
--token	MIKROSCOPE_TOKEN	The agent’s bearer token.
--port	9123	The agent’s HTTP port.
--subnet	MIKROSCOPE_SUBNET, else 172.30.10.0/30	The agent’s /30; its address is .2.
--in	none	plot: a recording prefix, or the path of a .jsonl file.
--svg	<in>.svg	plot: the output file.
--title	the prefix	plot: the chart’s title.

First recording runs the three commands end to end. A measured recording, with its sample count, gaps and clock skew, is on the evidence page. Recordings above 10 Hz and through the relay are not measured.

Transports

--transport	Path	On failure
auto (default)	direct if GET /healthz on http://<.2 of --subnet>:<--port> answers, else relay	an error, below
direct	plain HTTP from your machine to the veth, with --token as a bearer token	fails, no fallback
relay	/tool fetch output=user on the router, over the binary API; the API user needs the read,api,test policies	fails, no fallback

• When auto has to fall back and --api or --api-user is missing, record stops with an error that names --api, --api-user and MIKROSCOPE_API_PASSWORD and suggests install --expose. A missing password shows as a login error instead, api <host:port>: ….
• The --expose hint is for other HTTP clients. record always dials the .2 of --subnet and has no flag for the router’s LAN address.
• The router-side fetch sends no header, so an agent installed with a token can be recorded only over direct.
• Each relayed call takes either about 3 ms or about 1 s. A reply is capped at 64 512 B, so a relayed pull asks for at most 13 samples, and a reply that reaches the cap is refused rather than parsed truncated.

The relay tops out at about 26 samples a second

By arithmetic, from the code. The cap of 13 samples comes from the 64 512 B fetch limit, a line charged at the allocator size class of 3 456 B (the measured line is 3 230 B) and 134 % headroom, so a full pull of charged-size lines is about 45 kB and still fits with lines up to 134 % of that size. A full pull whose lines average more than the fetch limit divided by the cap reaches the limit and is refused; line sizes at the default per-source floors are not measured. A full pull is asked again at once, so the round trip of /tool fetch, not --poll, sets the throughput: with about 1 s for half the calls and about 3 ms for the rest, that is on the order of 26 samples a second, less when the slow calls cluster. Above that, use direct.

Network access covers which path a network allows, and API user the RouterOS user the relay needs.

Pull loop

1. record reads /healthz. The agent’s wall clock minus your machine’s is the clock skew. It prints the skew on stderr with the agent’s version, rate, newest sequence number, the oldest one the ring still holds, and the transport.
2. The recording starts live from the newest sample, or from the oldest the ring holds with --from-start.
3. Every --poll, it pulls /snapshot?since=<last seq>&max=<batch>. While a pull comes back full it asks again, up to 100 times per poll, so a ring that got ahead is drained rather than followed at a fixed pace.
4. A pull that fails is logged as pull: … and the recording carries on; the next pull asks from the same sequence number.
5. When --for elapses or you press Ctrl-C, one last pull drains what arrived meanwhile. record prints how many samples it kept, their sequence range, the gaps, the markers, the transport and the files.

Output files

Every file is created with mode 0600. An existing file with the same prefix is overwritten.

File	Contents
<out>.jsonl	Each sample line exactly as the agent sent it: raw tick and counter deltas, with every source the agent has. This is the recording; the other files are views of it.
<out>.csv	One wide row per sample, for a spreadsheet. The column set is sized from the first sample: one group per core and one per softnet queue.
<out>.markers.csv	wall_ns,wall_utc,seq,kind,label, one row per marker.
<out>.meta.json	Written at the start, so mark and plot can run later: started_utc, skew_ns (agent wall minus host wall), agent (its version), rate_hz, transport, and capabilities when the transport could fetch them.

The CSV holds a fixed subset of each sample. Everything else a sample carries — kernel-log events, PMU counters, temperatures, interrupts per line and the rest — is in the .jsonl only.

Columns	Content
seq, wall_ns, wall_utc, dt_ns	The sample’s sequence number, agent wall clock, and real interval.
busy_total	The mean of the per-core busy ratios, three decimals, computed by the CLI from the ticks.
c<N>_busy, c<N>_user, c<N>_nice, c<N>_system, c<N>_idle, c<N>_iowait, c<N>_irq, c<N>_softirq	Per core: the busy ratio, then the raw tick deltas.
ctxt, intr, irq_total	Context switches and interrupts from /proc/stat, and the delta summed over every row of /proc/interrupts.
softnet<N>_processed, softnet<N>_dropped, softnet<N>_time_squeeze	Per softnet queue, the receive path’s deltas.
mem_free_kb, mem_available_kb, mem_cached_kb, mem_slab_kb	Memory levels, in kB.
load1, threads_running, threads_total	Load average and thread counts.
pgfault, pgmajfault	Page-fault deltas.
self_cpu_us, self_rss_bytes	The agent’s own CPU time during the sample, in µs, and its resident memory at that moment, in bytes.

The busy ratio is the one number the CLI computes, and it inherits the kernel’s floor: a tick is 10 ms, so one core in a 100 ms sample resolves to 10 % steps. Resolution limits explains why, and why the ratio is capped at 1.

Markers

A marker is a row of <out>.markers.csv, of one of three kinds:

kind	Written when
note	You type a non-empty line while record runs in a terminal (it prints type a line and press Enter to add a marker; Ctrl-C stops), or run mark --out <prefix> "text". Nothing is read when standard input is not a terminal.
gap	The agent reports that samples were no longer in its ring. The label is samples <from>..<to> lost, and the summary record prints lists the same gaps.
log	--log-markers adds a line of the router’s own log (below).

Notes and gaps are stamped with your machine’s clock plus the skew measured at the start, so they sit on the agent’s time axis. mark --out cap "text" does the same with the skew stored in cap.meta.json: it stamps the current time in the agent’s clock, and no flag sets another time. mark needs that file and an existing cap.markers.csv, and its row carries seq 0.

Mark while the recording runs

record does not hold <out>.markers.csv open: it opens the file, appends and closes it for each marker, as mark --out <prefix> "text" does from a second shell. Markers typed in one terminal and added from another land in the file in the order they were made, on Linux, macOS and Windows alike. A text mark on a recording that has already stopped stamps the current time, which falls after the last sample, so plot draws nothing there.

Router log markers

--log-markers turns the router’s log lines of the recording’s window into log markers, labelled <topics>: <message>. Use it when a transient you did not cause needs explaining: scheduler runs, errors, interface events.

export MIKROSCOPE_API_ADDR=192.168.88.1:8728 MIKROSCOPE_API_USER=mikroscope MIKROSCOPE_API_PASSWORD=…
mikroscope record --for 60s --out burst
mikroscope mark --out burst --log-markers --router-tz Europe/Madrid

• mark --log-markers uses the window from started_utc in the meta file, shifted by its skew_ns into the agent’s clock, to the last sample in the .jsonl, and prints how many lines it added, the window and the topics. started_utc is stored to the second. It appends without checking what is already there: running it twice adds the same lines again.
• record --log-markers asks for the log once the recording has stopped, for the window from its start to its end in the agent’s clock, and appends the lines through the same path mark uses, so the summary’s marker total counts what the file holds. If the log request fails, record prints log markers: … and keeps the recording.

The CLI asks the router only for the window, with a ?>time= query starting one second before it, and only for the time, topics and message fields, because a router that logs a busy topic to disk can hold tens of thousands of rows. It then keeps the lines inside the window whose topics include one of --topics. An event logged by several logging actions on one topic arrives once per action, prefixed with the action’s name ([INFO]: …, [SYSTEM]: …); the prefix is stripped and the duplicates are kept once.

Log times carry no zone:

• Over the API, RouterOS prints a full date and time (verified). The parser also accepts the shorter forms without a year or without a date, and fills them in from the end of the window.
• Times are read in --router-tz and stamped as the agent’s wall clock, which is the router’s own clock, so no skew is applied. The default Local is your machine’s zone: if the router’s differs, pass it, or every log marker is off by the difference.
• Log times have one-second resolution, so a log marker places its event to the second, not to the sample.

Chart

plot --in cap reads cap.jsonl, and cap.markers.csv when it exists, and writes cap.svg (or --svg). It prints the file name with its sample and marker counts. The same input always produces the same bytes.

The chart is 1 200 units wide, with a title (--title, or the prefix) and a line giving the sample count, the duration and the core count. Three panels share one time axis, in seconds since the recording started:

1. CPU busy per core, % — 0 to 100, one line per core, each labelled at its end. The palette has eight colours, so the first eight cores are drawn.
2. softnet per second, all CPUs — dropped and time_squeeze, summed over every queue and bucketed into whole seconds. Drops are drawn in red.
3. memory available, MiB — MemAvailable per sample, with the y-axis bounded to the data.

Markers on the chart:

• Every marker is a dashed vertical line across all three panels, grey for notes and log lines and red for gaps, with a label chip above the first panel.
• Chips are laid out in up to six rows. A label longer than 40 characters is shortened; when no row has room it is shortened further until one does, and a chip that would fall under 12 characters is dropped. Its dashed line still marks the instant.
• Consecutive log markers, in time order, that fall in the same second are folded into one chip, <count>× <topics>: <first message>. A note or gap between them breaks the fold, and notes and gaps are never folded.
• Markers outside the recording’s time span are not drawn.

With a single sample the SVG says not enough samples to draw; with none, plot stops with record: no samples and writes no SVG. The palette is the chart’s own, validated for its light background: adjacent-pair colour-vision-deficiency ΔE 9.1, normal-vision ΔE 22.9. There is no dark variant.

plot also reads a capture the agent kept on a trigger: save GET /captures/<id> to a .jsonl file and pass it to --in. The capture’s header line carries no sequence number and is skipped.

Trigger lines

When a trigger fires on the agent, a {"trigger":{…}} line rides in the same pull as the samples, just before the sample it fired on. The collector recognises it; record does not.

A trigger line in a recording

record reads a trigger line as if it were a sample with sequence number 0. The line is written verbatim to the .jsonl, where plot skips it, and as a row of zeros with seq 0 to the .csv; it is counted once in the number of samples record reports. If it is the first line of the recording, the CSV header is sized for zero cores. record does not add it as a marker. To see what fired during a window, read /captures on the agent.

Triggered capture covers what fires, and how to fetch the full-rate window the agent kept around it.

See also

• First recording: record, mark and plot, step by step.
• Triggered capture: the samples around a condition, kept at full rate by the agent without a recording running.
• Network access: which of the direct and relay paths your network allows.
• API user: the RouterOS user the relay and --log-markers need.