# mikroscope Sub-second kernel telemetry from inside a MikroTik router, with the cost of the observer measured rather than claimed. Source: https://jmrplens.github.io/mikroscope/ ## Measured, not budgeted - [**2.85 %** — of one core at 10 Hz, the install default](/mikroscope/cost/) - [**31.3 MiB** — resident memory at 10 Hz](/mikroscope/cost/) - [**17.81 %** — of one core at 100 Hz, the CLI's cap](/mikroscope/cost/rate-ceiling/) - [**0 / 0** — gaps and drops, in every sink, across five runs](/mikroscope/cost/rate-ceiling/) The first three from the agent's own cgroup and `/metrics`, the fourth from the three sinks the collector forwarded to, on an RB5009 (4 × 1.4 GHz Cortex-A72, RouterOS 7.24.2), 60 s windows at steady state, 2026-09-15. The 10 Hz figures are above the ≤ 2 % and ≤ 16 MiB budget. ## A one-second average is a report about a second The RouterOS API reports `cpu-load` once a second. A core saturated for 100 ms and idle for the other 900 moves a four-core, one-second average by 2.5 %. That is arithmetic, not a measurement, and the figure is true: it just cannot say when. The agent reads `/proc/stat`, `/proc/interrupts`, `/proc/softirqs` and `/proc/net/softnet_stat` from inside the router, at 10 Hz by default, and ships raw tick deltas with the interval each one covers. It never computes a percentage; the window is yours. The floor is the kernel's, not the tool's. `/proc/stat` counts in ticks of 10 ms, so a 100 ms sample resolves one core to 10 % steps. On the RB5009 (RouterOS 7.24.2, Linux 5.6.3) there is no PSI and no schedstat to go finer: both files are absent, checked 2026-09-11. ## What it costs, at three rates Each row is one 60 s window with the ring already full. Memory differs by row because the ring and the memory limit do. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once The measured runs: | rate | floors | CPU of one core | µs/sample | RSS | slipped ticks | gaps / drops | | --- | --- | --- | --- | --- | --- | --- | | 10 Hz (default) | default | **2.85 %** | 2 856 | 31.3 MiB | **0** | 0 / 0 | | 50 Hz | default | **10.13 %** | 2 026 | 51.9 MiB | **0** | 0 / 0 | | 100 Hz | default | **17.81 %** | 1 781 | 76.5 MiB | 5 (0.08 %) | 0 / 0 | Nothing was lost at any of these rates: every sink reported 0 gaps and 0 drops, and the delivered rate matched the configured one to three figures. At the default floors and 100 Hz, a whole tick's sources were read in under 2 ms for 97.5 % of samples, inside a 10 ms period. [All five runs, including every source on every tick →](/mikroscope/cost/rate-ceiling/) ## The router's CPU from the kernel, its interfaces from the API ### Kernel tier · the agent · 10 to 100 Hz Global inside the container, so these are the router's own: per-core CPU ticks, interrupts, softirqs, softnet drops and time squeezes, `/proc/meminfo`, `/proc/vmstat`, load and disk I/O. A privileged container adds the kernel log as timestamped events and the global slab caches. ### API tier · the collector · 1 Hz The container has its own network namespace, so `/proc/net/dev` describes the container, not the router. Interface bytes and packets come from the RouterOS API and are merged by the collector, not interpolated. `privileged=yes` does not change that (checked 2026-09-12). ## Every write listed before it is made Download the archive for your platform from the release, or build the CLI from a checkout with `make build`. The router needs RouterOS 7.24 or later — the container step writes `privileged=`, an attribute earlier 7.x releases reject — with the `container` package and `device-mode container=yes`, which MikroTik gates behind a reset-button press or a power cycle. arm64, arm and x86_64; not MIPS, not TILE. 1. `mikroscope doctor` Read-only preflight; names the fix for anything missing. 2. `mikroscope plan` Every RouterOS command, nothing written. 3. `mikroscope install` Doctor, confirmation, the writes, then a probe of the agent. The image comes from your own Go toolchain, from the published agent tar, or from the registry the router pulls it from. 4. `mikroscope status` Ownership counts and the agent's health. 5. `mikroscope uninstall` Removes and verifies. **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. ## What is not claimed > **Not measured, so not claimed** > > Any rate on a board that is not this RB5009, and any traffic load heavier than this router's ordinary evening, about 30 Mbit/s. Cost scales with core speed, source set and ring size: measure it on your own device before you budget for it. The project has run on one device, an RB5009UG+S+ on RouterOS 7.24.2, arm64; the arm and x86_64 builds are cross-built and checked in CI and have never run on hardware, and seven of the ten sinks are exercised only against fakes. ## Where to go next - [What it is](/mikroscope/start/): Two programs, one container, four limits stated first - [Five minutes with a router](/mikroscope/start/walkthrough/): Install, record while you change something, draw the chart, from a real RB5009 recording - [Installing the agent](/mikroscope/install/): Four ways to get the agent onto the router, what install writes and in which order, and how uninstall removes only what it created - [The collector](/mikroscope/sinks/): Pull, merge, derive, fan out to ten sinks, and why a slow one never stops the loop - [How to read what it shows](/mikroscope/playbooks/): A production fault the API could not see, provoked faults, and the idle shape they are read against - [What the numbers do not say](/mikroscope/cost/limits/): Every limit on the figures above --- # What it is An agent inside the router reading the shared kernel, a CLI outside it, and the four limits that cannot be engineered away. Source: https://jmrplens.github.io/mikroscope/start/ mikroscope is two programs and one container. The **agent** is a static Go binary in a scratch container on the router. A RouterOS container shares the host kernel, so `/proc` inside it is the router's own `/proc`: `/proc/stat` per core, `/proc/interrupts`, `/proc/softirqs`, `/proc/net/softnet_stat`, `/proc/meminfo`, `/proc/vmstat`, `/proc/diskstats`, `/dev/kmsg`. It samples them on a ticker at 1 to 100 Hz (10 Hz by default; 10, 50 and 100 Hz measured), keeps the last 300 seconds in a ring, and serves them. It has no outbound connection and presents no credential; the only secret it holds is the optional token it requires of whoever reads it. The **CLI** runs on your machine. It installs and removes the agent, records a window with markers, draws a deterministic SVG of it, and runs as a collector that pulls from the agent, merges a RouterOS API tier at 1 Hz, and fans out to file, Prometheus and InfluxDB 3. It is released as an archive per platform — linux, macOS, Windows and FreeBSD on amd64, arm64 and arm — and the agent beside it as one image tar per architecture and as a registry image, on Docker Hub as `jmrplens/mikroscope-agent` and on GHCR as `ghcr.io/jmrplens/mikroscope-agent`, so installing needs a Go toolchain only when you want the agent built from your own tree. ## Four limits, stated before anything else 1. **The prerequisites the tool cannot remove.** RouterOS 7.24 or later with the `container` package and `device-mode container=yes` — which MikroTik gates behind a physical reset-button press or a power cycle. 7.24 is the floor because the container step writes `privileged=`, an attribute MikroTik added in that release; `--privileged=false` changes its value, not whether it is written. arm64, arm and x86_64; not MIPS, not TILE. 2. **The resolution floor is the kernel's, not the tool's.** `/proc/stat` ticks at 100 Hz, so a 100 ms window resolves one core to 10 % steps. The agent ships raw ticks so you choose the window. Do not expect PSI: the RB5009's kernel has neither PSI nor schedstat. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-11 · `/proc/pressure` and `/proc/schedstat` absent 3. **The container sees the router's CPU and memory, but its own network.** `/proc/net/dev`, `/proc/net/snmp` and `nf_conntrack_count` are per network namespace and describe the container; the conntrack count from the slab under `privileged` is the exception, and describes the router. Interface counters come from the RouterOS API and are merged, not faked. 4. **The observer costs something, and it is written down.** 2.85 % of one core at 10 Hz on an RB5009, 17.81 % at 100 Hz. [What it costs](/mikroscope/cost/) has the full table and the conditions. Every figure on this site comes from that one device, an RB5009UG+S+ on RouterOS 7.24.2, arm64: the arm and x86_64 builds are cross-built and checked in CI and have never run on hardware. **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. ## What it does not do It does not replace the RouterOS API — it merges with it. It does not claim a number it did not measure: where a source is absent on a board, the metric is absent rather than zero. And it does not decide for you what a percentage means, because it never computes one. ## See also - [The cost of the observer](/mikroscope/cost/): the budget, the measured result and how to measure it on your own device. - [The rate ceiling](/mikroscope/cost/rate-ceiling/): the five runs behind the figures in point 4. - [What the numbers do not say](/mikroscope/cost/limits/): what one device on one day cannot tell you. --- # Five minutes with a router Get the CLI, install the agent, record a window while you change something, add the router’s own log as markers, and draw the chart — with a real recording from an RB5009. Source: https://jmrplens.github.io/mikroscope/start/walkthrough/ The question this tool exists for: _I am about to change something on the router — what does it actually do, at a resolution where I can see it?_ This page is the shortest path from a router that has never seen mikroscope to a chart that answers it: six steps with the commands as they are, and one real recording from the RB5009 read line by line. ## Before you start The router needs what the tool cannot give it: RouterOS 7.24 or later with the `container` package and `device-mode container=yes`, which MikroTik gates behind a reset-button press or a power cycle. 7.24 is the floor because the container step writes `privileged=`, an attribute MikroTik added in that release; `--privileged=false` changes its value, not whether it is written, so an older 7.x rejects the command either way. Everything on this page was measured on 7.24.2. `doctor` reports the router's version in its first line but does not gate on it. [What the router needs](/mikroscope/install/prerequisites/) has the whole list, and `mikroscope doctor` checks the rest read-only and names the fix for anything missing. The CLI does not read `.env` itself, and only some flags take their default from a `MIKROSCOPE_*` variable: the connection and naming ones (`ROUTER`, `SSH_PORT`, `SSH_KEY`, `NAME`, `VETH`, `SUBNET`, `IFACE_LIST`, `ADDR_LIST`, `DISK`, `ARCH`, `TOKEN`, `LAN_ADDRESS`), the API ones (`API_ADDR`, `API_USER`, `API_PASSWORD`), the sink URLs such as `INFLUX_URL`, `INTERFACES` and `HOST_TAG`. The rest, among them `--ephemeral`, `--rate`, `--for`, `--topics` and `--prom`, have plain defaults, whatever the CLI's usage text says. Export the variables you need, or copy `.env.example` to `.env` and source it with `set -a; . ./.env; set +a`; [environment variables](/mikroscope/reference/environment/) lists every one. ## The path 1. **Get mikroscope.** ```sh wrap tar xzf mikroscope_1.0.0_linux_x86_64.tar.gz # a .zip on Windows ./mikroscope version ``` The release carries the CLI as an archive per platform — linux, macOS, Windows and FreeBSD on amd64, arm64 and arm — with `checksums.txt`, cosign signatures and SBOMs beside it. The agent travels separately, as one image tar per architecture (`mikroscope-agent-arm64.tar`, `mikroscope-agent-arm.tar`, `mikroscope-agent-amd64.tar`) and as a registry image, published both as `jmrplens/mikroscope-agent:1.0.0` on Docker Hub and as `ghcr.io/jmrplens/mikroscope-agent:1.0.0` on GHCR; step 2 takes one of the two. From a checkout instead: ```sh wrap git clone https://github.com/jmrplens/mikroscope && cd mikroscope && make build ``` That needs Go 1.27 and leaves the CLI in `bin/mikroscope`, which is how the commands below are written. It is also the only path that installs an agent built from your own tree, because `install` cross-compiles the agent from the module root. 2. **Install, once.** ```sh wrap export MIKROSCOPE_ROUTER=admin@192.168.88.1 bin/mikroscope plan --ephemeral bin/mikroscope doctor --ephemeral && bin/mikroscope install --ephemeral ``` `plan` prints every RouterOS command and writes nothing. `install` puts the agent image on the router, lists the objects, runs the same preflight again (`--no-doctor` skips it), asks `write the objects above to the router? [y/N]` (`--yes` skips the question), writes, and then probes the agent from your host. Where the image comes from is your choice, and it is the only difference between the three installs: - **your Go toolchain**, as above: the CLI runs `go build ./cmd/mikroscope-agent` with `CGO_ENABLED=0`, `GOOS=linux` and `GOARCH` from `--arch`, which is why it has to run from the module root; - **the published tar**, `--agent-tar mikroscope-agent-arm64.tar`: no Go toolchain and no checkout. The CLI reads the tar before it uploads it — it has to be a mikroscope agent image and its architecture has to match `--arch`, or the verb stops and names the asset to download instead; - **the registry**, `--remote-image jmrplens/mikroscope-agent:1.0.0`: the router pulls the image itself, nothing is uploaded, and `uninstall` has no file to account for. RouterOS takes the registry host from the global `/container/config registry-url`, which mikroscope never writes because every container on the device shares it, and which ships as `https://registry-1.docker.io` — so the Docker Hub reference above runs on a stock router as it stands. The GHCR reference, `ghcr.io/jmrplens/mikroscope-agent:1.0.0`, names a host of its own: `doctor` checks the setting against it and prints `/container/config/set registry-url=https://ghcr.io` when it does not match, or points at `--agent-tar`. The pull needs the router to reach the registry and the free RAM for the layers. `--arch` defaults to `arm64`, the RB5009's; it is not detected from the router, but `doctor` compares it with the architecture the router reports and names the `--arch` value to re-run with if they differ. On a router you only reach through WinBox or WebFig, there is a fourth way with no CLI on your side at all: `mikroscope plan --rsc --remote-image jmrplens/mikroscope-agent:1.0.0 --out install.rsc` writes the same commands, in the same order and with the same tags, as a RouterOS script to paste into the terminal or `/import`. [Installing the agent](/mikroscope/install/) has that path and its two caveats in full. `--ephemeral` puts the image tar and the container's root on the router's tmpfs RAM disk and creates the container with `start-on-boot=no`, so the agent does not come back after a reboot. `install` deletes the tar once the container has extracted it. Nothing was written to flash: `write-sect-since-reboot` stayed at 58 279 across install, run and removal (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). Drop it for a persistent install. The tmpfs disk has to exist; the RB5009 has one, and `doctor --ephemeral` checks for it and prints the `/disk/add` that creates it if it does not. Pass `--ephemeral` to `doctor` too: without it, `doctor` checks free flash instead. `install` ends by probing the agent from your host, and prints one line: ```text wrap direct transport ok: agent 1.0.0 (9ddd760) built 2026-09-16T08:38:27Z, 10 Hz, seq 29, 0 slipped, 7ms round trip ``` The first field is the agent's build identity: the version, which a release stamps from the `VERSION` file, then the commit and the build date. An unstamped `go build` inside a checkout reports that same version with the commit and time the Go toolchain records, so the field is never `dev` and never a bare hash. An agent from the tar or the registry reports what the release build stamped into it; an agent `install` built from your tree carries the CLI's own stamp, so the two report the same string. The rest of the line is the probe itself: the sampler rate, the sequence the agent had reached, the ticks it slipped and the round trip. On the RB5009 the probe answered after about three seconds with a 7 ms round trip, and 5 ms after an `upgrade` the same day (2026-09-12). The probe waits up to 30 s. If your host cannot reach the agent's /30, it first asks the router whether the container is running and only then suggests alternatives; see [reaching the agent](/mikroscope/install/reaching-the-agent/). 3. **Record while you do the thing.** ```sh wrap bin/mikroscope record --for 70s --out burst ``` Type a line and press Enter whenever you do something worth remembering; it becomes a marker with the agent's timestamp. That works when standard input is a terminal, and the CLI says so: `type a line and press Enter to add a marker; Ctrl-C stops`. From another shell, ```sh wrap bin/mikroscope mark --out burst "queue tree applied" ``` does the same: `mark` appends to `.markers.csv` while `record` holds the file open, so a note from a second terminal and a note typed into the recorder's own end up in the same file. `--for 0`, the default, records until Ctrl-C. At the end `record` prints the sample count, the sequence range, the gaps, the markers and the transport it used, and names any stretch of samples that was no longer in the agent's ring. 4. **Add the router's own log.** This step talks to the RouterOS API, not to the agent, so it needs an account on the router to talk to. A read-only one is enough, and [the API user](/mikroscope/security/api-user/) has the two commands that create the group and the user; the CLI never needs `admin` for this. ```sh wrap export MIKROSCOPE_API_ADDR=192.168.88.1:8728 MIKROSCOPE_API_USER=mikroscope MIKROSCOPE_API_PASSWORD=… bin/mikroscope mark --out burst --log-markers --router-tz Europe/Madrid ``` (Or put those three in `.env` — `cp .env.example .env` — and source it, as above.) Every log line of the recording's window — from its start to its last sample, in the agent's clock — whose topic is `system`, `interface` or `container` becomes a marker; add `firewall` or `script` with `--topics` when their lines are the story. The router reports errors and scheduler runs there, and the log often explains a transient you did not cause. `MIKROSCOPE_API_ADDR` has the flag `--api` and `MIKROSCOPE_API_USER` the flag `--api-user`; the password has no flag. `--router-tz` is the IANA zone the router's clock shows, because RouterOS log times carry no zone; it defaults to your machine's. `record --log-markers` does the same at the end of a recording, and `mark --log-markers` does it afterwards, as here ([the router's own log as markers](/mikroscope/record/#the-routers-own-log-as-markers)). 5. **Look.** ```sh wrap bin/mikroscope plot --in burst ``` This writes `burst.svg`: three panels on one time axis — busy ratio per core, softnet drops and time squeezes per second, memory available — with every marker as a dashed vertical drawn per panel, so no panel title is struck through. Log markers that fall in the same second fold into one line whose label reads `N×` and the first message; a gap in the samples is drawn as a red line; a label longer than 40 characters is cut to 37 and an ellipsis, and a chip with no room left shortens further until it fits rather than being laid over its neighbour, keeping only its rule when nothing readable is left; a marker outside the recording's span is not drawn. The same recording always yields the same bytes. `--title` sets the heading and `--svg` another output path. 6. **Keep it running (optional).** ```sh wrap bin/mikroscope forward --prom :9124 --influx "$MIKROSCOPE_INFLUX_URL" --interfaces bridge,ether1 ``` This runs the collector: the kernel tier from the agent and, with the `MIKROSCOPE_API_*` variables from step 4 in the environment, the RouterOS API tier beside it. With the default `--api-mode full` that tier reads `/system/resource`, `/system/resource/cpu`, `/system/health`, `monitor-traffic` every second for the interfaces named in `--interfaces`, every port's cumulative counters every 10 s, and what each interface is — its comment, type, interface lists, bridge and MTU — at start and every 5 min; [the RouterOS API tier](/mikroscope/sinks/api-tier/) says which of those the agent can read itself. Without those variables the API tier is disabled with a warning and the kernel tier still runs. Both go to a Prometheus exposition on `:9124` and to InfluxDB 3. From there, [import and check](/mikroscope/dashboards/import-and-check/) sets up the two Grafana dashboards, the datasource field an InfluxDB 3 import needs and the two Prometheus scrape jobs. ## What the chart shows [![RB5009UG+S+, 70 s at 10 Hz: 700 samples over 69.9 s on four cores, with three dashed markers — “baseline, router idle” at 12 s, “dashboards check started” at 30 s and “check finished” at 50 s. Per-core busy stays low with single-sample excursions to 100 %; the softnet panel shows time squeezes and a flat zero for dropped; memory available stays between 662 and 671 MiB.](https://raw.githubusercontent.com/jmrplens/mikroscope/main/site/src/assets/walkthrough/rb5009-walkthrough.svg)](https://raw.githubusercontent.com/jmrplens/mikroscope/main/site/src/assets/walkthrough/rb5009-walkthrough.svg) [Open the chart at full size](https://raw.githubusercontent.com/jmrplens/mikroscope/main/site/src/assets/walkthrough/rb5009-walkthrough.svg) (SVG, 1200 × 754) to read its labels on a phone. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-16 · a 70 s `record` at 10 Hz, 700 samples over 69.9 s, the router otherwise at rest, three notes typed into `record`'s terminal This is a real recording of a router at rest, the shape everything else is read against. The three dashed lines are the notes typed into `record`'s terminal during it; the chips carry them in full because they fit, and a longer one is shortened rather than laid over the next. - **t = 12 s, `baseline, router idle`.** Nothing is happening, and the panels say so: every core averages under 10 % over the whole recording, and over the quiet stretch that follows this note the four together average 3.9 %. - **t = 30 to 50 s, between `dashboards check started` and `check finished`.** A browser loading the two Grafana dashboards against this router's own collector. The four cores together average 5.0 % over that stretch, and the work arrives in two short bursts right after the note: core 0 at or above 50 % for 0.6 s from 32.3 s, and again for 0.3 s at 33.2 s. - **The work is in short excursions.** 76 of the 700 samples have a core at or above 50 %, in 46 separate stretches; 38 of those are one sample long, and the longest is 1.4 s, at the start of the recording and before the first note. The kernel's scheduler puts each excursion on whichever core is free. One isolated sample — one core at 100 % for 100 ms — moves a four-core, one-second average by 2.5 %. - **softnet: `dropped` flat at zero, `time_squeeze` between 0 and about 20 per second.** Nothing was lost in 70 s. The squeezes are this device's background rather than an event; what the collector calls a microburst is a cluster of them — three flagged samples on one core inside 60 s — and never a single one. - **memory available, 662 to 671 MiB.** About 9 MiB of ordinary churn across the recording, with no step at either end of the dashboards check. RouterOS's own `cpu-load`, at 1 s, reports this minute as a flat few per cent. The recording shows what the few per cent are made of: which core took each excursion, how long it lasted, and where the notes fall against it. > **True of this device, not of yours** > > One recording, on one router, at 10 Hz, with the router otherwise at rest. The per-panel figures > above are read off this chart, not re-measured. The three seconds before the probe answered and > the 7 ms round trip are what that install on that network showed, not a figure for yours. ## What step 2 wrote, and taking it off **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. ```sh wrap bin/mikroscope uninstall --ephemeral ``` Pass the same `--ephemeral`, `--disk` and `--name` you installed with: `uninstall` and `status` rebuild the plan from their own flags, and without `--ephemeral` they look for the image tar on flash instead of on tmpfs. `uninstall` removes every step of the plan in reverse order, by exact tag, then counts what is left owned by mikroscope on the router and fails, naming the objects, if anything remains. `bin/mikroscope status --ephemeral` prints the same ownership counts at any time, plus the agent's health when it is reachable. > **ssh is not free on the router** > > Every ssh connect costs the RB5009 20–27 % CPU for its duration. The CLI > therefore batches each read into one connect and never uses ssh as a data path; the samples > travel over HTTP to the veth, or over the RouterOS API relay. ## See also - [Installing the agent](/mikroscope/install/): every install option, and where each object goes. - [Record, mark, plot](/mikroscope/record/): the recording files, the transports and the markers in full. - [The API user](/mikroscope/security/api-user/): the read-only account step 4 and the API tier need, and the two commands that create it. - [The collector](/mikroscope/sinks/): what `forward` merges and where it sends it. - [Import and check](/mikroscope/dashboards/import-and-check/): the Grafana dashboards, their datasources and the scrape jobs. - [How to read what it shows](/mikroscope/playbooks/): the signatures a real fault and a provoked one leave on these panels. --- # The cost of the observer What the agent costs on an RB5009, measured from its own cgroup, and how to measure it on a device that is not this one. Source: https://jmrplens.github.io/mikroscope/cost/ An observer that costs 20 % of the thing it observes is not measuring the router, it is measuring itself. So this number is a first-class result here, not a footnote: it is reported by the agent on every sample and exposed on `/metrics`, and the image-size budget is asserted in CI. ## The budget, and what it actually costs The budget is **≤ 2 % of one core, ≤ 16 MiB RSS, ≤ 8 MiB image**. The image is 6.1 MiB. The other two depend on the rate and on how much you ask it to read, and the honest answer is a table rather than a number. At the install default — 10 Hz, default per-source floors, a 300 s ring — the agent costs **2.85 % of one core and 31.3 MiB RSS**, from its own cgroup: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once That is above the 2 % the budget asks for, with every source read — the perf timings, buddyinfo, the MTD ECC counters, the cgroup events and the port counters among them. ### What the alternative costs The budget says what the agent is allowed to cost. The other comparison, the one a reader usually wants, is against doing it the obvious way: a busybox shell loop reading the same file set at the same rate. On the same router that costs **2.4 % of one core**, while the reads themselves are about 0.77 ms per sample. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-11 · a busybox shell loop reading the full file set at 10 Hz, one fork per iteration, in a container on the router So most of the shell loop's cost is not the reading. It pays a fork per iteration and the agent pays none: one process starts, opens its files once and keeps them open. That is the whole of the difference, and it is why the agent is a binary rather than a script. > **Where the number comes from** > > Not from `top`, and not from a snapshot. The agent counts its own CPU microseconds and RSS out of > its cgroup and publishes them as `mikroscope_self_cpu_usec_total` and `mikroscope_self_rss_bytes`. > Two reads of `/metrics` 60 s apart, at steady state with the ring already full, is the > measurement. RouterOS `/tool profile` shows the same process as `mikroscope-agent`. ## Size the memory limit to the data The single most expensive mistake available here is giving the process less memory than its ring needs. The Go garbage collector responds to a tight soft limit by running more often, and the CPU cost jumps for a reason that has nothing to do with the sampling rate. The ring holds pre-encoded lines rather than structs for the same reason. If you raise `--buffer` or the rate, raise `--mem-limit-mb` and the container's `--memory-max` with it; the numbers used for each row of the measurements are listed beside them on [the rate ceiling](/mikroscope/cost/rate-ceiling/). The parse is not where the time goes. Parsing the seven global `/proc` files plus one delta took 27 µs and 239 allocations per sample on the amd64 development host (Go 1.27.1, three runs, 26.7–27.5 µs, 2026-09-11). On the RB5009's Cortex-A72 a whole tick — timers, JSON and the garbage collector included — costs 2 856 µs at 10 Hz with the default floors, and [the five runs](/mikroscope/cost/rate-ceiling/) give it at every rate. The parse was not measured on the A72 on its own. ## Measuring it on your own device Nothing here transfers to a board that is not an RB5009 — a different core count, a different clock, a different kernel and a different flash all move it. The procedure is three commands and takes a minute: ```sh curl -s http://172.30.10.2:9123/metrics | grep -E 'self_cpu_usec_total|self_rss_bytes' sleep 60 curl -s http://172.30.10.2:9123/metrics | grep -E 'self_cpu_usec_total|self_rss_bytes' ``` The difference in `self_cpu_usec_total` divided by 60 000 000 is the share of one core. Take it at steady state, with the ring full: a freshly started agent is still filling it and will read low. ## See also - [The rate ceiling](/mikroscope/cost/rate-ceiling/): the five runs, the memory flags each one used, and what sampling faster buys. - [What the numbers do not say](/mikroscope/cost/limits/): what these figures cannot tell you about another device or another load. - [What it is](/mikroscope/start/): the agent, the CLI and the four limits stated first. --- # The rate ceiling Five measured runs at 10, 50 and 100 Hz on an RB5009 — all of them lossless — and what sampling faster actually buys. Source: https://jmrplens.github.io/mikroscope/cost/rate-ceiling/ The question this page answers is the one worth asking before trusting any of it: **how fast can it sample before it starts losing data?** On the reference device the answer is that it does not, up to the CLI's own 100 Hz cap, with every source read on every tick. That is not an extrapolation from the 10 Hz figure. It is five runs. ## The five runs Every figure is from the agent's own cgroup and `/metrics`, with the full source set. Each row is one window with the ring already full: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once The measured runs: | rate | floors | CPU of one core | µs/sample | RSS | slipped ticks | gaps / drops | | --- | --- | --- | --- | --- | --- | --- | | 10 Hz (default) | default | **2.85 %** | 2 856 | 31.3 MiB | **0** | 0 / 0 | | 50 Hz | default | **10.13 %** | 2 026 | 51.9 MiB | **0** | 0 / 0 | | 100 Hz | default | **17.81 %** | 1 781 | 76.5 MiB | 5 (0.08 %) | 0 / 0 | | 50 Hz | `FLOOR_HZ=50` | **22.47 %** | 4 494 | 60.6 MiB | 6 (0.20 %) | 0 / 0 | | 100 Hz | `FLOOR_HZ=100` | **43.95 %** | 4 395 | 79.6 MiB | 14 (0.23 %) | 0 / 0 | `FLOOR_HZ` means every source is read on every tick, with no per-source floor at all — the worst case the agent can be asked for. Memory differs by row because the ring does. The 10 Hz row is the install default (300 s ring, `--mem-limit-mb 40 --memory-max 64M`); the 50 Hz rows used `--buffer 120 --mem-limit-mb 64 --memory-max 96M` and the 100 Hz rows `--buffer 120 --mem-limit-mb 80 --memory-max 128M`. Give the agent's garbage collector room or the cost jumps for reasons that have nothing to do with the rate. > **Nothing was lost at any rate, in any configuration** > > Sampled seconds covered wall clock to 1.0000 in all five runs, the delivered rate was the > configured rate to three figures, and every sink — file, Prometheus, InfluxDB — reported 0 gaps > and 0 drops. ## Two things in that table worth reading twice ### Per-sample cost falls as the rate rises A sample costs 2 856 µs at 10 Hz against 1 781 µs at 100 Hz. That is not a paradox, it is the floors working: the expensive sources are amortised over more samples. `/proc/slabinfo`, one of the expensive sources (13.8 kB), is read every 2nd tick at 10 Hz and every 17th at 100 Hz, so an average sample costs less while the _rate_ of slabinfo reads stays near its 6 Hz floor either way (5 Hz at 10 Hz, about 5.9 Hz at 100 Hz). With `FLOOR_HZ` there is no amortisation to be had and the per-sample cost is flat — 4 494 µs at 50 Hz and 4 395 µs at 100 Hz — so CPU scales linearly with the rate: 22.47 %, then 43.95 %. ### A slipped tick is not a lost sample The sample is still produced and still delivered, carrying its real `dt_ns`, so any rate computed from it stays correct. It is a smear, not a hole, and `mikroscope_tick_interval_seconds` is where you see it. At 100 Hz with everything on every tick, 99.5 % of ticks still landed within 11 ms of a 10 ms period and the worst was 15 ms. ## What binds is the read, not the CPU At the default floors a whole tick's sources are read in under 2 ms for 97.5 % of samples at 100 Hz, comfortably inside a 10 ms period. `FLOOR_HZ` pushes 1.4 % of reads past 5 ms, and those are the ticks that slip. The CPU headroom is larger than the timing headroom, which is why the ceiling is a statement about I/O rather than about the A72. ## What sampling faster actually buys Not CPU-percent resolution. The jiffie is 10 ms, so at 100 Hz a sample holds either 0 or 1 busy tick and the per-sample busy ratio has two possible values. Above roughly 20 Hz the tick counters stop being a percentage and become an occupancy indicator; the PMU is the resolution from there. What a higher rate does buy is everything that is not jiffie-quantised — softnet packet counts, interrupt deltas, PMU counters, the kernel log's own timestamps — and a tighter bound on how long a burst can hide between two samples. > **Not measured, so not claimed** > > Any rate on a board that is not this one, and what happens under a traffic load heavier than this > router's ordinary evening — about 30 Mbit/s. Before quoting a number for your > device, re-measure it there: two `/metrics` reads 60 s apart at steady state. ## See also - [The cost of the observer](/mikroscope/cost/): the budget these runs are measured against, and the three commands to measure it yourself. - [What the numbers do not say](/mikroscope/cost/limits/): what `/metrics` can and cannot recover at any of these rates. --- # What the numbers do not say The questions the measurements on this site cannot answer, and which metric to reach for instead of guessing. Source: https://jmrplens.github.io/mikroscope/cost/limits/ Every measurement on this site was taken on one device, on one day, under one load. This page is the list of things it therefore does not tell you — kept here, in the documentation, rather than left for a reader to discover by being wrong. ## What `/metrics` can recover, and what it cannot The histogram `mikroscope_cpu_busy_ticks` (integer busy ticks per sample and core, which keeps timing jitter out of the bucket choice) recovers **time above a threshold** to one sample. It cannot recover **contiguity**: a 2 s plateau at 30 % and twenty scattered 100 ms spikes at 30 % look identical in it. That is what `mikroscope_cpu_busy_run_seconds{threshold="0.5"|"0.9"}` is for — the length of each run of consecutive samples at or above the threshold, observed when the run ends, with the run still in progress in `…_run_open_seconds`. The trailing-window gauges (`window="1s"|"10s"|"60s"`, `stat="max"|"min"|"p95"`) show the peak whatever your scrape interval is, because the agent computes them over its own clock rather than over yours. To see the _shape_ of a transient rather than its envelope, `record` it, or let a trigger capture it. ## The sampler's own smear is published, not hidden How late the sampler woke after its ticker and how long the read took are `mikroscope_tick_wake_latency_seconds` and `mikroscope_tick_read_seconds`, with the interval actually achieved in `mikroscope_tick_interval_seconds`. If you suspect the sampler rather than the router, those three are where to look first. ## Gaps are reported, never papered over The agent keeps 300 s by default. A collector outage shorter than that is backfilled on reconnect through `since=`; a longer one is reported as a gap of known length — a marker in a recording, a counter on the collector. A chart with a hole in it is a chart telling the truth. > **Not measured, so not claimed** > > Any board that is not the RB5009 described on [the rate ceiling](/mikroscope/cost/rate-ceiling/). > Traffic heavier than about 30 Mbit/s. A 32-bit RouterOS build, which is > coming on a hEX S but has not arrived. Sustained operation beyond the windows stated. None of > these are claims this project is making, and none of them should be inferred from the ones it is. ## See also - [The rate ceiling](/mikroscope/cost/rate-ceiling/): the five runs and the conditions they were taken under. - [The cost of the observer](/mikroscope/cost/): the budget, and how to measure the cost on a device that is not this one. --- # Installing the agent What `mikroscope install` does to a RouterOS device and in which order, and how `upgrade` and `uninstall` change or remove it without touching anything they did not create. Source: https://jmrplens.github.io/mikroscope/install/ This page answers two questions: what `mikroscope install` does to your router, and how you get the router back. `install` puts a 6.1 MiB agent image in a container on the router; `uninstall` removes it again. On RB5009UG+S+, RouterOS 7.24.2, 2026-09-12, a scripted doctor → install → status → upgrade → uninstall round trip (`make roundtrip`, which passes `--ephemeral` to doctor, install and upgrade) left the router's `/export` byte-identical, its `#` header lines aside. Every object install creates carries the comment `mikroscope: (managed by mikroscope)`, except the envlist and the image file, which cannot; the envlist carries the tag as its `MIKROSCOPE_TAG` entry. Nothing is written before it is listed; removal selects by that tag plus the object's identity, never by pattern, and is verified by ownership counts. ## Before the first install Three things have to be true on the router, and the tool cannot make any of them true for you: a container-capable architecture on RouterOS 7.24 or later, the `container` package, and `device-mode container=yes` — the last one needs a physical button press or a power cycle. [What the router needs](/mikroscope/install/prerequisites/) covers all three. `mikroscope doctor` checks them read-only, in one ssh connect, and prints the exact command or physical step for anything missing. One decision comes before the first install and not after it: where the agent image comes from. A checkout builds it, `--agent-tar` takes the one the release publishes, `--remote-image` has the router pull it, and `plan --rsc` writes a script that installs without this CLI at all. [Four ways to install](/mikroscope/install/routes/) lays the four out. ## What install does, in order 1. **Gets the agent image.** From a checkout, the CLI builds it: `go build ./cmd/mikroscope-agent` for `linux/` (`--arch`, default `arm64`) with `CGO_ENABLED=0`, packed into an image tar without Docker, and the build path is relative, so run it from the checkout. `--agent-tar` takes the tar the release publishes instead, and checks it before it uploads it. `--remote-image` skips this step entirely: the router pulls the image itself and nothing is uploaded. [Four ways to install](/mikroscope/install/routes/) is the choice, with what each route needs. 2. **Prints the plan.** One line of options (a token shows as `token=(set)`, never its value), the tag, then every RouterOS command numbered, with the `scp` upload and its size before the container step. It ends with `nothing above has been written yet`. `mikroscope plan` and `install --dry-run` stop here, before any ssh connect. 3. **Runs `doctor`.** Any missing prerequisite stops the install with `N prerequisite(s) missing; nothing was written`. `--no-doctor` skips this step. 4. **Asks** `write the objects above to the router? [y/N]`. `--yes` skips the question. 5. **Asks the router about every step at once.** One connect asks, for each step, whether mikroscope's object is there and whether the effect exists under any other owner. A step already ours prints `ok … (already present)` and is skipped; an effect that exists without the tag stops the install, naming it; an absent one is created, one connect per write. 6. **Uploads the image and creates the container.** The tar goes up with `scp`; one command then writes the envlist, adds the container, waits up to 15 s for the container to appear, then 3 s more (RouterOS extracts the image at add time), deletes the tar and starts the container. With `--remote-image` there is no upload and no tar to wait for or delete: the container is added with `remote-image=` and started. Either way the container is added with `privileged=`, which RouterOS knows from 7.24: on an earlier 7.x this is the step that fails, and `--privileged=false` is the way through. 7. **Probes the agent** from your host and says which transport works: [Reaching the agent](/mikroscope/install/reaching-the-agent/). A second `install` on a router where every step is already ours creates nothing and goes straight to the probe. **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. Where each of those objects lives, what the envlist carries and which container settings are written is on [Where things go](/mikroscope/install/layout/). Why two of them are firewall list memberships is on [The two firewall traps](/mikroscope/install/firewall/). ## How ownership is decided The tag is the only thing a removal matches on, together with the object's own identity: the veth by name, the address by interface, a list membership by list and interface, an address-list entry by list and address. Neither `/container/envs` nor `/file` carries a comment, so the envlist is signed by an entry `MIKROSCOPE_TAG` whose value is the exact tag, and the uploaded image counts as mikroscope's only while that marker exists. The agent ignores the entry. Every `find` quotes its address and port attributes. Unquoted, RouterOS parses them as typed values and the comparison with the stored one comes back empty — an unquoted `dst-port=9123` matches nothing (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). Over ssh, RouterOS reports an error as text with exit status 0 and abandons the rest of a `;`-joined line at the first one. A write that prints anything is therefore treated as a failure. If the container step fails after the upload, the uploaded tar is taken back (`undo removed the uploaded …`), because without the marker it would count as foreign forever. What else the installer refuses is on [What the installer refuses](/mikroscope/security/installer/). ## Upgrade **What `upgrade` replaces** - a new image and the container - the envlist, rewritten from the flags `upgrade` is given - network objects stay Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `mikroscope upgrade` gets the image the same way `install` does — a build from the checkout, `--agent-tar`, or `--remote-image` and no image at all — checks that every step is present (otherwise `nothing to upgrade: run install first`), asks for confirmation, removes the container step, waits for RouterOS's asynchronous removal, creates the step again with the new image and probes the agent. Unlike `install`, it prints no plan and runs no `doctor`: its prompt is the same `write the objects above to the router? [y/N]` with nothing listed above it. `mikroscope plan` with the same flags shows the container command it will write. The envlist belongs to the container step, so `upgrade` writes it again from the flags `upgrade` itself is given. `--port`, `--rate`, `--buffer`, `--memory-max`, `--mem-limit-mb`, `--capture-mb`, `--triggers`, `--floor-hz`, `--privileged`, `--ephemeral` and `--expose` read no environment variable: pass them again or they return to their defaults. That is also the way to change them without touching the network objects. Two omissions are not harmless. An upgrade without `--ephemeral` re-creates the container with its image and root on the internal flash and `start-on-boot=yes`. An upgrade of an `--expose` install without `--expose` and without a token (`--token` or `MIKROSCOPE_TOKEN`) re-creates the agent with no token, while the two LAN firewall rules stay. ## Uninstall **What `uninstall` removes** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. `mikroscope uninstall` runs the removals newest first. A removal that fails or prints anything is reported as `skip` with what the router said, and the rest continue. A step whose selector finds nothing prints `gone` just the same, which is why the count, not the removal output, decides. Then it asks the ownership count of every step in one connect, prints one line per step, and either ends with `verified: nothing mikroscope created remains on the router` or fails with `uninstall left objects behind`, naming the steps still present. The container removal waits, because RouterOS does not: `/container/remove` returns before the container is gone, and a `/file/remove` of the image issued meanwhile did nothing, silently (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). So it stops the container, removes it, waits up to 20 s for it to vanish, retries the tar removal for up to 15 s, and removes the marker only once the file is gone. If it is not, the marker stays and the count says so. > **Uninstall with the flags you installed with** > > `uninstall`, `status` and `upgrade` build their selectors from their own flags, not from anything > stored on the router. Pass the same `--name`, `--veth`, `--subnet`, `--iface-list`, `--addr-list`, > `--disk` or `--ephemeral` and `--port` that `install` got. For an install with `--remote-image`, > pass that too: the container's ownership count is taken by what it was created from, and a > remote-image install has no tar to count. For an install with `--expose`, pass > `--expose --lan-address … --token …` as well: without `--expose` the plan has no firewall steps, > so `uninstall` neither removes the two rules nor counts them. ## Status `mikroscope status` prints the ownership count of every step, from one connect. When nothing is installed it ends with the `verified` line and probes nothing. Otherwise it probes the agent's `/healthz` with a 3 s timeout and prints its version, rate, sequence and oldest sequence, uptime, slipped ticks and round trip, then one line about the board: whether this build knows how to turn the kernel's port names (`eth0`, `eth1`, …) into RouterOS's on it. Where it does not, the line asks for the measurement that would add the board; [RouterOS ports and kernel names](/mikroscope/reference/port-names/) shows how to take it. If the agent does not answer, it prints `agent: not reachable from this host` with the error, and no board line; `status` still exits 0. ## Day to day ```sh mikroscope doctor # read-only mikroscope plan # every command, nothing written mikroscope install [--ephemeral] # doctor, confirmation, writes, probe mikroscope status # ownership counts + agent health mikroscope upgrade # new image, container only mikroscope uninstall # removes and verifies mikroscope image --arch arm64 # the tar, for side-loading by hand mikroscope plan --rsc --out install.rsc # the same writes, as a RouterOS script ``` `--router` takes `user@host` or an ssh config alias and is required by every verb that connects. ## Flags and the environment For these verbs, the flags that read their default from a variable are `--router`, `--ssh-port`, `--ssh-key`, `--name`, `--veth`, `--subnet`, `--iface-list`, `--addr-list`, `--disk`, `--arch`, `--token`, `--lan-address`, `--agent-tar` and `--remote-image` (`MIKROSCOPE_ROUTER`, `MIKROSCOPE_SSH_PORT` and so on); `.env.example` documents them. The CLI does not read `.env` itself: ```sh set -a; . ./.env; set +a ``` Every value that reaches a RouterOS command is bounded before the first connection: names, disks, the architecture, memory syntax, the token's characters, the port and rate ranges, the registry reference, the subnet, which must be an IPv4 /30 given at its network address, and `--triggers`, which the agent's own parser decides on — an unknown condition, a bad threshold, a quote or a semicolon fails the verb with exit 2 and writes nothing. The agent parses `TRIGGERS` again when it starts, because an envlist can be edited by hand on the router; a value it rejects there makes it exit and log `bad configuration`. The full list is on [Commands and flags](/mikroscope/reference/cli/) and [Environment variables](/mikroscope/reference/environment/). ## What ssh costs the router Each ssh connect costs the RB5009 20–27 % CPU for its duration. So the CLI batches every read into one connect — `doctor` is one, `status` is one, the state questions of `install` are one — and each write takes one more, plus the `scp` upload. ssh is never a data path: `record` and `forward` reach the agent over HTTP or the RouterOS API. > **Untested** > > Reboot survival of a persistent install: the reference router is production and is not rebooted > for tests. The byte-identical round trip on any device other than the RB5009, on any RouterOS > other than 7.24.2, or with install and upgrade run without `--ephemeral` (the scripted run passed > it to doctor, install and upgrade, not to status or uninstall). The round trip has not been run > against the current container settings: `privileged=yes`, `memory-max=64M`, and the envlist > entries `MEM_LIMIT_MB`, `CAPTURE_MB`, `TRIGGERS` and `FLOOR_HZ`; it ran with `memory-max=32M`. ## See also - [What the router needs](/mikroscope/install/prerequisites/): the architecture, the package and the device-mode step that needs your hands on the router. - [Four ways to install](/mikroscope/install/routes/): a checkout, the published tar, a registry pull, or a RouterOS script. - [Where things go](/mikroscope/install/layout/): disks, the envlist and the container settings. - [Reaching the agent](/mikroscope/install/reaching-the-agent/): the probe, and direct, relay and `--expose`. - [What the installer refuses](/mikroscope/security/installer/): objects it will not build on or remove. --- # Getting the CLI onto your machine How to have the `mikroscope` command on Linux, macOS or Windows — which archive to download for your own computer as opposed to the router, how to verify it, where to put it so the shell finds it, and how to check it works. Source: https://jmrplens.github.io/mikroscope/install/cli/ Two programs are released together and they run on two different machines: - **`mikroscope`**, the CLI and collector, runs on **your computer** — the laptop or server you type commands on. This page is about that one. - **`mikroscope-agent`** runs **on the router**, inside a container. `mikroscope install` puts it there for you; you never run it yourself, and the only time you download it by hand is the `--agent-tar` route. So the archive you want here is picked by **your** operating system and CPU, not by the router's. The router's architecture decides something else, and [What the router needs](/mikroscope/install/prerequisites/) has that table. ## Which archive | Your machine | Download | | ------------------------------------- | ---------------------------------------------- | | Linux, ordinary PC or server | `mikroscope__linux_x86_64.tar.gz` | | Linux on ARM (Raspberry Pi 4/5, …) | `mikroscope__linux_arm64.tar.gz` | | macOS, Apple silicon (M1 and later) | `mikroscope__darwin_arm64.tar.gz` | | macOS, Intel | `mikroscope__darwin_x86_64.tar.gz` | | Windows | `mikroscope__windows_x86_64.zip` | They are on the [latest release](https://github.com/jmrplens/mikroscope/releases/latest). Anything whose name starts with `mikroscope-agent` is the other program. ## Install it - **Linux** 1. Download the archive and the checksums: ```sh VERSION=1.0.4 curl -fsSLO https://github.com/jmrplens/mikroscope/releases/download/v$VERSION/mikroscope_${VERSION}_linux_x86_64.tar.gz curl -fsSLO https://github.com/jmrplens/mikroscope/releases/download/v$VERSION/checksums.txt ``` 2. Check it against the list before unpacking it: ```sh sha256sum --ignore-missing -c checksums.txt ``` 3. Unpack and put it where the shell looks: ```sh tar xzf mikroscope_${VERSION}_linux_x86_64.tar.gz mikroscope sudo install -m 0755 mikroscope /usr/local/bin/mikroscope ``` Without `sudo`, `mkdir -p ~/.local/bin && install -m 0755 mikroscope ~/.local/bin/` works as well, as long as `~/.local/bin` is on your `PATH`. - **macOS** 1. Download the archive for your CPU — `darwin_arm64` for Apple silicon, `darwin_x86_64` for Intel — and the checksums: ```sh VERSION=1.0.4 curl -fsSLO https://github.com/jmrplens/mikroscope/releases/download/v$VERSION/mikroscope_${VERSION}_darwin_arm64.tar.gz curl -fsSLO https://github.com/jmrplens/mikroscope/releases/download/v$VERSION/checksums.txt ``` 2. Check it: ```sh shasum -a 256 --ignore-missing -c checksums.txt ``` 3. Unpack, clear the quarantine flag the download put on it, and install it: ```sh tar xzf mikroscope_${VERSION}_darwin_arm64.tar.gz mikroscope xattr -d com.apple.quarantine mikroscope 2>/dev/null || true sudo install -m 0755 mikroscope /usr/local/bin/mikroscope ``` The binary is not signed or notarized, so without that `xattr` line macOS refuses to run it and says it "cannot be opened because the developer cannot be verified". - **Windows** 1. Download `mikroscope__windows_x86_64.zip` and `checksums.txt` from the release page. 2. Check it in PowerShell, against the line for your file in `checksums.txt`: ```powershell Get-FileHash .\mikroscope_1.0.4_windows_x86_64.zip -Algorithm SHA256 ``` 3. Unpack it somewhere permanent and put that folder on your `PATH`: ```powershell Expand-Archive .\mikroscope_1.0.4_windows_x86_64.zip -DestinationPath $HOME\mikroscope $env:PATH += ";$HOME\mikroscope" ``` That line lasts for the session. To keep it, add the folder in **Settings → System → About → Advanced system settings → Environment Variables**, or: ```powershell [Environment]::SetEnvironmentVariable("PATH", "$env:PATH;$HOME\mikroscope", "User") ``` > **ssh on Windows** > > Three of the four install routes reach the router over ssh, and the CLI uses the `ssh` and `scp` > on your `PATH`. Windows 10 and 11 ship OpenSSH: `Get-Command ssh` should find it, and > **Settings → System → Optional features** installs it if not. The fourth route, > [a RouterOS script](/mikroscope/install/routes/#a-routeros-script), needs no ssh at all. - **With Go** If you have Go 1.27 or later and would rather build it: ```sh go install github.com/jmrplens/mikroscope/cmd/mikroscope@latest ``` That puts `mikroscope` in `$(go env GOPATH)/bin`, which is `~/go/bin` unless you moved it, and that directory has to be on your `PATH`. A binary built this way reports the module version rather than a release stamp. A checkout builds both programs at once, which is the contributor's route: ```sh git clone https://github.com/jmrplens/mikroscope cd mikroscope make build # leaves bin/mikroscope and bin/mikroscope-agent ``` ## Check it ```sh mikroscope version ``` It prints the version, the commit and the build date. Then, with a router to point at: ```sh mikroscope doctor --router user@192.168.88.1 ``` `doctor` writes nothing. It reads the device, prints what it is, and marks each prerequisite `ok` or `MISSING` with the command that fixes it — including the one nobody can do remotely. That is the first thing to run, before any install route. > **The flags have environment variables** > > `--router`, `--ssh-port`, `--ssh-key`, `--arch` and most of the rest read a default from a > `MIKROSCOPE_*` variable, so a shell that exports them turns every command below into > `mikroscope doctor`. [Environment variables](/mikroscope/reference/environment/) is the list, and > `.env.example` in the repository is a template. ## See also - [What the router needs](/mikroscope/install/prerequisites/): the three prerequisites on the device, and the architecture table for the **router**. - [Four ways to install](/mikroscope/install/routes/): how the agent image reaches the router once you have the CLI. - [Commands and flags](/mikroscope/reference/cli/): every verb and every flag. --- # What the router needs The architecture, the container package and the device-mode step that needs a hand on the router, plus what your own host needs, and how `doctor` checks each one. Source: https://jmrplens.github.io/mikroscope/install/prerequisites/ This page lists what has to be in place before `mikroscope install` can write anything: three things on the router the tool cannot do for you, the resources and firewall lists it checks, and what the host you run it from needs. `mikroscope doctor` checks all of it read-only, in one ssh connect, and prints the exact command or physical step for anything missing. > **One step needs your hands on the router** > > `device-mode container=yes` cannot be set remotely. After the command, RouterOS waits five minutes > for someone to press the reset or mode button, or power-cycle the device. No ssh session, API call > or option of this tool can do that step. Plan to be at the router. ## A container-capable device The router must run RouterOS 7.24 or later on one of three architectures: arm64, arm (32-bit RouterOS on the hEX refresh line) or x86_64. Not MIPS, not TILE. 7.24 is the floor because the container step writes `privileged=`, an attribute RouterOS added in that release. `doctor` prints the version and checks nothing against it, so on an earlier 7.x the install runs as far as the container and fails there with RouterOS's error about an unknown `privileged` parameter — and a write that prints anything counts as a failure, so the install stops and takes the uploaded tar back with it. `--privileged=false` is the way through, at the price of everything the container's user namespace hides: [What privileged buys](/mikroscope/limits/privileged/) lists it. Tell the CLI which one with `--arch`: `arm64` (the default), `arm` or `amd64`. `doctor` compares it with the router's `architecture-name` and, on a mismatch, names the flag to re-run with. | Your device | `architecture-name` | `--arch` | The agent build | | -------------------------------------------------- | ------------------- | -------- | --------------------------- | | RB5009, CCR2004, hAP ax³ and other 64-bit ARM | `arm64` | `arm64` | `linux/arm64` | | hEX Refresh / hEX S (2025), any EN7562CT board | `arm` | `arm` | `linux/arm/v5` | | Other 32-bit ARM (hAP ac², hAP ax², …) | `arm` | `arm` | `linux/arm/v5` or `v7` | | CHR and x86 RouterOS | `x86_64` | `amd64` | `linux/amd64` | **32-bit ARM is two things, not one.** MikroTik's container documentation states that devices with the EN7562CT CPU — the hEX Refresh line — "support only arm32v5 container images", while its other 32-bit ARM boards run an ARMv7 userland. An ARMv5 binary runs on both; an ARMv7 one does not run on the first, and the way it fails is an `exec format error` in the container log after a successful install. So `--goarm` defaults to **5**, the level that starts everywhere, and the image declares the matching variant. `--goarm 7` builds the ARMv7 one for a board where that instruction set is wanted; what the difference costs has not been measured, because this project has no ARM hardware. `--remote-image` makes the question go away: the published index carries `linux/amd64`, `linux/arm64`, `linux/arm/v7` and `linux/arm/v5`, and the router picks its own. ## The container package Download the `container` package for your architecture and RouterOS version from mikrotik.com, upload it to the router and reboot; then `/system/package/enable container`. That is the fix `doctor` prints, and it counts the package as present only when it is installed and not disabled. ## device-mode container=yes MikroTik gates containers behind a physical step: 1. Run, on the router's console: ```text /system/device-mode/update container=yes ``` 2. The console answers: ```text update: please activate by turning power off or pressing reset or mode button in 5m00s ``` 3. Within those five minutes, press the reset or mode button, or power-cycle the router. If nobody does, the change is cancelled. After three failed attempts the router says `too many unsuccessful attempts … to reset attempt-count` and needs a power cycle before it accepts another. ## What doctor checks `doctor` prints `device:` with the board, the RouterOS version and the architecture, then one line per check marked `ok` or `MISSING`, with what it found in parentheses and, for a missing one, a `fix:` line. It ends with `doctor: every prerequisite is met`, or fails with `N prerequisite(s) missing; nothing was written`. `install` runs the same checks first unless you pass `--no-doctor`. The checks doctor runs: | Check, as printed | Passes when | The fix it names | | --- | --- | --- | | registry-url is https:// | with `--remote-image`, `/container/config registry-url` names the reference's registry host. Without `--remote-image` doctor does not ask: the setting is global to the device and mikroscope never writes it | `/container/config/set registry-url=https://` on the router, which applies to every container on it, or install from a tar with `--agent-tar` | | container package installed and enabled | a `container` package exists with `disabled=no` | download, upload, reboot; then `/system/package/enable container` | | device-mode container=yes | `/system/device-mode` reports `container=yes` | `/system/device-mode/update container=yes`, then the reset or mode button, or a power cycle, within 5 minutes | | architecture matches --arch | the router's `architecture-name` is the one `--arch` maps to (`arm64`, `arm`, `x86_64`) | re-run with the `--arch` it names | | free memory ≥ <--memory-max> | `free-memory` is at least what `--memory-max` asks for, 64 MiB by default | free memory on the router, or ask for less with `--memory-max` | | free flash ≥ (image tar + extracted root) | without `--disk`: `free-hdd-space` is at least twice the image plus 4 MiB | free flash, or install with `--disk tmpfs` or `--ephemeral` where a tmpfs disk exists | | disk exists | with `--disk` or `--ephemeral`: a disk with that slot exists; its free space is not checked | `/disk/add type=tmpfs tmpfs-max-size=64M slot=tmpfs` for a RAM disk, or name an existing disk with `--disk` | | interface list exists (raw rule trap) | the `--iface-list` list (default `LAN`) exists | `/interface/list/add name=…`, or pass the list your `in-interface-list=!…` drop rule uses | | address list has entries (raw rule trap) | the `--addr-list` list (default `LANs`) has at least one entry | pass the list your `drop local if not from default IP range` rule uses; an empty list is fine only if there is no such rule | | veth name is free or ours | always reported `ok`, with the count found | none: a collision is caught by `install` itself | The flash check uses the real tar size under `install`. `doctor` on its own assumes a 7 MiB image, so it asks for 18.0 MiB. Twice the image because the tar and the root extracted from it are on the disk together until `install` deletes the tar; with `--remote-image` no tar is uploaded, so the check asks for the 4 MiB of headroom alone. The memory threshold follows `--memory-max`: it asks for at least what that flag asks for, which is 64 MiB by default, so `--memory-max 128M` on a router with 70 MiB free is caught here rather than by a container that will not start. The `registry-url` check runs only with `--remote-image`, and only when the reference carries a registry host. `/container/config` is global to the device and shared with every other container on it, so mikroscope reads that setting and never writes it. RouterOS ships it as `https://registry-1.docker.io`, so the Docker Hub reference, `--remote-image jmrplens/mikroscope-agent:1.0.0`, needs nothing set there on an untouched router, and the GHCR reference is the one that needs the setting changed first. [Four ways to install](/mikroscope/install/routes/#a-registry-pull) has the command to set it by hand. The two list checks exist because of two raw firewall rules that drop every packet a container sends; [The two firewall traps](/mikroscope/install/firewall/) explains them. `doctor` marks an empty address list missing even on a router that has no such rule; there, `--no-doctor` is the way past it, and it skips every other check with it. ## What your host needs - **ssh to the router as a user that can write**, your own admin access. The CLI runs the system `ssh` with `BatchMode=yes` and `ConnectTimeout=15`, so it cannot answer a password prompt: use a key or an ssh agent. `--router` takes `user@host` or an ssh config alias; `--ssh-port` and `--ssh-key` (`MIKROSCOPE_SSH_PORT`, `MIKROSCOPE_SSH_KEY`) fall back to your ssh configuration when unset. The image goes up with `scp`, on the two routes that upload one. The RouterOS script route needs no ssh at all. - **An agent image, by one of four routes.** The CLI can build one (`go build ./cmd/mikroscope-agent` from a checkout, which needs Go 1.27), take the tar the release publishes (`--agent-tar`, no toolchain and no checkout), or let the router pull the image itself (`--remote-image`, nothing uploaded). The fourth route needs no CLI on your machine at all: `plan --rsc` writes a RouterOS script you paste into the router. [Four ways to install](/mikroscope/install/routes/) has the commands, what each route needs and how to verify a download. - **A way to reach the agent** once it runs: a route to the container's /30 through the router, the relay over the RouterOS API, or `--expose`. See [Reaching the agent](/mikroscope/install/reaching-the-agent/). - **A RouterOS API user**, only for the relay transport, `--log-markers` and the collector's API tier. It stays on your machine; its policy is on [The API user](/mikroscope/security/api-user/). > **Untested** > > An install on arm or x86_64: every installation so far ran on one arm64 RB5009, and the hEX S that > will test 32-bit RouterOS has not arrived. Any RouterOS other than 7.24.2. The container step > writes `privileged=`, which RouterOS added in 7.24, and envlist entries with `key=`, where `name=` > fails on 7.24.2; how an earlier 7.x takes either was not tried. ## See also - [Installing the agent](/mikroscope/install/): what `install` does once these are in place. - [Four ways to install](/mikroscope/install/routes/): the four routes the agent image can take to the router, and how to verify a download. - [The two firewall traps](/mikroscope/install/firewall/): why the two list checks exist. - [Where things go](/mikroscope/install/layout/): the disk the flash check is about, and `--ephemeral`. --- # Four ways to install The four routes the agent image can take to the router — a Go checkout, the published tar, a registry the router pulls from, or a RouterOS script you paste in — what each one needs, what each one writes, and how to verify a downloaded release. Source: https://jmrplens.github.io/mikroscope/install/routes/ The agent is a container image, and the four routes below differ in one thing only: how that image reaches the router. Everything else `install` writes — the veth, the address, the two list memberships, the envlist, the container and its tag — is the same whichever route you take, and so are the prerequisites: [What the router needs](/mikroscope/install/prerequisites/) comes first in all four, because `device-mode container=yes` needs a hand on the device and no route works around it. ## Which one | Route | Needs | Prefer it when | | ------------------------------------------------ | ---------------------------------------------------------------- | ----------------------------------------------------------------- | | **[A registry pull](#a-registry-pull)** — start here | the router can reach Docker Hub; nothing else | almost always: one command, nothing to choose, nothing uploaded | | [A RouterOS script](#a-routeros-script) | a terminal on the router; `--remote-image` | you reach the router through WinBox or WebFig and not over ssh | | [A checkout, with Go](#a-checkout-with-go) | Go 1.27 and the repository; ssh to the router | you are working on mikroscope and want the agent from your tree | | [The published tar](#the-published-tar) | the release assets, the right one for the board; ssh to the router | the router cannot reach a registry | **Take the first one unless something stops you.** A registry pull is a single command with nothing to pick: the published image index carries every platform a MikroTik container can be, so the router matches its own and no one has to know whether the board is 64-bit ARM or one of the two 32-bit kinds. Nothing lands on the flash, and `uninstall` has no file to account for. The tar is last on purpose. It is the right route for a router with no way out to a registry, and it is the only one where **you** pick the architecture — the way that goes wrong is an image that installs, starts and dies with `exec format error` in the container log. If you take it, read [which tar](#which-tar) before downloading. ## A registry pull The recommended route, and the shortest. One command, nothing to download, nothing uploaded: ```sh mikroscope install --router user@192.168.88.1 \ --remote-image jmrplens/mikroscope-agent:1.0.4 ``` Nothing is uploaded, no tar lands on the device, and `uninstall` has no file to account for: the container step becomes `/container/add remote-image="jmrplens/mikroscope-agent:1.0.4" …` and the plan prints `the router pulls … (nothing is uploaded)` where the upload line would be. The release publishes the image twice, as `jmrplens/mikroscope-agent:1.0.4` on Docker Hub and as `ghcr.io/jmrplens/mikroscope-agent:1.0.4` on GHCR. Both carry `linux/amd64`, `linux/arm64`, `linux/arm/v7` and `linux/arm/v5`, and RouterOS picks the one its architecture needs — which is why this route asks nothing about the board: the two kinds of 32-bit ARM MikroTik ships are both in the index. The reference above is the Docker Hub one, and it carries no registry host: the router pulls it from whatever `/container/config registry-url` already names, and RouterOS ships that setting as `https://registry-1.docker.io`. On a router where nobody has changed it, the command above needs nothing set first — on the RB5009 this project is measured on, that setting reads `https://registry-1.docker.io`. The GHCR reference is the alternative, and it needs `/container/config/set registry-url=https://ghcr.io` on the device first, which is a change to every container on it. It needs two things the other routes do not: the router has to reach the registry, and it has to have room in RAM for the layers while it extracts them. > **The registry host is a global router setting** > > RouterOS takes the registry host from `/container/config registry-url`, which is global to the > device and shared with every other container on it, and only the rest of the reference goes into > `remote-image=`. mikroscope never writes that setting — pointing your router's registry somewhere > else to install a probe would be a change to somebody else's containers. `doctor` reads it > instead, and when the reference names a host the setting does not match it names the one command > to run — `/container/config/set registry-url=https://ghcr.io` for the GHCR reference. Install > with `--agent-tar` if you would rather not change it. A reference with no host — `jmrplens/mikroscope-agent:1.0.4` — leaves the registry to whatever the router is already configured for, and `doctor` then checks nothing about it. `--remote-image` reads its default from `MIKROSCOPE_REMOTE_IMAGE`, and `upgrade` takes it too; `image` refuses it, because there is no tar to write. ## A RouterOS script For a router you reach through WinBox or WebFig, or where you do not want ssh from another machine at all: ```sh mikroscope plan --rsc \ --remote-image jmrplens/mikroscope-agent:1.0.4 \ --out install.rsc ``` The file holds the same commands `install` runs, in the same order, each object tagged the same way, so `status` and `uninstall` from the CLI recognise them afterwards. Read it, then paste it into the router's terminal, or upload it and `/import` it. Without `--out` it goes to standard output. It carries its own header: the tag it writes, what to check before running it, and, at the end, `/container/print where name~"mikroscope"` and the `/healthz` URL the agent answers on. Two caveats, both stated in the script itself: - **It cannot upload anything.** A script running on the router has no way to put the image there, so pair it with `--remote-image`. Without one, the header says instead which filename to put on the device first — the name the container step expects — and how to regenerate the script for a registry pull. - **With `--token`, the file is a credential.** The envlist line carries the token in clear, because the router needs it in clear. The CLI writes the file `0600`; what you do with it afterwards is the exposure. Nothing in the script checks anything. There is no `doctor`, no question about what the objects it creates would collide with, and no confirmation prompt: it writes. Run `mikroscope doctor` from a machine that can, or read [What the router needs](/mikroscope/install/prerequisites/) and check the three prerequisites by hand, before you paste it. All four routes were run end to end against the reference RB5009UG+S+ (RouterOS 7.24.2, arm64) on 2026-09-17, one after another, each installing under its own name, veth and `/30` so that nothing already on the device was touched, and each removed again before the next. In every one the agent answered `/healthz` from the collector host: the checkout build and the published `mikroscope-agent-arm64.tar` at a 2 ms round trip, the router's own pull of `jmrplens/mikroscope-agent:1.0.4` from Docker Hub at 2 ms, and the `plan --rsc` script — uploaded and `/import`ed, with no CLI involved in the install itself — at 15 ms on its first samples. `uninstall` then verified by ownership count in each case, and the router's `/export` after all four was byte-identical to the one taken before them. > **Untested** > > `--remote-image` against **GHCR**. `/container/config registry-url` is a single global RouterOS > setting that mikroscope reads and never writes, and the reference router points at Docker Hub; > pointing it at ghcr.io to test that path would change the registry for every other container on > the device. The image is published to both registries and CI starts it from GHCR on all three > architectures, but no router has pulled it from there. Nor has any route been run on arm or on > x86_64 hardware. ## A checkout, with Go ```sh git clone https://github.com/jmrplens/mikroscope cd mikroscope make build bin/mikroscope install --router user@192.168.88.1 ``` `plan`, `install`, `upgrade` and `image` build the agent themselves: `go build ./cmd/mikroscope-agent` for `linux/` (`--arch`, default `arm64`) with `CGO_ENABLED=0`, packed into an image tar without Docker. The build path is relative, so run the CLI from the checkout. This is the only route that installs an agent built from your own tree, which is why it is the one to use while changing the agent. `make build` leaves the CLI in `bin/mikroscope`. With no Go toolchain on `PATH`, the verb stops before anything is written and names the other two routes and the Go version it wanted. ## The published tar The route for a router that cannot reach a registry. Two assets: the CLI archive for the machine you run it from — which is [Getting the CLI](/mikroscope/install/cli/), and has nothing to do with the router — and one agent image tar, for the **router's** architecture. No Go toolchain, no checkout. ### Which tar | Your MikroTik | `architecture-name` | Agent image tar | | ----------------------------------------------- | ------------------- | ------------------------------- | | RB5009, CCR2004, hAP ax³, other 64-bit ARM | `arm64` | `mikroscope-agent-arm64.tar` | | hEX Refresh / hEX S (2025), any EN7562CT board | `arm` | `mikroscope-agent-armv5.tar` | | Other 32-bit ARM (hAP ac², hAP ax², …) | `arm` | `mikroscope-agent-armv7.tar`, or the v5 one | | CHR, x86 RouterOS | `x86_64` | `mikroscope-agent-amd64.tar` | `mikroscope doctor --router …` prints the architecture off the device, so run it first and let it tell you. **If you are not sure which 32-bit ARM board you have, take the v5 tar**: MikroTik's container documentation says EN7562CT boards "support only arm32v5 container images", and an ARMv5 image runs on every 32-bit ARM MikroTik ships, while an ARMv7 one does not run on those. 1. Download `mikroscope_1.0.4__.tar.gz` (`.zip` on Windows) and the agent image tar from the table above, together with `checksums.txt` and `checksums.txt.sigstore.json`. 2. Verify them, below, before unpacking anything. 3. Unpack the CLI and install: ```sh tar xzf mikroscope_1.0.4_linux_x86_64.tar.gz ./mikroscope install --router user@192.168.88.1 \ --arch arm64 --agent-tar mikroscope-agent-arm64.tar ``` The CLI reads the tar before it uploads it, which is what catches the wrong download: it prints what it read, as `using mikroscope-agent-arm64.tar: linux/arm64, agent KiB`, and on the ARMv7 image it adds the note that an EN7562CT board needs the v5 one instead. It wants a one-image `manifest.json`, the config that manifest names, one layer, and `/mikroscope-agent` as the entrypoint; anything else fails as `this is not a mikroscope agent image`. Then the image's architecture has to be the one `--arch` says, or the verb fails naming the asset to download instead — an `amd64` image on an arm64 board would otherwise install, start, and die with `exec format error` in the container log. On a tar it accepts it prints what it read, as `using mikroscope-agent-arm64.tar: linux/arm64, agent KiB`. `--agent-tar` reads its default from `MIKROSCOPE_AGENT_TAR`, and `upgrade` and `image` take it too. From here on the install is the upload route: the tar goes up with `scp`, RouterOS extracts it at add time, and `install` deletes it. > **Two assets have similar names** > > `mikroscope-agent-arm64.tar` is the side-loadable container image, the one `--agent-tar` wants. > `mikroscope-agent_1.0.4_linux_arm64.tar.gz` is an archive of the bare agent binary, for reading or > running it outside a container; `--agent-tar` rejects it. ### Verifying the download `checksums.txt` covers every archive and every agent image tar, and it is the one file the signature vouches for. The signature is keyless: the identity is the workflow run that produced it, recorded in a public transparency log, so there is no key to fetch. ```sh cosign verify-blob \ --certificate-identity-regexp 'https://github.com/jmrplens/mikroscope/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --bundle checksums.txt.sigstore.json \ checksums.txt sha256sum --ignore-missing -c checksums.txt ``` `--ignore-missing` is what lets you check the two files you downloaded against a list that covers the whole release. Each archive also ships an SPDX SBOM (`.spdx.json`) with a signature bundle of its own, verified the same way. ## See also - [What the router needs](/mikroscope/install/prerequisites/): the three prerequisites every route shares, and what `doctor` checks. - [Installing the agent](/mikroscope/install/): what `install` does once the image is decided. - [Where things go](/mikroscope/install/layout/): the disk the tar and the container root use, and what `--remote-image` leaves off it. - [Reaching the agent](/mikroscope/install/reaching-the-agent/): the probe that runs afterwards. --- # The two firewall traps The two raw rules in the MikroTik default firewall that silently drop every packet a container sends, what `install` adds so they do not, and what to pass when your lists have other names. Source: https://jmrplens.github.io/mikroscope/install/firewall/ This page answers why an agent that is running can still be unreachable on a router with the MikroTik default firewall, and what `install` changes in that firewall so it is not. It adds two list memberships and nothing else; without `--expose` it writes no firewall rule. ## Two rules that drop everything a container sends The MikroTik default firewall carries two raw rules that silently drop every packet a container sends: - `drop the rest (in-interface-list=!LAN)` matches a packet that enters on an interface outside the `LAN` interface list, and a new veth is outside it. - `drop local if not from default IP range (src-address-list=!LANs)` matches a source address outside the `LANs` address list, and the container's /30 is outside it. The drop is silent. From your host the agent does not answer, and that looks the same as a container that is not running — which is why the probe after `install` asks the router whether the container runs before it suggests anything else. ## What install adds `install` adds the veth to your `LAN` interface list and the container's /30 to your `LANs` address list. Both are additions to lists that already exist, both carry the tag, and `uninstall` removes both by that tag plus the list and the member. With the defaults the two writes are: | Step | Command | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | interface-list membership | `/interface/list/member/add list="LAN" interface="veth-mikroscope" comment="mikroscope:mikroscope (managed by mikroscope)"` | | address-list membership | `/ip/firewall/address-list/add list="LANs" address=172.30.10.0/30 comment="mikroscope:mikroscope (managed by mikroscope)"` | On RB5009UG+S+, RouterOS 7.24.2, 2026-09-11 these two memberships were enough for a host on the LAN to reach the agent directly through the router. A membership is not scoped to mikroscope. Any other rule on your router that matches the `LAN` interface list or the `LANs` address list matches the container's traffic too, for as long as it is installed. Read your own rules with that in mind. ## When your lists have other names Pass the names your rules use: - `--iface-list` (`MIKROSCOPE_IFACE_LIST`): the list your `in-interface-list=!…` drop rule uses. - `--addr-list` (`MIKROSCOPE_ADDR_LIST`): the list your `drop local if not from default IP range` rule uses. `doctor` checks both before `install` writes anything. An interface list that does not exist is reported with the fix `/interface/list/add name=…`, or the flag. An address list with no entries is reported too; its fix text says an empty list is fine only if no such rule exists, but the check still counts as missing, and only `--no-doctor` gets past it, skipping every other check with it. `uninstall`, `status` and `upgrade` need the same two flags again, because the selectors are built from them; with other names `upgrade` finds the memberships missing and refuses with `nothing to upgrade: run install first`. ## When the membership already exists If the veth is already in the interface list, or the /30 is already in the address list, and that entry does not carry mikroscope's tag, `install` stops at that step and names it: the effect exists, but mikroscope did not create it and will not later remove it. Pick another `--veth` or `--subnet`, or remove your entry by hand if it is yours. `uninstall` never touches it. ## The rules `--expose` adds Only `--expose` writes firewall rules: a dst-nat on the router's LAN address and a forward accept placed before the first forward drop, both tagged, both removed by `uninstall --expose --lan-address --token …`: the dst-nat selector matches the LAN address, and `--expose` is refused without a token even though no selector uses it. What that opens and why the token becomes mandatory is on [What --expose opens](/mikroscope/security/expose/); how to use it is on [Reaching the agent](/mikroscope/install/reaching-the-agent/). > **True of this device, not of yours** > > One firewall was measured: the RB5009's, on 2026-09-11, where the two memberships were enough. A > firewall with other drop rules in `raw`, `input` or `forward` may drop the container's traffic > somewhere else, and `install` adds nothing for that case beyond the two memberships. ## See also - [Reaching the agent](/mikroscope/install/reaching-the-agent/): what to do when the memberships are not enough. - [What the router needs](/mikroscope/install/prerequisites/): the `doctor` checks for both lists. - [What --expose opens](/mikroscope/security/expose/): the two rules, and who can reach the agent afterwards. --- # Where things go Where each object `install` creates lives on the router, which disk holds the image and the root, what the envlist carries, and the container settings install writes. Source: https://jmrplens.github.io/mikroscope/install/layout/ This page answers where `install` puts things: the addresses, the disk that holds the image and the container root, what goes into the envlist and nothing else, and the settings the container is created with. Every value here is what `mikroscope plan` prints for your flags, so the plan is the way to check it for your router before anything is written. ## The objects and their defaults | Object | Default | Flag | | -------------- | -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------- | | veth + /30 | `veth-mikroscope`, `172.30.10.0/30` (router `.1`, agent `.2`) | `--veth`, `--subnet` | | router address | `172.30.10.1/30` on the veth | derived from `--subnet` | | image tar | `.tar`, uploaded with `scp`, **deleted right after extraction**; none at all with `--remote-image` | `--disk`, `--ephemeral`, `--remote-image` | | container root | `mikroscope/` on the internal flash | `--disk tmpfs` for a RAM disk, `--ephemeral` | | envlist | `-env` | `--rate`, `--buffer`, `--port`, `--token`, … | | tag | `mikroscope: (managed by mikroscope)` on every object that takes a comment; as `MIKROSCOPE_TAG` in the envlist | `--name`, default `mikroscope` | `--subnet` must be an IPv4 /30 given at its network address; the router takes `.1` and the agent `.2`. `--name` is up to 32 characters and `--veth` up to 64, letters, digits, `_`, `.` and `-`, starting with a letter or digit. The default /30 and veth name were chosen not to collide with a hand-installed sampler on the reference device; if `172.30.10.0/30` is in use on yours, pick another. With `--disk`, the image and the root move together: `/.tar` and `/mikroscope/`. The value is a RouterOS disk slot: empty for the internal flash, `tmpfs`, `disk1`, `usb1` and so on. ## Persistent or ephemeral **Persistent is the default**: the root on the internal flash, `start-on-boot=yes`, and `restart-policy=on-failure` bounded to five retries ten seconds apart, so a broken image cannot loop at boot. **`--ephemeral`** puts the tar and the root on the router's tmpfs disk, if it has one — the RB5009 does — with `start-on-boot=no`: zero flash writes, and nothing survives a reboot. Verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11: `write-sect-since-reboot` read 58 279 before the install and 58 279 after the removal — the same value, not a small increase. With `--ephemeral` on a router that has no tmpfs disk, `doctor` reports it missing and names the command that adds one. `--disk tmpfs` without `--ephemeral` also puts both on the tmpfs disk, but keeps `start-on-boot=yes`; only `--ephemeral` changes the boot setting. > **Untested** > > Reboot survival, in either mode: the reference router is production and is not rebooted for tests. ## Why the tar is deleted at install RouterOS extracts the image when the container is added, so once the container exists the tar has no further use. A tar left on the device is what `uninstall` would later have to find in a `/file` index that lagged for minutes after a container removal. So `install` waits up to 15 s for the container to appear, then 3 s more, deletes the tar, and only then starts the container. It does not check that the extraction finished; on the RB5009 a 1.8 MiB tar was extracted within the same second as the add (RouterOS 7.24.2, 2026-09-11). The container is created with `ignore-remote-image-change=yes`. With the default `no`, RouterOS watched the image and, once the tar was removed, stopped and removed the container and re-extracted it minutes later (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). Until the tar is deleted, it and the root extracted from it share the disk, which is why `doctor` asks for twice the image plus 4 MiB of free flash. With `--remote-image` none of this happens. RouterOS pulls the layers itself, no tar lands on the device, there is nothing to wait for and nothing to delete, `doctor` asks for the 4 MiB alone, and `uninstall` has no file to account for — the container's ownership count is the container and the envlist. The container root still goes where `--disk` and `--ephemeral` say. ## What the envlist carries The envlist holds the agent's configuration and the ownership marker, and nothing else: The entries install writes into the agent's envlist: | Key | Written | From | Holds | | --- | --- | --- | --- | | `MIKROSCOPE_TAG` | always | `--name` | the ownership marker `mikroscope: (managed by mikroscope)`, written first and removed last; the agent ignores it | | `RATE_HZ` | always | `--rate`, default `10`, 1–100 | the sampler rate, in Hz | | `BUFFER_S` | always | `--buffer`, default `300`, 10–3600 | the ring's length, in seconds | | `PORT` | always | `--port`, default `9123`, 1–65535 | the agent's HTTP port | | `ADDR` | always | `--subnet` | the agent's address, the `.2` of the /30; the agent binds only there | | `MEM_LIMIT_MB` | always | `--mem-limit-mb`, default `40`, 8–1024 | the agent's Go soft memory limit, in MiB | | `FLOOR_HZ` | only when above 0 | `--floor-hz`, default `0`, 0–1000 | one cadence for every level source, in Hz | | `CAPTURE_MB` | always | `--capture-mb`, default `4`, 0–256 | the triggered-capture budget, in MiB; `0` turns captures off | | `TRIGGERS` | only when set | `--triggers` | the trigger conditions; unset, the agent uses its default set | | `TOKEN` | only when set | `--token` | the bearer token the agent requires, from `--token` or `MIKROSCOPE_TOKEN`, with or without `--expose` | No sink address, no sink token and no API credential goes into it. Any RouterOS user with `read` can list every container's envlist over the API (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11), so what is written here is readable by all of them — the token included, when you set one. The plan prints the envlist with the token masked as `(token)`. [What runs where](/mikroscope/security/) says which credential lives where; the agent reads more variables than `install` writes, and [Environment variables](/mikroscope/reference/environment/) lists them. ## The container settings The container is added with: - `file=` the uploaded tar, or `remote-image=` the reference without its registry host when the router pulls it; - `interface=` the veth, `root-dir=` the root, `envlist=` the envlist, and the tag as its comment; - `logging=yes`, so what the agent prints reaches the router log; - `start-on-boot=yes`, or `no` with `--ephemeral`; - `restart-policy=on-failure restart-max-count=5 restart-interval=10s`; - `memory-max=64M`, enforced as the container's cgroup limit (`--memory-max`); - `privileged=yes` (`--privileged=false` to opt out), which drops the container's user namespace so the kernel log, `/proc/slabinfo` and the MTD ECC counters are readable, and does not widen its network or PID namespace — [What privileged buys](/mikroscope/limits/privileged/) has the detail; - `ignore-remote-image-change=yes`, for the reason above. The agent catches SIGTERM: RouterOS kills a container that does not, immediately. It honours the default stop time of 10 s. ## Size the memory to the ring `--memory-max` and `--mem-limit-mb` have to move with `--rate` and `--buffer`. The ring holds `rate × buffer` lines of about 2.4 kB each; the Go soft limit wants about twice that and has to sit comfortably under `memory-max`. The agent's startup check counts 2 560 bytes a line plus `--capture-mb`: above `memory-max` it refuses to start, and above half of `--mem-limit-mb` it warns. The defaults, 40 MiB under `64M`, are sized for 10 Hz and a 300 s ring. The triggered-capture budget counts against both limits in that check, as the ring does. The measured runs used `--mem-limit-mb 40 --memory-max 64M` at 10 Hz, `--buffer 120 --mem-limit-mb 64 --memory-max 96M` at 50 Hz and `--buffer 120 --mem-limit-mb 80 --memory-max 128M` at 100 Hz. What a tight limit costs is on [The cost of the observer](/mikroscope/cost/); the runs themselves are on [The rate ceiling](/mikroscope/cost/rate-ceiling/). ## See also - [Installing the agent](/mikroscope/install/): the order these objects are created in, and how they are removed. - [The rate ceiling](/mikroscope/cost/rate-ceiling/): what each rate costs with the memory flags above. - [What privileged buys](/mikroscope/limits/privileged/): the one container setting that is a real privilege grant. - [What runs where](/mikroscope/security/): what else can read the envlist. --- # Reaching the agent How the host running `record` or `forward` reaches an agent that listens only on its veth address, what the probe after `install` tells you when it cannot, and what direct, relay and `--expose` each cost. Source: https://jmrplens.github.io/mikroscope/install/reaching-the-agent/ The agent listens on the veth address only — `http://172.30.10.2:9123` with the defaults — and makes no outbound connection, ever. Something has to come to it. This page answers how your host gets there, what `install` checks for you, and which of the three ways to use when the first one does not work. ## What the probe after install tells you After `install` and `upgrade`, the CLI probes the agent from your host: a TCP connect to the agent's address and port, then `GET /healthz`, retried a second after each failed attempt (each attempt times out after 2 s) for up to 30 s. When it answers, the CLI prints the agent's version, rate, sequence number, slipped ticks and the round trip, as in `direct transport ok: agent 1.0.0, 10 Hz, seq 29, 0 slipped, 7ms round trip`. The version is the CLI's own: `install` stamps the agent it builds with `internal/version.Version`, and both the Makefile and the release take that from the VERSION file, so an agent put there by 1.0.0 reports `1.0.0`. An agent the router pulled reports the tag it was published under. On the RB5009 (RouterOS 7.24.2, 2026-09-12) the agent answered 3 s after install, with a 5–7 ms round trip. When it does not answer, the CLI asks the router — one more connect — whether the container carrying the tag is running, because a veth is up only while its container runs: - **Not running**: it says so and points at the router log, `/log/print where topics~"container"`. The firewall is not the problem yet. - **Running**: this host cannot reach the agent's address. It suggests running the collector on a host the router routes to the veth from, or `install --expose --lan-address --token …`. The relay is not in that message; it is the third way, below. Either way the command fails with `agent installed but not reachable from this host`, and everything it created stays on the router. ## Direct, the default Your host reaches the /30 through the router, with plain HTTP to the agent's address. On RB5009UG+S+, RouterOS 7.24.2, 2026-09-11 the two list memberships `install` adds were enough; [The two firewall traps](/mikroscope/install/firewall/) explains them. The host needs its packets for the /30 to go to the router: a host whose default gateway is the router already sends them there. With a token set, the direct transport sends `Authorization: Bearer ` from `--token` or `MIKROSCOPE_TOKEN`. `/healthz` never needs it. ## Relay, through the RouterOS API `record` and `forward` with `--transport relay` run `/tool fetch` on the router over the binary API, and the router, which does reach its own veth, fetches from the agent. - It needs a RouterOS user with the `read,api,test` policy: on RouterOS 7.24.2 `/tool fetch` requires `test`, and without it the call comes back `not enough permissions (9)` rather than empty. [The API user](/mikroscope/security/api-user/) has the commands. The CLI takes `--api host:port` (`MIKROSCOPE_API_ADDR`), `--api-user` (`MIKROSCOPE_API_USER`) and the password from `MIKROSCOPE_API_PASSWORD` only, never from a flag. - Each call returns at most 64 512 B; anything longer is truncated silently by RouterOS. The relay therefore asks for at most 18 samples per pull and refuses a reply that reaches the cap rather than parse it truncated. `forward` warns at start when that cannot keep up with the agent's rate. - Each call takes either ~3 ms or ~1 s; about half the calls took ~1 s (RB5009, RouterOS 7.24.2, 2026-09-11). - This build's relay sends only the URL to `/tool fetch`, no header, so it does not present a token. Every path but `/healthz` returns 401 without one: an agent installed with a token needs the direct transport. `--transport auto`, the default, tries direct first; if `/healthz` does not answer and `--api` and `--api-user` are set, it tries the relay; otherwise it fails naming `install --expose`. ## --expose, on the router's LAN address **What `install --expose` adds** - two firewall rules, tagged - a token becomes mandatory - `uninstall` and `status` see the two rules only when given `--expose` again Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `install --expose --lan-address --token …` adds a dst-nat from the router's LAN address on the agent port to the veth, and a forward accept for that flow placed before the first forward drop (appended when the forward chain has no drop). Every LAN host can then reach the agent, so the token is mandatory: `install` refuses `--expose` without a token or without an IPv4 LAN address. The token may hold letters, digits, `_`, `.` and `-`, up to 128 characters. Both rules are removed by `uninstall` — given `--expose` again, as [Installing the agent](/mikroscope/install/) warns. Verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11: the pair works, and both rules are removable by tag. A client you point at `http://:9123` uses it: Prometheus, `curl`. The CLI's own transports do not. `record`, `forward` and the probe after `install` build the agent's address from `--subnet` and `--port`, and this build has no flag that points them at the exposed address. What a LAN-wide listener opens is on [What --expose opens](/mikroscope/security/expose/). ## Choosing | Way | Needs | Costs and limits | | ---------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------- | | direct | a route from your host to the /30 through the router | only the two list memberships `install` already adds | | relay | an API user with `read,api,test`; `--transport relay` or `auto` | 64 512 B and 18 samples per call, ~1 s in about half the calls, no token | | `--expose` | `--lan-address` and a token; two firewall rules on the router | reachable from the whole LAN; not used by `record`, `forward` or the probe | > **True of this device, not of yours** > > Every reachability result here is the RB5009's, on its own firewall and LAN, in September 2026. > The relay's ~1 s share and the 64 512 B cap were measured on RouterOS > 7.24.2 and may differ on another version. ## See also - [The two firewall traps](/mikroscope/install/firewall/): the memberships the direct transport depends on. - [The API user](/mikroscope/security/api-user/): the user the relay needs, and where to restrict it. - [What --expose opens](/mikroscope/security/expose/): the two rules and the token. - [The collector](/mikroscope/sinks/): what pulls from the agent once it is reachable. --- # Record, mark, plot How to pull a window of full-rate samples to your machine, mark the moments in it, add the router’s own log, and draw it as a deterministic SVG. Source: https://jmrplens.github.io/mikroscope/record/ The collector keeps a router under watch; `record` is for the other question — _I am about to change something, what does it actually do?_ This page answers how to take a recording from your machine, what the four files it writes hold, how markers get into it and in whose clock, how the router's log becomes part of it, and what `plot` draws. ## Three verbs ```sh mikroscope record --for 5m --out cap # cap.jsonl, cap.csv, cap.markers.csv, cap.meta.json; type lines to mark mikroscope mark --out cap "queue tree applied" # a note stamped with the current time (see below) mikroscope mark --out cap --log-markers # the router's own log lines, over the API mikroscope plot --in cap # cap.svg, deterministic ``` All three verbs share one flag set, so every flag below is accepted by each of them; the notes say which verb acts on it. | Flag | Default | What it does | | --------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--out` | `capture-` | Output prefix: `.jsonl`, `.csv`, `.markers.csv`, `.meta.json`. `mark` needs the prefix of an existing recording. | | `--for` | `0` | `record`: stop after this long. `0` records until Ctrl-C. | | `--from-start` | off | `record`: backfill everything the agent's ring holds before going live. | | `--poll` | `500ms` | How often to pull the agent's ring. | | `--batch` | `0` | Samples per pull. `0` is twice what one `--poll` interval produces at the agent's rate, at least 20; the relay caps a pull at 18. | | `--transport` | `auto` | `auto`, `direct` (HTTP to the veth) or `relay` (`/tool fetch` over the RouterOS API). | | `--log-markers` | off | `record`: adds the router log lines of the window once the recording stops. `mark`: adds the log lines of the recording's window. | | `--topics` | `system,interface,container` | Log topics kept as markers; add `firewall` or `script` when their lines are the story. | | `--router-tz` | `Local` | IANA zone the router's clock shows. RouterOS log times carry no zone. | | `--api` | `MIKROSCOPE_API_ADDR` | RouterOS API `host:port`, for the relay and for `--log-markers`. | | `--api-user` | `MIKROSCOPE_API_USER` | The API user. The password is read only from `MIKROSCOPE_API_PASSWORD`; there is no flag for it. | | `--token` | `MIKROSCOPE_TOKEN` | The agent's bearer token. | | `--port` | `9123` | The agent's HTTP port. | | `--subnet` | `MIKROSCOPE_SUBNET`, else `172.30.10.0/30` | The agent's /30; its address is `.2`. | | `--in` | none | `plot`: a recording prefix, or the path of a `.jsonl` file. | | `--svg` | `.svg` | `plot`: the output file. | | `--title` | the prefix | `plot`: the chart's title. | ## What a recording has been measured to deliver Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-12 · a 60 s `record` at 10 Hz, 600 samples over 59.9 s, a RouterOS script loop started over ssh and the router log added as markers That recording yielded exactly 600 samples, 0 gaps and a clock skew of −7 ms. In it, a scripted RouterOS loop (`:for … 400 000`) showed as one core's worth of load at 100 % from t = 21.0 to 25.8 s, with the sample at 25.8 s reading about 60 %, onset and offset resolved to 100 ms, and the router's log markers explained a 2 s plateau at 15–17 s that nobody had caused: the `ensure-ipv6-nd-prefix` scheduler. [Five minutes with a router](/mikroscope/start/walkthrough/) runs `record` and `plot` end to end on a different capture: 70 s at 10 Hz with the router otherwise at rest, with three notes typed into `record`'s terminal. > **Not measured, so not claimed** > > A recording above 10 Hz. The lossless 50 Hz and 100 Hz runs on [the rate > ceiling](/mikroscope/cost/rate-ceiling/) were the collector's, not `record`'s. Both use the same > batch sizing, but a recording at those rates has not been measured on its own. Nor has a recording > through the relay at any rate. ## How `record` reaches the agent `--transport auto` tries the direct path first: a `GET /healthz` to `http://<.2 of --subnet>:<--port>`. If that does not answer, it opens the RouterOS API and asks the router to fetch from the agent itself. If `--api` or `--api-user` is missing, it stops with an error that names `--api`, `--api-user` and `MIKROSCOPE_API_PASSWORD` and suggests `install --expose`; a missing password surfaces instead as a login error, `api : …`. The `--expose` hint helps other HTTP clients, not `record`: `record` always dials the `.2` of `--subnet`, and has no flag for the router's LAN address. `direct` and `relay` force one path and fail rather than fall back. - **direct** is plain HTTP from your machine to the veth. It sends `--token` as a bearer token. - **relay** runs `/tool fetch output=user` on the router over the binary API, so the API user needs the `read,api,test` policies. Each relayed call takes either about 3 ms or about 1 s. A reply is capped at 64 512 B, so a relayed pull asks for at most 18 samples, and a reply that reaches the cap is refused rather than parsed truncated. The router-side fetch carries no header: an agent installed with a token can only be recorded over the direct path. > **A full relayed pull can still reach the cap** > > By arithmetic, read from the code on 2026-09-15 and not measured. The cap of 18 samples comes from > the 64 512 B fetch limit, a mean line of 2 560 B, and 134 % headroom; > the mean is the 2 439 B measured on the RB5009 (RouterOS 7.24.2, > 2026-09-12), rounded up. A full pull of mean lines is about 46 kB. A full pull whose lines average > more than 134 % of that mean would still reach the limit and be refused, and how long lines run at > today's default per-source floors was not measured. At 18 samples a pull and the default 500 ms > `--poll` the relay carries 36 samples a second; above that, use the direct path. [Reaching the agent](/mikroscope/install/reaching-the-agent/) covers which path a network allows. The API user is described on [its own page](/mikroscope/security/api-user/). Once connected, `record` reads `/healthz`. The agent's wall clock minus your machine's is the **clock skew**, printed on stderr with the agent's version, rate, newest sequence number, the oldest one the ring still holds, and the transport. The recording then starts live from the newest sample, or from the oldest the ring holds with `--from-start`. Every `--poll` it pulls `/snapshot?since=&max=`. While a pull comes back full it asks again, up to 100 times per poll, so a ring that got ahead is drained rather than followed at a fixed pace. A pull that fails is logged as `pull: …` and the recording carries on; the next pull asks from the same sequence number. When `--for` elapses or you press Ctrl-C, one last pull drains what arrived meanwhile, and `record` prints how many samples it kept, their sequence range, the gaps, the markers, the transport and the files. ## The four files Every file is created with mode `0600`, and an existing file with the same prefix is overwritten. - **`.jsonl`** — each sample line exactly as the agent sent it: raw tick and counter deltas, with every source the agent has. This is the recording; the other files are views of it. - **`.csv`** — one wide row per sample, for a spreadsheet. The column set is sized from the first sample: one group per core and one per softnet queue. - **`.markers.csv`** — `wall_ns,wall_utc,seq,kind,label`, one row per marker. - **`.meta.json`** — written at the start, so that `mark` and `plot` can run later: `started_utc`, `skew_ns` (agent wall minus host wall), `agent` (its version), `rate_hz`, `transport`, and `capabilities` — what the agent established about the board — when the transport could fetch it. The CSV holds a fixed subset of each sample. Everything else a sample carries — kernel-log events, PMU counters, temperatures, interrupts per line and the rest — is in the `.jsonl` only. | Columns | Content | | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | `seq`, `wall_ns`, `wall_utc`, `dt_ns` | The sample's sequence number, agent wall clock, and real interval. | | `busy_total` | The mean of the per-core busy ratios, three decimals, computed by the CLI from the ticks. | | `c_busy`, `c_user`, `c_nice`, `c_system`, `c_idle`, `c_iowait`, `c_irq`, `c_softirq` | Per core: the busy ratio, then the raw tick deltas. | | `ctxt`, `intr`, `irq_total` | Context switches and interrupts from `/proc/stat`, and the delta summed over every row of `/proc/interrupts`. | | `softnet_processed`, `softnet_dropped`, `softnet_time_squeeze` | Per softnet queue, the receive path's deltas. | | `mem_free_kb`, `mem_available_kb`, `mem_cached_kb`, `mem_slab_kb` | Memory levels, in kB. | | `load1`, `threads_running`, `threads_total` | Load average and thread counts. | | `pgfault`, `pgmajfault` | Page-fault deltas. | | `self_cpu_us`, `self_rss_bytes` | The agent's own CPU time during the sample, in µs, and its resident memory at that moment, in bytes. | The busy ratio in the CSV is the one number the CLI computes, and it inherits the kernel's floor: a tick is 10 ms, so one core in a 100 ms sample resolves to 10 % steps. [The resolution floor](/mikroscope/limits/) explains why, and why the ratio is capped at 1. ## Markers, and whose clock they are in A marker is a row in `.markers.csv` of one of three kinds: - **`note`** — a line of text you added. While `record` runs in a terminal it prints `type a line and press Enter to add a marker; Ctrl-C stops`, and every non-empty line you type becomes a note. When standard input is not a terminal, nothing is read from it. - **`gap`** — written when the agent reports that samples were no longer in its ring, labelled `samples .. lost`. The same gaps are listed in the summary `record` prints at the end. - **`log`** — a line of the router's own log, added with `--log-markers` (below). Notes and gaps are stamped with your machine's clock plus the skew measured at the start, so they sit on the agent's time axis, not your machine's. `mark --out cap "text"` does the same with the skew stored in `cap.meta.json`: it stamps the current time in the agent's clock, and no flag sets another. On a finished recording the note therefore falls after the last sample, and `plot` does not draw it. `mark` needs that file and an existing `cap.markers.csv`, and its row carries `seq` 0. > **Mark while the recording runs, not after** > > `record` does not hold `.markers.csv` open: it opens the file, appends and closes it for > each marker, which is the same thing `mark --out "text"` does from a second shell. So a > marker typed in one terminal and a marker added from another land in the file in the order they > were made, on Linux, macOS and Windows alike. What a text `mark` > cannot do is mark a moment in a recording that has already stopped: it stamps the current time, > which falls after the last sample, and `plot` draws nothing there. ## The router's own log as markers The log often explains a transient you did not cause: scheduler runs, errors, interface events. `--log-markers` turns the log lines of the recording's window into `log` markers, labelled `: `. ```sh export MIKROSCOPE_API_ADDR=192.168.88.1:8728 MIKROSCOPE_API_USER=mikroscope MIKROSCOPE_API_PASSWORD=… mikroscope record --for 60s --out burst mikroscope mark --out burst --log-markers --router-tz Europe/Madrid ``` - `mark --log-markers` uses the window from `started_utc` in the meta file, shifted by its `skew_ns` into the agent's clock, to the last sample in the `.jsonl`, and prints how many lines it added, the window and the topics. `started_utc` is stored to the second. It appends without checking what is already there: running it twice adds the same lines again. - `record --log-markers` asks for the log once the recording has stopped, for the window from its start to its end in the agent's clock, and appends the lines through the same path `mark` uses, so they land in `.markers.csv` and the summary's marker total counts what the file holds. If the log request itself fails, `record` prints `log markers: …` and keeps the recording. The CLI asks the router only for the window, with a `?>time=` query starting one second before it, and only for the `time`, `topics` and `message` fields: the RB5009 held 66 217 log rows, because its `dns` topic logs to disk. It then keeps the lines inside the window whose topics include one of `--topics`. A router with several logging actions on one topic carries each event once per action, prefixed with the action's name (`[INFO]: …`, `[SYSTEM]: …`); the prefix is stripped and the duplicates are kept once. RouterOS log times carry no zone. RouterOS 7.24.2 prints a full date over the API; the parser also accepts the shorter forms without a year or without a date, and fills them in from the end of the window. They are read in `--router-tz` and stamped as the agent's wall clock — the router's own clock — so no skew is applied to them. The default `Local` is your machine's zone: if the router's differs, pass it, or every log marker lands off by the difference. Log times have one-second resolution, so a log marker places its event to the second, not to the sample. ## The chart `plot --in cap` reads `cap.jsonl`, and `cap.markers.csv` when it exists, and writes `cap.svg` (or `--svg`). It prints the file name with its sample and marker counts. The same input always produces the same bytes. The chart is 1 200 units wide, with a title (`--title`, or the prefix) and a line giving the sample count, the duration and the core count. Three panels share one time axis, in seconds since the recording started: 1. **CPU busy per core, %** — 0 to 100, one line per core, each labelled at its end. The palette has eight colours, so the first eight cores are drawn. 2. **softnet per second, all CPUs** — `dropped` and `time_squeeze`, summed over every queue and bucketed into whole seconds. Drops are drawn in red. 3. **memory available, MiB** — `MemAvailable` per sample, with the y-axis bounded to the data. Every marker is a dashed vertical line across all three panels, grey for notes and log lines and red for gaps, with a label chip above the first panel. Chips are laid out in up to six rows to keep them apart; a label longer than 40 characters is shortened. Consecutive log markers, in time order, that fall in the same second are folded into one chip, `× : `, so a chatty scheduler does not bury the chart; a note or gap between them breaks the fold, and notes and gaps are never folded. Markers outside the recording's time span are not drawn. With a single sample the SVG says `not enough samples to draw`; with none, `plot` stops with `record: no samples` and writes no SVG. The palette is the chart's own, validated for its light background: adjacent-pair colour-vision-deficiency ΔE 9.1, normal-vision ΔE 22.9. There is no dark variant. `plot` also reads a capture the agent kept on a trigger: save `GET /captures/` to a `.jsonl` file and pass it to `--in`. The capture's header line carries no sequence number and is skipped. ## Triggers during a recording When a trigger fires on the agent, a `{"trigger":{…}}` line rides in the same pull as the samples, just before the sample it fired on. The collector recognises it; `record` does not. > **A trigger line in a recording** > > `record` reads a trigger line as if it were a sample with sequence number 0. The line is written > verbatim to the `.jsonl`, where `plot` skips it, and as a row of zeros with `seq` 0 to the `.csv`; > it is counted once in the number of samples `record` reports. If it is the first line of the > recording, the CSV header is sized for zero cores. `record` does not add it as a marker. To see > what fired during a window, read `/captures` on the agent. [Triggered capture](/mikroscope/record/triggers/) explains what fires, and how to fetch the full-rate window the agent kept around it. ## See also - [Five minutes with a router](/mikroscope/start/walkthrough/): a real recording from the RB5009, its markers and its chart, step by step. - [Triggered capture](/mikroscope/record/triggers/): the samples around a condition, kept at full rate by the agent without a recording running. - [Reaching the agent](/mikroscope/install/reaching-the-agent/): which of the direct and relay paths your network allows. - [The API user](/mikroscope/security/api-user/): the RouterOS user the relay and `--log-markers` need. --- # Triggered capture How the agent keeps the full-rate samples around a condition you configured, what fires, how to fetch a capture, and why the set of captures is a sample of events rather than a census. Source: https://jmrplens.github.io/mikroscope/record/triggers/ Everything on `/metrics` is a lossy summary, and a recording only exists if someone started it before the moment. This page answers what the agent does instead: which conditions make it keep the samples around a moment at full rate, how to configure them, how to fetch what it kept, what the collector does with the news, and what a capture cannot tell you. ## What a capture is, and what it is not The one lossless thing the agent can do on its own is keep the samples that already exist, at full rate, around the moment they matter — and only the agent can, because only the agent has every sample. The ring already holds the last 300 s by default, so the seconds before a fire cost nothing to keep; the seconds after cost only the wait. It decides nothing about meaning. A condition is a comparison you configured. The field it compared and the value that tripped it travel in the capture's header, so you can see what was compared. The capture is the same raw delta lines `/snapshot` ships. Nothing is turned into a percentage or a verdict. A capture does not copy those lines. The ring stores each sample as a pre-encoded, immutable line, and a capture pins the lines it needs, so firing costs a copy of the entry headers once per fire and nothing per tick. The design estimates that at about 3 µs for a 10 s window at 10 Hz; it has not been measured on the device. Conditions are evaluated in the sampler's own loop, between reading a sample and pushing it into the ring, never in a second goroutine. There is no expression language, on purpose: a parser is a dependency and an attack surface, and an operator-writable expression on the sampler's hot path is a way to make the router slow. ## Configuring it The agent reads six variables from its container's envlist. Two of them have an `install` flag. | Agent variable | `install` flag | Default | Accepted | What it sets | | ---------------------- | -------------- | -------------------------------------------------- | ------------------------------------- | ---------------------------------------------------------------------------- | | `TRIGGERS` | `--triggers` | `softnet-drop,oom,kmsg<=3,reset,irq-err,flash-bad` | the conditions below, comma-separated | Which conditions arm a capture. | | `CAPTURE_MB` | `--capture-mb` | `4` | `0`–`256` | The budget of pinned ring bytes, in MiB. `0` turns the feature off. | | `CAPTURE_PRE_S` | none | `5` | `1`–`60` | Seconds kept before the sample that fired. | | `CAPTURE_POST_S` | none | `5` | `1`–`60` | Seconds kept after it. | | `CAPTURE_POLICY` | none | `first` | `first`, `last` | On a full budget: `first` refuses the new capture, `last` evicts the oldest. | | `TRIGGER_REFRACTORY_S` | none | `10` | `0`–`3600` | Quiet time per condition after it fires. | `install` always writes `CAPTURE_MB`, and writes `TRIGGERS` only when `--triggers` is given; without it the agent uses its default set. It writes none of the other four, so an installed agent runs with their defaults. `mikroscope plan` shows the envlist entries before anything is written. `--triggers` goes through the agent's own parser before the first connection: `Finish` hands it to `agent.ParseTriggers`, so an unknown condition, a threshold out of range, a quote or a semicolon fails the verb with exit status 2 and writes nothing. The agent parses the list again when it starts, because an envlist can be edited by hand on the router; a value it rejects there makes it refuse to run, with one line on its standard output, which RouterOS puts in its log. ## The conditions The default set is the conditions that take no threshold except `squeeze` — `softnet-drop`, `oom`, `reset`, `irq-err`, `flash-bad`, each firing when the kernel counts something it normally does not — plus `kmsg<=3`. The level conditions are not in it: their thresholds are yours to choose. | Condition | Fires when | `field` in the header | Threshold | In the default set | | -------------- | -------------------------------------------------------------------------- | ---------------------------- | ------------ | ------------------ | | `softnet-drop` | any softnet queue dropped a packet in the sample | `softnet[N].dropped` | none | yes | | `squeeze` | any softnet queue ran out of budget (`time_squeeze`) in the sample | `softnet[N].time_squeeze` | none | no | | `oom` | the kernel OOM-killed something (`vm.oom_kill` moved) | `vm.oom_kill` | none | yes | | `reset` | a counter went backwards in a way that is not a 32-bit wrap | `resets` | none | yes | | `irq-err` | the `Err` row of `/proc/interrupts` moved | `irq_err` | none | yes | | `flash-bad` | a YAFFS partition's bad-block count rose since the previous sample | `flash[].bad_blocks` | none | yes | | `kmsg<=N` | a kernel-log record at severity N or more severe (0 is emergency, 3 error) | `events.level` | `0`–`7` | `kmsg<=3` | | `busy>=X` | any core's busy ratio is at or above X | `cpu[N].busy_ratio` | `0.05`–`1` | no | | `slip>=X` | the sample's interval was at least X sampler periods | `dt_ns/period` | `1.1`–`100` | no | | `memfall>=N` | `MemAvailable` fell by N or more in one tick | `mem.MemAvailable fall (MB)` | `1`–`100000` | no | Where a condition covers several cores, queues or partitions, the header names the first one that matched. `memfall` compares `/proc/meminfo`'s kB divided by 1 024, so its N is in MiB although the field calls it MB. `kmsg<=N` needs the kernel log, which the agent can read only in a privileged container ([what privileged buys](/mikroscope/limits/privileged/)); `flash-bad` needs a YAFFS partition. A condition whose source is absent never fires. `squeeze` is available but not a default. On the reference RB5009 time squeezes are background: a rule that fires on any squeeze fires 92 times in twenty minutes there (RouterOS 7.24.2, 2026-09-15). The collector's microburst rule therefore asks for an episode of three deviations. A capture can also be armed by hand, with `POST /capture` (below). Its cause is `manual` and its `field` is the reason you gave. ## How a fire becomes a capture 1. A condition is true on sample S. If that condition fired fewer than `TRIGGER_REFRACTORY_S` × rate samples ago (that many seconds at the nominal rate), the fire is **suppressed** (reason `refractory`). If another capture is still collecting its window, the fire is **suppressed** (reason `pending`): one capture collects at a time, whatever condition armed it. 2. Otherwise a capture is armed for the window from S − `CAPTURE_PRE_S` × rate to S + `CAPTURE_POST_S` × rate, and a `{"trigger":{…}}` line is queued for the stream. 3. When the sample at the end of the window is in the ring, the capture pins the ring's lines for that window. If the ring no longer holds any of them, the capture is **refused** (`empty`). 4. If the window's bytes alone exceed the budget, it is **refused** (`budget`). If the budget is full, `first` refuses it (`budget`) and `last` evicts the oldest captures until it fits. A capture whose window holds fewer than `(pre + post) × rate + 1` samples is kept with `complete: false` rather than silently short. That happens when the ring did not hold the whole window: a fire within `CAPTURE_PRE_S` of the agent starting, or a ring (`BUFFER_S`) shorter than the window. When the agent stops, it collects a pending capture with what the ring holds, but it cannot serve it: captures are in memory, and the HTTP server stops with the agent. Captures live in the agent's memory. Nothing writes them to disk, so a restart of the container, an `upgrade` or a reboot loses the ones not yet downloaded. ### What a capture weighs A capture's size is its window's sample count times the line size. The mean line measured on the RB5009 (RouterOS 7.24.2, 10 Hz, every source of that date, 2026-09-12) was 2 439 B; lines at the default floors were not measured. That gives, by arithmetic and not by measuring captures: | Rate | Default window (5 s + 5 s) | Captures in the 4 MiB default | | ------ | --------------------------- | ----------------------------- | | 10 Hz | 101 samples, about 250 kB | 17 | | 50 Hz | 501 samples, about 1.2 MB | 3 | | 100 Hz | 1 001 samples, about 2.4 MB | 1 | A bigger board — more cores, more interrupt lines — has longer lines. The `bytes` field of each capture is the real figure. The budget is memory the agent holds beyond its ring: a pinned line stays alive after the ring has moved past it. So the agent counts it at start in the same check as the ring. When the agent can read the container's `memory.max` and about `rate × buffer × 2.56 kB` plus `CAPTURE_MB` exceeds it, the agent refuses to start, naming the three settings to lower or `--memory-max` to raise. When `MEM_LIMIT_MB` is above 0, it warns when twice that exceeds its Go soft memory limit; the agent's own default for `MEM_LIMIT_MB` is 14, and `install` writes 40. [The cost of the observer](/mikroscope/cost/) explains why that second ratio matters. ## Reading captures over HTTP Four endpoints on the agent. Each needs the bearer token when the agent has one, and each answers `404` with `captures disabled (CAPTURE_MB=0)` when the feature is off. | Request | Answer | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `GET /captures` | The index, as JSON. | | `GET /captures/` | One `{"capture":{…}}` header line, then the sample lines verbatim, as NDJSON. | | `DELETE /captures/` | Frees that capture's share of the budget; `204`. | | `POST /capture?reason=…` | Arms a manual capture at the newest sample: `{"id":N,"armed":true}`, or `409` when a capture is pending or the manual trigger is in its refractory window. The reason defaults to `operator`. | ```sh curl -s -H "Authorization: Bearer $MIKROSCOPE_TOKEN" http://172.30.10.2:9123/captures curl -s -H "Authorization: Bearer $MIKROSCOPE_TOKEN" http://172.30.10.2:9123/captures/3 > cap3.jsonl mikroscope plot --in cap3.jsonl curl -s -X DELETE -H "Authorization: Bearer $MIKROSCOPE_TOKEN" http://172.30.10.2:9123/captures/3 ``` The index carries the `policy`, `budget_bytes`, the `bytes` held, the capture still `pending` if there is one, the configured `triggers`, and one entry per capture: `id`, `cause`, `condition`, `field`, `value`, `threshold`, `fire_seq`, `fire_mono_ns`, `fire_wall_ns`, `first_seq`, `last_seq`, `samples`, `bytes` and `complete`. The sample lines of `GET /captures/` are byte-identical to what `/snapshot` serves for the same samples, so a tool that reads a snapshot needs no new parser; `plot` skips the header line. The CLI has no verb for captures: use any HTTP client. > **Captures need the direct path** > > The relay transport pulls through `/tool fetch`, which returns at most 64 512 B and sends no token. A capture at the defaults is about 250 kB. Fetch captures from a host that > reaches the agent directly, or through `--expose`: [reaching the > agent](/mikroscope/install/reaching-the-agent/) covers both. ## The trigger line, and what the collector does with it When a capture is armed, the agent places one line before the sample it fired on, in `/stream` and in `/snapshot?since=` (not in `/snapshot?seconds=`): ```text {"trigger":{"id":3,"cause":"busy>=0.95","field":"cpu[2].busy_ratio","value":1,"threshold":0.95,"seq":48213,"wall_ns":1789000000000000000}} ``` The values above are illustrative. The line is a kind of its own, like the `{"gap":…}` line, not a field on the sample, so the sample schema is unchanged. The agent keeps the last 64 of them for pullers, so a puller more than 64 fires behind never sees the older ones; a suppressed fire produces none. A manual capture fires on the newest sample already in the ring, so a puller that has already received that sample gets no trigger line for it; read `/captures` instead. `forward` recognises the line, never mistakes it for a sample, counts it, and hands it to every sink as an annotation: the `mikroscope_trigger` measurement in InfluxDB and table in SQL, `mikroscope_collector_triggers_total{cause}` on the collector's Prometheus exposition, and the line itself in the file sink. Both Grafana dashboards carry a `triggers` annotation, off by default in the toggle bar: on InfluxDB it reads the `mikroscope_trigger` rows, on Prometheus the agent's `mikroscope_trigger_fired_total`. The capture itself stays on the agent, under `/captures/`. `record` does not recognise the line yet — [Record, mark, plot](/mikroscope/record/#triggers-during-a-recording) says what it does with it. ## Counting what was not captured The agent's `/metrics` carries the families that say how much the captures did not see. Every condition and reason pair is rendered from the start, at 0 until it happens, so a dashboard can show "0 so far". | Family | Type | Meaning | | ------------------------------------------------------- | ------- | -------------------------------------------------------------------------------------------------------- | | `mikroscope_trigger_fired_total{condition}` | counter | Times each condition armed a capture; `condition="manual"` appears once a manual capture has been armed. | | `mikroscope_trigger_suppressed_total{condition,reason}` | counter | Times a condition was true and nothing was armed: `refractory` or `pending`. | | `mikroscope_capture_refused_total{reason}` | counter | Captures collected and then not kept: `budget` or `empty`. | | `mikroscope_captures_held` | gauge | Captures currently retained. | | `mikroscope_capture_bytes` | gauge | Ring bytes the retained captures pin. | | `mikroscope_capture_budget_bytes` | gauge | The budget, from `CAPTURE_MB`. | | `mikroscope_capture_bytes_served_total` | counter | Bytes handed out over `/captures/`. | The collector cannot recompute these from the samples, so the Prometheus dashboard expects a scrape job on the agent itself that keeps only the agent-only families; [Prometheus](/mikroscope/sinks/prometheus/) has the job. ## What it cannot do Stated because each of these will happen: - **The capture set is a sample of events, never a census.** The refractory window and the byte budget bound a trigger storm, and one capture collects at a time. `mikroscope_trigger_suppressed_total` and `mikroscope_capture_refused_total` are how much was not seen. - **Full rate is not full detail.** A capture holds samples at the sampler's rate: at 10 Hz nothing shorter than 100 ms is reliably visible, and a busy ratio still moves in the kernel's tick steps. [The resolution floor](/mikroscope/limits/) sets that limit, not the capture. - **A window can be short.** One the ring did not wholly hold is served with `complete: false`. One cut off by the agent stopping is collected but never served, because the captures stop with the agent. - **Downloading costs the router.** A download runs on the same core as the sampler, and is counted in `mikroscope_capture_bytes_served_total` the way any puller is charged. > **Not measured, so not claimed** > > The cost of a fire on the device — 3 µs is the design's estimate. How the agent behaves under a > sustained trigger storm. Any capture at 50 Hz or 100 Hz. The capture sizes in the table above, > which are arithmetic from the line size, not the sizes of captures taken on the RB5009. ## See also - [Record, mark, plot](/mikroscope/record/): a recording you start, with markers, and the chart `plot` draws from a capture too. - [The agent's HTTP endpoints](/mikroscope/reference/http/): `/captures`, `/stream` and `/snapshot` beside the rest. - [Prometheus metric families](/mikroscope/reference/metrics/): the trigger and capture families with every other one the agent exposes. - [The resolution floor is the kernel's](/mikroscope/limits/): what full rate can and cannot resolve. --- # The collector What `mikroscope forward` does between the agent and your stores — pull, merge, derive, fan out — and what it promises when a store is slow. Source: https://jmrplens.github.io/mikroscope/sinks/ _Where the data comes from and where it goes_ — The router runs the agent in a container that reads the shared kernel and serves it over a veth. The collector on your machine pulls that, merges the RouterOS API tier into it, derives, and writes to every sink you named. `mikroscope forward` is the collector. It pulls the kernel tier from the agent, samples the RouterOS API tier, stamps both in the agent's clock, runs the derive stage over them and writes the merged timeline to every sink you name. This page answers what one run does, how it keeps up with the agent, which clock each record carries, and what "dropped" means when a destination stops answering. ```sh mikroscope forward --prom :9124 --influx "$MIKROSCOPE_INFLUX_URL" --interfaces bridge,ether1 ``` `forward` with no sink is an error, not a silent no-op: it would read the router and throw the data away. At least one of `--file`, `--prom`, `--influx`, `--loki`, `--otlp`, `--graphite`, `--elastic`, `--sql`, `--telegraf` or `--stdout` is required, and more than one at a time is the normal arrangement. ## What one run does 1. **Ask the agent for its health.** The reply carries the agent's wall clock, its rate, its newest sequence number and its capability hash. The difference between the agent's clock and the collector's is the skew; the rate sizes the pull batch and the derive stage's trailing baselines. 2. **Hand every sink the device-info stream.** The agent's `/capabilities` — board, kernel, ceilings, cadences — goes out once as its own record. See [the device-info stream](/mikroscope/sinks/device-info/). 3. **Pull the ring every `--poll`.** The first pull starts after the agent's newest sample, so `forward` does not replay what the ring held before it started. Each pull asks for samples after the last sequence number seen. A trigger marker rides among the samples in sequence order and is forwarded as an annotation, never decoded as a sample. 4. **Derive, then fan out.** Every kernel sample goes through [the derive stage](/mikroscope/sinks/derive/), and the sample, its derived values and any [detection](/mikroscope/sinks/detections/) it raised go to every sink in the same order. 5. **Sample the API tier every `--api-every`** (1 s by default) when API credentials are configured. See [the RouterOS API tier](/mikroscope/sinks/api-tier/). 6. **Re-measure the skew every minute.** A jump of more than 50 ms is logged — a router clock step, an NTP correction — and counted as a skew jump. The same health read re-checks the capability hash. 7. **On Ctrl-C or at the end of `--for`**, pull once more, close every sink with a final flush, and print what each one did. No interpolation happens anywhere in that loop: every consumer sees the cadence each source really has. ## Pulling fast enough `--poll` (default 500 ms) is how often the ring is pulled and `--batch` how many samples one pull asks for. The default batch is twice what one poll interval produces at the agent's rate, and never under 20. A fixed 20 samples per 500 ms poll would cap a pull at 40 Hz and lose 1 − 40/50 of a 50 Hz agent's samples, which is why the batch is sized from the agent's rate. A pull is repeated while it comes back full — up to 100 times — so the cursor catches up within one poll instead of advancing one batch per poll. A short reply is the ring's edge. The relay transport caps a pull at 18 lines, and the cap is computed rather than chosen: `/tool fetch` returns 64 512 B at most, the mean ring line is taken as 2 560 B (the measured 2 439 B, rounded up), and the cap allows 134 % of that mean so a batch of above-average lines still fits — 18 lines, about 46 kB. A reply that reaches the fetch limit anyway is refused with `relay reply hit the 64512-byte fetch limit; lower the batch` rather than parsed truncated. At the default 500 ms poll that is 36 samples/s, and above that the collector falls behind. `forward` computes what the effective batch and the poll allow per second and warns at start when that is below the agent's rate — for the relay against an agent at 100 Hz, the arithmetic gives: ```text warning: at most 18 samples per pull every 500ms is 36/s, below the agent's 100 Hz; the collector will fall behind and report gaps. Raise --batch, lower --poll, or use the direct transport ``` The cap and the warning are read from the code on 2026-09-15, not re-measured against a device. A collector that falls further behind than the agent's ring (300 s by default) receives a gap line instead of the samples, and every sink records the gap. ## Whose clock each record carries | Record | Timestamp | | ----------------------------- | ----------------------------------------------------------------- | | Kernel sample, derived values | the agent's own wall clock, as the sample carries it | | Detection | the wall clock of the sample that raised it | | Trigger marker | the agent's wall clock of the fire | | API-tier sample | the collector's clock plus the measured skew | | Gap | the collector's clock when the pull that found it returned | | Device-info record | the collector's clock: board facts have no timestamp of their own | ## Which one should I use? ten destinations, capitalised by the sentence it opens: the honest answer is that most readers want one of the first two. The rest exist so that mikroscope fits what you already run rather than asking you to run something new. | If you… | Use | It carries | Dashboard | | ----------------------------------------------------------- | -------------- | ----------------------------------------------- | --------- | | want the whole thing, with the dashboards, and have nothing yet | `--influx` | every measurement, as line protocol | **yes**, generated | | already run Prometheus | `--prom` | every family, recomputed from the samples | **yes**, generated | | want to capture a window and look at it later | `--file` | the merged timeline as JSONL, nothing to install | no | | keep long-term data in PostgreSQL or TimescaleDB | `--sql` | DDL and INSERTs for `psql`, no driver | **yes**, generated | | want the kernel log and the detections where your logs are | `--loki` | **events only** — kmsg, detections, gaps | no | | already run an OpenTelemetry pipeline | `--otlp` | metrics as OTLP/HTTP | no | | already run Graphite or Elasticsearch | `--graphite`, `--elastic` | every measurement, in that product's shape | **yes**, a smaller one | | already run Telegraf | `--telegraf` | every measurement, as line protocol | no | | want to pipe it into something of your own | `--stdout` | line protocol or NDJSON on standard output | no | Nothing stops you naming several at once, and that is the normal arrangement: `--file` beside a store gives you a capture to go back to, and `--loki` beside `--influx` puts the kernel log where a log query can reach it while the numbers go to the store the dashboards read. Two of these do not carry the same thing as the rest. **Loki takes events, not metrics** — the kernel-log records, the detections and the gaps — so a Loki-only run has no CPU or memory numbers in it at all. **`--prom` is scraped, not pushed**: `forward` serves `/metrics` and Prometheus comes to it, which means the collector has to be reachable from the Prometheus host. ## The ten sinks | Flag | Destination | URL or credential from the environment | Shape | Page | | -------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------ | ----------- | ---------------------------------------------------------------------- | | `--file path.jsonl` | JSONL file | — | synchronous | [the file](/mikroscope/sinks/other/#the-file) | | `--prom :9124` | Prometheus `/metrics` on the collector host | — | in memory | [Prometheus](/mikroscope/sinks/prometheus/) | | `--influx URL` | InfluxDB 3 line protocol | `MIKROSCOPE_INFLUX_URL`, `MIKROSCOPE_INFLUX_TOKEN` | queued | [InfluxDB 3](/mikroscope/sinks/influxdb/) | | `--sql path` or `--sql -` | PostgreSQL / TimescaleDB statements, for `psql` | — | synchronous | [SQL](/mikroscope/sinks/other/#sql-for-postgresql-and-timescaledb) | | `--stdout lp` or `--stdout json` | standard output | — | queued | [stdout](/mikroscope/sinks/other/#standard-output) | | `--loki URL` | Loki push API: events, not metrics | `MIKROSCOPE_LOKI_URL`, `MIKROSCOPE_LOKI_TOKEN`, `MIKROSCOPE_LOKI_TENANT` | queued | [Loki](/mikroscope/sinks/other/#loki) | | `--otlp URL` | OTLP/HTTP metrics, JSON encoding | `MIKROSCOPE_OTLP_URL`, `MIKROSCOPE_OTLP_TOKEN` | queued | [OTLP](/mikroscope/sinks/other/#otlp) | | `--graphite host:port` | carbon plaintext over TCP | `MIKROSCOPE_GRAPHITE_ADDR` | queued | [Graphite](/mikroscope/sinks/other/#graphite) | | `--elastic URL` | Elasticsearch or OpenSearch `_bulk` | `MIKROSCOPE_ELASTIC_URL`, `MIKROSCOPE_ELASTIC_AUTH` | queued | [Elasticsearch](/mikroscope/sinks/other/#elasticsearch-and-opensearch) | | `--telegraf URL` | a Telegraf listener over HTTP, TCP or UDP | `MIKROSCOPE_TELEGRAF_URL`, `MIKROSCOPE_TELEGRAF_TOKEN` | queued | [Telegraf](/mikroscope/sinks/other/#telegraf) | Sink credentials never come from a flag: a flag is visible in `ps` and in a shell history. Each sink token is read from its `MIKROSCOPE_*` variable only. The agent's own bearer token is the exception: `forward` takes it as `--token`, default `MIKROSCOPE_TOKEN`. `--host-tag` (`MIKROSCOPE_HOST_TAG`, default `router`) puts the same host tag or label on every point in every sink. A sink that was asked for and cannot be constructed — a port already bound, a file that cannot be opened — fails the run. A sink silently absent is worse than no data, because the absence is invisible. ## A slow sink never stops the loop The collector's pull loop must never wait on a destination. Every sink that talks to a remote renders into memory and hands the bytes to a bounded queue that a goroutine of its own drains once a second: - The queue holds `--queue-seconds` (default 60) seconds' worth of a byte budget: 64 KiB per second for InfluxDB, Loki, OTLP, Elasticsearch, Telegraf and stdout, 256 KiB per second for Graphite, whose one-line-per-value format is bulkier. - Past the budget the **oldest** batch is evicted and counted; the newest is always kept, because fresh telemetry beats stale. - A failed delivery backs off 2 s, doubling to 60 s, and logs at most one line per minute. Everything else is in the counters. - Each HTTP post carries a 10 s timeout, and a dead pooled connection is an error that is retried and counted, not a silent resend. Three sinks are not queued. The file and SQL sinks write synchronously through a 64 KiB buffer, because a local file does not stall the way a remote does; a write error counts one error and one drop. The Prometheus sink updates in-memory state under a lock and serves it on scrape. > **A pipe into psql can block the collector** > > The SQL sink has no queue, so with `--sql -` feeding `| psql`, a `psql` that falls behind fills > the pipe and the next write blocks the pull loop instead of dropping. Each `INSERT` is its own > transaction, which is the realistic way for `psql` to fall behind a 10 Hz agent. Not measured. > Write to a file and apply it afterwards. ### What the counters count `forward` prints `written`, `dropped` and `errors` per sink, and the unit differs by shape: - **Queued sinks count batches.** `written` is one batch the destination accepted, `dropped` one batch evicted by the byte budget, `errors` one failed attempt — a batch that fails three times and then lands is 3 errors and 1 written. Elasticsearch adds one `dropped` per document the cluster refused inside a 200 reply. - **File, SQL and Prometheus count events**: one per sample, trigger, API read, gap, detection or device record accepted. ## What `forward` prints At start, on standard error: one `sink: ` line per sink, the API tier's settings, and the agent's version, rate, sequence number, skew, transport and effective batch. Every minute, on standard error, a running report: ```text forwarded kernel, api, gap(s), trigger(s), detection(s), last seq ; : written, dropped, errors ``` On exit, on **standard output**, the totals and one line per sink: ```text forwarded kernel samples, api samples, gap(s), skew jump(s) : written, dropped, errors ``` Two properties of that output can surprise a consumer: the exit summary goes to the same stream the `--stdout` sink writes, so `forward --stdout=lp | telegraf` ends every run with lines the consumer cannot parse; and a wrong `--token` is logged per pull as `401 Unauthorized` while the run still ends at its `--for` deadline with exit status 0 and `forwarded 0 kernel samples`. ## One dimension, one name A processor is `cpu` everywhere — InfluxDB tag, SQL column, Prometheus label, OTLP attribute — never `core`. The wire carries the kernel's own unit and names it in the field (`_khz`, `_kb`, `_ticks`, `_pages`, `_sectors`); each sink converts once, to that store's convention, and converts a value and its ceiling identically. So temperature is `celsius` beside `critical_celsius`, and block-device busy time is `io_s`. ## What has been measured On the reference RB5009 on 2026-09-12, this eight-minute run forwarded 4 800 kernel and 479 API samples with 0 gaps and 0 drops: ```sh mikroscope forward --for 8m --prom :9124 --influx … --interfaces bridge,ether1,PPPoE_DIGI --conntrack-every 10s ``` In the five rate runs of 2026-09-15 the collector wrote to three sinks at once, and every one reported 0 gaps and 0 drops at 10, 50 and 100 Hz: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once > **Not measured, so not claimed** > > Loki, OTLP, Graphite, Elasticsearch, Telegraf, SQL and stdout have been tested against local > receivers that assert the bytes each protocol accepts (development host, amd64, 2026-09-12), not > fed from the RB5009 into a running backend. The byte sizes quoted for them on [the other > sinks](/mikroscope/sinks/other/) come from test fixtures, not from a device. ## See also - [Prometheus](/mikroscope/sinks/prometheus/): the collector's `/metrics` and the two scrape jobs the dashboard expects. - [InfluxDB 3](/mikroscope/sinks/influxdb/): the write URL, the measurements and what InfluxDB 3 Core refuses. - [The RouterOS API tier](/mikroscope/sinks/api-tier/): what the collector still asks the router, and how to ask less. - [Reaching the agent](/mikroscope/install/reaching-the-agent/): the direct and relay transports the pull runs over. --- # Prometheus The collector’s own `/metrics` — every agent family recomputed from the samples, plus the API tier, the derive stage and the collector’s counters — and the two scrape jobs that go with it. Source: https://jmrplens.github.io/mikroscope/sinks/prometheus/ `--prom :9124` makes the collector serve Prometheus text on `GET /metrics` at that address. This page answers what that exposition carries, what it cannot carry and has to come from the agent instead, and how to scrape both without counting anything twice. ## Recomputed from the samples The kernel-tier families on the collector are rendered by the same code the agent runs — its cumulative counters, its busy-tick histogram and its trailing windows — fed by the samples the collector received. A deployment whose collector reaches the agent only through the relay still gets metrics that do not depend on who scrapes or when. On top of them the collector adds what only it has: the RouterOS API tier's gauges, the derive stage's values, its detection and gap counters, and the device-info families from the agent's `/capabilities`. > **Sized for 10 Hz, whatever the agent runs at** > > The collector sizes its `mikroscope_cpu_busy_ticks` histogram and its sample ring from a constant > 10 Hz, not from the connected agent, so the bucket layout does not change when it reconnects to an > agent configured differently. Read from the code, not measured, the same constant sets more than > that. The ring behind the trailing windows does follow the agent: the collector reads the rate > from its health check and sizes the ring to 60 s of it, so `window="60s"` spans a minute at any > rate. The softnet trailing mean behind `mikroscope_softnet_burst_samples_total` does not — its > weight is 1/600, a 60 s memory at 10 Hz and shorter above it, so the collector's burst baseline > tightens as the agent samples faster; the agent sizes its own from its real rate. An interrupt > line is pruned from the top-K families after 36 000 samples out of every top-K: an hour at 10 Hz, > 12 min at 50 Hz, 6 min at 100 Hz. ## Two scrape jobs The Prometheus dashboard expects two jobs: the collector, which has every family the agent has plus its own, and the agent itself, keep-relabelled to the families only the sampler can produce — its tick timing histograms, the trigger and capture counters, slipped ticks: ```yaml - job_name: "mikroscope" scrape_interval: 5s static_configs: [{ targets: [":9124"] }] - job_name: "mikroscope-agent" scrape_interval: 5s static_configs: [{ targets: ["172.30.10.2:9123"] }] metric_relabel_configs: - source_labels: [__name__] regex: "mikroscope_(tick_.*|trigger_.*|capture.*|captures_held|slipped_total)" action: keep ``` Scraping the agent without the keep list would double every counter the collector also exposes. Point Prometheus at the collector host, or at the agent directly if it can reach the veth. ### What only the agent can say The collector renders no `mikroscope_slipped_total` and no `mikroscope_tick_interval_seconds`, `mikroscope_tick_wake_latency_seconds` or `mikroscope_tick_read_seconds`: it never ran the sampler, and a 0 there would be a claim about a ticker it does not own. The agent's capture index and its `mikroscope_trigger_*` counters live on the agent as well; the collector counts the trigger markers it saw in `mikroscope_collector_triggers_total{cause}`. ## What the collector adds ### The collector's own counters | Family | Type | Carries | | --------------------------------------------- | ------- | ----------------------------------------------------------------------------------- | | `mikroscope_collector_gaps_total` | counter | ring gaps the collector saw: samples lost between pulls | | `mikroscope_collector_triggers_total{cause}` | counter | capture triggers the agent fired, per cause; present once one has been seen | | `mikroscope_collector_detections_total{rule}` | counter | detection events per rule, every one of the eleven rules at 0 from the first scrape | | `mikroscope_collector_bursts_total` | counter | samples the derive stage flagged as a sub-sample burst | Detections and bursts are counters so a Prometheus-only user learns of an event despite a missed scrape, and every rule is rendered at 0 from the start because a family that appears only after its first event cannot be read as "none so far". ### The derive stage | Family | Type | Carries | | -------------------------------------------------------- | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_derived_memory_pressure` | gauge | the allocator's escalation ladder at the newest sample, 0 to 4 | | `mikroscope_derived_cycles_per_packet` | gauge | PMU cycles per packet processed, summed over cores; absent without a PMU, in a sample with no packets, or after a counter reset | | `mikroscope_derived_instructions_per_packet` | gauge | PMU instructions per packet, same conditions | | `mikroscope_derived_cache_misses_per_packet` | gauge | PMU cache misses per packet, same conditions | | `mikroscope_derived_packets_per_interrupt` | gauge | packets per device interrupt; absent when the timer row was not in the sample's top-K | | `mikroscope_derived_fastpath_share{interface,direction}` | gauge | fast-path share of the traffic the interface hands the CPU, between the last two counter polls; not a share of the wire; `rx` only while `fp-tx-byte` has never counted | These are the newest sample's values, a level at scrape time; the full series is in the stores that keep every sample. What each one means, and when it is withheld, is on [what the collector derives](/mikroscope/sinks/derive/). ### The API tier Present only once the API tier has delivered a sample; with `--api-mode off` or without API credentials none of these families exists. | Family | Type | Carries | | ------------------------------------------------------------------------------ | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_api_up` | gauge | 1 while the API tier delivers samples | | `mikroscope_api_cpu_load` | gauge | RouterOS `cpu-load` from `/system/resource` | | `mikroscope_api_memory_bytes{kind}` | gauge | `free` and `total` from `/system/resource` | | `mikroscope_api_uptime_seconds` | gauge | RouterOS uptime | | `mikroscope_api_core_percent{cpu,kind}` | gauge | per-core `load`, `irq` and `disk` percent from `/system/resource/cpu` | | `mikroscope_api_health{name}` | gauge | each `/system/health` reading | | `mikroscope_api_interface{interface,kind}` | gauge | `rx_bps`, `tx_bps`, `rx_pps`, `tx_pps` from `monitor-traffic`, plus each loss rate the router returned | | `mikroscope_api_interface_info{interface,label,type,role,bridge,default_name}` | gauge | always 1; one series per interface from the configuration inventory: its comment, RouterOS type, interface lists, bridge and factory name | | `mikroscope_api_interface_counter_total{interface,counter}` | counter | every per-port cumulative counter the router returned, under RouterOS's own counter name | | `mikroscope_api_conntrack_entries` | gauge | the connection count, when `--conntrack-every` asks for it | `mikroscope_api_up` is never rendered as 0: before the first API sample, and when the tier is off, the family is absent. The conntrack count, the port counters and the fast-path shares arrive on slower cadences than the scrape. The collector holds the last value of each between polls, so a scrape that lands between two polls still sees the family instead of a series that blinks in and out. ### What each interface is The API tier reads what every interface is — its comment, RouterOS type, interface lists, the bridge it is a port of, its factory name and its MTU — from three configuration-only reads at collector start and again every `--labels-every` (5 min by default). On `/metrics` that inventory is one info series per interface. None of it is a label on the rate or counter series: a comment is edited by a human, and a changed label would start a fresh series for every rate and every one of the sixty-odd counters of that port on every edit. Join it in a query instead: ```text mikroscope_api_interface_counter_total * on(interface) group_left(label, type, role) mikroscope_api_interface_info ``` Every interface in the inventory gets a series, with or without a comment; an empty `label` value is how Prometheus spells "none", so the label set is the same on all of them. `type` says what the counters of that interface mean: an `ether` port in a bridge counts its wire, including the frames the switch chip forwarded in hardware, while the `bridge` counts its CPU side. Neither is a subset of the other — on the reference RB5009 (RouterOS 7.24.2, 2026-09-16) ether1 received 255.8 GB on the wire and handed 29.7 GB to the CPU — so do not sum a port and its bridge. `mikroscope_api_interface_counter_total` has one series per port and counter the router reports: 9 ports × about 60 counters on the reference RB5009. A counter a port does not report has no series, and a loss rate the router did not return has no `kind`. Keys that parse as integers but count nothing — `mtu`, `actual-mtu`, `l2mtu`, `max-l2mtu`, `sfp-shutdown-temperature` — are sizes and configuration and get no counter series; the MTU is part of the inventory. On the RB5009 with RouterOS 7.24.2 (2026-09-15) `monitor-traffic` returns `rx-drops`, `tx-drops` and `tx-queue-drops` and no error keys at all. ## Device-info families The collector's exposition carries the same device-info families as the agent's own `/metrics`, from what it fetched from `/capabilities`: `mikroscope_device_info`, the ceilings the board publishes (`mikroscope_thermal_critical_celsius`, `mikroscope_thermal_polling_seconds`, `mikroscope_cpu_frequency_limit_hertz`, `mikroscope_cpu_frequency_step_hertz`, `mikroscope_cpu_frequency_governor_info`, `mikroscope_cpu_frequency_cluster`, `mikroscope_self_cgroup_memory_max_bytes`) and `mikroscope_source_cadence_hz{source,reason}`. See [the device-info stream](/mikroscope/sinks/device-info/). > **Not measured, so not claimed** > > The collector's exposition has been scraped by a Prometheus 3.14 every 5 s with the RB5009 feeding > the collector, on 2026-09-12 and again on 2026-09-15. No other scrape interval and no other > Prometheus version is recorded. ## See also - [Prometheus metric families](/mikroscope/reference/metrics/): every family the agent and the collector render, with its labels. - [Import and check](/mikroscope/dashboards/import-and-check/): the dashboard these two scrape jobs feed, and how to check it panel by panel. - [What the collector derives](/mikroscope/sinks/derive/): what the `mikroscope_derived_*` gauges mean and when they are absent. - [The collector](/mikroscope/sinks/): what one `forward` run does before anything reaches `/metrics`. --- # InfluxDB 3 The InfluxDB 3 line-protocol sink — the write URL and token, how batches are delivered and dropped, every measurement it writes, and what InfluxDB 3 Core refuses. Source: https://jmrplens.github.io/mikroscope/sinks/influxdb/ `--influx URL` writes the merged timeline as InfluxDB line protocol to InfluxDB 3's `/api/v3/write_lp`. It is the sink that keeps every sample at the agent's rate, and the one the InfluxDB dashboard reads. This page answers how to point it at a database, what it does when the database is slow or refusing, and which measurements land there. ## The write URL and the token ```sh export MIKROSCOPE_INFLUX_URL="http://host:8181/api/v3/write_lp?db=mikroscope&precision=nanosecond" export MIKROSCOPE_INFLUX_TOKEN=… mikroscope forward --influx "$MIKROSCOPE_INFLUX_URL" --host-tag rb5009 ``` The flag reads its default from `MIKROSCOPE_INFLUX_URL`. The token is read from `MIKROSCOPE_INFLUX_TOKEN` only, never from a flag, and sent as `Authorization: Bearer `. > **Quote the URL** > > When the URL lives in a file you `source`, quote it: `&` is a shell operator, and an unquoted > `…?db=mikroscope&precision=nanosecond` is cut at the `&`. ## Delivery - **One batch per second.** Every event is rendered into the current batch as it arrives; a goroutine moves it to the queue and posts it once a second, with a 10 s timeout per post. - **A bounded queue.** `--queue-seconds` (default 60) × 64 KiB. Past it the oldest batch is dropped and counted; the newest is never dropped. - **Backoff.** A failed post backs off 2 s, doubling to 60 s, and logs at most one line per minute. A reply that is not 2xx is an error, with the first 512 bytes of the body in that line. - **No silent resend.** A request whose pooled connection turns out dead is reported as an error and retried by the sink, rather than re-sent by the HTTP client on its own. That automatic resend of a batch the server had already committed is the leading, unverified explanation for the duplicate rows of the 2026-09-13 overnight run: a 50 Hz run that wrote duplicated sequence numbers into InfluxDB. Whether the sink's own retry avoids them has not been measured. A 10 Hz kernel sample renders to about 1.2 KiB of line protocol, measured on the RB5009 (RouterOS 7.24.2, kernel 5.6.3, 2026-09-12). So one second of budget holds about 50 samples — about 5 s of a 10 Hz backlog — and the default 60 s holds about 5 minutes. Not measured above 10 Hz, and not re-measured against the current source set. Counters are in batches: `written` one batch InfluxDB accepted, `dropped` one batch evicted, `errors` one failed attempt. ## Measurements Every measurement is named `mikroscope_` and carries `host=<--host-tag>`. Kernel-tier rows are stamped with the agent's wall clock in nanoseconds; the table names the exceptions. Field lists are on [InfluxDB and SQL measurements](/mikroscope/reference/measurements/). ### The kernel tier | Measurement | Tags | Carries | | --------------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_cpu` | `cpu` | the tick deltas per mode, `busy_ratio`, and the sample's real interval `dt_ns` | | `mikroscope_cpufreq` | `cpu` | `khz`, with the core's `max_khz` on the same row where it is published | | `mikroscope_softnet` | `cpu` | `processed`, `dropped`, `time_squeeze` deltas | | `mikroscope_irq` | `irq`, `name` | the interrupt count summed over CPUs, for the top-K sources | | `mikroscope_irq_cpu` | `irq`, `name`, `cpu` | the same count per CPU, rows with a non-zero delta only | | `mikroscope_softirq` | `kind`, `cpu` | softirq deltas per vector and CPU, non-zero only | | `mikroscope_sample` | — | `seq`, `dt_ns`, `mono_ns` | | `mikroscope_stat` | — | `/proc/stat`'s deltas: `ctxt`, `intr`, `forks`, `irq_total`, `irq_err` | | `mikroscope_mem` | — | `/proc/meminfo` levels, in `_kb` | | `mikroscope_load` | — | load averages, `running`, `threads`, `procs_blocked` | | `mikroscope_vm` | — | `/proc/vmstat` counter deltas: faults, reclaim scans and steals, stalls, `oom_kill`, swap | | `mikroscope_vm_level` | — | `/proc/vmstat` levels: `nr_free_pages`, `nr_dirty`, `nr_writeback`, slab pages | | `mikroscope_buddy` | `node`, `zone` | free blocks per order and `free_pages`, on the samples the free lists changed | | `mikroscope_self` | — | the agent's own CPU, RSS, cgroup memory and `memory.max`; its cgroup events where cgroup2 was read; `resets`; `kmsg_dropped` | | `mikroscope_psi` | — | stall microseconds, only where the kernel has PSI — not on the RB5009 | | `mikroscope_thermal` | `zone` | `celsius`, with the zone's `critical_celsius` where it declares one | | `mikroscope_slab` | `cache` | active objects, with `limit` where the kernel publishes one (`nf_conntrack`) | | `mikroscope_mtd` | `device`, `partition` | flash ECC counters as read, with the partition's thresholds where published | | `mikroscope_flash` | `device` | YAFFS page writes, reads, erasures, GC; `bad_blocks` and `free_chunks` levels | | `mikroscope_disk` | `device` | block-device read and write deltas, `io_s`, `inflight` | | `mikroscope_perf` | `counter`, `cpu` | PMU count, with `enabled_ns` and `running_ns` on the same row | | `mikroscope_kmsg` | `level`, `port`, `kind`, `label`, `role` | a count of kernel-log records per level, not the text; a record naming a port is counted per level, port and kind, with its `label` and `role` where known | | `mikroscope_derived` | — | the derive stage's values beside the sample | Counters are written as unsigned deltas; levels as the absolute value the kernel reported. They are separate measurements where one source has both (`mikroscope_vm` against `mikroscope_vm_level`), because a delta is an event rate and a level is a depth, and one measurement invites a dashboard to sum a level or rate a gauge. A source the deployment cannot read writes no row, never a zero row: PSI is absent on the reference kernel, and slab, kernel-log, MTD and PMU rows need `privileged=yes`. The level sources the agent stores on change appear only on the samples that carry them; see [each source at its own floor](/mikroscope/limits/source-floors/). `running_ns` below `enabled_ns` on a `mikroscope_perf` row means that count is a multiplexed, scaled-down estimate. Kernel-log text belongs in [Loki](/mikroscope/sinks/other/#loki); what a metrics store can answer is when the router started producing warnings, on which port, and of what kind. `kind` is the classification of a record that names a port: `link-up`, `link-down`, `stp-blocking` and its siblings (`listening`, `learning`, `forwarding`, `disabled`), `own-address` — the bridge received a frame with its own MAC as source address, the layer-2 loop signature — or `other`. A record that names no port keeps the tagless shape and carries `level` alone. A normal link-up is followed by `stp-blocking`, `stp-learning` and `stp-forwarding` on its bridge port: four records, not four faults. ### The API tier, and what the collector adds | Measurement | Tags | Carries | Clock | | --------------------------- | ---------------------------------------------- | -------------------------------------------------------------------------------------------- | ------------------------- | | `mikroscope_api_system` | — | `cpu_load`, `free_memory`, `total_memory`, `free_hdd`, `uptime_s` | collector + skew | | `mikroscope_api_core` | `cpu` | RouterOS's per-core `load`, `irq`, `disk` percent | collector + skew | | `mikroscope_api_health` | `name` | each `/system/health` value | collector + skew | | `mikroscope_api_iface` | `interface`, `label`, `type`, `role`, `bridge` | `monitor-traffic` rates, and each loss rate the router returned | collector + skew | | `mikroscope_api_ifcounters` | `interface`, `label`, `type`, `role`, `bridge` | every per-port cumulative counter the router returned, `-` folded to `_` | collector + skew | | `mikroscope_api_ifinfo` | `interface`, `label`, `type`, `role`, `bridge` | `default_name` and `mtu`: what each interface is, at start and every `--labels-every` | collector + skew | | `mikroscope_api_conntrack` | — | `entries`, when `--conntrack-every` asks | collector + skew | | `mikroscope_derived_iface` | `interface` | the fast-path share of what the interface hands the CPU, beside the byte deltas it came from | collector + skew | | `mikroscope_gap` | — | `from`, `to`: the sequence range that never arrived | collector, when noticed | | `mikroscope_detection` | `rule`, `key` | `value`, `threshold`, `seq`, `message` | the sample that raised it | | `mikroscope_trigger` | `cause` | `id`, `seq`, `value`, `threshold`, `field`: the capture is on the agent | the agent, at the fire | | `mikroscope_device` | `board`, `kernel` | `cores`, `privileged`, `cgroup`, `sources`, `hash`, and the ceilings the board publishes | collector | | `mikroscope_device_thermal` | `zone` | `critical_celsius`, `polling_ms` | collector | | `mikroscope_device_cpufreq` | `cpu` | `cluster`, `min_khz`, `max_khz`, `governor`, `steps` | collector | | `mikroscope_device_cadence` | `source`, `reason` | `hz` | collector | Every interface row carries what that interface is: `label` is its RouterOS comment, `type` is RouterOS's own type (`ether`, `bridge`, `vlan`, `pppoe-out`, `wg`, `veth`, `loopback`), `role` is its interface lists sorted and comma-joined (`WAN`, `LAN,VPN` — a bridge member in no list of its own takes its bridge's lists, which is how RouterOS firewall rules match it), and `bridge` is the bridge it is a port of. Each is omitted rather than sent empty, so the series key of an interface without one stays stable, and a panel can group the wire ports (`type=ether`) apart from the bridge's CPU side (`type=bridge`), or WAN apart from LAN. A loss field is written only when the router returned that key. `mikroscope_api_ifinfo` carries those five tags for every interface the collector read, in the `--interfaces` list or not, with `default_name` (the factory name of a physical port, an empty string for a bridge, VLAN or tunnel) as a string field and `mtu` where the router reports one above zero. It is written once before the first kernel pull and again on every `--labels-every` re-read, 5 minutes by default, and it is the table a panel joins to say what an interface is. `mikroscope_api_ifcounters` carries counters only: `mtu`, `l2mtu`, `max-l2mtu` and `sfp-shutdown-temperature` parse as integers but are sizes and configuration, not counts, so they are not fields there; the MTU is in `mikroscope_api_ifinfo`. `mikroscope_derived_iface`'s `fp_rx_share` is the fast-path share of the traffic an interface hands the CPU: `fp-rx-byte` over `driver-rx-byte` on a switch port, whose `rx-byte` is the wire total, and over `rx-byte` on a software interface. It is not a share of the wire — frames the switch chip forwards in hardware are in neither number. `fp_tx_share` is withheld, and the `tx_bytes` denominator beside it written as 0, while the cumulative `fp-tx-byte` is 0, which it was on every interface of the reference RB5009 on 2026-09-16 after hundreds of GB transmitted. ## InfluxDB 3 Core refuses things Learned the hard way, against InfluxDB 3 Core: - **A node holds at most five databases.** A sixth write fails with 422 — the first real target, on 2026-09-12, answered `422: would exceed limit of 5 databases`. The sink backs off and keeps trying, so the symptom is a rising `errors` count, not a failed run. The measured runs used an InfluxDB 3 Core instance of their own. - **Every query must be time-bounded.** - **A column's type is immutable once written.** - **A column exists only once a row has carried it.** `mikroscope_kmsg` has no `kind` column until the first kernel-log record naming a port is written, and a query that filters on it before then fails at planning time — which is why the layer-2 loop alert rule's SQL form needs a store that has already held one classified port record. The Grafana datasource for InfluxDB 3 needs two secure fields, not one; see [import and check](/mikroscope/dashboards/import-and-check/). > **Not measured, so not claimed** > > Every recorded run wrote to InfluxDB 3 Core; no write to InfluxDB 2's `/api/v2/write` or to > InfluxDB 3 Enterprise is recorded. The duplicate rows of 2026-09-13 have an explanation that is > leading and unverified, and disabling the HTTP client's own resend has not been measured against a > repeat of that run. ## See also - [InfluxDB and SQL measurements](/mikroscope/reference/measurements/): every field of every measurement, with its unit. - [Import and check](/mikroscope/dashboards/import-and-check/): the InfluxDB dashboard, its datasource and the store probe. - [The collector](/mikroscope/sinks/): the queue, the backoff and the counters every queued sink shares. - [The device-info stream](/mikroscope/sinks/device-info/): what the four `mikroscope_device*` measurements hold. --- # The file and the other sinks The JSONL file, standard output, SQL, Loki, OTLP, Graphite, Elasticsearch and Telegraf — what each one carries, how it delivers, and what its protocol cannot promise. Source: https://jmrplens.github.io/mikroscope/sinks/other/ Besides [Prometheus](/mikroscope/sinks/prometheus/) and [InfluxDB 3](/mikroscope/sinks/influxdb/), `forward` writes to eight more destinations. This page answers, for each one, which parts of the timeline it receives, how it is delivered, and what the destination's protocol means it cannot tell you. The queue, backoff and counters they share are on [the collector](/mikroscope/sinks/). ## What each sink receives | Sink | Kernel samples | Kernel-log text | API tier | Derived and detections | Triggers, gaps, device | | ------------- | ------------------------- | ------------------------------- | ----------------------------------- | ------------------------------------------------ | ---------------------- | | file | verbatim agent lines | inside the sample lines | `{"api":…}` lines | `{"derived":…}`, `{"detection":…}` | all three, as lines | | stdout `json` | verbatim agent lines | inside the sample lines | `{"api":…}` lines | as the file | all three, as lines | | stdout `lp` | the InfluxDB measurements | counts per level, port and kind | the InfluxDB measurements | the InfluxDB measurements | all three | | SQL | one table per source | `mikroscope_event` rows | tables, plus `mikroscope_api_error` | `mikroscope_derived`, `mikroscope_detection` | all three | | Loki | no | one line per record | per-command errors only | detections only | all three, as lines | | OTLP | sums and gauges | no | gauges, and an error count | gauges, a detection sum | all three | | Graphite | one path per value | no | paths | paths | numeric parts only | | Elasticsearch | one document per tick | one document per record | one document per read, no errors | on the kernel document, a document per detection | all three | | Telegraf | the InfluxDB measurements | counts per level, port and kind | the InfluxDB measurements | the InfluxDB measurements | all three | Kernel-log records exist only when the container runs with `privileged=yes`; see [what privileged buys](/mikroscope/limits/privileged/). ## The file ```sh mikroscope forward --file timeline.jsonl ``` The file is opened and **truncated**, with mode `0600`. Every line is one JSON object, and its first key says what it is: - a kernel sample — the agent's own line, byte for byte, as `/snapshot` served it; - `{"trigger":…}` — the agent's capture marker, also verbatim; - `{"derived":…}` — the derive stage's values, on the line after the sample they belong to, so a reader that wants only raw samples skips that kind; - `{"detection":…}`, `{"device":…}`, `{"api":…}` and `{"gap":…}`. Writes are synchronous through a 64 KiB buffer, and the counters are in events. A write the filesystem refuses counts one error and one drop. ## Standard output ```sh mikroscope forward --stdout=lp | telegraf --config … mikroscope forward --stdout=json | jq ``` `--stdout lp` renders InfluxDB line protocol with the InfluxDB sink's own encoder, so a pipe shows exactly what `--influx` would post. `--stdout json` writes the same line kinds as the file. Any other value is refused before the run starts. Standard output can stall — a slow reader fills the pipe buffer and the write then blocks with nothing to bound it — so this sink is queued: one batch per second, each batch written in a single call of whole lines so a reader never sees a partial record, 64 KiB × `--queue-seconds` of budget, oldest dropped first. A batch that a stalled reader lets grow past the whole budget is closed early. That budget was sized for a line-protocol sample of about 1.2 KiB; it has **not** been measured for `json`, whose lines carry sources line protocol leaves out and are larger. - A broken pipe does not reach the counters: Go leaves `SIGPIPE` on standard output uncaught, so the process exits. - On shutdown the sink waits at most 3 s for the reader to accept the last batch, so a reader that stopped reading cannot hang `forward`. - `forward` prints its exit summary on standard output as well, so a pipe into `telegraf` ends every run with lines the consumer cannot parse. ## SQL for PostgreSQL and TimescaleDB ```sh mikroscope forward --sql out.sql --for 10m && psql -f out.sql mikroscope forward --sql - | psql # see the caution below ``` `--sql` writes PostgreSQL text — a DDL header, then one `INSERT` per row — to a file, or to standard output with `-`. It is driverless on purpose: speaking the PostgreSQL wire protocol needs a third-party driver, so the SQL text is the interface and `psql` owns the connection. The cost is that the sink cannot know whether a row was stored; it counts the events it wrote. The header is applyable on its own and idempotent: - `SET standard_conforming_strings = on;` — so a backslash in a kernel message can never turn into an escape and swallow the statements after it; - `CREATE TABLE IF NOT EXISTS` for every table, each with a primary key that starts with `time, host`; - with `--sql-hypertable`, a TimescaleDB `create_hypertable` call on `time` for each table, with `if_not_exists => TRUE`. Every `INSERT` ends in `ON CONFLICT DO NOTHING`, so applying the same file twice is a no-op rather than a duplicate-key abort. A row is one immutable instant of a counter delta, never a running total a later file revises. | Table | Key after `time, host` | Holds | | ---------------------------------------------------------------------------------------------------------- | -------------------------- | -------------------------------------------------------------------------------------- | | `mikroscope_cpu` | `cpu` | `user_ticks` … `steal_ticks`, `busy_ratio`, `dt_ns` | | `mikroscope_softnet` | `cpu` | `processed`, `dropped`, `time_squeeze` | | `mikroscope_irq` | `irq` | `name`, `count` summed over CPUs | | `mikroscope_mem` | — | levels: `free_kb`, `available_kb`, `cached_kb`, `slab_kb`, `sunreclaim_kb` | | `mikroscope_load` | — | levels: load averages, `running`, `threads`, `procs_blocked` | | `mikroscope_stat` | — | deltas: `ctxt`, `intr`, `forks`, `irq_total`, `irq_err`, `pgfault`, `pgmajfault` | | `mikroscope_self` | — | `cpu_us` delta; `rss`, `cgroup_mem` levels; cgroup events, NULL without cgroup2; `seq` | | `mikroscope_buddy` | `node, zone, block_order` | `free_blocks`, one row per zone and order | | `mikroscope_mtd` | `device` | `partition`, ECC counters as read, thresholds NULL where unpublished | | `mikroscope_psi` | — | stall microseconds; no row where the kernel has no PSI | | `mikroscope_thermal` | `zone` | `celsius`, `critical_celsius` | | `mikroscope_slab` | `cache` | `active_objs`, `limit_objs` (NULL for every cache but `nf_conntrack`) | | `mikroscope_disk` | `device` | read and write deltas, `io_s`; `inflight` is a level | | `mikroscope_flash` | `device` | wear deltas; `bad_blocks` and `free_chunks` are levels | | `mikroscope_event` | `kernel_seq` | one kernel-log record: `level`, `facility`, `time_usec`, `message`, and `port`, `kind` | | `mikroscope_api_system` | — | `cpu_load`, memory, `free_hdd`, `uptime_s`, `version` | | `mikroscope_api_core` | `cpu` | RouterOS's `load`, `irq`, `disk` percent | | `mikroscope_api_health` | `name` | `value` | | `mikroscope_api_iface` | `interface` | `label`, rates, and five loss columns that are NULL where the router returned no key | | `mikroscope_api_conntrack` | — | `entries`, the last value repeated at the API cadence | | `mikroscope_api_ifinfo` | `interface` | what each interface is: `default_name`, `type`, `role`, `bridge`, `label`, `mtu` | | `mikroscope_api_ifcounter` | `interface, counter` | `value`, long form, under RouterOS's own counter name | | `mikroscope_api_error` | `message` | one per API command that failed | | `mikroscope_gap` | `seq_from, seq_to` | the lost range, at the collector's clock | | `mikroscope_trigger` | `id` | `cause`, `field`, `value`, `threshold`, `seq` | | `mikroscope_derived` | — | `seq`, `mem_pressure`, `burst`, `suspect`, per-packet values NULL where not computed | | `mikroscope_derived_iface` | `interface` | the four byte deltas and the two fast-path shares | | `mikroscope_detection` | `rule, key` | `seq`, `value`, `threshold`, `message` | | `mikroscope_device`, `mikroscope_device_thermal`, `mikroscope_device_cpufreq`, `mikroscope_device_cadence` | —, `zone`, `cpu`, `source` | [the device-info stream](/mikroscope/sinks/device-info/) | Columns never need quoting: the tick columns are `user_ticks` and friends because `user` is reserved, and `block_order` because `order` is. `dt_ns` rides on `mikroscope_cpu` only, so a rate over any other delta table joins `mikroscope_cpu` on `(time, host)` for the real interval rather than assuming the nominal period. `mikroscope_api_ifinfo` holds one row per interface, written at collector start and on every `--labels-every` re-read (5 minutes by default), so a query joins it on `interface` to give any interface series a type, a role and the port's comment. `default_name` is the factory name of a physical port and empty for a bridge, VLAN or tunnel; `mtu` is RouterOS's `actual-mtu`, NULL where the router reports none. In `mikroscope_event`, `port` is the port the record names — its current RouterOS name where the API tier's inventory supplies one, the board's default name otherwise — and `kind` is what happened to it: `link-up`, `link-down`, the `stp-*` states, `own-address` (the bridge received a frame carrying its own MAC as source, the layer-2 loop signature) or `other`. Both are NULL for a record that names no port. A source the deployment cannot read emits no row. A value that was not measured is NULL, never 0: RouterOS 7.24.2 returns no interface error keys at all, and a 0 there would claim a measurement that was never made. Text is coerced where PostgreSQL would refuse it — a NUL byte is dropped, invalid UTF-8 becomes U+FFFD — and a NaN or infinite float becomes NULL. `TIMESTAMPTZ` resolves to 1 µs, so two samples closer than that would collide on the primary key; at 10 Hz they are 100 ms apart. Compared with InfluxDB the SQL sink carries fewer sources: no CPU frequency, PMU, softirq, `/proc/vmstat` level, per-CPU interrupt or kernel-log count tables, no `/proc/vmstat` counter table beyond `pgfault` and `pgmajfault` (which ride on `mikroscope_stat`, so the `pgscan_*`, `pgsteal_*`, `pgalloc`, `pgfree`, `allocstall`, `compact_stall`, `oom_kill`, `pswpin` and `pswpout` deltas are missing), no `mikroscope_sample`, and a narrower `mikroscope_mem`. It carries two that InfluxDB does not: the kernel-log text in `mikroscope_event`, and the API tier's errors. > **A pipe into psql can block the collector** > > The SQL sink is synchronous, with no queue. With `--sql -` feeding `psql`, a `psql` that falls > behind — each `INSERT` is its own transaction and commit — fills the pipe, and the next write > blocks the collector's pull loop instead of dropping. Not measured. Write a file and apply it > afterwards, or wrap the file in `BEGIN`/`COMMIT` by hand. Size, from the sink's own test fixture on 2026-09-12 — two cores, one softnet queue, one interrupt, no privileged sources — not from the device: a kernel event renders to 1 375 B of SQL and an API event to 1 138 B, so 10 Hz plus the 1 Hz API tier is about 14 KiB/s of file after a 5.6 KiB header. The same two events in line protocol are 716 B and 608 B, about 1.9× smaller, though part of that is content the SQL rows carry and line protocol did not. With the privileged sources present the kernel event grows to 2 749 B. That header is the fixture's: the header for all thirty-two tables the sink declares, computed from the schema strings rather than measured, is 7 757 B, about 7.6 KiB. ## Loki ```sh export MIKROSCOPE_LOKI_URL=http://host:3100/loki/api/v1/push mikroscope forward --loki "$MIKROSCOPE_LOKI_URL" --loki-tenant team-a ``` Loki receives the timeline's **events**, not its samples. A sample is a measurement and belongs in a metrics store; 10 Hz of numbers in a log store is a slower, larger copy of it. What arrives is what happened once, at a known moment. A bearer token comes from `MIKROSCOPE_LOKI_TOKEN`, and `--loki-tenant` (`MIKROSCOPE_LOKI_TENANT`) sets `X-Scope-OrgID` for a multi-tenant Loki. Streams carry three labels and no more — `host`, `source` and `level` — because Loki indexes labels and cardinality is a cost: | `source` | `level` | One line per | | ----------- | ----------------------------------- | --------------------------------------------------------------------------- | | `kmsg` | the record's own: `emerg` … `debug` | kernel-log record, stamped at its tick | | `gap` | `warn` | lost sequence range, stamped when the collector noticed it | | `api` | `err` | API-tier command that failed | | `detection` | `warn` | [detection](/mikroscope/sinks/detections/), at the sample that raised it | | `trigger` | `info` | capture trigger, at the fire; the capture stays on the agent | | `device` | `info` | device-info record: board, kernel, cores, privileged, cgroup, sources, hash | A kernel record's line is the message followed by logfmt pairs — `level`, `facility`, `prio`, `kseq`, `us` (the kernel's own microseconds since boot), `seq` (the sample), and, when the record names a port, `iface` (the kernel's name for it), `ros_iface` (RouterOS's name for it), `port_event` (what happened: `link-up`, `link-down`, an `stp-*` state, `own-address` or `other`), `label` (the port's comment, quoted) and `role` (its interface lists). `ros_iface` is the port's current RouterOS name where the API tier's inventory supplies one and the board's default name without it; `label` and `role` come from that inventory too, so a run with no API tier has neither. The port rides in the line, not in a label, because a stream per port multiplies the stream count for a field LogQL extracts on demand: ```text {source="kmsg"} | logfmt | ros_iface="ether2" {source="kmsg"} | logfmt | port_event="own-address" ``` Each record is stamped with its tick's wall clock, never with its own since-boot stamp, which would date it 1970 plus uptime and be refused by Loki. Records of one tick are 1 ns apart, counting up, because a Loki stream is ordered by timestamp alone and on the reference device a "blocking state" then "learning state" pair arrives inside one 100 ms tick at the same level: their order is the signal. The offset is at most 63 ns, bounded by the agent's cap of 64 records per tick. One push per second, 64 KiB × `--queue-seconds` of budget. On the RB5009 (RouterOS 7.24.2, kernel 5.6.3, 2026-09-12) the kernel log ran at 1.49 /s while the layer-2 reflection was live and 0.03 /s after it was fixed, and a rendered line measures about 150 B with its JSON envelope, so one second of budget holds hours of that traffic. Not measured during a kernel-log storm. ## OTLP ```sh mikroscope forward --otlp http://collector:4318/v1/metrics ``` `--otlp` posts OpenTelemetry metrics to an OTLP/HTTP receiver in the JSON encoding, one request per second. A bearer token comes from `MIKROSCOPE_OTLP_TOKEN`. JSON rather than protobuf is a dependency decision: protobuf would add a code generator and a runtime, and every OTLP receiver accepts `application/json` on the same endpoint. The resource carries `host.name` (the host tag) and `service.name=mikroscope`; the scope carries the collector's version. The mapping is the reason this sink is cheap to consume. mikroscope ships raw deltas, and OTLP has an exact home for them: every counter is a **Sum** with `AGGREGATION_TEMPORALITY_DELTA` and `isMonotonic=true`. For the kernel-sample counters `startTimeUnixNano` = the sample's wall clock minus its real interval, and `timeUnixNano` = its wall clock — so the receiver is told the interval each delta covers instead of guessing the nominal one. The other sums do not carry an interval: `mikroscope.api.errors`, `mikroscope.collector.gaps` and `mikroscope.collector.gap.samples` have no start time, and `mikroscope.trigger.fired` and `mikroscope.detection` have a start time equal to their time. Every level is a **Gauge**. Nothing from the kernel tier is pre-divided except `mikroscope.cpu.busy_ratio`; the `mikroscope.derived.*` gauges are the collector's [derive stage](/mikroscope/sinks/derive/). | Kind | Metric | Attributes | | ----- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | Sum | `mikroscope.cpu.ticks` | `cpu`, `mode` | | Sum | `mikroscope.context_switches`, `mikroscope.interrupts`, `mikroscope.forks`, `mikroscope.self.cpu.time` | — | | Sum | `mikroscope.self.throttled_periods`, `mikroscope.self.throttled_time`, `mikroscope.self.oom_kills` (with cgroup2 only) | — | | Sum | `mikroscope.irq.count`; `mikroscope.irq.total`, `mikroscope.irq.errors` | `irq`, `name`; — | | Sum | `mikroscope.softnet`; `mikroscope.sched` | `cpu`, `kind` | | Sum | `mikroscope.softirq`; `mikroscope.vm.events` | `kind` | | Sum | `mikroscope.psi.stalled` | `resource`, `scope` | | Sum | `mikroscope.flash`, `mikroscope.disk`; `mikroscope.disk.io_time` (ms) | `device`, `kind`; `device` | | Sum | `mikroscope.api.errors`; `mikroscope.collector.gaps`, `mikroscope.collector.gap.samples` | `tier`; — | | Sum | `mikroscope.trigger.fired`; `mikroscope.detection` | `cause`; `rule` | | Gauge | `mikroscope.cpu.busy_ratio`, `mikroscope.cpu.frequency` | `cpu` | | Gauge | `mikroscope.sample.dt`, `mikroscope.sample.seq`, `mikroscope.threads`, `mikroscope.procs_blocked` | — | | Gauge | `mikroscope.memory` (KiB), `mikroscope.self.memory`, `mikroscope.vm.pages` | `kind` | | Gauge | `mikroscope.load` | `window` | | Gauge | `mikroscope.thermal.temperature` | `zone` | | Gauge | `mikroscope.slab.objects`, `mikroscope.slab.limit` | `cache` | | Gauge | `mikroscope.memory.buddy_free_blocks` | `node`, `zone`, `order` | | Gauge | `mikroscope.mtd.ecc`; `mikroscope.mtd.bitflip_threshold`, `mikroscope.mtd.ecc_strength` | `device`, `partition`, `kind`; `device`, `partition` | | Gauge | `mikroscope.flash.blocks`; `mikroscope.disk.io_in_progress` | `device`, `kind`; `device` | | Gauge | `mikroscope.api.cpu_load`, `mikroscope.api.uptime`; `mikroscope.api.memory` | —; `kind` | | Gauge | `mikroscope.api.core`; `mikroscope.api.health` | `cpu`, `kind`; `name` | | Gauge | `mikroscope.api.interface` | `interface`, `kind`, and `label`, `type`, `role` where the inventory has them | | Gauge | `mikroscope.api.interface.counter`; `mikroscope.api.conntrack.entries` | `interface`, `counter`, and `label`, `type`, `role` where the inventory has them; — | | Gauge | `mikroscope.derived.memory_pressure`, `mikroscope.derived.cycles_per_packet`, `….instructions_per_packet`, `….cache_misses_per_packet`, `….packets_per_irq` | — | | Gauge | `mikroscope.derived.fastpath_share` | `interface`, `direction` | | Gauge | `mikroscope.device.cores`; `mikroscope.device.thermal.critical`, `mikroscope.device.thermal.polling` (s) | `board`, `kernel`, `hash`; `zone` | | Gauge | `mikroscope.device.cpu.frequency_max`, `mikroscope.device.cpu.frequency_min`; `mikroscope.device.source_cadence` | `cpu`; `source`, `reason` | The reclaim and swap members of `mikroscope.vm.events` are omitted at zero: for a delta Sum an absent point and a zero point mean the same. The per-port counters are gauges of a running total, not Sums, because the sink has no start time for a counter RouterOS keeps since boot. A gauge whose value is NaN or infinite is dropped. Kernel-log records are not emitted: their home is OTLP logs on `/v1/logs`, which this sink does not implement. An OTLP **partial success** — a 2xx whose body rejects some data points — counts as written and is logged, not retried. The rejection is deterministic (a Prometheus OTLP receiver refusing a point older than its window is the common case), so the same batch would be rejected identically. Size, from a two-core test fixture on 2026-09-12 and not from the device: one kernel sample renders to 8 132 B of OTLP JSON against 716 B of line protocol, about 11×, the price of repeating attribute keys and quoting every 64-bit integer. One second of 10 Hz samples plus one API sample is 68 868 B, so the 64 KiB-per-second budget holds about one second of backlog per second and the default 60 s about 57 batches. The four-core RB5009, with its real interrupt top-K, renders larger; not measured. ## Graphite ```sh mikroscope forward --graphite carbon:2003 --graphite-prefix mikroscope ``` `--graphite` (`MIKROSCOPE_GRAPHITE_ADDR`) writes carbon's plaintext protocol — `path value timestamp`, one line per value — over one persistent TCP connection. Graphite has no labels, so every dimension is a path node under `..`; `--graphite-prefix` defaults to `mikroscope`. In a node only ASCII letters, digits, `_`, `-` and `:` survive, every other byte becomes `_`, and an empty value becomes `none`, so a path's depth never changes. | Paths | From | | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | | `sample.{seq,dt_ns}`, `stat.{ctxt,intr,forks,procs_blocked,irq_total,irq_err}` | the sample | | `cpu..{user,nice,system,idle,iowait,irq,softirq,steal,busy_ratio,freq_khz}` | `/proc/stat`, cpufreq | | `softnet..{processed,dropped,time_squeeze}`, `irq...count`, `softirq..count` | softnet, interrupts, softirqs | | `mem.{total,free,available,cached,slab,sunreclaim,dirty,writeback}_kb`, `load.{load1,load5,load15,running,threads}` | meminfo, loadavg | | `vm.{pgfault,pgmajfault,pgscan_kswapd,pgscan_direct,pgsteal_kswapd,pgsteal_direct,allocstall,oom_kill}`, `vmg.{nr_free_pages,nr_dirty,nr_writeback}` | vmstat | | `self.{cpu_us,rss_bytes,cgroup_mem,throttled,throttled_us,oom_kill}` | the agent's own cost | | `psi.*_us`, `sched..{run_ns,wait_ns}` | only where the kernel has them | | `thermal..celsius`, `slab..{active_objs,limit_objs}`, `buddy...order_` | thermal, slab, buddyinfo | | `mtd..`, `flash..`, `disk..{reads,read_sectors,writes,write_sectors,io_s,inflight}` | flash and block devices | | `api.system.`, `api.core..{load,irq,disk}`, `api.health.`, `api.conntrack.entries` | the API tier | | `api.iface..{rx_bps,tx_bps,rx_pps,tx_pps,,fp_rx_share,fp_tx_share}`, `api.ifcounter..` | the API tier and the derive stage | | `derived.{mem_pressure,cycles_per_packet,instructions_per_packet,cache_misses_per_packet,packets_per_irq}` | the derive stage | | `trigger.`, `detection.` — the value 1 at each event | triggers and detections | | `device.{cores,conntrack_max,cgroup_mem_max}`, `device.thermal..*`, `device.cpufreq..*`, `device.cadence..hz` | the device-info stream | | `collector.gap.{samples,from,to}` | gaps, at the collector's clock | What the protocol cannot promise, stated because each one changes what a Graphite panel means: - **Whole-second timestamps.** Whisper's finest retention is one second, so at 10 Hz nine of every ten samples land on a slot that already holds a value and carbon keeps the last one written. That is a valid reading for a level (`mem`, `load`, `thermal`, `freq_khz`, `slab`, `vmg`) and an understatement for a delta: a sum over `cpu.0.user` sees about a tenth of the ticks the agent shipped. Summing each second's deltas in the sink would fix the delta paths and not the level ones; that trade has not been made. A consumer that needs every tick has the InfluxDB or file sink. - **No reply.** `written` counts batches handed to the socket, not points carbon stored. A write into a socket carbon has already closed succeeds once and fails on the next, so the sink retries once on a fresh connection; the batch that went into the closed socket is lost, one per carbon restart. - **Thermal zones by index**, not by name, because a zone's type string is not unique; the name is kept in the file and InfluxDB sinks. - **Less detail**: per-CPU interrupt counts are dropped, softirqs are summed over CPUs, `cpu.total` is not emitted (`sumSeries` over `cpu.*.user` gives it), `pgalloc`, `pgfree` and the swap counters are omitted, and kernel-log records are dropped because Graphite stores numbers only. Board, kernel and governor strings have no Graphite form. The byte budget is 256 KiB per queued second, four times the others: a four-core RB5009-shaped tick with every source present renders to 6 411 B in 131 lines in the sink's test fixture (development host, 2026-09-12, not the device), so 10 Hz is about 63 KiB/s. Not measured above 10 Hz. ## Elasticsearch and OpenSearch ```sh export MIKROSCOPE_ELASTIC_AUTH=elastic:… # or an API key mikroscope forward --elastic http://opensearch:9200 --elastic-index 'mikroscope-%Y.%m.%d' ``` `--elastic` (`MIKROSCOPE_ELASTIC_URL`) writes through the bulk API both products share. `/_bulk` is appended to a cluster root, keeping any query string. `MIKROSCOPE_ELASTIC_AUTH` holding `user:password` is sent as basic auth, anything else as `Authorization: ApiKey`; credentials embedded in the URL are redacted from the sink's printed name. One document per event, with `kind` telling them apart: - `kernel` — one agent tick: per-CPU ticks with `busy` and `busy_ratio`, `stat`, `mem`, `load`, `vm`, `self`, each optional source only when it was read (no key, never a zero), and the derive stage's values under `derived`; - `event` — one kernel-log record: `priority`, `level`, `facility`, `seq`, `time_usec`, `message`, and, when the record names a port, `iface`, `ros_iface`, `port_event` (what happened to the port), `label` and `role`; - `api` — one API read: `system`, `cores`, `health`, `ifaces`, `iface_counters`, the last conntrack count, the fast-path shares under `fastpath`, and `inventory` on the rounds that read what each interface is. Each `ifaces` entry carries `label`, `type`, `role` and `bridge`, and each `iface_counters` entry `comment`, `type`, `role` and `bridge`, where the inventory has them. The API tier's per-command errors are not written; - `gap` (`from`, `to`, `lost`), `device`, `detection` and `trigger`. Every document carries `@timestamp` from the agent's clock (the collector's for gaps and device records) and `host`. The index name expands `%Y`, `%m` and `%d` — only those — against that timestamp, so a batch queued across midnight lands in the day it was sampled, and is lower-cased because the cluster refuses an upper-case index name for the whole request. The action is `index` with an `_id` built from kind, host, the document's timestamp in nanoseconds and, where one exists, the sequence number that tells documents at the same instant apart, so a batch the cluster applied but whose response was lost is re-sent without duplicating anything. The timestamp is part of the identity because the agent's sequence starts again from 1 on every launch: without it a restarted agent's samples would overwrite the day's earlier ones, each with a 201. A bulk request answers 200 even when every item in it was refused. The sink reads the per-item verdicts, adds one to `dropped` for each refused document, and logs the first reason once a minute — separately from delivery failures, because a mapping conflict and an unreachable cluster call for different actions. Batches close once a second or at 1 MiB, whichever comes first. Size, from the sink's two-core fixture on 2026-09-12 and not from the device: 1 077 B of NDJSON for a kernel sample with no optional source, 1 952 B across two documents with the optional sources and one kernel-log record. A four-core sample from the RB5009 has not been rendered in this format. ## Telegraf ```sh mikroscope forward --telegraf http://host:8186/telegraf mikroscope forward --telegraf tcp://host:8094 ``` `--telegraf` (`MIKROSCOPE_TELEGRAF_URL`) ships the same line-protocol records as the InfluxDB sink — it calls that encoder rather than copying it — so Telegraf's own outputs can fan them out to destinations this repository has no sink for. Telegraf passes a timestamp through unchanged. The endpoint's scheme picks the transport: - `http://` or `https://` — post to an `http_listener_v2` or `influxdb_v2_listener` input. A bare `host:port` is read as HTTP, and an HTTP endpoint with no path is given `/telegraf`, `http_listener_v2`'s default: a listener answers 404 on `/` and the body does not say why. `MIKROSCOPE_TELEGRAF_TOKEN` holding `user:password` is sent as basic auth, anything else as `Authorization: Token …`. - `tcp://` — newline-delimited records to a `socket_listener`, a fresh connection per batch. - `udp://` — datagrams of at most 1 432 bytes, cut at record boundaries so no datagram carries half a line. There is no acknowledgement: `written` counts batches the local kernel accepted, a datagram lost in flight is invisible, and a retry after a mid-batch failure can deliver some records twice. Prefer `http://` or `tcp://` for anything that matters. The budget is the InfluxDB sink's 64 KiB per queued second: at about 1.2 KiB per 10 Hz sample (RB5009, 2026-09-12), about 5 minutes of backlog at the default 60 s. The shared encoder writes the `/system/health` readings in Go map order, so records within a batch are not ordered stably — 12 renders of an 8-name map gave 7 orders (2026-09-12). Every record carries its own timestamp, so nothing is lost or mis-timed. > **Not measured, so not claimed** > > Apart from the file sink, which was one of the three sinks in the 2026-09-15 rate runs, none of > these sinks was part of the measured rate runs, and none has been fed from the RB5009 into a > running Loki, OTLP receiver, carbon, Elasticsearch, OpenSearch, Telegraf or TimescaleDB in a > recorded run. Each of those has been tested against a local receiver that asserts the bytes its > protocol accepts (development host, amd64, 2026-09-12). ## See also - [The collector](/mikroscope/sinks/): the queue, backoff and counters these sinks share. - [InfluxDB 3](/mikroscope/sinks/influxdb/): the measurements `--stdout lp` and `--telegraf` carry. - [Detections](/mikroscope/sinks/detections/): what the detection lines, rows and documents say. - [Environment variables](/mikroscope/reference/environment/): every `MIKROSCOPE_*` a sink reads. --- # The RouterOS API tier What the collector still asks RouterOS over its binary API, why most of it is optional, and the flags and presets that decide how much to ask. Source: https://jmrplens.github.io/mikroscope/sinks/api-tier/ Beside the kernel tier it pulls from the agent, `forward` can hold one persistent session to the RouterOS binary API for the API tier (a second one when the kernel tier itself comes through the relay) and read what the container cannot see. This page answers what that tier reads, which of it the agent already covers, how to choose how much of it to run, and what a missing value means. ## What the agent already reads, and what it cannot Most of what the API tier can fetch, the agent reads itself: - `/system/resource` and `/system/resource/cpu` are RouterOS's one-second average of the same `/proc/stat` jiffies the agent differences at 10 Hz. - `/system/health`'s temperature is `/sys/class/thermal`, readable from the container. - The conntrack count is the global `nf_conntrack` slab cache, which the agent reads under `privileged=yes`; see [conntrack without the API](/mikroscope/playbooks/conntrack/). What is left is **per-interface bytes and packets**. They live in the router's network namespace and stay out of reach whatever the container is given; see [the router's CPU, the container's network](/mikroscope/limits/namespaces/). ## What `cpu-load` actually averages `/system/resource` reports `cpu-load` as an integer percent, and this page calls it a one-second average. That is measured, not assumed: the API series was correlated against the agent's own per-core busy ratio, which is the same `/proc/stat` jiffies read at 10 Hz, over two separate hours. The best fit is a **1.0 s trailing mean with 0.6 s of delay, r = 0.9825** over 3 499 API samples, and **1.1 s with 0.1 s of delay, r = 0.9734** over 3 594 in the second hour. Widening the window only makes the fit worse — 1.5 s gives 0.955, 2 s gives 0.919, 5 s gives 0.822, 8 s gives 0.791. A sixty-second average is ruled out twice over. Its correlation is 0.238, and the step response has no ramp in it: at the sharpest load step in the day the kernel went from 5 % to 27 % in one second and `cpu-load` went from 5 to 26 in that same second, then from 22 % to 6 % on the way down just as fast. A one-minute average would have needed a minute to travel either way. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-15 · `/system/resource` polled at 1 Hz against the agent's per-core busy ratio at 10 Hz, two separate hours So the number is what it claims to be. What it still cannot do is resolve anything shorter than its own second, which is the whole reason this project reads the kernel. ## Choosing how much to ask `--api-mode` picks a preset. It is `full` unless you say otherwise, and an explicit `--api-every`, `--no-health` or `--conntrack-every` still wins over it. | Mode | What it does | Use it when | | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | `off` | no API tier: `--api-every 0` | production, when per-interface traffic already comes from somewhere else | | `slow` | one round every 10 s, without `/system/health` and without the conntrack count; `/system/resource`, `monitor-traffic` and the port counters still run on each round | production, when you want interface rates too | | `full` | one round every second, with `/system/health`; port counters every 10 s; the conntrack count only if `--conntrack-every` asks for it | experiments and `record` runs | `off` is the mode that gives per-interface traffic up; `slow` is the one that keeps it cheaply. Two dashboard panels are blank by configuration under `slow` rather than by device — `/system/health` and the RouterOS connection count — and they name the flag that fills them. > **The off preset stops the API tier, not every API session** > > The relay transport pulls the agent's ring through `/tool fetch` over the same binary API. With > `--transport auto`, a collector that cannot reach the agent directly falls back to the relay and > opens an API session for it whatever `--api-mode` says. `--transport direct` never does. ## What one round asks Every API-tier command runs on the tier's one session, one at a time, with a 15 s timeout each. The slower cadences are checked on each round, so none of them runs more often than `--api-every`. | Command | Asks for | Runs | Off when | | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------- | | `/system/resource/print` | `cpu-load`, `free-memory`, `total-memory`, `free-hdd-space`, `uptime`, `version` | every round | the tier is off | | `/system/resource/cpu/print` | `cpu`, `load`, `irq`, `disk` | every round | the tier is off | | `/system/health/print` | `name`, `value` | every round | `--no-health`, or `slow` | | `/interface/monitor-traffic` with `interface=` and `once` | the four traffic rates and whichever loss rates the router returns | every round, one call for all interfaces | `--interfaces` is empty | | `/interface/print` with `.proplist=name,default-name,type,comment,actual-mtu` | what each interface is: its current name, the board's name for it, its type, its comment and its MTU | at start, before the first kernel pull, then every `--labels-every` (5 min) | the tier is off | | `/interface/list/member/print` with `.proplist=list,interface` | which interface lists name each interface, which is its role | with the read above | the tier is off | | `/interface/bridge/port/print` with `.proplist=interface,bridge` | which bridge each port belongs to | with the read above | the tier is off | | `/interface/ethernet/print stats` and `/interface/print stats-detail` | every numeric counter of **every** interface | every `--counters-every` (10 s) | `--counters-every 0` | | `/ip/firewall/connection/print count-only` | the connection count | every `--conntrack-every` | `--conntrack-every 0`, the default | A command that fails leaves its part of the sample empty and records why; the rest of the round still stands. The failures are logged on standard error as `api tier: …`, written as lines to Loki and rows to SQL, and counted on OTLP — never turned into a value. An `/interface` read that fails keeps the inventory already held — a transient error does not blank every panel's label — and is reported as `inventory: …`; the list and bridge reads are best effort, and without them the inventory still carries names, types and comments. The API-tier sample is stamped with the collector's clock plus the skew measured against the agent, so it lands on the agent's timeline. ## What each interface is The three inventory reads answer what the counters cannot: what the numbers belong to. Per interface they give its current name, the board's default name (the factory `ether5` of a physical port, empty for a bridge, a VLAN or a tunnel), RouterOS's own type (`ether`, `bridge`, `vlan`, `pppoe-out`, `wg`, `veth`, `loopback`), the comment, the interface lists it belongs to — sorted and comma-joined, `WAN` or `LAN,VPN` — the bridge it is a port of, and the MTU. A bridge member that is in no list of its own takes its bridge's lists, because that is how a RouterOS firewall rule matches it, and a bridge port that names an interface list rather than an interface is not labelled. On the reference RB5009 (RouterOS 7.24.2, 2026-09-16) the three reads return 17 interfaces. `ether1` is an `ether` in the `LAN` list, a port of `bridge`, labelled "TrueNAS - High Performance Storage", MTU 9000; `ether5` is an `ether` in `WAN` and in no bridge, labelled "DIGI ONT"; `PPPoE_DIGI` is `pppoe-out`, `WAN`, MTU 1480; `VLAN_DIGI` is `vlan`, `WAN`; `wg_devices` and `wg_trastero` are `wg` in `LAN,VPN`; `ether6` and `ether7` carry the comment "Unused". This is configuration, not telemetry, so it is read once at start — before the first kernel pull, so a kernel-log record is labelled from the first line — and then on the slow `--labels-every` cadence, never per poll. An edited comment, or a port moved between lists, reaches the dashboard within minutes rather than at the next collector restart, and a comment removed in RouterOS disappears here too: the read replaces the inventory wholesale. None of the five properties it asks for can carry a secret. Every `monitor-traffic` rate row and every per-port counter row then carries `label` (the comment), `type`, `role` and `bridge`, so a panel says what is plugged in rather than a port number, and says which numbers may be compared. Prometheus is the exception by design: a comment is edited by a human, and a changing label would spawn a new series for every edit, so the collector's `/metrics` carries one info series per interface — `mikroscope_api_interface_info{interface,label,type,role,bridge,default_name} 1`, for every interface whether or not it has a comment — and a query joins it: ```text mikroscope_api_interface_counter_total * on(interface) group_left(label, type, role) mikroscope_api_interface_info ``` The inventory also gives the kernel log its names. A kernel record names a port as `eth1`, the board's port table maps that to the RouterOS default name, and the inventory maps the default name to the current one, so an operator who renamed `ether5` to `WAN` reads `WAN` on the panel, with the port's label and role beside it. Without the API tier a kernel record keeps the board's default name and gets no label. ## Flags | Flag | Default | Meaning | | ------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------- | | `--api host:port` | `MIKROSCOPE_API_ADDR` | the RouterOS binary API, e.g. `192.168.88.1:8728` | | `--api-user` | `MIKROSCOPE_API_USER` | the API user; its password only from `MIKROSCOPE_API_PASSWORD` | | `--api-mode` | `full` | `off`, `slow` or `full`, as above | | `--api-every` | `1s` | the round's cadence; `0` disables the tier | | `--interfaces` | `MIKROSCOPE_INTERFACES`, empty | comma-separated interfaces for `monitor-traffic`, e.g. `bridge,ether1` | | `--counters-every` | `10s` | how often to read every port's cumulative counters; `0` never | | `--labels-every` | `5m` | how often to re-read what each interface is — comment, type, interface lists, bridge; `0` means the default | | `--conntrack-every` | `0` | how often to ask the connection count; `0` never, because it is a table scan | | `--no-health` | off | skip `/system/health` | Without `--api`, `--api-user` and `MIKROSCOPE_API_PASSWORD`, or when the session cannot be opened within 10 s, `forward` logs `api tier disabled: …` and runs the kernel tier alone. That is a warning, not a failure: the kernel tier is the point. ## The user it needs In any mode the API tier's commands are reads: a user with the `read` and `api` policies is enough. `test` is needed only for `/tool fetch`, which the relay transport and nothing else in the API tier uses. The group, the address restriction and why the credential never lives on the router are on [the API user](/mikroscope/security/api-user/). ## A value that is absent is absent The API tier keeps exactly what the router returned and invents nothing for what it did not. **Loss rates.** `monitor-traffic` may return `rx-drops`, `tx-drops`, `tx-queue-drops`, `rx-errors` and `tx-errors` per second, and a sink writes each one only when it came back. On the RB5009 with RouterOS 7.24.2 (2026-09-15) it returns the three drop rates and **no error keys at all**. **Port counters.** The per-port counters are a map of RouterOS's own field names, not a fixed set of fields, and only plain unsigned integers that count something are kept. `mtu`, `actual-mtu`, `l2mtu`, `max-l2mtu` and `sfp-shutdown-temperature` parse as integers but are sizes and configuration, and every sink renders this map as a counter family, so they are dropped rather than left for each consumer to know about; the MTU travels in the inventory instead. The two commands are merged per interface: `/interface/ethernet/print stats` brings the MAC's typed errors, the collision family, the frame-size buckets and the driver counters; `/interface/print stats-detail` brings the fast-path `fp-*` counters, `link-downs`, `tx-queue-drop` and the kernel-side totals. A board without collision counters produces no collision entries rather than a row of zeroes that reads as "no collisions". The reason is a measurement. On 2026-09-15 the reference router's `ether1` had 652 364 `rx-overflow` events, growing, and `monitor-traffic` returns no error key for that port at all: that count reaches a consumer only through the port counters. What those overflows turned out to be is [below](#what-rx-overflow-on-the-nas-port-is). The counters cover **every** interface the router lists, not only `--interfaces`: the port a fault lives on is often one nobody thought to monitor, and the layer-2 loop of 2026-09-12 was on a port absent from the `monitor-traffic` list. They are cumulative since boot or the port's last reset, so a consumer differences them. The derivation they enable, measured on `ether1` (2.5 GbE to a NAS, 2026-09-16, since the port's last counter reset): 255.8 GB of `rx-bytes` on the wire, of which 29.7 GB of `driver-rx-byte` reached the CPU — the rest the switch chip forwarded in hardware, and no counter inside the container has a number for it. The collector turns the fast-path counters beside them into [a share of the traffic each interface hands the CPU](/mikroscope/sinks/derive/#beside-every-counter-poll). **A switch port and a bridge count different things**, which is why the type travels with every row. An `ether` in a bridge counts its wire, including the frames the switch chip forwarded without the CPU; the `bridge` counts its own CPU side; a VLAN or a PPPoE link counts what the CPU sent and received. `ether1` and `bridge` are two planes, neither a subset of the other: drawn side by side without their type they read as peers, and summed they double-count. Never add them. **Interface labels.** What each row is labelled with — the comment, the type, the role and the bridge — is the inventory above, and an interface the inventory does not list carries none of them rather than an invented blank identity. ## What `rx-overflow` on the NAS port is The port counters are the only place the reference router's `ether1` overflows appear, and over fifteen hours they say what kind of event they are. Measured 2026-09-15 between 07:13 and 22:20 UTC, over 4 471 consecutive 10 s counter polls of `ether1` (2.5 Gbps to a NAS, MTU 9000): - 126 443 `rx-overflow` events in all, present in 40 % of the intervals; per interval the median is 29, the p99 about 1 036 and the largest 3 747. Over the run that is 0.53 % of the packets the NAS sent. - Rank correlation over the 10 s deltas: 0.85 against the part of the NAS's receive the switch forwarded in hardware (`rx-bytes` minus `driver-rx-byte`), 0.00 against the part it sent to the CPU (`driver-rx-byte`). - Where it was going: `ether8` (NGINX, 1 Gbps) carries most of the volume, `ether4` (Mastodon, 1 Gbps) is the most frequent destination; the SFP+ cage, `ether2`, `ether3` and the CPU path show nothing. - The frames were large: the 1024-and-up frame-size bucket on `ether1` has a median of 9 331 per interval with overflow against 1 336 per interval without. - The load was not high: the median NAS receive in an interval with overflow is about 9 Mbit/s as a 10 s mean. Bursts, not sustained load. - Nothing on the CPU side: softnet dropped 0, `time_squeeze` correlates 0.04 and the `switch0` interrupts 0.05 against the overflows, with the agent's 10 Hz data binned to 10 s. No pause frames on `ether1` in either direction. Read together that is consistent with 2.5 Gbps line-rate bursts switched inside the chip toward 1 Gbps ports with no flow control in effect. Not verified: the switch chip's exact counter semantics, and the NAS's own retransmit count — the counters are the port's, not the conversation's. The kernel tier cannot see any of it by construction. A frame the switch chip forwards in hardware never reaches the CPU, so no `/proc` file on the router has a number for it; it takes the per-port counters, and only the API has those. ## What the tier costs the router Measured on the reference RB5009 (RouterOS 7.24.2, 2026-09-16) with `/tool profile duration=60s cpu=total`, once with the collector stopped and once with it running `--interfaces bridge,ether1,PPPoE_DIGI --counters-every 10s --api-every 1s`. The first five one-second snapshots of each profile are dropped: they hold the SSH connect that asked for it. | Profile row | Collector stopped | Collector running | | ---------------- | ----------------- | ----------------- | | total | 5.93 % | 6.04 % | | `interface-mgmt` | 0.40 % | 0.87 % | | `config-db` | about 0 % | 0.15 % | The total moves by 0.11 points, which is inside the traffic noise of a minute; the two rows that answer the tier's questions move by about half a point between them. No `api` process row appears in either profile: the API process relays, and the work lands on the subsystem that answers. So a tier running a round every second costs the router about 0.5 % of its total CPU. That is small, and the project still treats the API as the costly path. Per-port data comes from the container wherever the container can see it, and configuration is read at start and on the slow labels cadence, never per poll. ## The conntrack count `--conntrack-every 10s` asks `/ip/firewall/connection/print count-only` at that cadence: 1.3 ms at 6 212 entries on the RB5009 (date not recorded). It is off by default because it is a table scan over an API session, and under `privileged=yes` the agent's `nf_conntrack` slab count is the same population read from a file at the sampler's rate. The two do not match exactly — they are sampled at different instants, and the slab counts objects the allocator still holds — but they track: the API said 6 212 the day before the slab said 6 287. > **Not measured, so not claimed** > > The API tier has run against one RouterOS version, 7.24.2, on one board. Which loss keys and which > counters another version or another board returns is that router's statement to make; the sinks > carry whatever comes back and nothing else. Nor does the correlation above say whether RouterOS > computes `cpu-load`'s one-second window on a wall clock or on jiffies. ## See also - [The API user](/mikroscope/security/api-user/): the RouterOS group and address restriction the tier's user needs. - [The collector](/mikroscope/sinks/): where the API-tier sample joins the kernel timeline. - [RouterOS ports and kernel names](/mikroscope/reference/port-names/): matching the API's `ether2` to the kernel's `eth1`. - [Conntrack without the API](/mikroscope/playbooks/conntrack/): reading the connection count from the slab instead. --- # What the collector derives The values the collector computes beside the raw samples — memory pressure, per-packet PMU cost, packets per interrupt, the burst flag and the fast-path share — and when each one is withheld. Source: https://jmrplens.github.io/mikroscope/sinks/derive/ The collector has a derive stage. This page answers what it computes, from which inputs, when a value is withheld rather than written, and where each sink puts it. The discrete events the same stage raises are on [detections](/mikroscope/sinks/detections/). ## Why the collector, and not the agent or the dashboard The derive stage exists for the derivations that need state across samples, that join the kernel tier to the API tier, or that must be computed once so every sink that carries them carries the same numbers (which sinks have a place for them is in the table below) — none of which a dashboard query does, and none of which the agent should pay for. The agent ships raw tick deltas and never divides; the collector may. Nothing in this stage runs on the router. Two rules govern it: - **A derived value is written beside its inputs, never instead of them**, so the store can recompute it if the derivation is later found wrong. - **A detection is a discrete event** on the timeline, a "look here" — never a continuous series and never a verdict. ## Beside every kernel sample ### `mem_pressure` The allocator's own escalation ladder as one ordinal. Every input is already shipped and plotted on its own; what the ordinal adds is that the ladder is ordered, so one series says how bad it got. The highest rung reached in the sample's `/proc/vmstat` deltas wins: | Value | Rung | From the sample's `/proc/vmstat` deltas | | ----- | ------------------------------------------- | --------------------------------------- | | 0 | none | none of the below | | 1 | kswapd scanned | `pgscan_kswapd` > 0 | | 2 | direct reclaim: a thread scanned itself | `pgscan_direct` > 0 | | 3 | an allocation stalled or a page swapped out | `allocstall` > 0 or `pswpout` > 0 | | 4 | the OOM killer ran | `oom_kill` > 0 | ### The per-packet PMU ratios `cycles_per_packet`, `instructions_per_packet` and `cache_misses_per_packet`: PMU counts summed over cores, divided by the packets softnet processed in the sample, summed over cores: the forwarding cost of the router in the one unit that lets two configurations be compared. A ruleset change that halves cycles per packet is a real win; one that halves busy time while traffic also halved is not. Absent without a PMU, in a sample with no packets, and in a sample with a counter reset (`suspect`), where a delta is a lower bound and not a measurement. ### `packets_per_irq` Packets processed per device interrupt — every `/proc/interrupts` row but the timer and the inter-processor interrupts — which is the NAPI coalescing depth. The device count is the sample's total interrupts minus the timer rows and the `IPI` rows present in its top-K. Absent in a sample with no packets or with a counter reset (`suspect`), and when the timer row is not in the sample's top-K, because then the device count cannot be separated from the total. ### `burst` A dropped packet, or more softnet squeezes than that CPU usually has — above its trailing 90th percentile **and** at least 3 — in a sample whose packet count was at or below its trailing median. It is the kernel's own evidence of a burst shorter than the sample interval, which is the only way this tool can see inside one. The flag is true when any CPU qualifies. The trailing baselines span ten seconds of wall clock at any rate: 100 samples at 10 Hz, 500 at 50 Hz, 1 000 at 100 Hz, bounded between 10 and 2 000 samples. No sample is flagged until its CPU has at least ten samples of history. Why a percentile and not "any squeeze", measured on the reference RB5009 over 3 476 samples on 2026-09-15: about 11.2 % of samples carry one squeeze as background and 2 % carry two or more. "Any squeeze" would flag the device's norm several times a minute. Three flagged samples on one CPU within 60 s is what raises the `microburst` [detection](/mikroscope/sinks/detections/#microburst); one flagged sample stays a data point. ### `suspect` True when the agent reported a counter reset in the sample: a counter that went backwards without a 32-bit wrap. The raw row is kept; the per-packet values above — the three PMU ratios and `packets_per_irq` — are withheld. ## Beside every counter poll On each API round that carried the per-port counters, the collector computes, over the deltas since the previous poll, the fast-path share of the traffic each interface hands the CPU: of the bytes that reached the CPU on that interface, the part RouterOS counted through the fast path rather than the slow path. The denominator is not the same counter on every interface, because RouterOS does not count the same thing on every type — the first counter present in both polls wins: | Interface | `fp_rx_share` is | Why that denominator | | --------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | a switch port (`ether…`, `sfp-sfpplus1`) | Δ`fp-rx-byte` / Δ`driver-rx-byte` | its `rx-byte` is the wire total, including frames the switch chip forwarded in hardware; `driver-rx-byte` is what reached the CPU | | a software interface (bridge, VLAN, PPPoE) | Δ`fp-rx-byte` / Δ`rx-byte` (or `rx-bytes`) | it has no driver counters, and its `rx-byte` is already what the CPU sent and received | `fp_tx_share` is the same quotient in the other direction — Δ`fp-tx-byte` over Δ`driver-tx-byte`, Δ`tx-byte` or Δ`tx-bytes` — and is withheld on the reference device, below. Each share is capped at 1, with the four byte deltas written on the same row; `rx_bytes` and `tx_bytes` there are the share's denominators, not the wire totals. **It is not a share of the wire.** A frame the switch chip forwarded in hardware is in neither number: it never reached the CPU. The CPU's share of a port's wire bytes — `driver-rx-byte` against `rx-byte` — is a different question, and the stacked "Where a port's receive bytes went" dashboard panel answers it from the raw counters; the derive stage does not compute it. Measured on the reference RB5009 (RouterOS 7.24.2, 2026-09-16) from the counters accumulated since boot: every switch port reads about 100 %, because `fp-rx-byte` there equals `driver-rx-byte` to within a few kB — every byte a port hands the CPU is counted at the fast-path-capable driver, so a port's line says little. The software interfaces are the lines the share informs on: `bridge` fast-pathed 211.9 GB of 663.0 GB (32 %), `PPPoE_DIGI` 99.97 %. **The tx share is withheld while `fp-tx-byte` has never counted.** On that router `fp-tx-byte` stands at 0 on every interface after hundreds of GB transmitted, which reads as a counter RouterOS does not maintain here rather than as a fast path that forwarded nothing; a share computed from it would be a fabricated 0 %. While an interface's cumulative `fp-tx-byte` is 0, the collector writes no `fp_tx_share` for it, and its `tx_bytes` and `fp_tx_bytes` are 0. A share is also absent for a direction that moved no bytes, whose counter went backwards, or whose fast-path counter the port does not report, and the whole row is absent when neither direction's denominator could be differenced. The first poll of each interface seeds and writes nothing. A delta that could not be computed — its counter absent, gone backwards, or withheld with the tx share — is written as 0 beside the absent share: the InfluxDB and SQL rows and the Elasticsearch `fastpath` object carry `rx_bytes`, `fp_rx_bytes`, `tx_bytes` and `fp_tx_bytes` unconditionally. Read from the code (`internal/derive/derive.go`), not observed. In that case read the share, not the delta. The share is per poll, and on an interface moving a few packets it swings: the dashboard walk of 2026-09-15 saw it go 0–100 % between polls on such interfaces, so the panel "Fast-path share of the traffic each interface hands the CPU" weights it by bytes per bin — the bin's fast-path bytes over the bin's bytes — rather than plotting the per-poll value. The Prometheus form of that panel is the collector's per-poll gauge and keeps the noise. ## Where each sink puts it | Sink | Beside kernel samples | Beside counter polls | | ------------------------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------- | | InfluxDB, Telegraf, stdout `lp` | `mikroscope_derived`: `mem_pressure`, `burst`, `suspect`, and the ratios that could be computed | `mikroscope_derived_iface{interface}` | | SQL | `mikroscope_derived`, ratios NULL where not computed | `mikroscope_derived_iface` | | file, stdout `json` | a `{"derived":…}` line after its sample | not written; the raw counters are in the `{"api":…}` line | | Prometheus | `mikroscope_derived_*` gauges for the newest sample; `mikroscope_collector_bursts_total` | `mikroscope_derived_fastpath_share{interface,direction}` | | OTLP | `mikroscope.derived.*` gauges | `mikroscope.derived.fastpath_share{interface,direction}` | | Graphite | `derived.*` paths | `api.iface..fp_rx_share`, `fp_tx_share` | | Elasticsearch | `derived` on the kernel document | `fastpath` on the API document | | Loki | — | — | > **Not measured, so not claimed** > > No comparison of the per-packet PMU cost across two router configurations on the reference device > is recorded, and that comparison is the use it exists for. The burst baseline is tuned against one > RB5009 on one day; on another board or another traffic mix its percentile will be that device's > own, and whether ten seconds is the right span there has not been measured. No interface of that > router counts `fp-tx-byte`, so no tx share has ever been produced from live counters, and why > RouterOS leaves that counter at 0 there is not established. ## See also - [Detections](/mikroscope/sinks/detections/): the eleven rules the same stage runs, and what each one may not claim. - [The RouterOS API tier](/mikroscope/sinks/api-tier/): the port counters the fast-path share is computed from. - [The resolution floor is the kernel's](/mikroscope/limits/): why the PMU is the resolution below the jiffie. - [A packet flood](/mikroscope/playbooks/packet-flood/): softnet drops and squeezes on a real router. --- # Detections The eleven rules the collector’s derive stage runs, each with its exact condition, the evidence it needs from the deployment, and what it may not claim. Source: https://jmrplens.github.io/mikroscope/sinks/detections/ A detection is a discrete event the collector puts on the timeline: a "look here", never a continuous series and never a verdict. This page answers, for each of the eleven rules, exactly when it fires, what the deployment has to provide for it to fire at all, and what it cannot tell you. The derived values some rules build on are on [what the collector derives](/mikroscope/sinks/derive/). ## What a detection carries Every detection has the same fields: `rule`; `key` — the CPU, core, zone or port it is about, empty for a device-wide rule; `seq` and `wall_ns` of the sample that raised it; `value`, the quantity the rule compared; `threshold`, what it compared against; and a `message` in words. The thresholds are the rules' own and are written into every event. **Once per rule and key per 10 s.** After a rule fires for a key, the same rule and key are suppressed for 10 s of the samples' wall clock, so a condition that persists fires every 10 s rather than on every sample. The stage counts what it suppressed, but no sink exports that count. The rules run in the collector process and their history lives there. A collector that restarts starts every trailing window, bin and previous value from nothing. ## Where detections land | Sink | Form | | ------------------------------- | ------------------------------------------------------------------------------------ | | InfluxDB, Telegraf, stdout `lp` | `mikroscope_detection{rule,key}` with `value`, `threshold`, `seq`, `message` | | SQL | a `mikroscope_detection` row | | file, stdout `json` | a `{"detection":…}` line | | Prometheus | `mikroscope_collector_detections_total{rule}`, every rule at 0 from the first scrape | | Loki | a line in the `source="detection"`, `level="warn"` stream | | OTLP | a `mikroscope.detection{rule}` delta sum of 1 | | Graphite | `detection.` = 1 at the event's second | | Elasticsearch | a document with `kind: detection` | The dashboards draw every detection as an annotation, and one of the [alert rules](/mikroscope/dashboards/alerts/) fires on any detection. ## The rules | Rule | Key | Fires when | Needs | | ----------------- | --------- | --------------------------------------------------------------------------------------------------- | ----------------------------------- | | `counter-reset` | — | the sample reports a counter that went backwards without a 32-bit wrap | any deployment | | `agent-restart` | — | the sequence number went backwards | any deployment | | `agent-oom` | — | the container's own cgroup recorded an OOM kill | cgroup2 in the container | | `microburst` | `cpu` | three `burst` samples on one CPU within 60 s | softnet | | `reboot` | — | a kernel-log record's since-boot clock is lower than the previous record's | [needs `privileged=yes`](/mikroscope/limits/privileged/) | | `link-flap` | port | two or more link up/down records on one port within 60 s | [needs `privileged=yes`](/mikroscope/limits/privileged/) | | `conntrack-cliff` | — | `nf_conntrack` fell below half its previous stored value | [needs `privileged=yes`](/mikroscope/limits/privileged/) | | `conntrack-high` | — | occupancy above 80 % of `nf_conntrack_max` **and** rising over the last 60 s | [needs `privileged=yes`](/mikroscope/limits/privileged/) | | `thermal-high` | zone | a zone within 15 % of its own declared critical trip | a thermal zone that declares a trip | | `thermal-rising` | zone | three consecutive one-minute rises of more than 1 °C each | a thermal zone | | `ipc-collapse` | `core` | a core's one-second IPC below half its trailing median **while** its cycle rate is above its median | the PMU | ### `counter-reset` **Fires when** the kernel sample's `resets` is above 0: the agent found a counter lower than its previous read without a 32-bit wrap to explain it, and used the counter's post-reset value as that tick's delta, a lower bound. `value` is the number of such counters, `threshold` 0. **Needs** nothing beyond a sample. The same condition marks the sample `suspect`, and the per-packet derived values are withheld for it. **May not claim** which counter reset, or why. Every delta in that sample is a lower bound. ### `agent-restart` **Fires when** a sample's sequence number is lower than the previous sample's. `value` is the new sequence number, `threshold` the previous one. **Needs** the collector to have seen at least one sample before the restart. The agent's sequence starts again from 1 on every launch, so this is what a restart looks like from the outside. > **A running forward does not see this today** > > From reading the code, not from a run: `forward` keeps its pull cursor, the last sequence number > it received, and never resets it. A restarted agent's ring answers > `/snapshot?since=` with nothing, and no gap, until its new sequence passes the old cursor; by then every > sample it returns has a sequence number above the previous one. So a `forward` that keeps running > across an agent restart receives nothing from the new agent for as long as the old one had been > running — a day at 10 Hz for an agent that ran a day — and this rule cannot fire in it. The rule is > exercised only by the derive stage's unit tests; no forward or end-to-end test covers it. > Restarting `forward` after the agent restarts resumes the data, but then there is no previous > sequence number and the rule does not fire either. **May not claim** why the agent restarted. A collector restarted at the same time has no previous sequence number and sees nothing. ### `agent-oom` **Fires when** the container's own cgroup records an OOM kill in the sample — a process inside mikroscope's container was killed by the kernel. `value` is the number of kills. **Needs** cgroup2 readable in the container; without it the agent reports no cgroup events and this rule cannot fire. **May not claim** anything about the numbers around it: every number in that window is suspect. Sizing the container's memory to the agent's ring is on [the cost of the observer](/mikroscope/cost/). ### `microburst` **Fires when** a CPU's sample carries the [`burst`](/mikroscope/sinks/derive/#burst) flag — a drop, or squeezes above that CPU's trailing 90th percentile and at least 3, while its packet count was at or below its trailing median — and that CPU now has at least three flagged samples within the last 60 s. `value` is the number of flagged samples in the window, `threshold` 3; the message carries the latest sample's squeezes, drops, packets and the trailing median. **Needs** `/proc/net/softnet_stat`, which every deployment reads, and ten samples of history per CPU before its first flag. The baselines span ten seconds of wall clock at any sampler rate. **False positives, measured.** Squeezing is this device's background, not an event. On the reference RB5009 `time_squeeze` is 0 in 87.3 % of per-CPU samples, 1 in 11.2 %, 2 in 1.2 % and 3 in 0.21 %, while softnet dropped nothing at all in those same 24 h. A trailing window of a distribution that is seven-eighths zeroes has a 90th percentile of 1, so "above p90" is satisfied by any 2 — which is why the floor, and not the percentile, is what the rule runs on. Replayed over 6 h of stored samples, a floor of 2 fires 77.7 /h and a floor of 3 fires 0.5 /h, still flagging 88 samples for the burst counter and for `derived.burst`. A drop flags on its own, at any squeeze count. **May not claim** the size of the burst, the flow or the interface that caused it. It says the kernel ran out of budget more than it usually does while carrying fewer packets than usual — evidence of something shorter than the sample interval. ### `reboot` **Fires when** a kernel-log record's timestamp, microseconds since boot, is lower than the previous record's. `value` and `threshold` are the new and previous timestamps in seconds. **Needs** `privileged=yes`, which the kernel log requires, and a collector that keeps running across the reboot while the agent comes back. It needs no RouterOS API credentials. A collector that keeps running is not enough on its own: the agent that comes back after the reboot is a new process, and its samples reach `forward` only once their sequence passes the old cursor (see [`agent-restart`](#agent-restart)). The rule can then fire only on a kernel-log record whose since-boot time is still below the last one seen before the reboot. This is read from the code, not observed. **May not claim** that every reboot is seen. The agent reads the kernel log from the end at start, so the first record after a reboot is one logged after the agent came up; if that record's since-boot time is later than the last record before the reboot, the clock did not go backwards and nothing fires. ### `link-flap` **Fires when** a kernel-log record that names an interface is classified `link-up` or `link-down` — the same classifier that puts a `kind` on every port record — and that port now has two or more such records within the last 60 s. `key` is the port's current RouterOS name where the API tier's interface inventory supplies one, the board's default name where the agent's port table maps the kernel name, and the kernel name otherwise. `value` is the number of records in the window, `threshold` 2. **Needs** `privileged=yes`. A RouterOS name needs the board to be in the agent's port table, and the current name needs the API tier as well; see [RouterOS ports and kernel names](/mikroscope/reference/port-names/). **May not claim** a fault. A cable pulled and reseated within a minute is a down and an up record, and fires. No provoked flap has been captured with this rule running; the flaps measured for the port table on 2026-09-15 were not. ### `conntrack-cliff` **Fires when** the `nf_conntrack` slab cache's active-object count is below half its previous stored value. `value` is the new count, `threshold` the previous one. **Needs** `privileged=yes`, for `/proc/slabinfo`. The slab is read at about 6 Hz and stored on change, so "previous" is the previous stored sample, not the previous tick. **May not claim** a fault either: the message says "a flush or a reset", and a deliberate flush of the connection table fires it. ### `conntrack-high` **Fires when** `nf_conntrack` active objects are above 0.8 of the kernel's `nf_conntrack_max`, **and** the count is higher than the oldest stored value in the last 60 s. `value` is the occupancy as a fraction, `threshold` 0.8. **Needs** `privileged=yes`, the ceiling published by the kernel, and at least two stored samples within the last 60 s. **May not claim** when the table will be full: no time-to-full is attached, on purpose. For scale, the reference router's table sat at 0.63 % of its 966 656 ceiling on 2026-09-12. ### `thermal-high` **Fires when** a zone's reading is at or above 0.85 of that zone's own lowest declared critical trip point. `value` is the reading in °C, `threshold` 0.85 × the trip. **Needs** a thermal zone that declares a critical trip. A zone that declares none never fires; nothing is compared against a compiled number. **May not claim** that cooling has failed, or anything about a zone the board does not report. ### `thermal-rising` **Fires when** a zone's last four completed one-minute means each exceed the one before by more than 1 °C — three consecutive rises. `value` is the rise from the first of the four means to the last, `threshold` 3. A one-minute bin closes on the first sample at least 60 s after it opened, and the rule is checked each time one closes, so the earliest it can fire is after about four minutes of readings. **Needs** a thermal zone. The means are over the readings the samples carried; temperature is read at the zone's declared polling cadence, 1 Hz on the reference device. **Resolution.** The reference device's sensor quantises to about 0.42 °C, so 1 °C per minute is 2.4 steps and resolvable. **May not claim** a cause, or a rise slower than 1 °C per minute. ### `ipc-collapse` **Fires when**, on one core, a one-second bin closes with instructions per cycle below half the median of that core's trailing bins **while** its cycle rate is above the median of its trailing rates. `key` is `core`, `value` the bin's IPC, `threshold` half the median. **Needs** the PMU's `cycles` and `instructions` per CPU (under `privileged=yes`), and twenty completed one-second bins of history for that core before it can fire; the trailing history holds up to sixty. **May not claim** idleness, or anything pooled: the conjunction with the cycle rate is what separates a memory-stall regime from a core going quiet, and the rule is per core, never across cores. > **Deliberately not provoked** > > Of the eleven rules, only `microburst` has a recorded behaviour on the reference device. The > others are exercised by the derive stage's unit tests against constructed samples. None of an OOM > kill inside the container, a reboot, a link flap, a conntrack flush or storm, a thermal excursion > or an IPC collapse has been provoked on the RB5009 with these rules running: it is the owner's > production router, reboots wait for a maintenance window, and a provoked conntrack storm risks > locking out the path being worked through. ## See also - [What the collector derives](/mikroscope/sinks/derive/): the `burst` flag and the other values written beside the samples. - [Alert rules](/mikroscope/dashboards/alerts/): the Grafana rules built on the detections and the fault counters. - [A loop only the kernel could see](/mikroscope/playbooks/loop/): what the kernel log caught on the production router. - [What privileged buys](/mikroscope/limits/privileged/): the sources half of these rules depend on. --- # The device-info stream What the agent establishes about the board with no RouterOS API — identity, ceilings, cadences — and how the collector hands it to every sink as facts rather than samples. Source: https://jmrplens.github.io/mikroscope/sinks/device-info/ At start the agent works out what it can about the board it runs on, with no RouterOS API and no credential, and serves it on `/capabilities`. The collector hands that to every sink as its own record. This page answers what the record holds, when it is sent, and where each sink puts it. ## What it carries - **Identity**: the device tree's model string (the board), the kernel, the core count, whether the container is privileged and whether it has cgroup2, the enabled sources, the port table's provenance, and a hash of the kernel, the core count and the enabled sources. - **The device's own ceilings**, read once at agent start: each thermal zone's critical trip point and polling delay; each core's cpufreq range, frequency ladder, governor and cluster; the kernel's conntrack ceiling; the container's own `memory.max`. - **Cadences**: the rate each level source is read and stored at, and the named reason it is not the sampler rate. A ceiling the board does not publish is left out, never filled with a number from somewhere else — the 966 656 conntrack ceiling and the 64 MiB container cap are the board's own. ### The reasons a cadence gives The cadence reasons a source can report: | `reason` | What it means | | --- | --- | | `rate` | read at the full sampler rate; nothing the device declares justifies less | | `declared` | the device publishes its own refresh cadence, and reading faster returns the same value with new dither | | `policy` | a setting says the value cannot move on its own: a `userspace` cpufreq governor | | `budget` | a measured parse cost | | `change` | read every tick, stored only when it moves | | `override` | `FLOOR_HZ` is set, and every level source is on its one cadence | Counters are never floored; only level sources carry a cadence. The floors themselves and how they were measured are on [each source at its own floor](/mikroscope/limits/source-floors/). ## When it is sent When `forward` starts, whenever the agent's capability hash changes — an agent restarted with another source set, for example — and otherwise every five minutes. The collector checks the hash on the health read it makes every minute to re-measure the clock skew, so a change reaches the sinks within about a minute. Both the direct and the relay transport can fetch `/capabilities`; when the fetch fails the collector logs it, sends nothing, and tries again at the next health read. Nothing sent is absence, not a board with no facts. **Why it repeats.** These facts are rows with the collector's timestamp, so a store holds them only at the instants they were sent, and a dashboard window that contains no emission contains no facts at all. Sent once at start, the four device panels read "No data" over every window after it: measured on the reference deployment on 2026-09-17, where the last device row was 26 hours old and those panels had been empty for as long. Five minutes puts the facts inside any window worth reading them over and costs twelve rows an emission — one identity, one per thermal zone, one per core with cpufreq facts, one per level source — against the 864 000 sample rows a day that `--hz 10` produces. The repeat is marked as one, and the sinks split on it: the stores write it like any other row, which is the whole point, and the streams meant for a reader — Loki, stdout, a recording — skip it, because a log is for change. So Loki still carries exactly one `device` line per set of facts. The hash covers only the kernel string, the core count and the enabled source names; the board, `privileged`, cgroup, the port table's provenance, every ceiling and every cadence are left out of it. A restarted agent whose only change is a ceiling or a cadence — a new `--memory-max`, or a `FLOOR_HZ` with the same source set — keeps its hash, so the change is not what triggers the send; it reaches the sinks at the next five-minute repeat, and the row that carries it is stamped then rather than when it happened. This is read from the code (`capsHash` in `internal/agent/source.go`), not observed. These are facts, not samples. They have no clock of their own, so the sinks that timestamp records stamp them with the collector's clock when they were sent; the file and stdout `json` lines carry no timestamp, and Prometheus renders them at scrape. They are never mixed into a sample row. ## Where each sink puts it | Sink | Form | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | InfluxDB, Telegraf, stdout `lp` | `mikroscope_device{board,kernel}`, `mikroscope_device_thermal{zone}`, `mikroscope_device_cpufreq{cpu}`, `mikroscope_device_cadence{source,reason}` | | SQL | the tables `mikroscope_device`, `mikroscope_device_thermal`, `mikroscope_device_cpufreq`, `mikroscope_device_cadence` | | file, stdout `json` | a `{"device":…}` line holding the capabilities as fetched | | Elasticsearch | a document with `kind: device` | | Graphite | the numeric facts under `device.*`; board, kernel and governor have no Graphite form | | OTLP | gauges: `mikroscope.device.cores` with board, kernel and hash as attributes, the thermal and cpufreq ceilings, `mikroscope.device.source_cadence` | | Loki | one `source="device"`, `level="info"` line: board, kernel, cores, privileged, cgroup, sources, hash | | Prometheus | the same device-info families the agent's own `/metrics` carries | ### InfluxDB and SQL | Measurement or table | Identified by | Holds | | --------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_device` | `board`, `kernel` | `cores`, `privileged`, `cgroup`, `sources` (comma-joined, sorted), `hash`, `conntrack_max`, `cgroup_mem_max`, `ports_from` | | `mikroscope_device_thermal` | `zone` | `critical_celsius`, `polling_ms` | | `mikroscope_device_cpufreq` | `cpu` | `cluster` (the lowest-numbered core that changes frequency with this one), `min_khz`, `max_khz`, `governor`, `steps` | | `mikroscope_device_cadence` | `source`, `reason` | `hz` | In InfluxDB `board` and `kernel` are tags, `unknown` when empty; in SQL they are columns, NULL when empty, and a ceiling that was not published is NULL. `steps` is the frequency ladder as a space-joined list of kHz. ### Prometheus | Family | Carries | | ------------------------------------------------------------------------------ | ---------------------------------------------- | | `mikroscope_device_info{board,kernel,cores,privileged,cgroup,ports_from,hash}` | always 1 | | `mikroscope_thermal_critical_celsius{zone}` | the zone's lowest critical trip | | `mikroscope_thermal_polling_seconds{zone}` | the zone's polling delay | | `mikroscope_cpu_frequency_limit_hertz{cpu,bound}` | the hardware clock range, `min` and `max` | | `mikroscope_cpu_frequency_step_hertz{cpu,step}` | every frequency the driver will use | | `mikroscope_cpu_frequency_governor_info{cpu,governor}` | always 1 | | `mikroscope_cpu_frequency_cluster{cpu}` | the cluster, named by its lowest-numbered core | | `mikroscope_self_cgroup_memory_max_bytes` | the container's own `memory.max` | | `mikroscope_source_cadence_hz{source,reason}` | each level source's cadence | On the reference RB5009, read from `related_cpus` and `affected_cpus` inside the container on 2026-09-14 (RouterOS 7.24.2), the cpufreq clusters are `{0,1}` and `{2,3}`, so a per-core frequency panel is really two series. That is this board's topology and nothing else's: the agent reads the clusters per device, and no other board has been measured. The conntrack ceiling reaches Prometheus as `mikroscope_slab_limit_objects` for the `nf_conntrack` cache rather than as a device-info family. ## See also - [The agent's HTTP endpoints](/mikroscope/reference/http/): `/capabilities`, where this record comes from. - [Each source at its own floor](/mikroscope/limits/source-floors/): the cadences and the measurements behind them. - [InfluxDB 3](/mikroscope/sinks/influxdb/): the other measurements the device records sit beside. --- # Five dashboards, one panel list What the InfluxDB 3, Prometheus, PostgreSQL, Graphite and Elasticsearch dashboards contain, section by section, in the order they are read, and why the five stores do not carry the same panels. Source: https://jmrplens.github.io/mikroscope/dashboards/ `dashboards/` holds one Grafana dashboard per store, both generated by `mikroscope dashboards gen` from a single panel list in `internal/dashboards`. This page answers what is on them: which sections, which panels in each, what the Overview shows before anything is expanded, and which panels exist on only one of the two stores. Importing them and checking them against a live Grafana is on [Import and check](/mikroscope/dashboards/import-and-check/); the alert rules generated beside them are on [Alert rules](/mikroscope/dashboards/alerts/). ## What `gen` writes - dashboards/ - mikroscope-influxdb.json 171 panels, InfluxDB 3 (SQL) - mikroscope-prometheus.json 133 panels, Prometheus - mikroscope-postgres.json 156 panels, PostgreSQL / TimescaleDB - mikroscope-graphite.json 41 panels, Graphite - mikroscope-elasticsearch.json 30 panels, Elasticsearch - mikroscope-alerts-influxdb.yaml 10 rules - mikroscope-alerts-prometheus.yaml 11 rules - mikroscope-alerts-postgres.yaml 8 rules ```sh mikroscope dashboards gen # writes the four files into ./dashboards mikroscope dashboards gen --out /tmp/dash # or into another directory ``` All five dashboards are in Grafana's shareable export format: the datasource is a `${DS_MIKROSCOPE}` placeholder declared in `__inputs`, there is no `id`, and the `uid` is fixed — `mikroscope-influxdb` and `mikroscope-prometheus` — so a re-import updates the same dashboard in place instead of creating a second one. A test asserts that two generations of the InfluxDB dashboard are identical, and regenerating on 2026-09-15 reproduced all four committed files byte for byte. The InfluxDB dashboard queries InfluxDB 3 in SQL, and every query is bounded by `$__timeFilter`, because InfluxDB 3 Core refuses unbounded scans; where a column is named `cluster` the SQL quotes it, because `cluster` is a reserved word in DataFusion. The Prometheus dashboard queries a Prometheus that scrapes the collector's `/metrics` — which carries everything the agent's does, recomputed from the samples, plus the collector's own derived and detection families — and the agent itself for the few families only the sampler produces. Prometheus 3 renders a histogram's zero bucket boundary as `le="0.0"`, so a query reading that bucket matches `le=~"0|0.0"`. The two scrape jobs are on [Import and check](/mikroscope/dashboards/import-and-check/#prometheus-two-scrape-jobs). ## What it looks like One capture per section of the InfluxDB dashboard, in the order the dashboard puts them. Each is a link to the file at full size. > **These are a demonstration database, not a router** > > Every capture below is the InfluxDB dashboard over a run of the same canned fake agent the > end-to-end suites use, written into a container store by > `test/e2e/docker`'s fill step and photographed by `site/scripts/gen-dashboard-captures.mjs`. > Nothing in them comes from a real device, the host is called `rb5009` because the fake agent > imitates that board's captured `/proc`, and the figures are whatever the fake publishes — read > them as "this is the shape of the page", never as a measurement. A panel that is empty in a > capture is one the fake agent produces nothing for; on a real router with the API tier running, > several of them fill in, and two whole sections that the store probe moved into > "not available" come back. One capture per section of the InfluxDB dashboard, over a demonstration database filled by the fake agent, on [the page](/mikroscope/dashboards/): - Overview (12) - CPU and scheduler (11) - Memory and load (9) - Connections (9) - Interface traffic (13) - Detections and captures (5) - Network receive path (9) - Forwarding cost (derived) (4) - Interrupts and softirqs (12) - Temperature and clock (10) - Kernel log (7) - CPU: how long a core stayed busy (3) - Memory: fragmentation (3) - Memory: reclaim and page faults (8) - Memory: detail and cross-checks (5) - Hardware counters (PMU) (11) - Flash wear (7) - NAND health (ECC) (2) - RouterOS API cross-checks — CPU and memory (9) - The observer (11) - The observer: sampler timing and self events (3) - This device (3) - Not available on this device (5) ## One list, two stores Every panel is declared once, with its SQL and its PromQL side by side, so a panel added to one store is added to the other. Where a panel has no query for one store, the generator **drops it from that store's dashboard** rather than shipping it to read "No data" forever. A section all of whose panels are dropped emits no row at all. That is why the counts differ, and they differ in both directions. Most of the thermal and clock family, the flash wear section, the slab census and several memory panels have SQL and no PromQL, so they exist only on InfluxDB. The sampler's timing histograms, the trigger suppressions and the capture budget live on the agent's `/metrics` only. The busy run still in progress and the age of each held reading are Prometheus exposition families, on the collector's `/metrics` too, with no InfluxDB field. Neither group is written to InfluxDB, so the panels reading them exist only on Prometheus. Each panel below that is on one store only says which. Panels per section, per store: | Section (Grafana row) | InfluxDB 3 | Prometheus | PostgreSQL | Graphite | Elasticsearch | | --- | --- | --- | --- | --- | --- | | Overview | 12 | 12 | 12 | 10 | 6 | | CPU and scheduler | 11 | 9 | 11 | 1 | no row | | Memory and load | 9 | 9 | 9 | 7 | 5 | | Connections | 9 | 3 | 9 | 1 | 1 | | Interface traffic | 13 | 11 | 9 | 2 | no row | | Detections and captures | 5 | 4 | 5 | no row | no row | | Network receive path | 9 | 7 | 9 | 3 | 3 | | Forwarding cost (derived) | 4 | 4 | 4 | no row | no row | | Interrupts and softirqs | 12 | 10 | 12 | 2 | 2 | | Temperature and clock | 10 | 2 | 10 | 2 | 2 | | Kernel log | 7 | 6 | no row | no row | no row | | CPU: how long a core stayed busy | 3 | 4 | 3 | no row | no row | | Memory: fragmentation | 3 | 1 | no row | no row | no row | | Memory: reclaim and page faults | 8 | 6 | 8 | 2 | 1 | | Memory: detail and cross-checks | 5 | 3 | 5 | 1 | 1 | | Hardware counters (PMU) | 11 | 7 | 11 | no row | no row | | Flash wear | 7 | no row | 7 | 1 | 1 | | NAND health (ECC) | 2 | 2 | 2 | no row | no row | | RouterOS API cross-checks — CPU and memory | 9 | 9 | 9 | 1 | no row | | The observer | 11 | 9 | 10 | 3 | 3 | | The observer: sampler timing and self events | 3 | 7 | 3 | no row | no row | | This device | 3 | 3 | 3 | no row | no row | | Not available on this device | 5 | 5 | 5 | 5 | 5 | | **Total** | **171** | **133** | **156** | **41** | **30** | The counts are those of the committed files, which carry the compiled defaults. `import` and `check` ask the datasource what it holds first and can move panels into or out of the last row; see [Not available on this device](#not-available-on-this-device) below. ## Reading order The sections are ordered by how often they are opened, not by taxonomy (owner, 2026-09-13: "usually the first thing you want to see is CPU, RAM, connections and the like"; in the original, "normalmente lo primero que se quiere ver es cpu, RAM, conexiones y cosas así"). First the four questions an operator arrives with — how busy is it, how much memory is left, how many connections is it holding, how much traffic is moving — then what the collector and the agent flagged, then the families that explain the first four when one looks wrong, then the deep tiers a reader goes to deliberately, and last what mikroscope costs the router it is measuring. Every section but the Overview ships **collapsed**. Grafana keeps a collapsed row's panels inside the row object and runs none of their queries until someone expands it, so the first render asks the store for the Overview's twelve panels and not for all 171. The defaults are a 3-hour range (`now-3h`) and a 5-minute refresh. The 5-minute refresh is kept for the case where someone expands a section: the slab census and the PMU and per-sample cost heatmaps ship one row per sample, and `maxDataPoints` does not apply to raw SQL. The 3-hour range is kept because `now-15m` opened on 108 empty panels with the agent stopped, and with it running a 15-minute window of 100 ms samples drew unreadable walls of noise. A live investigation sets its own range and refresh. ## The Overview The one open section. It answers "is this router healthy right now" and, before that, "can I believe these numbers". No panel in it is a heatmap or a join across measurements, and every panel returns an aggregate; three tiles ("Reboots in the window", "Sample continuity" and "Ticks never delivered, this window") compute it with a window function over the window's raw samples. Most of them are copies of panels that also live in a later section, because a Grafana panel belongs to exactly one row. "Connections tracked right now", "Load average (1 min) against the core count", "Reboots in the window" and "Detections in the window" live only here, as does "OOM kills in the window" on Prometheus, where the Reclaim section's copy has no PromQL and is dropped. The copies share their SQL; on Prometheus the Overview's "OOM kills in the window" and the Observer's "Ticks never delivered, this window" differ from their counterparts (the Observer's carries a third query, the sampler's slipped ticks). 1. **CPU busy per core**, **Memory in use, against the kernel's own total** (a gauge of `(MemTotal − MemAvailable) / MemTotal`, orange at 75 %, red at 90 %) and **Connections tracked right now** (the `nf_conntrack` slab's active objects; blank on an unprivileged container, and the tile says so). 2. **Interface throughput — rx above, tx below** (API tier; blank with `--api-mode off`, and the tile names the flag), **Die temperature by zone** and **Load average (1 min) against the core count**, where the core count is measured from the store rather than assumed. 3. **Sample continuity**, full width: each bin, never narrower than a minute, classified as continuous, ticks missing or agent restarted from the first difference of the sample sequence number. On Prometheus, which has no sequence number, the lane approximates this from the collector's gap counter and resets of the agent's sample counter, and its description says so. Every other panel on the dashboard should be read against this lane. 4. **Packets dropped in the kernel RX path (window total)**, **OOM kills in the window**, **Reboots in the window** and **Detections in the window** — each zero on a healthy device and colored on its own thresholds. 5. **Ticks never delivered, this window**, beside the count of agent restarts. > **One screen, measured once** > > On 2026-09-14 an eleven-panel Overview measured 1 052 px tall in a 1 080 px browser viewport — one > desktop screen with nothing cut off. The Overview carries twelve panels, and the grid wraps "Ticks > never delivered" onto its own full-width line; that height has not been measured in a browser, > though the committed JSON of both dashboards makes it 31 grid units, row header included. On a > phone it is not one screen: Grafana stacks a 24-column row into one column below about 768 px, and > an eight-panel Overview rendered 2 188 px tall against an 844 px viewport on 2026-09-12. ## The four everyday questions ### CPU and scheduler `/proc/stat` ticks and the sampler's own cadence. Every per-core series comes from the data (`GROUP BY cpu`, `by (cpu)`), so a 2-core and an 8-core board each draw their own lines. - CPU busy per core - Busy per core, p95 of one-second means - Worst sample interval, relative to the window's median - Per-core busy as states — which core paid, and when - Where the ticks went — device share by mode - softirq share of busy time, per core - Busy-tick distribution per sample - Share of samples the tick counter called completely idle - Cycles retired in samples /proc/stat called idle — InfluxDB only - nice, irq and iowait ticks in the window - Tick accounting closes — InfluxDB only ### Memory and load The levels from `/proc/meminfo` and `/proc/loadavg`, averaged or maxed over a bin and never summed. Every "percent of RAM" divides by the kernel's own emitted total. - Memory in use, against the kernel's own total - Load average, all three windows - Runnable threads out of total - Memory by category, as a share of total - Free memory — three definitions, against the ceiling - Commit headroom — Committed_AS as a share of CommitLimit - Load average (1 min) per core - Threads on the whole router - Slab — the conntrack and route tables the netns hides ### Connections The router's connection table from the global slab allocator (`/proc/slabinfo`, privileged only), next to the API tier's count where it is polled. An empty section here means an unprivileged agent, not an idle router; the "Slab caches reporting" tile exists to say which. - Connection table, two ways — slab objects against the RouterOS API count - Connection churn floor — peak-to-trough swing inside each bin — InfluxDB only - Packet-buffer and large-allocation caches — InfluxDB only - Every slab cache, normalized to its own window minimum — InfluxDB only - Connection count distribution over time (nf_conntrack) — InfluxDB only - Slab census — latest, min, max and spread per cache — InfluxDB only - Connection table occupancy - Connections as RouterOS counts them (API poll) - Slab caches reporting (is this a privileged deployment?) — InfluxDB only ### Interface traffic `/interface/monitor-traffic`, the MAC's per-port counters and `/system/health` from the RouterOS API — the only per-interface counters mikroscope has, because the container's `/proc/net/dev` describes its own veth — beside the configuration the API tier reads at start and every `--labels-every`: what each interface is. The rates arrive already per second and are never summed, and neither are the per-port counters, because RouterOS counts a different thing on each type. A switch port counts its wire, hardware-forwarded frames included; the bridge counts its CPU side; a VLAN or a PPPoE counts what the CPU sent and received. On the reference RB5009 (RouterOS 7.24.2, 2026-09-16) ether1 had received 255.8 GB on the wire and 29.7 GB of it had reached the CPU: ether1 and bridge are different planes, and neither is a subset of the other. The rate panels' interface set is whatever `--interfaces` selected, so an interface missing from one may be idle, unconfigured or not selected, and the three look the same. - Interface throughput — rx above, tx below - Interface packet rate — rx above, tx below - Mean packet size per interface - CPU cost of forwarding: cpu-load against the busiest interface's packet rate — InfluxDB only - Port errors per bin — typed, from the MAC counters - Where a port's receive bytes went — switched in hardware, CPU fast path, CPU slow path — InfluxDB only - Link flaps — link-downs per interface, per bin - Frame size mix over the window, per Ethernet port - Interface drops — rx, tx and tx-queue - What each interface is: type, role, bridge and label - Interface inventory and window summary - /system/health sensors, as RouterOS reads them - API tier coverage — which sources delivered, and when "What each interface is: type, role, bridge and label" is the table to read before two interface series are compared: one row per interface the router has — RouterOS type, interface lists, the bridge it is a port of, its comment, its default name and its MTU — from three configuration reads, never per poll. "Frame size mix over the window, per Ethernet port" is one row per port and carries no total: each `tx-rx-*` counter holds both directions of its own port, so a frame switched from ether1 to ether8 is counted on both, and a sum over ports counts every switched frame twice. "Interface inventory and window summary" carries no error columns, because `monitor-traffic` on RouterOS 7.24.2 returns no error rate; the MAC's typed error counters have their own panel. ## What was flagged ### Detections and captures Events, not levels: what the collector's [derive stage](/mikroscope/sinks/derive/) and the agent's [triggered capture](/mikroscope/record/triggers/) said about the window. Empty is the healthy state, and the event panels are marked known-empty so that `check` does not fail on it. - Detections per bin, by rule - Memory pressure state - Detections in this window — InfluxDB only - Trigger fires and suppressions per bin - Captures held on the agent, and the budget they pin — Prometheus only - Trigger markers in this window — InfluxDB only In "Trigger fires and suppressions per bin", the suppressions are on Prometheus only: they live on the agent's `/metrics`. ### The red dashed lines on every panel A dashboard with detections in its window draws a vertical red dashed line, with a small triangle at the foot of the axis, across **every** panel at the instant of each one. They are annotations, not data: not a gap in the record, not a slipped tick, not a break in the series. They are there so that whatever panel you happen to be reading can be read next to what the collector flagged at that moment — a memory-stall regime on one core, a microburst on one queue, a link that flapped — without having to scroll to the Detections section to find out that anything happened at all. Two layers ship with every dashboard: | Layer | Colour | Default | What each marker is | | -------------- | ------ | ------- | ------------------------------------------------------------------------------------------------------- | | **detections** | red | on | one row of `mikroscope_detection`: the rule, its key and its message, from the collector's derive stage | | **triggers** | orange | off | one capture marker from the agent: the condition that fired, the field and the value | Hovering a marker shows the message the row carries, so the line answers *what* as well as *when*. The triggers layer is off by default so that a quiet dashboard stays quiet; turn it on when you are working with [triggered captures](/mikroscope/record/triggers/). **Turning them off.** Both layers are checkboxes in the submenu row under the dashboard title — the same row a dashboard's variables sit in, which on the InfluxDB, Prometheus and PostgreSQL dashboards holds nothing else. Click the layer's name to hide its markers. That is a view setting: it lasts for the session, and saving the dashboard keeps it. Nothing about the underlying rows changes, and the Detections section still counts them. On InfluxDB each annotation is one event row with its message; on Prometheus it is `increase(…[1m]) > 0` at a 1-minute step, so an annotation there marks the minute, not the instant. The section captures further up are taken with both layers off, because the canned fake agent that fills the demonstration database fires a detection every few seconds and twenty minutes of that photographs as a solid red wash. This is the same overview with the detections layer on, over a demonstration store carrying three of them — the density a real deployment produces, and the tile at the bottom right counts the same three: *The detections layer, on: one red dashed line per detection, on every panel at once.* ## Why one of the four looks the way it does ### Network receive path `/proc/net/softnet_stat`, which is global even inside the container's network namespace. No absolute packet-rate band survives here: the squeeze regime is banded as a multiple of the window's own median. - RX path: packets processed per second, per core - Squeeze rate: softirq budget exhaustions per second, per core - Squeeze pressure: budget exhaustions per 1 000 packets - Receive-path balance across cores - Packets dropped in the kernel RX path (window total) - Squeeze regime per core, as a multiple of this window's median - Burst distribution: packets per sample (all cores) — InfluxDB only - Squeeze against throughput (1 s points, whole window) — InfluxDB only - Packets per NET_RX poll, per core ### Forwarding cost (derived) The per-packet ratios and the fast-path share the collector derives across subsystems — PMU against softnet, softnet against interrupts, RouterOS port counters against each other. The fast-path share is the share of the traffic an interface hands the CPU, not a share of the wire: `fp-rx-byte` over `driver-rx-byte` on a switch port, over `rx-byte` on a software interface, with hardware-switched frames in neither number. That split is the "Where a port's receive bytes went" panel's, from the raw counters. On the reference RB5009 (RouterOS 7.24.2, 2026-09-16) the switch ports read about 100 % — `fp-rx-byte` equals `driver-rx-byte` there to within a few kB — while bridge had fast-pathed 211.9 GB of 663.0 GB since boot (32 %) and PPPoE_DIGI 99.97 %. The tx line is usually absent: `fp-tx-byte` stayed 0 on every interface of that router after hundreds of GB transmitted, so the collector withholds the tx share while the cumulative counter is 0 rather than draw a 0 % it did not measure. Per bin the share is bytes-weighted — the bin's fast-path bytes over the bin's bytes — because an interface moving a few packets per poll swings 0–100 % from poll to poll, as that router showed on 2026-09-15. The Prometheus form is the collector's per-poll gauge and keeps that noise. - Cycles, instructions and cache misses per packet - Packets per device interrupt (NAPI coalescing depth) - Fast-path share of the traffic each interface hands the CPU - Sub-sample bursts: squeezed samples whose packet count looked ordinary ### Interrupts and softirqs `/proc/interrupts` and `/proc/softirqs` deltas. The device name is recovered from the raw `/proc/interrupts` text in the query, not matched against a driver name. - Interrupts per second by source (top-K) - Per-line interrupt load on the device with the most lines - Receive-path IRQ imbalance (max / mean across a device's lines) - Which core takes each interrupt - Top-K interrupt coverage - Interrupt mix right now - Top-K membership — which interrupt sources the agent was watching — InfluxDB only - NET_RX softirq invocations per core - Softirq invocations by kind (all CPUs) - NET_RX burst size distribution per sample — InfluxDB only - µs of softirq CPU per invocation - NET_TX and TASKLET softirq invocations per second ### Temperature and clock `/sys/class/thermal` and cpufreq. Every thermal threshold is measured from the zone's own critical trip point, read from the board: the headroom panel's thresholds sit at 10 °C and 5 °C of remaining margin below it. No panel carries a temperature the board did not publish. - Die temperature by zone - Temperature by source — kernel zones, and the RouterOS sensor when it is polled — InfluxDB only - Thermal slope, °C per minute by zone - Sensor step dwell — where each zone actually sat — InfluxDB only - Latest temperature by zone — InfluxDB only - Thermal headroom — how far each zone is from its own critical trip — InfluxDB only - Gap between the hottest and coolest thermal zone — InfluxDB only - Core clock per core — governor state — InfluxDB only - Is the clock pinned? — InfluxDB only - Clock-weighted work: effective Hz per core — InfluxDB only ### Kernel log `/dev/kmsg`, privileged only. The log is not absent when it is empty, it is silent, and silence is the healthy state. - Worst kernel-log severity in each bin - Kernel-log records per second by severity - Warning-or-worse kernel-log rate now - Kernel records in the window, by severity - Kernel records per sample — burst distribution against the per-tick cap — InfluxDB only - Port events from the kernel log, per port and kind - Port events in the window, per port Two of them read the records that name a network port. Each such record carries a kind — link-up, link-down, `stp-`, own-address (the bridge receiving a frame with its own MAC as the source, the layer-2 loop signature) or other — classified by the agent as it reads `/dev/kmsg`, and by the collector for records from an agent that does not classify. The collector then replaces the board's default port name with the interface's current RouterOS name and attaches its comment and interface lists from the API tier's inventory; without the API tier the record keeps the default name and gets no label. "Port events from the kernel log, per port and kind" counts them per bin, one series per port and kind, and "Port events in the window, per port" is the census beside it: one row per port, with the label, the role and a column per kind. This is the per-port view the container has for free — the kernel log costs the router nothing and dates each transition to the microsecond, where the RouterOS per-port counters need an API read every poll — and it is blind to anything the kernel never hears of, the hardware-switched frames included. A link-up is followed by stp-blocking, stp-learning and stp-forwarding on its bridge port: four records, not four faults. Both panels are marked known-empty, because a quiet set of ports is the healthy state. The Prometheus form reads the collector's `/metrics`, where every port record carries a `kind` because the collector classifies whatever reaches it unclassified; the agent's own exposition carries the label only when the agent itself classifies, which the one on the reference router does not on 2026-09-16. On InfluxDB the `mikroscope_kmsg` table is created by the first kernel record written, and its `kind` column by the first port record written with one. On a router whose kernel has not spoken since the collector started, the table does not exist and InfluxDB 3 refuses the query when it plans it; the collapsed section keeps that query from running until someone opens it. The code calls this a mitigation, not a fix. `import` with the probe routes those panels into the not-available row instead. ## The deep tiers ### CPU: how long a core stayed busy Contiguity, which no per-sample histogram can recover: a 2 s plateau and twenty scattered spikes bin the same. On InfluxDB the run lengths are a query over the raw samples; on Prometheus they are the upper edge of the largest populated bucket of `mikroscope_cpu_busy_run_seconds`, clamped at 60 s, so 60 reads as "longer than a minute", not as a measurement. - Longest run at or above 90 % busy, per cpu - Longest run at or above 50 % busy, per cpu - Busy run in progress right now — Prometheus only - Blocked tasks and forks ### Memory: fragmentation `/proc/buddyinfo`, the one memory number `/proc/meminfo` cannot give. - Free memory by block order (pages) — InfluxDB only - Free blocks per order (count) - Largest block order with any free block — InfluxDB only ### Memory: reclaim and page faults The `/proc/vmstat` deltas, summed or rated over a bin and never averaged — a separate section from the levels so the two kinds of reducer cannot be mixed. - Reclaim efficiency — pgsteal ÷ pgscan - Did the kernel have to reclaim at all? - Allocation distress — stalls and swap - OOM kills in the window — InfluxDB only - Page churn — allocate, free, and the net - Page faults — minor and major per second - Minor-fault bursts — distribution per sample — InfluxDB only - Context switches and all interrupts per second ### Memory: detail and cross-checks Writeback, the LRU, the small levels, and two panels that validate the instrument itself. These are read once per new device, not during an incident. - Writeback backlog — dirty pages and pages in flight - LRU balance — active vs inactive - Mapped, kernel stacks and page tables - vmstat pages vs meminfo kB — unit cross-check — InfluxDB only - Kernel stack per thread — InfluxDB only ### Hardware counters (PMU) `perf_event_open` from the privileged container. Every panel divides two raw counts; the clock-normalized ones divide by the frequency the kernel reported for that core in that sample. - IPC per core (instructions retired / cycles) - Beneath the tick floor: PMU cycles against /proc/stat busy ticks — InfluxDB only - Instructions retired while /proc/stat reported the core idle — InfluxDB only - Work the jiffie threw away (selected range) — InfluxDB only - Core cycles per bus cycle - Unhalted-cycle fraction of the clock, per core - Distribution of cycles per sample, as a share of the clock (all cores pooled) — InfluxDB only - Cache-miss rate per core (misses / references) - Cache misses per 1 000 instructions (MPKI), per core - Branch mispredictions per 1 000 instructions per core - PMU counters this CPU actually opened ### Flash wear `/proc/yaffs`, the only NAND wear signal on a RouterBOARD. InfluxDB only: the section has no row on the Prometheus dashboard. No panel can show a share of the partition, because the agent does not parse the partition's block range, so there is no denominator for one. - Flash page traffic per YAFFS partition (pages/s) - Write amplification — GC copies per page write - Flash housekeeping — erasures, garbage collections and GC copies per bin - YAFFS free-chunk drift within the window (chunks, relative to the first sample) - YAFFS partition state over the window - Bad blocks retired during the window, per YAFFS partition - Page writes and erasures per day, at the window's rate ### NAND health (ECC) The MTD ECC counters under `/sys/class/mtd`, privileged only — the flash's leading indicator, where the YAFFS bad-block count is the post-mortem. - ECC corrections since boot, per partition, against the bitflip threshold - Uncorrectable ECC failures and bad blocks, per partition ### RouterOS API cross-checks — CPU and memory `/system/resource` and `/system/resource/cpu`: the independent tier the kernel figures are checked against. RouterOS recomputes these once per second, so every field is 1 Hz at best and integer-quantized; no panel here claims a sub-second reading. - RouterOS cpu-load vs kernel busy — do the two tiers agree? - Cross-tier residual: cpu-load − kernel busy, distribution - Per-core load, RouterOS's own accounting - Per-core mean over the window: RouterOS load next to kernel busy - Per-core IRQ time as RouterOS accounts it, against kernel softirq+system - Per-core disk time (RouterOS) — max over the window - RAM used, as RouterOS accounts it - Free memory: RouterOS free-memory vs the kernel's two answers - RouterOS uptime ## What observing costs, and whether it ran ### The observer What mikroscope costs the router it is measuring, and whether it was running. On InfluxDB, continuity is derived from the sequence number of every sample (Prometheus approximates it, as in the Overview), which found 4 493 missing ticks and one restart in the 2026-09-11/12 capture that the collector's gap record said nothing about. - Agent CPU cost against its 2 % budget - Observer effect: agent share of all busy CPU on the router - CPU per sample: mean and worst tick - Where the agent's cost actually lives (per-sample distribution over time) — InfluxDB only - Agent memory against the container cap - Headroom under the container memory cap - CPU budget used (window mean) - Ticks the agent took vs ticks the store received - Sample continuity - Ticks never delivered, this window - Gaps and restarts in this window — InfluxDB only ### The observer: sampler timing and self events The sampler's own smear — how late it woke and how long the read took — and the cgroup events the agent records about itself. The three heatmaps read the agent's histograms, which are never shipped as samples, so they exist only on Prometheus. - Tick interval distribution, relative to the nominal period — Prometheus only - Wake latency: how late the sampler ran after its ticker — Prometheus only - Read duration: how long every source took to read — Prometheus only - Counter resets the agent saw - The container's own throttling and OOM kills - How each level source is read - Age of each held reading — Prometheus only ### This device The [device-info stream](/mikroscope/sinks/device-info/): what the agent established about the board at start with no RouterOS API — identity, ceilings, the frequency ladder. - This device, as the agent established it - Thermal zones: the board's own trip points and polling cadence - CPU clock: range, ladder, governor and clusters ## Not available on this device The last row is titled "Not available on this device — measurements this kernel or board does not produce (open to read why)" and is collapsed. A panel whose measurement the store does not hold is moved out of its own section into this row, where its description says what it is waiting for. In the committed files — the compiled defaults, which are what plain `gen` writes and what a manual upload into Grafana gets — the row holds the five panels the reference RB5009 (RouterOS 7.24.2, kernel 5.6.3 arm64) cannot produce: - Pressure stall (PSI), where the kernel exposes it - Block-device queue depth (requests in flight) - Block-device requests per second (reads and writes completed) - Block-device busy percent (io_s / wall time) - Block-device throughput (sectors → bytes per second) The PSI panel is empty there because that kernel has no `/proc/pressure`. The block-device panels are empty there because the agent drops a block device whose reads, writes and in-flight count are all zero in a tick, and on that router every listed device stays at zero, so no sink ever creates the table. Two sections are declared for these panels, **Pressure stall (PSI)** and **Block devices**, and ship no row while every panel in them is absent. On a kernel built with PSI, or a board with USB or eMMC storage that moves, `import` finds the measurement and the section appears in its place with no edit to the generator. `import` also works the other way: on a store that lacks a measurement the reference device had — or a field added after that store was first written — the panel moves into this row with its query removed, so it shows its explanation and not a red error badge. How the probe decides is on [Import and check](/mikroscope/dashboards/import-and-check/#the-probe). ## How the dashboards handle ratios, holes and figures - **Ratios are never the agent's.** The agent ships raw counters only, never percentages. A ratio on these dashboards is computed either in the panel's own query or by the collector's derive stage (the Forwarding cost section and the `mikroscope_derived*` families). The API tier's interface rates are the exception: they arrive from RouterOS already computed. - **They do not draw short holes.** A line is broken where two neighbouring points are more than 5 minutes apart. The threshold has to exceed the widest bin a reader selects — at a 2-day range a 12-column panel's bin is about 4 minutes — so a hole shorter than 5 minutes is still drawn as an interpolated line. Read holes from Sample continuity, not from the shape of a line. - **Their figures are one router's.** Descriptions that quote a figure attribute it to the reference RB5009 with the date it was measured and say yours will differ. No title, query or threshold names a device, its core count, its interfaces or its memory size; the only fixed numbers are mikroscope's own budget targets (2 % of one core, 16 MiB). > **Grafana versions** > > The dashboards were checked on Grafana 12.3.2 (2026-09-12) and Grafana 13.2.1 (browser passes on > 2026-09-12 and 2026-09-14, `check` and render on 2026-09-15, `check` again on 2026-09-16). The panel options are written to the > schema Grafana 13.2.1 expects — the xychart's mark, for one, moved between Grafana 11 and 13 — and > `__requires` declares Grafana 11.0.0. No version other than those two has been tried. ## See also - [Import and check](/mikroscope/dashboards/import-and-check/): the datasource, the two Prometheus scrape jobs, and what `check` does and does not verify. - [Alert rules](/mikroscope/dashboards/alerts/): the rules generated beside the dashboards, and where each threshold comes from. - [Detections](/mikroscope/sinks/detections/): the events behind the Detections section and the annotations. - [Prometheus metric families](/mikroscope/reference/metrics/): what each Prometheus panel reads. --- # Import and check How to bind the dashboards to a datasource, what import and check ask the store first, and what check proves against a live Grafana and what it cannot see. Source: https://jmrplens.github.io/mikroscope/dashboards/import-and-check/ This page answers how to get the five dashboards into a Grafana, what the datasource needs before they can return anything, and what `mikroscope dashboards check` tells you once they are there. The short version of the last part: `check` proves that every panel's query returns rows through Grafana's own query API. It does not prove that a reader can read the result. ## The InfluxDB 3 datasource The database comes first: `influxdb3 create database mikroscope` on the InfluxDB 3 node, with a token that can read and write it — the same database and token the sink writes to, on [InfluxDB 3](/mikroscope/sinks/influxdb/). The datasource is type `influxdb`, `version: SQL`, `dbName: mikroscope`, with **both** secure fields set: - `httpHeaderValue1` = `Bearer ` (the HTTP path) - `token` = `` (the FlightSQL path) Without the second, panels fail with `flightsql: Unauthenticated` (Grafana 12.3.2, 2026-09-12). ## Prometheus: two scrape jobs The Prometheus dashboard expects two scrape jobs. The first scrapes the collector (`mikroscope forward --prom :9124`), which carries every family the agent has, recomputed from the samples it received, plus the collector's own derived and detection families. The second scrapes the agent directly and keeps only the families only the sampler can produce: its tick timing histograms, the trigger and capture counters, and slipped ticks. ```yaml - job_name: "mikroscope" scrape_interval: 5s static_configs: [{ targets: [":9124"] }] - job_name: "mikroscope-agent" scrape_interval: 5s static_configs: [{ targets: ["172.30.10.2:9123"] }] metric_relabel_configs: - source_labels: [__name__] regex: "mikroscope_(tick_.*|trigger_.*|capture.*|captures_held|slipped_total)" action: keep ``` Scraping the agent without the keep list would double every counter the collector also exposes. `172.30.10.2:9123` is the agent's address on the default install; how a Prometheus host reaches it is on [Reaching the agent](/mikroscope/install/reaching-the-agent/). ## The PostgreSQL datasource The SQL sink writes a **script**, not rows: `forward --sql out.sql` and then `psql -f out.sql`, or `--sql - | psql`. So the database has the schema and the data only after that script has been applied — a datasource pointed at an empty database answers every panel with "relation does not exist". The datasource is Grafana's `grafana-postgresql-datasource`, with the database and the user the script was loaded as. `sslmode` is yours to choose; `postgresVersion` only decides which syntax the plugin may emit, and every query in this dashboard is plain SQL. The panels are the InfluxDB ones, rewritten: the bucket macro, the percentiles, the casts and the column names the SQL sink had to change because `user`, `from` and `to` are reserved words. Ten queries are not rewritten and say so — `mikroscope_buddy` and the RouterOS interface counters are wide in InfluxDB and long in the SQL schema, and a pivot is a different question. ## The Graphite datasource Graphite has no labels: every dimension is a path node, so a query IS a path — and the first two nodes are yours. `--graphite-prefix` (default `mikroscope`) and `--host-tag` are therefore **dashboard variables**, read from Graphite's own metric tree, and the dashboard asks for them at the top rather than being hard-coded to whoever generated it. The datasource is type `graphite`; nothing else is needed. From the CLI, `check` cannot read a browser's variable picker, so it takes them: ```sh mikroscope dashboards check --store graphite --datasource-uid \ --var prefix=mikroscope --var host=rb5009 ``` ## The Elasticsearch datasource Type `elasticsearch`, with the index the `--elastic-index` you forwarded with produces and `@timestamp` as the time field. This dashboard is the smallest of the five, and the reason is in the documents rather than in the queries: the sink writes a sample's per-core and per-device readings as **arrays** — `cpu` is an array of four objects — and a dynamically mapped array is a multi-valued field with no correspondence between its members. `avg(cpu.busy_ratio)` is the mean over the cores, which is a real number; "core 2's busy ratio" is not expressible at all without a nested mapping the sink does not declare. So the Elasticsearch panels are the scalar aggregates, and the per-core ones are absent rather than wrong. ## Importing by hand Grafana → Dashboards → New → Import, upload `dashboards/mikroscope-influxdb.json` or `mikroscope-prometheus.json`, and pick the datasource when Grafana asks for `DS_MIKROSCOPE`. A file uploaded this way carries the **compiled defaults**: the five panels the reference device cannot produce sit in the not-available row, and every other panel ships with its query, whether your store holds its measurement or not. On InfluxDB a panel whose table or column is missing then shows InfluxDB 3's planning error, as a red badge, when its section is opened. `import` from the CLI avoids that. ## Importing from the CLI ```sh export GRAFANA_URL=http://grafana:3000 GRAFANA_TOKEN=… mikroscope dashboards import --store influxdb --datasource-uid mikroscope dashboards check --store influxdb --datasource-uid --window 15m ``` | Flag | Default | Used by | What it does | | ------------------ | -------------- | ------------- | ------------------------------------------------------------------------- | | `--store` | `influxdb` | import, check | `influxdb` or `prometheus`; also the datasource plugin id sent to Grafana | | `--grafana` | `$GRAFANA_URL` | import, check | Grafana's base URL | | `--datasource-uid` | none, required | import, check | the datasource `DS_MIKROSCOPE` is bound to | | `--no-probe` | off | import, check | skip asking the datasource what it holds; use the compiled defaults | | `--window` | `15m` | check | length of the query window | | `--end` | now | check | the window's right edge, RFC 3339 | | `--out` | `dashboards` | gen | the directory `gen` writes the four files into | The token is read only from `GRAFANA_TOKEN`; there is no flag for it. It is a Grafana service account token with permission to write dashboards. The datasource UID is the last path segment of the datasource's settings URL in Grafana, `/connections/datasources/edit/`. Without a Grafana URL, a token and a datasource UID, both commands stop with `import/check need --grafana, GRAFANA_TOKEN and --datasource-uid`. `import` posts the dashboard to Grafana's `/api/dashboards/import` with the datasource input resolved to your UID, `overwrite` on, into the General folder (`folderId` 0). The dashboard's `uid` is fixed, so importing again replaces the same dashboard at the same URL. It prints that URL. ## The probe Before generating, `import` and `check` ask the datasource which of mikroscope's measurements it holds, through Grafana's `/api/ds/query`: - **InfluxDB 3** ```sql SELECT table_name, column_name FROM information_schema.columns WHERE table_schema = 'iox' ``` Columns and not only tables: InfluxDB 3 refuses a query naming a missing column at planning time exactly as it refuses a missing table, and a store written before a field existed has the table and not the field. A panel that reads a field added later declares it, and the probe checks it. - **Prometheus** ```text group by(__name__) ({__name__=~"mikroscope_.+"}) ``` An instant query that returns one series per metric name that exists, with no samples to transfer. A histogram counts as present when its `_bucket`, `_count` or `_sum` series is. It prints `datasource holds N measurements`; on InfluxDB, N counts tables plus `table.column` pairs, so it is larger than the number of measurements. It then generates against the answer: - A panel whose measurements and required fields are all present ships in its own section with its query — including a panel the reference device could not produce. - A panel with anything missing moves into the collapsed "Not available on this device" row **with its query removed**. It runs nothing, so it cannot paint a red `table … not found` badge; its no-value text names what this store does not hold. - A probe that fails — an error from Grafana, or an answer with no `mikroscope_` names in it — is a warning, not an error. `import` and `check` print `warning: could not ask which measurements it holds`, with the reason, and carry on with the compiled defaults, so you are told which dashboard you got. `--no-probe` skips the question and uses the compiled defaults, which is also what plain `gen` does, since it has no datasource to ask. ## What `check` verifies `check` generates the dashboard exactly as `import` would — probe included — and then, for every panel, including every panel nested inside a collapsed row, sends each of its queries through Grafana's `/api/ds/query` against your datasource over the window, and counts the rows that come back. The request carries the step Grafana would compute for that panel: the window divided by 900 data points, raised to the panel's own minimum interval where it has one. Without that step, an `increase(x[$__interval])` target returns an empty frame, because a step below the scrape interval leaves fewer than two points in the range. It prints one line per panel and a verdict: ```text ok rows= frames= none rows= frames= FAIL rows= frames= every panel returns data ( known-empty tolerated) ``` - **ok**: the panel returned at least one row and no query reported an error. - **none**: the panel is marked known-empty and did not qualify as ok. Two kinds of panel carry the mark: those whose emptiness is the healthy state (for example the detection and trigger panels, the two port-event panels, the gaps table, the opt-in conntrack poll, the sub-sample burst panel and the worst kernel-log severity timeline), and those in the not-available row. A known-empty panel is tolerated whether it returned no rows or an error. - **FAIL**: anything else — no rows, or an error on any of the panel's queries even if another returned rows. With one or more failures `check` exits non-zero with `N panel(s) return no data (K known-empty tolerated)`. A dashboard is not done until every panel that is not known-empty returns rows. ## What `check` does not verify > **Beyond the row count** > > `check` counts rows. It does not see a legend, an axis, a unit, a colour, a threshold or the grid. > On 2026-09-12 `dashboards check --store influxdb --window 12h` passed on all 125 panels it walked > while about 90 of them were unreadable in a browser: legends reading "value core 0", two xycharts > stuck on "Loading plugin panel…", a continuity lane that stayed green over 2 170 missing ticks. > Readability is established by rendering the dashboard in Chromium; `check` passing says nothing > about it. What else is outside its reach, from the code: - **The dashboard stored in Grafana.** `check` regenerates the dashboard locally and runs those queries. It does not read back what `import` stored, so a dashboard edited in Grafana's UI is not what it checks. - **Whether a number is right.** One row is a pass. Wrong arithmetic that returns rows passes. - **Annotations.** Only panels are walked; the detections and triggers annotation queries are not run. - **Alert rules.** The provisioning files `gen` writes are not loaded or evaluated. - **What the browser does to a query.** Some variables are substituted in the browser, not by the server `check` talks to. On 2026-09-14 the InfluxDB datasource escaped `$__interval_ms` in five panels, in the browser, into SQL InfluxDB 3 could not parse; the rendered dashboard showed it. - **The real panel width.** `check` pretends every panel is 900 data points wide, the value the browser sent for this dashboard's graphs on 2026-09-12. A narrower panel gets a wider bin. ## Checking against a finished capture `--window` is the length of the query window and `--end` moves its right edge, so the panels can be checked against a capture that has already finished rather than against an idle now: ```sh mikroscope dashboards check --store influxdb --datasource-uid \ --window 1h --end 2026-09-13T08:30:00Z ``` A panel answers differently over a window with data than over one without, and a check is only as good as the window it is pointed at. ## What has been verified **2026-09-16**, on the owner's Grafana 13.2.1, against the reference deployment: the agent on the reference RB5009 (RouterOS 7.24.2, privileged, the default triggers); `forward --prom :9124 --influx … --interfaces bridge,ether1,PPPoE_DIGI --counters-every 10s` for 30 minutes into an isolated InfluxDB 3 Core; a Prometheus 3.14 scraping the collector every 5 s, plus the agent directly for the families the collector cannot recompute. | Store | Window | Panels | Failing | Known-empty tolerated | | ---------- | ---------- | -----: | ------: | -------------------------------------------------------------------------------------------------------------------- | | InfluxDB 3 | 30 minutes | 171 | 0 | 10 (the two port-event panels, the opt-in conntrack poll, the two trigger panels, PSI, the four idle block devices) | | Prometheus | 30 minutes | 133 | 0 | 9 (the two port-event panels, the two conntrack API panels, PSI, the four block devices) | The two port-event panels' SQL was validated the same day against a synthetic table in the same InfluxDB 3, because the live store holds no `kind` column until the first port record classified by kind is written to it. The headless row-by-row walk of both dashboards in Chromium at 1600x1000 — 0 error badges, 0 "No data" — is from **2026-09-15** and covers 168 InfluxDB panels and 130 Prometheus ones. It has not been repeated, so no render pass is claimed for the three panels it did not cover: the two port-event panels and "What each interface is: type, role, bridge and label". **2026-09-12**, on the owner's Grafana 12.3.2. The InfluxDB dashboard against an isolated InfluxDB 3 Core fed by `mikroscope forward` from an RB5009: every panel returned rows, 158–316 per panel over 10 minutes. The Prometheus dashboard against a Prometheus 3.14 scraping the collector every 5 s: every panel returned rows, 228–2 052 over 5 minutes. > **Empty by configuration is not known-empty** > > The 10 and 9 above are that deployment over that window. Only panels marked in the generator, or > moved into the not-available row by the probe, are tolerated. A panel that is empty because of how > the collector was run — the interface panels under `--api-mode off`, the slab panels on an > unprivileged agent — is not marked: on a store that never held its measurement the probe moves it > into the not-available row, and on a store that held it earlier but not in the window, `check` > fails it. ## See also - [Five dashboards, one panel list](/mikroscope/dashboards/): every section and panel, and which store carries each. - [Alert rules](/mikroscope/dashboards/alerts/): the provisioning files `gen` writes, which `check` does not run. - [Prometheus](/mikroscope/sinks/prometheus/): the collector's `/metrics` the first scrape job reads. - [InfluxDB 3](/mikroscope/sinks/influxdb/): the write URL, the token and the store's own limits. --- # Alert rules The Grafana alert rules generated beside the dashboards, what each one fires on, where its threshold comes from, and what has not been tested about them. Source: https://jmrplens.github.io/mikroscope/dashboards/alerts/ `mikroscope dashboards gen` writes, beside each dashboard, a Grafana unified-alerting provisioning file with the rules that follow from the dashboards' own fault counters, from the kernel log the agent reads, and from the collector's detections. This page answers what those rules are, what each one fires on and what silence means for it, how to install the file, and where every threshold comes from. Every threshold is zero (a counter that should not move), one sample (the silent-agent rule), or a share of a ceiling the device itself published. None is a number compiled in for one router. ## The files | File | Rules | Query language | | ---------------------------------------------- | -----------------------------------------------: | -------------- | | `dashboards/mikroscope-alerts-influxdb.yaml` | 10 | InfluxDB 3 SQL | | `dashboards/mikroscope-alerts-prometheus.yaml` | 11 | PromQL | The InfluxDB file has one rule fewer because "The sampler is slipping ticks" has no SQL form: the slipped-tick counter is exposed on the agent's `/metrics` and is not written to InfluxDB. Each file is `apiVersion: 1` with one rule group, `mikroscope`, in a folder named `mikroscope`, organisation 1, evaluated every minute. The rules are provisioned rather than built into the dashboards, so an operator who wants none copies nothing. ## Installing them Provisioning files do not resolve a dashboard's `${DS_MIKROSCOPE}` input, so the datasource is a literal placeholder, `DS_UID_PLACEHOLDER`, that you replace with your datasource's UID before Grafana reads the file: ```sh sed 's/DS_UID_PLACEHOLDER//g' dashboards/mikroscope-alerts-influxdb.yaml \ > /etc/grafana/provisioning/alerting/mikroscope-alerts-influxdb.yaml ``` The provisioning directory is Grafana's; `/etc/grafana/provisioning/alerting/` is the path the generated file's own header names. Use the file that matches the datasource the UID belongs to. Every rule has the same shape, the one Grafana's own rule editor writes: 1. **A** — the query, against your datasource, with a relative time range of the last 600 s. Every SQL query and every Prometheus counter query also bounds its own window (2 minutes, 5 minutes or 1 hour, below); the two Prometheus gauge rules, thermal and conntrack, read the latest value. 2. **B** — reduce A to one number per series with `last`, dropping non-numeric values. 3. **C** — compare B against the threshold. C is the rule's condition. Each rule carries the labels `severity` (`critical` or `warning`) and `source: mikroscope`, a `summary` annotation, and `execErrState: Error`. What Grafana then does with a rule whose query fails is Grafana's behaviour, set out in its own documentation, and has not been tested here. ## The rules The alert rules: | Rule (uid) | Fires when | Threshold (C) | Severity | `for` | No data means | Stores | | --- | --- | --- | --- | --- | --- | --- | | `mikroscope-agent-silent` | fewer than 1 new sample reached the store in the last 2 minutes | < 1 | critical | 2m | Alerting | InfluxDB only | | `mikroscope-softnet-drops` | `softnet_stat` dropped a packet in the last 5 minutes | > 0 | critical | 0s | OK | InfluxDB only | | `mikroscope-oom-kill` | `/proc/vmstat` `oom_kill` moved in the last 5 minutes | > 0 | critical | 0s | OK | InfluxDB only | | `mikroscope-detections` | any detection in the last 5 minutes | > 0 | warning | 0s | OK | InfluxDB only | | `mikroscope-thermal-near-critical` | a zone at or above 85 % of its own critical trip | > 0 | critical | 1m | OK | InfluxDB only | | `mikroscope-conntrack-near-limit` | `nf_conntrack` active objects above 0.8 of the kernel's limit | > 0.8 | warning | 5m | OK | InfluxDB only | | `mikroscope-ticks-slipped` | the sampler slipped a tick in the last 5 minutes | > 0 | warning | 5m | OK | Prometheus only | | `mikroscope-agent-oom` | the agent's own cgroup recorded an OOM kill in the last 5 minutes | > 0 | critical | 0s | OK | InfluxDB only | | `mikroscope-l2-loop` | an own-address record on any port in the last 5 minutes | > 0 | critical | 0s | OK | both | | `mikroscope-port-link-down` | a link-down record on any port in the last 5 minutes | > 0 | warning | 0s | OK | both | | `mikroscope-ecc-failure` | the NAND reported an uncorrectable ECC failure in the last hour | > 0 | critical | 0s | OK | InfluxDB only | The InfluxDB form of `mikroscope-conntrack-near-limit` is broken; the "what has not been tried" note at the end of this page says why. "No data means" is the rule's `noDataState`. The silent-agent rule is the one where silence is the fault, so no data fires it; for every other rule no data is the healthy reading. Each rule's title, and under it its `summary` annotation verbatim, as generated: - **mikroscope agent stopped delivering samples.** "No new samples reached the store in the last two minutes: the agent stopped, the collector stopped, or the path between them did. Every other rule is blind while this one fires." - **Packets dropped in the kernel receive path.** "softnet_stat dropped a packet: a per-CPU backlog was full. Unambiguous loss inside the router, invisible to every SNMP and RouterOS counter. Zero is the expected reading." - **The kernel OOM-killed a process.** "/proc/vmstat oom_kill moved: the kernel killed a process to get memory back. Which process is not knowable from the container (no PID namespace)." - **The collector's derive stage flagged an event.** "A detection rule fired (counter-reset, agent-restart, agent-oom, microburst, reboot, link-flap, conntrack-cliff, conntrack-high, thermal-high, thermal-rising, ipc-collapse). The rule, key, value and threshold are in the Detections section and on the dashboard as an annotation." - **A thermal zone is within 15 % of its own critical trip.** "The reading is at or above 85 % of the zone's declared critical trip point (105 C on the reference RB5009). The ceiling is the board's own, read from /sys, not a number compiled in." - **The connection table is above 80 % of nf_conntrack_max.** "nf_conntrack active objects over the kernel's own ceiling. Past the ceiling the router drops new connections. The limit is the sysctl the agent read, not a compiled number." - **The sampler is slipping ticks.** "Ticks finished after the next was due. The rate is not being delivered: the device is starved, the source set is too expensive for the rate, or the container's CPU quota throttled the agent (see the observer's throttling counter)." - **mikroscope's own container was OOM-killed.** "The kernel killed a process inside the agent's cgroup: the capture ring and the captures are gone, and every number in the window is suspect. Raise --memory-max or lower RATE_HZ, BUFFER_S or CAPTURE_MB." - **The bridge received its own address back: a layer-2 loop signature.** "The kernel log reported `received packet on with own address as source address`: a frame the router sent came back in, which is what a loop through a downstream switch or access point looks like. The port label says which cable. Read from /dev/kmsg by the agent, no API. On the reference RB5009 this ran at 1.49 records/s for hours on 2026-09-12 while every RouterOS counter looked healthy. The InfluxDB form needs a store that has held at least one port record classified by kind." - **A port's link went down.** "The kernel log reported a link-down on a port: a cable pulled, a peer rebooted or powered off, a renegotiation. The collector's link-flap detection covers the repeated case; this is the single event. Read from /dev/kmsg by the agent, no API; the port, its comment and its role are on the Kernel log section's port events." - **The NAND reported an uncorrectable ECC failure.** "ecc_failures rose on an MTD partition: a read the error correction could not fix, i.e. data loss on the flash. Any increment is an incident." ## The queries - **Prometheus** ```text # mikroscope-agent-silent (< 1) sum(increase(mikroscope_samples_total[2m])) # mikroscope-softnet-drops (> 0) sum(increase(mikroscope_softnet_total{kind="dropped"}[5m])) # mikroscope-oom-kill (> 0) sum(increase(mikroscope_vm_events_total{event="oom_kill"}[5m])) # mikroscope-detections (> 0) sum(increase(mikroscope_collector_detections_total[5m])) # mikroscope-thermal-near-critical (> 0) count(mikroscope_thermal_celsius >= on(zone) 0.85 * mikroscope_thermal_critical_celsius) # mikroscope-conntrack-near-limit (> 0.8) max(mikroscope_slab_active_objects{cache="nf_conntrack"} / mikroscope_slab_limit_objects{cache="nf_conntrack"}) # mikroscope-ticks-slipped (> 0) sum(increase(mikroscope_slipped_total[5m])) # mikroscope-agent-oom (> 0) sum(increase(mikroscope_self_oom_kills_total[5m])) # mikroscope-l2-loop (> 0) sum(increase(mikroscope_kmsg_port_records_total{kind="own-address"}[5m])) # mikroscope-port-link-down (> 0) sum(increase(mikroscope_kmsg_port_records_total{kind="link-down"}[5m])) # mikroscope-ecc-failure (> 0) sum(increase(mikroscope_mtd_ecc_failures_total[1h])) ``` `mikroscope_slipped_total` comes from the agent scrape job, the one with the keep list on [Import and check](/mikroscope/dashboards/import-and-check/#prometheus-two-scrape-jobs); the other ten read the collector's `/metrics`. `mikroscope_kmsg_port_records_total` is among them: both expositions are written by the same renderer, and the collector's copy is the one the keep list leaves in place, the one that classifies a record the agent did not and names each port as RouterOS names it now. - **InfluxDB 3** ```sql -- mikroscope-agent-silent (< 1) SELECT count(1) AS value FROM mikroscope_cpu WHERE time >= now() - interval '2 minutes' -- mikroscope-softnet-drops (> 0) SELECT coalesce(sum(dropped), 0) AS value FROM mikroscope_softnet WHERE time >= now() - interval '5 minutes' -- mikroscope-oom-kill (> 0) SELECT coalesce(sum(oom_kill), 0) AS value FROM mikroscope_vm WHERE time >= now() - interval '5 minutes' -- mikroscope-detections (> 0) SELECT count(1) AS value FROM mikroscope_detection WHERE time >= now() - interval '5 minutes' -- mikroscope-thermal-near-critical (> 0) SELECT count(1) AS value FROM (SELECT zone, max(celsius) AS c, max(critical_celsius) AS crit FROM mikroscope_thermal WHERE time >= now() - interval '2 minutes' AND critical_celsius IS NOT NULL GROUP BY zone) WHERE c >= 0.85 * crit -- mikroscope-conntrack-near-limit (> 0.8) SELECT max(active) * 1.0 / nullif(max(limit_objs), 0) AS value FROM mikroscope_slab WHERE time >= now() - interval '2 minutes' AND cache = 'nf_conntrack' AND limit_objs IS NOT NULL -- mikroscope-agent-oom (> 0) SELECT coalesce(sum(oom_kill), 0) AS value FROM mikroscope_self WHERE time >= now() - interval '5 minutes' AND oom_kill IS NOT NULL -- mikroscope-l2-loop (> 0) SELECT coalesce(sum(count), 0) AS value FROM mikroscope_kmsg WHERE time >= now() - interval '5 minutes' AND kind = 'own-address' -- mikroscope-port-link-down (> 0) SELECT coalesce(sum(count), 0) AS value FROM mikroscope_kmsg WHERE time >= now() - interval '5 minutes' AND kind = 'link-down' -- mikroscope-ecc-failure (> 0) SELECT coalesce(sum(delta), 0) AS value FROM (SELECT max(ecc_failures) - min(ecc_failures) AS delta FROM mikroscope_mtd WHERE time >= now() - interval '1 hour' AND ecc_failures IS NOT NULL GROUP BY "partition") ``` ## Where the thresholds come from - **Zero, for a counter that should not move.** Kernel RX drops, kernel OOM kills, detections, slipped ticks, the agent's own OOM kills, uncorrectable ECC failures, and the kernel-log port records whose kind is `own-address` or `link-down`. A healthy device reads zero on each. - **One sample, for the silent agent.** Fewer than one sample in two minutes is none at all, at any configured rate. - **A share of the board's own thermal trip.** 85 % of the lowest critical trip point each zone declares, which the agent reads from `/sys/class/thermal` and ships beside every reading. A zone that declares no critical trip is left out of the query rather than compared against a made-up ceiling. - **A share of the kernel's own connection limit.** 0.8 of the `nf_conntrack` limit the agent read. The occupancy comes from `/proc/slabinfo`, which needs a privileged container. The rules do not alert on softnet squeezes. Measured on the reference RB5009 on 2026-09-15 over 3 476 samples, about 11.2 % of samples carry one squeeze, and alerting on "squeeze > 0" would page forever. What reaches the detections alert instead is the `microburst` detection: three burst samples on one CPU within 60 s. A burst sample is one in which a softnet queue dropped a packet, or ran out of budget more often than that CPU's trailing 90th percentile and at least three times, while the sample's packet count was at or below its trailing median. See [Detections](/mikroscope/sinks/detections/). ## What the rules cannot see - **Which process.** The OOM rule says the kernel killed something. From inside the container there is no PID namespace to say what. - **Which port.** The two port-event queries sum over ports, so a firing rule says a loop signature or a link-down happened, not on which cable. The port, its comment and its interface lists are on the Kernel log section's two port-event panels. - **An unprivileged agent's blind spots.** The connection-table and ECC rules read sources that need a privileged container. Without one those measurements never reach the store, and on Prometheus a query over a missing metric returns no data — which these rules read as OK. - **Anything while the silent-agent rule fires.** Every other rule but the slipped-ticks rule, which reads the agent directly, reads the same stream; with no samples arriving they read zero or no data, and both are OK. > **What has not been tried** > > These files have not been loaded into Grafana and watched while a rule evaluated, fired or > resolved; `dashboards check` runs the dashboards' panel queries and not these. Several consequences > follow from the code and have not been observed. **InfluxDB tables that appear only after their > first event:** the collector creates `mikroscope_detection` with the first detection it writes, > and InfluxDB 3 refuses a query naming a missing table when it plans it, so on a store that has > never held a detection the detections rule's query should fail, and its `execErrState: Error` > applies instead of OK; the same applies to any table or column the deployment has never written, > such as `mikroscope_mtd` on an unprivileged agent. **The two port-event rules have never been seen > firing:** neither has been watched against a live loop or a live link-down, on either store. Their > InfluxDB form reads the `kind` column of `mikroscope_kmsg`, which a store holds only once the > collector has written a first port record classified by kind — on the reference deployment on > 2026-09-16 the store had no `kind` column yet, so that query fails at planning there for the same > reason as the missing-table case above. Their Prometheus form reads > `mikroscope_kmsg_port_records_total{kind=…}` from the collector's `/metrics`, which carries a > `kind` on every port record whatever the agent shipped; the agent's own exposition carries the > label only when the agent classifies, and the one on the reference router does not, so a > Prometheus scraping the agent alone sees no `kind` there and the query returns no data, which > these rules read as OK. **The InfluxDB conntrack rule names a column no > InfluxDB store holds:** its query reads `limit_objs`, which is the column name in the SQL > (Postgres/Timescale) sink, while the InfluxDB sink writes the slab ceiling as the field `limit` > (the dashboards' own occupancy panel reads `limit`). So on every InfluxDB store, privileged or > not, that query should fail at planning and the rule cannot evaluate. This is a defect in the > generator, not a property of the device. **The InfluxDB ECC rule across partitions:** its query > takes the largest `ecc_failures` reading in the hour minus the smallest, over every partition > together, with no grouping by partition — so on a board whose partitions sit at different non-zero > levels the difference is non-zero without any new failure. Every partition reads zero on the > reference RB5009, so that device cannot show it. The flash panel's own description adds that a > board shipping with factory-marked bad blocks shows a non-zero level that is normal for it, and > that the change is the event, not the level. ## See also - [Detections](/mikroscope/sinks/detections/): the eleven rules behind the detections alert, and what each may not claim. - [Import and check](/mikroscope/dashboards/import-and-check/): the datasource UID these files need, and the Prometheus scrape jobs. - [Five dashboards, one panel list](/mikroscope/dashboards/): the panels whose fault counters these rules are built from. - [Prometheus metric families](/mikroscope/reference/metrics/): the families the Prometheus queries read. --- # How to read what it shows Seven readings of kernel data from one RB5009 — a production fault found by accident, three events provoked on purpose, two that needed no provoking, and the idle shape they are read against. Source: https://jmrplens.github.io/mikroscope/playbooks/ This section answers the question that comes after installing the agent: _the numbers are arriving — what does a fault look like in them?_ Each page is one shape, read from data the agent collected on the reference router, with the commands that produced it so you can reproduce it on your own device. Every number in these pages comes from one campaign on the reference router, an aarch64 kernel on a board with 1 GB of RAM: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container Where a later page adds a figure from another date, it says so beside the figure. Where a fault was provoked deliberately, the page says exactly how. ## Before you provoke anything The scenarios were chosen so that nothing they do can break the router's uplink or cut an administrator's own path to it. Keep that property on your own device: find out which port you are connected through first, ```text /interface/bridge/host/print where mac-address="" ``` and leave that port, the WAN port and anything carrying a service alone. ## The seven readings Read the idle shape first. Without it, every other page looks like an anomaly. | Page | How it came about | Where it shows | Signature | | -------------------------------------------------------------------- | ------------------------------------------------ | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | [The shape of an idle router](/mikroscope/playbooks/idle/) | 60 s at idle | per-core busy, `time_squeeze`, `events` | 6–8 % busy across four cores, squeeze never zero, zero kernel events | | [A loop only the kernel could see](/mikroscope/playbooks/loop/) | real, found by accident on the production router | `events` (the `kmsg` source) | `events` at 1.49 /s, 2.00–2.01 s apart; the API reported a healthy device | | [RouterOS ports and kernel names](/mikroscope/reference/port-names/) | a dead port toggled off and on | `events` | the kernel prints `eth5` where RouterOS says `ether6` | | [A CPU-bound workload](/mikroscope/playbooks/cpu/) | a console loop that ends by itself | per-core busy, temperature, frequency | ~29 % total, one core at 99.8 %, ~20 s of migration before it settles | | [A packet flood](/mikroscope/playbooks/packet-flood/) | `ping -f` at the router's own LAN address, 10 s | `switch0` interrupts, softirqs, `time_squeeze` | `switch0` 5.5 k → 34 k per 5 s bucket, all on the one core the IRQ is pinned to | | [Flash wear](/mikroscope/playbooks/flash-wear/) | nothing provoked; the router writes on its own | the `yaffs` source, MTD ECC counters | 2 page writes per 30 s at idle, traced to the `dns` topic logging to disk | | [Conntrack without the API](/mikroscope/playbooks/conntrack/) | a cross-check against the API, no storm | the `nf_conntrack` slab cache | the router's real connection count where the container's own namespace reports 0 | ## Two checks before you trust a reading **The sampler was not starved.** `mikroscope_slipped_total` should be 0 over the window you are reading. A slipped tick is one whose read finished after the next tick was due, and the sampler's own accounting is then the first thing to distrust. **The kernel log was kept whole.** `mikroscope_kmsg_dropped_total` counts loss events, not records: one per tick that hit the agent's cap of 64 records, and one per kernel ring overrun, which can stand for many records. While it is non-zero, the per-level counts in `mikroscope_kmsg_records_total` are a lower bound, and so is an event rate taken from them. > **The commands on these pages carry no token** > > The `curl` examples talk to the agent at `http://172.30.10.2:9123` without credentials, which is > how a default install answers. An agent started with a token — mandatory with `--expose` — > returns 401 on every path but `/healthz` until each request carries the header > `Authorization: Bearer `. ## What the agent costs while you do this Measure the observer, and measure it honestly — including the part where measuring changes the answer. Pulling a 60-second `/snapshot` is not free: the agent has to hand over ~600 pre-encoded lines, about 1.5 MB, and `self.cpu_us` inside those samples _includes the cost of serving them_. Reading the agent's cost out of a large snapshot therefore overstates it, and doing it repeatedly overstates it more. Use `/metrics` instead. It is small, its counters are cumulative, and it is independent of who scrapes it and when — scrape it twice and divide: ```sh U=http://172.30.10.2:9123/metrics get() { curl -s "$U" | awk -v k="$1" '$1==k{print $2}'; } c0=$(get mikroscope_self_cpu_usec_total); t0=$(date +%s) sleep 180 c1=$(get mikroscope_self_cpu_usec_total); t1=$(date +%s) echo "$c0 $c1 $t0 $t1" | awk '{printf "%.2f %% of one core\n", 100*($2-$1)/1e6/($4-$3)}' ``` Two things to expect: - **Cost and memory rise until the ring fills.** With the default 300 s ring at 10 Hz the agent holds 3 000 pre-encoded samples; a figure taken in the first minute after install is measured on a nearly empty heap and will be optimistic. Wait out `BUFFER_S` before quoting a steady-state number. - **`mikroscope_slipped_total` is the number that actually matters.** A sampler that costs a little more but never slips is telling you the truth; one that slips is not. For scale: at the install default the agent costs 2.85 % of one core and 31.3 MiB RSS. That figure was measured on 2026-09-15 with the full source set and three sinks at once, not during the campaign these readings come from: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once ## See also - [The cost of the observer](/mikroscope/cost/): the budget, the measured cost and the procedure above in full. - [The resolution floor is the kernel's](/mikroscope/limits/): why one core's busy percentage in a 100 ms sample moves in 10 % steps, and what reads beneath that. - [Detections](/mikroscope/sinks/detections/): the rules the collector runs over these same signals. - [Five minutes with a router](/mikroscope/start/walkthrough/): a recording with markers, the other way to read a transient. --- # The shape of an idle router Sixty seconds of the reference RB5009 doing nothing in particular — the busy floor, the squeeze that is never zero, and the silence of a healthy kernel log. Source: https://jmrplens.github.io/mikroscope/playbooks/idle/ This page answers what "normal" looks like, so that the other pages have something to be anomalous against. Know this shape before you go looking for anomalies, or you will find them everywhere. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container ## Sixty seconds at idle Three 20-second buckets: ```text t busy% per-core busy% ctxt sirq squeeze temp MHz events 0 7.1 10.5 7.9 4.3 5.8 41793 325 296 36.2 1400 0 1 8.4 8.0 7.8 9.9 7.8 60518 251 223 36.2 1400 0 2 6.5 7.1 5.2 6.7 6.8 58259 199 179 36.3 1400 0 ``` The percentages in that table were computed after the fact from the raw tick deltas the agent ships; the agent never turns them into percentages. ## What to read - **6–8 % busy across four cores is this router's floor**, not a problem. It is DNS, DHCP, WireGuard, the bridge and its own housekeeping. - **`time_squeeze` is never zero** (~200 per 20 s). A nonzero squeeze is normal; what matters is the _change_ under load, which [the packet flood](/mikroscope/playbooks/packet-flood/) shows. Alerting on "squeeze > 0" would page you forever. - **`arch_timer` and `switch0` are always the top two interrupt sources.** - **Zero kernel events is the healthy state.** Any steady stream of `events` at idle deserves the treatment in [the loop case study](/mikroscope/playbooks/loop/) — that is exactly how it started. ## The background squeeze and the burst flag The squeeze floor was measured again on 2026-09-15, on the same device, over 3 476 samples: about 11.2 % of samples carry one squeeze as background, and 2 % carry two or more. Measured on this router that day: a rule that flags any squeeze fires 92 times in twenty minutes, and means nothing. So the collector's `burst` flag is a deviation from the device's own norm: a dropped packet, or more squeezes than that CPU usually has (above its trailing 90th percentile and at least 3) in a sample whose packet count was at or below its trailing median. The trailing baselines span ten seconds of wall clock at any rate. Three flagged samples on one CPU within 60 s raise the `microburst` detection; one flagged sample stays a data point. > **True of this device, not of yours** > > This floor is one RB5009 with its clock pinned at 1400 MHz, running this owner's DNS, DHCP, > WireGuard and bridge. A router with different services, a scaling clock or a different board has a > different idle shape. Take the same sixty seconds on yours before reading any other page against > it. ## See also - [A packet flood](/mikroscope/playbooks/packet-flood/): what the squeeze does when it does change. - [A loop only the kernel could see](/mikroscope/playbooks/loop/): what a steady stream of events at idle turned out to be. - [Detections](/mikroscope/sinks/detections/): the `microburst` rule and the others, with what each may not claim. --- # A loop only the kernel could see A layer-2 loop on the reference router’s production network that RouterOS’s log and port monitor never showed, found by the agent’s kernel-log source, diagnosed from a rate, and fixed with a firmware update confirmed two independent ways. Source: https://jmrplens.github.io/mikroscope/playbooks/loop/ This fault was not provoked. It was found by accident on 2026-09-12, while measuring something else, on the owner's production RB5009UG+S+ (RouterOS 7.24.2, kernel 5.6.3), and it was diagnosed and fixed the same day. It is the clearest answer this project has to _"why not just poll the RouterOS API?"_: for its whole life the API reported a healthy device. The page follows the diagnosis in the order it happened, because the order is the method. ## The symptom `/snapshot` carried a steady stream of `events` — the agent's `kmsg` source — at **1.49 /s**: ```text [6] br0: port 2(eth1) entered blocking state [4] br0: received packet on eth1 with own address as source address (addr:00:00:5e:00:53:5d, vlan:0) [6] br0: port 2(eth1) entered learning state ``` The address in the second line is the router's own: the MAC of `sfp-sfpplus1`, coming back into the bridge. It is shown here as `00:00:5e:00:53:5d`, from the block RFC 7042 reserves for documentation — on your device it is your bridge's address, and that is how you recognise the line. ## Step 1 — establish that it is real, and get a number A single alarming log line is an anecdote. A _rate_ is a measurement, and a rate is what tells you later whether a fix worked. Count events per second over a window long enough to be stable: ```sh curl -s "http://172.30.10.2:9123/snapshot?seconds=120" \ | python3 -c ' import sys, json, collections rows=[json.loads(l) for l in sys.stdin if l.strip()] secs=sum(r["dt_ns"] for r in rows)/1e9 ev=[e for r in rows for e in r.get("events",[])] print(f"{len(ev)/secs:.2f} events/s over {secs:.0f}s") c=collections.Counter(e["msg"].split("(")[0][:60] for e in ev) for k,v in c.most_common(): print(f" {v:5d} {k}")' ``` `?seconds=` takes 1 to 3600, and the agent can only return what its ring still holds — 300 s at the default `BUFFER_S`. The window's length comes from summing each sample's own `dt_ns`, not from the number you asked for. ## Step 2 — read the timing, not just the text The gaps between the reflected frames were **2.00–2.01 s**, every time. That is not a coincidence to be noted and moved past: 2 s is the STP _hello interval_. The mechanism was therefore "the router emits a BPDU and receives its own BPDU back", which is a loop, not a misbehaving client. Timing is usually where the diagnosis actually lives: ```sh # gaps between consecutive occurrences of one message ... | python3 -c ' import sys, json ts=sorted(e["us"]/1e6 for l in sys.stdin if l.strip() for e in json.loads(l).get("events",[]) if "own address" in e["msg"]) print([round(ts[i+1]-ts[i],2) for i in range(len(ts)-1)][:12])' ``` `us` is the kernel's own timestamp for the record, in microseconds since boot on the monotonic clock — not the time the agent read it — so the gaps are the kernel's, not the sampler's. ## Step 3 — confirm the API really is blind Worth doing explicitly, because it decides where you spend the next hour: ```text /log/print where topics~"bridge" or topics~"stp" or topics~"interface" /interface/bridge/port/monitor [find] once ``` Both came back clean: zero log rows on those topics, every port `designated-port` / `in-bridge`. RouterOS was not hiding the fault; it genuinely does not surface this class of kernel event. ## Step 4 — find corroborating evidence outside the kernel log A single source, however convincing, is one source. Two independent observations pointing the same way is a diagnosis. The MAC in the message belonged to `sfp-sfpplus1`, so the obvious suspicion was the SFP+ segment — but the bridge's own tables said otherwise: ```text # hosts learned per port :foreach p in=[/interface/bridge/port/find] do={ \ :local n [/interface/bridge/port/get $p interface]; \ :put ($n . " hosts=" . [:len [/interface/bridge/host/find interface=$n]]) } ``` | Port | Learned MACs | Traffic | RSTP `edge` | | -------------- | ------------ | ------------------------------- | ----------- | | `sfp-sfpplus1` | 60 | 23.8 GB rx | true | | **`ether2`** | **0** | **9.15 GB rx / 22.4 M packets** | **false** | | others | 0–2 | — | true | `ether2` was passing 22 million packets on a healthy 1 Gbps link and the bridge had learned **nothing** behind it — which is what a bridge does on a port where it keeps seeing its own addresses. It was also the only port receiving BPDUs. Both anomalies on the same port. The kernel log had said `eth1`, not `ether2`. Establishing that those are the same port cost real time that day; [RouterOS ports and kernel names](/mikroscope/reference/port-names/) is how to do it in one safe step, and why the agent does it for you on this board. ## Step 5 — make a change that tests the hypothesis The hypothesis was "the device on `ether2` has a second path to the router through the other APs, which hang off the switch". Two changes were made on the switch first, and the honest result is that **they did nothing**: | Change | Event rate | | --------------- | ----------------------------------- | | baseline | 1.49 /s | | switch change 1 | 1.67 /s | | switch change 2 | 1.47 /s | | (noise band) | ±0.2 /s | One further reading in the same series, 1.50 /s, was also inside that band. That is a useful outcome, not a wasted step: it ruled out the switch with evidence. The lesson for a playbook is to **always re-measure after a change, even one you expect to work**, and to know your noise band before you interpret a difference. ## Step 6 — the fix, and confirming it two ways The firmware of the mesh APs (Deco units) was updated; the cabling was not touched. Then: ```text events 9 -> 0.03/s (baseline 1.49/s) ``` and of those nine, _none_ was the loop message — they were the APs coming back (`eth1: phy link up`, `eth1: set isolation from 0 to 1`). The loop message, which had appeared every 2 s (30 times a minute, with twice as many blocking/learning records around it), appeared **zero** times in 300 s. The second, independent confirmation is the one worth internalising: the bridge's view became _coherent_. | | Before | After | | ---------------------- | ------ | -------- | | MACs on `ether2` | 0 | **24** | | MACs on `sfp-sfpplus1` | 60 | **36** | | `ether2` `edge` | false | **true** | The same 60 devices, redistributed 36/24 instead of 60/0. While the loop existed, the router was learning nearly every device on the wrong port; the Zigbee coordinator `00:4B:12:96:80:33`, for instance, moved from the SFP+ to `ether2`, where it actually lives. A fix that makes a _second_ unrelated measurement fall into place is a fix you can trust. > **Not established** > > The firmware update removed the loop, and the two measurements above confirm it. Why the previous > firmware looped is not established: the fix was observed, not explained. ## The signature **A real fault, not provoked** · 2026-09-12 - A steady, periodic stream of kernel `events` at idle, where the healthy state is zero. - `received packet on with own address as source address`, with the router's own MAC in it. - Inter-arrival times of 2.00–2.01 s: the STP hello interval. - A busy port on which the bridge has learned no hosts, with `edge=false` while its neighbours are `true`. - `/log/print` and `/interface/bridge/port/monitor` both clean. ## What to take from it - Alarming text is a lead; a rate is evidence; a rate before and after is a conclusion. - Read the inter-arrival times. They often name the protocol for you. - Prefer a change that discriminates between hypotheses over a change that merely might fix things. - Distrust a fix that only your primary instrument can confirm. ## See also - [RouterOS ports and kernel names](/mikroscope/reference/port-names/): how `eth1` became `ether2`, and what the agent ships to spare you the hour. - [The shape of an idle router](/mikroscope/playbooks/idle/): why zero events is the baseline this stream stood out against. - [The router's CPU, the container's network](/mikroscope/limits/namespaces/): why the kernel log is visible from the container and the interface counters are not. --- # A CPU-bound workload A console loop that saturates one core of the reference RB5009 for a minute — why the device total reads 29 %, how the scheduler’s migration shows at 10 Hz, and what the temperature and frequency did. Source: https://jmrplens.github.io/mikroscope/playbooks/cpu/ This page answers what a single-threaded bottleneck looks like in the per-core data, and why the device total hides it. The workload was provoked on purpose. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container ## How it was provoked With no configuration change and no external dependency — a console loop that burns one core and ends on its own: ```text :local i 0; :while ($i < 4000000) do={ :set i ($i + 1) } ``` It ran 58.8 s. ## What the agent recorded In 5-second buckets. The agent ships raw tick deltas; the percentages below were computed from them afterwards. ```text t busy% per-core busy% ctxt/s temp MHz 0 22.3 31.8 7.4 40.4 9.6 9192 36.5 1400 1 30.9 11.7 11.4 89.7 10.6 11325 37.0 1400 2 29.9 14.3 9.4 19.6 76.0 9476 37.0 1400 3 29.1 7.4 9.4 17.6 81.6 10805 37.0 1400 4 29.3 3.1 9.4 4.9 99.8 9257 37.2 1400 8 28.4 2.2 7.0 4.6 99.8 11947 37.2 1400 11 33.0 11.8 11.2 9.0 99.7 12222 37.3 1400 ``` ## What to read - **Total ~29 % is one core of four.** A device total is a trap; always look at the per-core row. "29 % CPU" here means "one core is saturated and three are idle", which for a single-threaded bottleneck is the whole story. - **The scheduler took ~20 s to settle.** Buckets 0–3 show the work moving between cores (40 % → 90 % → 76 % → 82 %) before pinning on core 3 at 99.8 %. If you sample at 1 s or slower you see a vague plateau; at 10 Hz you see the migration. This is the resolution earning its cost. - **Temperature followed: 36.5 → 37.3 °C.** One saturated core is worth ~0.8 °C on this passively cooled board. Small, but it tracks, and it is read from `/sys/class/thermal` with no API call. - **Frequency stayed flat at 1400 MHz.** On this device that is a _configuration_, not an observation: the owner has pinned the clock to maximum, which `/system/routerboard/settings` reports as `Warning: cpu not running at default frequency`. On a device that scales, the `freq_khz` field is where you would see a busy tick that was earned slowly. The agent reads the frequency every tick and stores `freq_khz` only when it changes, plus a heartbeat once every 60 s, so on a pinned clock you get one row a minute, and on a scaling clock you get every step that lasts at least one tick. ## Check the sampler before the CPU Before you conclude anything from a CPU number, confirm `mikroscope_slipped_total` is 0. A slipped tick is one whose read finished after the next tick was due, and the sampler's own accounting is then the first thing you should distrust. ## The signature **Provoked on purpose** · 2026-09-12 - A device total near 100 % divided by the core count — here ~29 % on four cores — with one per-core column at 99.7–99.8 %. - Before it settles, the load visibly hopping between cores for tens of seconds. - A temperature rise of under a degree that tracks the load. > **Not measured, so not claimed** > > A multi-threaded load, a load long enough to reach thermal equilibrium, and any change of clock > frequency: the reference device's clock is pinned, so what a scaling governor does under this loop > was not observed. ## See also - [The resolution floor is the kernel's](/mikroscope/limits/): why a 100 ms sample resolves one core in 10 % steps, and the PMU counters that read beneath it. - [Five minutes with a router](/mikroscope/start/walkthrough/): a recorded script loop pinning one core, with markers naming what else the router was doing. - [Each source at its own floor](/mikroscope/limits/source-floors/): why temperature is read at the zone's declared 1 Hz and frequency is stored only when it changes. --- # A packet flood Ten seconds of ICMP flood at the reference RB5009’s own address — the switch interrupt that stands in for an interface counter, the one core that paid, and the time_squeeze counter worth watching. Source: https://jmrplens.github.io/mikroscope/playbooks/packet-flood/ This page answers what traffic the router's CPU has to handle looks like from inside a container that cannot see the router's interfaces. The flood was provoked on purpose. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container ## How it was provoked ICMP at the router's own LAN address, from a LAN host, bounded to 10 s. This is traffic the router's CPU must handle, unlike LAN-to-LAN traffic, which the switch chip forwards without the CPU ever seeing it: ```sh ping -f -w 10 192.168.0.1 # 67 793 packets, ~6.8 kpps, 0 % loss ``` `192.168.0.1` is the reference router's LAN address; use your own router's. ## What the agent recorded In 5-second buckets, before and during: ```text t busy% per-core busy% sirq squeeze top IRQ 2 4.8 3.8 8.0 3.0 4.2 51 50 switch0=5337 3 5.5 3.9 10.4 3.9 3.9 50 45 switch0=5567 4 9.6 23.3 2.8 3.3 8.8 149 138 switch0=34922 5 14.3 24.9 19.6 3.5 9.2 153 136 switch0=33832 ``` ## What to read - **`switch0` interrupts went 5.5 k → 34 k per bucket**, a 6× rise. This is the closest thing to a per-interface counter that exists inside the container: the network namespace hides `/proc/net/dev`, but the interrupt the NIC raises is global. It will not give you bytes; it will give you onset, offset and which core paid. - **The cost landed on one core** (core 0 at 23–25 %) because that IRQ is pinned. A flood that saturates a single core while three idle is a common and confusing shape — the total says 14 %, the router feels wedged. - **softirq counts tripled (50 → 150)**, and `time_squeeze` rose with them. `time_squeeze` is the one to watch: it counts times the softirq handler ran out of budget with work still queued. Rising squeeze with flat throughput is the signature of a router at its packet-rate ceiling. - **Hard-IRQ time does not appear in `/proc/stat`'s `irq` column** on this kernel — it is always 0 (no `IRQ_TIME_ACCOUNTING`). That work is inside `system`. Do not read the `irq` field and conclude the router has no interrupt load. ## Where these signals live - The interrupt lines are `mikroscope_irq_total{irq,name,cpu}` on `/metrics`, per core, for the sources that appeared in any sample's top-K; `mikroscope_irq_delivered_total` is every source summed, the denominator for what share the top-K accounts for. On InfluxDB the per-core distribution is `mikroscope_irq_cpu`, with a `cpu` tag. Which line a NIC raises and what it is called belong to the board and its driver: match on the `name` label or on the rate, not on `switch0` written into a query. - The squeeze is `mikroscope_softnet_total{cpu,kind="time_squeeze"}`, beside `kind="processed"` and `kind="dropped"`; the softirqs are `mikroscope_softirq_total{cpu,kind}`. - The squeeze has no budget to be divided by: `/proc/sys/net/core/*` (`netdev_budget`, `netdev_max_backlog`) is absent in the container's namespace. The collector flags a sample as `burst` when a softnet queue dropped a packet or squeezed more than that CPU usually does (above its trailing 90th percentile and at least 3), in a sample whose packet count was at or below its trailing median — the kernel's evidence of a burst shorter than the sample interval. Read [the idle shape](/mikroscope/playbooks/idle/) for why "any squeeze" is not that rule. ## Your instrument is part of the system An aside that demonstrates it: running `/system/routerboard/settings/print` over SSH produced kernel events of its own — ```text [4] rb_ioctl, cmd: 0x5212, arg: 0x0 [4] rb: RB_GET_CF_INFO ``` Reading RouterOS configuration leaves a trace in the kernel log. When you are correlating events with your own actions, remember your instrument is part of the system. On this device each SSH connect also costs 20–27 % CPU for its duration. ## The signature **Provoked on purpose** · 2026-09-12 - One interrupt line rising several-fold with a sharp onset and offset. - The cost on the one core that line is pinned to, while the device total stays modest. - softirq counts and `time_squeeze` rising together, against a squeeze that is never zero at idle. > **Deliberately not provoked** > > A flood that reached the router's packet-rate ceiling: this one delivered ~6.8 kpps with 0 % loss. > "Rising squeeze with flat throughput" is how the ceiling is expected to read, and it was not > observed here. Nor was traffic the switch chip forwards between LAN ports, which the CPU never > sees. ## See also - [The shape of an idle router](/mikroscope/playbooks/idle/): the squeeze and interrupt baseline this flood rose from. - [The router's CPU, the container's network](/mikroscope/limits/namespaces/): why the interrupt is visible and the interface counters are not. - [What the collector derives](/mikroscope/sinks/derive/): the `burst` flag and the per-packet costs. --- # Flash wear The reference RB5009 writes to its NAND at idle without being asked — how to see it in the YAFFS counters, trace it to a logging rule, and read the ECC counters that warn before a block is lost. Source: https://jmrplens.github.io/mikroscope/playbooks/flash-wear/ This page answers what is writing to the router's flash, and whether the flash is wearing. Nothing needed provoking: the RB5009 writes to NAND on its own, and the agent's `yaffs` source shows it. The MTD figures are from 2026-09-14. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container ## What the router writes at idle Over one 30-second window at idle, the Main partition took **2 page writes and 14 page reads**; over another, zero. The cause is findable: ```text /system/logging/print where action="disk" # TOPICS ACTION 30 dns disk ``` The `dns` topic logs to disk, which is also why this device's log holds tens of thousands of rows. If the `pw` (page writes) or `er` (erasures) deltas climb, ask what started writing. ## What the fields mean | Field | Meaning | Read it as | | ----------- | ------------------- | ---------------------------------------------------------------------------------------------- | | `er` | erasures | the counter that maps to flash _lifetime_ | | `pw` / `pr` | page writes / reads | the workload | | `gcc` | GC copies | write amplification: `gcc` ≫ `pw` means the filesystem is working hard for each byte you store | | `gc` | garbage collections | how often the filesystem collected at all | | `bad` | bad blocks | a level, and it must stay 0 | | `free` | free chunks | headroom | On `/metrics` the counters are `mikroscope_flash_operations_total{device,kind}` with `kind` one of `page_writes`, `page_reads`, `erasures`, `gc_copies` and `gcs`; the levels are `mikroscope_flash_bad_blocks` and `mikroscope_flash_free_chunks`. At the default per-source floors the counters are read every tick and stored only when a counter moved or the free-chunk level changed, which on the reference device is about 0.04 times a second (a few times a minute). A missing `flash` row in a sample means neither happened, not that the source is absent. ## Boot and Main Both YAFFS devices are reported. The split is instructive: after two weeks, the **Main** partition showed 83 812 page writes and 1 579 erasures, while the **Boot** partition showed **6** page writes and 16 erasures for the device's whole life — Boot is written only by a firmware upgrade. ## The ECC counters: the warning before the loss Under `privileged=yes` the MTD ECC counters are readable too, from `/sys/class/mtd`: `corrected_bits`, `ecc_failures` and `bad_blocks`. All are zero on a healthy device. `corrected_bits` climbing is aging NAND; `ecc_failures` is data loss. The YAFFS bad-block count is the post-mortem — a block is retired only after the ECC has failed on it. The corrected-bit count is the leading indicator, because it climbs as the cells weaken. The kernel publishes the ceiling as well: `bitflip_threshold` is the corrected bits per ECC step at which it moves the data off a block, and `ecc_strength` is the most bits per step the code can correct at all. On the reference RB5009 on 2026-09-14 there were three partitions — `RouterBoard NAND 1 Boot` (8 MiB), `RouterBoard NAND 1 Main` (1 GiB) and `RouterBoot` (1 MiB SPI) — with `corrected_bits`, `ecc_failures`, `bad_blocks` and `bbt_blocks` all 0, and `bitflip_threshold` 12 and `ecc_strength` 16 on the NAND. In a sample they are `mtd` rows (`corr`, `fail`, `bad`, `bbt`, `bitflip_threshold`, `ecc_strength`); on `/metrics`, `mikroscope_mtd_ecc_corrected_bits_total{device,partition}`, `mikroscope_mtd_ecc_failures_total`, `mikroscope_mtd_blocks{kind="bad"|"bbt"}`, `mikroscope_mtd_bitflip_threshold` and `mikroscope_mtd_ecc_strength`. They are cumulative since boot and shipped as read, never differenced, because they move on the scale of a device's lifetime. The agent reads them every 10 s — an arbitrary, generous cadence, not a measured floor. ## Why `--ephemeral` exists This is the source that justifies `--ephemeral`: a deployment with its root and image on tmpfs adds nothing to these counters at all. ## The signature **Needed no provoking** · 2026-09-12 - `pw` and `er` deltas at idle that you did not cause: something is configured to write. Look at `/system/logging` actions set to `disk` first. - `gcc` far above `pw`: the filesystem is paying write amplification. - `corrected_bits` rising, or any `ecc_failures` or new `bad` blocks: the flash itself, not the workload. > **Not measured, so not claimed** > > A wearing flash. Every MTD ECC counter on the reference device read 0 on 2026-09-14, so what a > rising `corrected_bits` looks like over time, and how far ahead of a retired block it warns, were > not observed on this project's hardware. ## See also - [What privileged buys](/mikroscope/limits/privileged/): why the ECC counters need `privileged=yes`. - [Where things go](/mikroscope/install/layout/): what `--ephemeral` puts on tmpfs and what it gives up. - [Alert rules](/mikroscope/dashboards/alerts/): the uncorrectable ECC failure alert. --- # Conntrack without the API The container’s own namespace reports zero tracked connections, but the global slab allocator does not — how to read the router’s real conntrack population and its ceiling from files, and why no storm was provoked to show it. Source: https://jmrplens.github.io/mikroscope/playbooks/conntrack/ This page answers how many connections the router is tracking, without a table scan over an API session. The ceiling and the timeouts were both read on 2026-09-14. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container ## The namespace reports zero The container's own network namespace reports `nf_conntrack_count` = 0 no matter how busy the router is. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-12 · `privileged=yes` does not change the network namespace ## The slab allocator does not The slab allocator is global. Under `privileged=yes` the agent reads `/proc/slabinfo` and reports the `nf_conntrack` cache's active object count, which **is** the router's real conntrack population: ```text slab: {'nf_conntrack': 6240, 'skbuff_head_cache': 1008, 'skbuff_fclone_cache': 400, 'TCP': 327, 'UDP': 180, 'TCPv6': 117, 'UDPv6': 125, 'sock_inode_cache': 645, 'kmalloc-1k': 1152, 'kmalloc-2k': 905} ``` In a sample it is the `slab` map; on `/metrics` it is `mikroscope_slab_active_objects{cache="nf_conntrack"}`. The family is absent without `privileged=yes`, because `/proc/slabinfo` is root-only. A cache the kernel does not have is omitted, not reported as a permanent 0 — the agent also asks for `dst_cache` and `ip_dst_cache`, which do not appear in the output above. `/proc/slabinfo` is the most expensive file the agent parses, so it is read on a 6 Hz budget floor, rounded to whole ticks: every 2nd tick (5 Hz) at the 10 Hz default, every 17th at 100 Hz. 6 Hz is also about the rate `nf_conntrack` was measured changing at. It is stored on change, plus a heartbeat once every 60 s. ## Cross-check it once Cross-check it against the API once, so you trust it thereafter: ```text /ip/firewall/connection/print count-only # 6 212 the day before; slab said 6 287 ``` Two slab readings that day were recorded against it, 6 582 and 6 287; the snippet above is a third moment (6 240). The two will not match exactly — they are sampled at different instants, and the slab counts objects the allocator still holds — but they track. The difference in cost is the point: a file read several times a second versus a table scan over an API session. ## The ceiling is readable too Next to `nf_conntrack_count` in `/proc/sys/net/netfilter`, `nf_conntrack_max` is not per-namespace. It reads **966 656** from inside the container — the same number a read-only `/ip/firewall/connection/tracking/print` reports as `max-entries` on the reference router (2026-09-14). So "how full is the connection table" is answerable without the API: the slab population over that ceiling. The agent reads the ceiling once at start, since it is a sysctl a human edits and not a counter, and ships it beside the population it bounds: `mikroscope_slab_limit_objects` on `/metrics`, `limit` on the InfluxDB slab row. The collector runs two detections on the population and its ceiling: `conntrack-cliff`, when `nf_conntrack` falls below half its previous stored value, and `conntrack-high`, when occupancy is above 80 % of `nf_conntrack_max` **and** rising over the last 60 s, with no time-to-full attached. The same subtree has a trap. The conntrack _timeouts_ there read Linux defaults (`tcp_timeout_established` 432 000 s, against RouterOS's `1d`; read 2026-09-14), so they must never be presented as the router's configuration. ## The neighbouring caches They are worth watching for their own sake: - `skbuff_head_cache` / `skbuff_fclone_cache` — packet buffers in flight. A spike here during a traffic event is memory pressure from the network path, not from anything you installed. - `TCP` / `UDP` / `sock_inode_cache` — sockets the router itself holds. - `kmalloc-1k` / `kmalloc-2k` — where large-allocation storms show up. ## The signature **Needed no provoking** · 2026-09-12 - `nf_conntrack` climbing toward `nf_conntrack_max` while the namespace's own count stays 0. - `nf_conntrack` falling below half its previous stored value, which is what `conntrack-cliff` flags; the detection says where to look, not why it fell. - `skbuff_*` rising with a traffic event, which places the memory pressure on the network path. > **Deliberately not provoked** > > A conntrack storm. Generating thousands of connections against a production router risks tripping > its own firewall or a CrowdSec-style bouncer and locking out the very path you are working > through. The cross-check above gives the same confidence in the counter without that risk, but > what a storm looks like in these caches was not observed. ## See also - [The router's CPU, the container's network](/mikroscope/limits/namespaces/): which counters the network namespace hides, and the slab exception. - [What privileged buys](/mikroscope/limits/privileged/): the root-only files, `/proc/slabinfo` among them. - [Detections](/mikroscope/sinks/detections/): `conntrack-cliff` and `conntrack-high` in full. --- # The resolution floor is the kernel’s Why a CPU reading cannot be finer than the kernel’s 10 ms tick, the one source that reads beneath it, and how far back the agent can remember. Source: https://jmrplens.github.io/mikroscope/limits/ This page answers the question that decides how to read every CPU number mikroscope produces: what is the smallest change it can show, and why is that limit set by the kernel rather than by the agent. It also says which source does see beneath that limit, what the reference kernel does not have, and how long a sample survives in the agent before it is gone. ## Ticks, not time A tick lasts 10 ms. `/proc/stat` does not count time, it counts `USER_HZ` ticks, 100 per second. A 100 ms sample can therefore hold 10 ticks per core, so the busy ratio of one core resolves to 10 % steps, and the average of four cores to 2.5 %. Over 1 s the resolution of one core is 1 %. That is arithmetic from the tick, not a property of the agent, and no rate or setting changes it. What the agent does about it is refuse to hide it: it ships the raw ticks and the real interval of every sample (`dt_ns`), never a percentage, so the window you divide over is yours to choose. Tick accounting is also quantised at the edges. A 100.3 ms interval can carry 11 ticks (seen on the amd64 development host, 2026-09-11/12), which would make a busy ratio above 1. The ratio the agent derives is capped at 1; the ticks themselves stay raw. Sampling faster does not refine this. At 100 Hz a sample holds 0 or 1 busy tick, so the per-sample busy ratio has two possible values; above roughly 20 Hz the tick counters are an occupancy indicator rather than a percentage. What a higher rate does buy is on [the rate ceiling](/mikroscope/cost/rate-ceiling/). ## The one source beneath it: the CPU's own counters One source reads below the tick: the CPU's performance monitoring unit, through `perf_event_open`, which the agent collects as the `perf` source when the container is privileged. It is the only source in mikroscope that does not come from a file. Measured on the RB5009 and recorded on 2026-09-12: in a 100.4 ms sample where `/proc/stat` reported **zero busy ticks on all four cores**, the PMU counted 2.2–4.6 million cycles and 0.8–2.1 million instructions retired, at a 3.8–5.7 % cache-miss rate. The jiffie rounds that work away; the counter does not. The number worth watching is instructions per cycle, and the agent does not compute it: it ships the raw counts and you divide. It is worth the division because it separates a core doing work from a core stalled on memory, which no tick counter can express. Measured over 2 s on the same device, the same day, it ranged from 0.381 on cpu0 to 0.992 on cpu1. What the source is, on the reference device (RB5009, RouterOS 7.24.2, kernel 5.6.3, Cortex-A72 r0p1, 2026-09-12, from inside a privileged container): - The agent asks for seven counters: `cycles`, `instructions`, `cache-references`, `cache-misses`, `branch-instructions`, `branch-misses` and `bus-cycles`. The probe of 2026-09-12 opened `cycles`, `instructions`, `cache-misses`, `branch-misses` and `bus-cycles` system-wide on 4 of 4 CPUs. In the agent's own data over the 24 h ending 2026-09-12, six reported on 4 of 4 cores, `cache-references` among them, and `branch-instructions` produced no rows at all. - The generic `stalled-frontend` and `stalled-backend` events return `ENOENT` on the A72. They would need raw PMU event codes, so the agent does not ask for them. - A counter that cannot be opened is absent, never zero. Which counters open depends on the CPU, so read the `counter` label of `mikroscope_perf_events_total{counter,cpu}` rather than assuming a set. - Without `privileged=yes` the whole family is absent: the counters are opened system-wide, which the unprivileged container cannot do. See [what privileged buys](/mikroscope/limits/privileged/). The collector uses the same two counters for one of its detections, `ipc-collapse`: per core, once at least 20 s of history exists, the one-second instructions-per-cycle falling below half its trailing 60 s median while the cycle rate is above its own median. That is described with the other rules on [detections](/mikroscope/sinks/detections/). ## No finer clock from the kernel Do not expect a finer clock from PSI or `schedstat`. The RB5009's RouterOS 7.24.2 kernel (Linux 5.6.3) has neither `/proc/pressure` nor `/proc/schedstat`, and its `/proc/stat` `irq` column is always 0, so hard-IRQ time is counted inside `system`: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-11 · `/proc/pressure` and `/proc/schedstat` absent Nor is there a tracing path. A privileged discovery container on the same device on 2026-09-12 found no eBPF, kprobes or ftrace: no BTF, no debugfs, no tracefs, and the mount points do not exist. A re-probe on 2026-09-14 found `/sys/fs/bpf` present and empty, and `/proc/modules` listing 238 loaded modules, but no `/lib/modules`, no kernel headers and no compiler on the device: a kernel feature that is missing cannot be added from a container. The agent detects what a kernel has at start and reports it on `/capabilities`: the `sources` map says which sources this deployment actually reads. Fields for an absent source are absent from every sample and every sink, never zero. The agent does read PSI and `schedstat` where a kernel has them. > **Untested** > > The PSI and `schedstat` paths have never run on the RB5009, because its kernel lacks both. The > `schedstat` path ran only on the amd64 development host (kernel 6.12.107, 2026-09-12); the PSI > parser has been tested only against synthetic input. The PMU opened on that development host > (2026-09-12), but which counters it opened was not recorded; on no CPU but the RB5009's Cortex-A72 > is the counter set known. The hEX S, a 32-bit RouterOS build on an ARM64 chip, has not been > measured at all. ## How far back the agent remembers The other hard bound is depth, not resolution. The agent keeps its samples in a ring of `--buffer` seconds, 300 s by default and 10–3600 s allowed, which holds rate × buffer samples. Nothing older exists anywhere on the router. A collector or recorder outage shorter than the ring is backfilled on reconnect: it asks for `since=` and receives every sample it missed. An outage longer than the ring is reported as a gap of known length, never papered over. The agent answers with a `{"gap":{"from":…,"to":…}}` line naming the sequence numbers that are gone, before the samples it still holds; `record` writes it as a marker reading `samples N..M lost`, and `forward` counts it in `mikroscope_collector_gaps_total` and hands it to every sink. A longer ring costs memory in the agent, and the agent refuses one that cannot fit. At start it estimates the ring at 2 560 bytes a line (the mean line was measured at 2 439 B on the RB5009 on 2026-09-12, without the PMU, buddyinfo and MTD sources; a board with more cores or interrupt lines, or more sources, costs more), adds the triggered-capture budget, and exits with an error if the total exceeds the container's `memory.max`. If the total is more than half the Go soft memory limit it starts but logs a warning, because a heap that tight keeps the garbage collector running. How to size both limits is on [the cost of the observer](/mikroscope/cost/). ## See also - [Each source at its own floor](/mikroscope/limits/source-floors/): the sources that are not read on every tick, and the named reason for each. - [What privileged buys](/mikroscope/limits/privileged/): the PMU, the kernel log and the slab caches, and what the container still cannot see. - [The rate ceiling](/mikroscope/cost/rate-ceiling/): what sampling faster buys once the tick has stopped being a percentage. - [What the numbers do not say](/mikroscope/cost/limits/): what `/metrics` can and cannot recover from these samples. --- # The router’s CPU, the container’s network Which kernel files a RouterOS container sees as the router’s and which as its own, why privileged does not change that, and the conntrack count that gets through anyway. Source: https://jmrplens.github.io/mikroscope/limits/namespaces/ A RouterOS container shares the router's kernel, but not all of the kernel's view. This page answers which files inside the container describe the router and which describe only the container itself, what that means for per-interface traffic and connection counts, and why no container setting mikroscope could choose moves the line. ## CPU and memory are the router's The CPU, interrupt, memory and block-device files are global: inside the container they are the router's own, and they are read on every tick (a block device's row is stored only when it did something). Established on the RB5009 (RouterOS 7.24.2, kernel 5.6.3, 2026-09-11): - `/proc/stat`, per core - `/proc/interrupts` and `/proc/softirqs` - `/proc/net/softnet_stat`: drops and time squeezes in the kernel's receive path. It lives under `/proc/net` but counts per CPU, not per namespace. - `/proc/meminfo`, `/proc/vmstat` and `/proc/loadavg` - `/proc/diskstats` An ordinary container also reads these files as the router's, read on 2026-09-12: the two thermal zones under `/sys/class/thermal`, `scaling_cur_freq` per core, the NAND wear counters in `/proc/yaffs` and `/proc/buddyinfo`. The device tree's `model` string (`RB5009`) is not namespaced either, and it is how the agent identifies the board without the RouterOS API. ## The network is the container's own `/proc/net/dev`, `/proc/net/snmp`, `/proc/net/netstat` and `nf_conntrack_count` are per network namespace. Inside the container they describe the container's own veth: 4 packets while the router forwarded millions. The agent does not read them as router data, and `/capabilities` lists them under `namespaced` so a consumer can see they were left out on purpose rather than missed. **`privileged=yes` does not change this.** It drops the container's _user_ namespace, not its network namespace: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-12 · `privileged=yes` does not change the network namespace So per-interface bytes and packets come from the RouterOS API instead, and the collector merges them with the kernel tier on the agent's clock. They are not interpolated from anything the container can see. How that tier is polled, and how little of it is left, is on [the RouterOS API tier](/mikroscope/sinks/api-tier/). The same boundary shows up in other places, each measured on the RB5009: | What | What the container sees | Measured | | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------- | | `/sys/class/net` | only `lo` and the veth; no device for any front-panel port, privileged or not | 2026-09-14 | | `/sys/class/mdio_bus`, `/sys/class/phy` | `mdio_bus` holds only `fixed-0`; `phy` is empty | 2026-09-14 | | `netdev_budget`, `netdev_max_backlog` and the other global `net.core` entries | absent; per-namespace entries such as `somaxconn` are present | 2026-09-14 | | `/proc/net/*` files created by modules | the container's own: `fib_trie` shows only the veth's `/30`, `snmp6` counts its packets | 2026-09-15 | | `nf_conntrack_max` | the router's real ceiling, 966 656, the same as RouterOS reports as `max-entries` | 2026-09-14 | | conntrack timeouts | Linux defaults (`tcp_timeout_established` 432 000 s against RouterOS's `1d`) | 2026-09-14 | The last two rows sit in the same directory and split in opposite directions. `nf_conntrack_max` is read once at start and shipped as the connection table's ceiling. The timeouts are never presented as the router's configuration, because they are not. The absent `netdev_budget` is also why the squeeze panels have no budget denominator: that is the namespace's doing, not the agent's. ## Conntrack: the count that gets through The connection count is the exception. Under `privileged=yes` the agent reads `/proc/slabinfo`, and the slab allocator is global: the `nf_conntrack` cache's active-object count is the router's real conntrack population. On the RB5009 on 2026-09-12 it read 6 582 in the discovery container and 6 287 from the agent the same day, while the container's own namespace reported 0; the RouterOS API had counted 6 212 the day before. That replaces an API table scan with a file read, and it is why the collector never polls the conntrack count unless `--conntrack-every` is set (off by default in every `--api-mode`). With the ceiling above, "how full is the connection table" is answerable from the container alone. What it cannot tell you is what the connections are. `/proc/slabinfo` counts objects in a cache and nothing else: no protocol, no address, no state. A conntrack flood and a legitimate burst of many connections (a torrent, say) look the same in it. Without `privileged=yes` the file is unreadable and the count is gone; see [what privileged buys](/mikroscope/limits/privileged/). How to cross-check it once against the API is on [conntrack without the API](/mikroscope/playbooks/conntrack/). ## Mounting the host's paths does not cross the boundary Whether bind-mounting the router's own paths into the container gets past the namespaces was tested directly on the RB5009 on 2026-09-15, with the owner's consent: a privileged container with host `/proc`, `/sys` and `/` mounted. - Host `/proc` mounts but reads **zero PIDs**. RouterOS renders a fresh procfs at the mount point, so the PID namespace holds and per-process CPU for RouterOS's own processes stays out of reach. - Host `/sys` mounts but has no `class/net`. Sysfs networking is per network namespace and the mount does not carry the host's, so per-interface counters stay API-only. - Host `/` works, and exposes the RouterOS flash filesystem: configuration and files, not live telemetry. A privileged container with `/` mounted reads the whole configuration, secrets included. mikroscope's installer mounts nothing into the container and must never do this. Namespaces are kernel boundaries, and a filesystem mount does not cross them. > **True of this device, not of yours** > > Every row on this page was read on one RB5009 running RouterOS 7.24.2. Which files are namespaced > is a property of the kernel and of how RouterOS builds its containers, and it has not been checked > on another RouterOS version or another board. ## See also - [The RouterOS API tier](/mikroscope/sinks/api-tier/): where per-interface traffic comes from, since the container cannot see it. - [What privileged buys](/mikroscope/limits/privileged/): the user namespace it drops, and the two it leaves in place. - [Conntrack without the API](/mikroscope/playbooks/conntrack/): reading the slab count and checking it against RouterOS once. - [RouterOS ports and kernel names](/mikroscope/reference/port-names/): naming a port the container cannot see as a device. --- # What privileged buys Why the agent’s container runs with privileged=yes by default, the sources that setting makes readable, and the network, processes and sensors it still does not reach. Source: https://jmrplens.github.io/mikroscope/limits/privileged/ `install` creates the agent's container with `privileged=yes`. This page answers what that setting changes on the router, which of the agent's sources depend on it, what it does not open however it is configured, and what you give up by turning it off. ## The default, and why The container runs `privileged=yes` unless you pass `--privileged=false` to `install` or `upgrade`, which recreates the container. It is a boolean flag, so it takes `=false`; a separate `false` argument is not read as its value. The setting exists on RouterOS 7.24 and later. Reading the device's internal state is what mikroscope is for, and privileged is what makes the root-only kernel files readable. An ordinary RouterOS container is placed in a user namespace where its root maps to host uid 32768, so `/proc/slabinfo` and `/dev/kmsg` come back `EACCES`. `privileged=yes` drops that user namespace. It is still a real grant of privilege, which is why `plan` and `install --dry-run` print it with the rest of the container's settings before anything is written. ## What it adds Found readable on the RB5009 (RouterOS 7.24.2, kernel 5.6.3) on 2026-09-12, and read by the agent except where the table says otherwise: | Source | What it gives | Where it shows | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | | `/dev/kmsg` | the kernel ring buffer as timestamped events, naming the router's real interfaces | `mikroscope_kmsg_records_total{level}`, and Loki | | `/proc/slabinfo` | the global slab caches: `nf_conntrack` (the router's real connection count), `skbuff_*`, `TCP`/`UDP` sockets, `kmalloc-1k`/`-2k` | `mikroscope_slab_active_objects{cache}` | | `/sys/class/mtd` ECC counters | `corrected_bits` and `ecc_failures` per NAND partition | `mikroscope_mtd_ecc_corrected_bits_total`, `mikroscope_mtd_ecc_failures_total` | | the PMU, via `perf_event_open` | cycles, instructions, cache references and misses, branch misses, bus cycles, per core, system-wide | `mikroscope_perf_events_total{counter,cpu}` | | `/proc/pagetypeinfo` | readable, but the agent does not read it | nowhere | The PMU needs privileged for a different reason from the files: its counters are opened system-wide (`pid = -1`), which takes `CAP_PERFMON` or `CAP_SYS_ADMIN` against the host. `perf_event_paranoid` reads 2 on the reference device and does not block a privileged container. What the PMU shows that no tick counter can is on [the resolution floor](/mikroscope/limits/). The kernel log is read without blocking and capped at 64 records per tick. It is opened at the end of the buffer, so a boot-time backlog is never replayed as if it had just happened. `mikroscope_kmsg_dropped_total` counts loss events, not records: one per tick that hit the 64-record cap (that tick's one extra read is discarded, and the rest of the backlog is read on later ticks), and one per kernel ring overrun, which can stand for many records. While it is non-zero, the per-severity count under-counts. Whether this deployment got them is not left to inference. `/capabilities` carries `"privileged": true` only when both `/proc/slabinfo` and `/dev/kmsg` opened, and `mikroscope_device_info{privileged="true"}` says the same on `/metrics` and in every sink's device-info stream. Without privileged, the slab, PMU and MTD families are absent, not zero. The kernel-log family appears only once a record has been seen, so its absence alone cannot tell an unprivileged deployment from a quiet kernel; the `privileged` flag can. ## What it does not add It adds **no** network access and **no** view of RouterOS's processes. Privileged drops the user namespace and leaves the network and PID namespaces in place: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-12 · `privileged=yes` does not change the network namespace So per-interface counters still come from the RouterOS API, and per-process CPU is out of reach even with the host's `/proc` mounted into the container. Both are measured on [the router's CPU, the container's network](/mikroscope/limits/namespaces/). It adds no extra capabilities either: an unprivileged container already holds all 38. What changes is the user namespace those capabilities are confined to, not the set. And it cannot give you more sensors than the board has. On the RB5009, `/sys/class/hwmon` is empty even privileged. The two thermal zones, `cpu-thermal` and `soc-thermal`, are the entire sensor set of the base model, and both are readable without privileged: there is no voltage, current or fan reading to be had. RouterOS's own `/system/health` on that board reports exactly one sensor, `cpu-temperature`, which is the `soc-thermal` zone truncated to whole degrees: a mean offset of +0.55 °C over the 8 minutes where both series existed (2026-09-14). On a board that does have voltage, current or fan sensors the API still adds them. > **True of this device, not of yours** > > The sensor set, the capability count and the uid mapping were read on one RB5009 on RouterOS > 7.24.2. A board with hwmon sensors, or a later RouterOS that builds containers differently, has > not been checked. ## Running without it `--privileged=false` keeps everything that is readable from an ordinary container: every global `/proc` file in the samples, the thermal zones, `scaling_cur_freq`, `/proc/yaffs`, `/proc/buddyinfo`, `/proc/diskstats`, and the agent's own cost. What goes: - The kernel log. The [layer-2 loop](/mikroscope/playbooks/loop/) this project found on its own reference router was visible only there. - The router's connection count from the slab cache. The RouterOS API's `count-only` table scan is the remaining source, which the collector runs only when `--conntrack-every` is set. - The NAND ECC counters. The YAFFS wear counters remain. - The PMU, and with it everything beneath the 10 ms tick. > **Privileged is not a mount** > > A privileged container with the host's `/` mounted reads the whole RouterOS configuration, secrets > included (tested on the RB5009 on 2026-09-15). The installer does not mount any host path into the > agent's container, and nothing here needs one. ## See also - [The router's CPU, the container's network](/mikroscope/limits/namespaces/): the two namespaces privileged leaves in place, file by file. - [What runs where](/mikroscope/security/): the agent's container and what it can reach, from the security side. - [Conntrack without the API](/mikroscope/playbooks/conntrack/): the slab count privileged makes readable, and how to check it. - [Flash wear](/mikroscope/playbooks/flash-wear/): the YAFFS counters that need no privilege and the ECC counters that do. --- # Each source at its own floor Which sources the agent reads on every tick and which it reads or stores less often, the named reason each one reports, and FLOOR_HZ, the one setting that turns every floor off. Source: https://jmrplens.github.io/mikroscope/limits/source-floors/ Not every source changes as fast as the sampler ticks, and recording a value more often than the hardware refreshes it is storage for no information. This page answers which sources the agent reads and stores at the sampler rate and which it does not, the reason each one gives for the difference, where those cadences came from, and how to turn every one of them off to measure your own device. ## Counters are never floored CPU ticks, interrupts, softirqs, the vmstat counters, the PMU and the memory levels (which moved about 24 times a second on the reference device) are read and shipped on every tick. For a counter a delta of zero is real information, the core was idle, so there is nothing to skip. Three more are read every tick with their own rule: - **NAND wear (`/proc/yaffs`) and block I/O (`/proc/diskstats`)** are counters and cheap to read. A device's row is stored only when its delta is non-zero, which on the reference device is a few times a minute. The RB5009 also lists sixteen idle `nbd` devices, and a row per tick for each would be payload and nothing else. - **The kernel log (`/dev/kmsg`)** is drained every tick and never slowed. An event's value is its timestamp, and a marker delayed by a second no longer lines up with the spike it explains. ## A level source is slowed only for a named reason A level source (a temperature, a clock frequency, a cache population) may be read or stored less often than every tick, but only for a reason the agent names. It is never slowed because it was seen changing slowly, since that would measure one night on one device. Each level source reports its cadence and reason on `/capabilities` under `cadences`, on `/metrics` as `mikroscope_source_cadence_hz{source,reason}`, and to every sink in the [device-info stream](/mikroscope/sinks/device-info/), so a consumer reads the true cadence of a field instead of inferring it from the data. The cadence reasons a source can report: | `reason` | What it means | | --- | --- | | `rate` | read at the full sampler rate; nothing the device declares justifies less | | `declared` | the device publishes its own refresh cadence, and reading faster returns the same value with new dither | | `policy` | a setting says the value cannot move on its own: a `userspace` cpufreq governor | | `budget` | a measured parse cost | | `change` | read every tick, stored only when it moves | | `override` | `FLOOR_HZ` is set, and every level source is on its one cadence | `rate` describes the read cadence only, so `/proc/buddyinfo` reports `rate` although it is stored on change. The MTD counters report `budget` although no parse cost was measured for them. Both are how the code labels them, and neither matches the table above exactly. ## Source by source | Source | Read | Stored | `reason` | | ------------------ | ------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------- | | thermal zones | at the zone's declared `polling_delay`, else every tick | every read | `declared`, or `rate` where the board declares none or the sampler rate is no faster than the declared cadence | | `scaling_cur_freq` | every tick | on change, or heartbeat | `change`, or `policy` under a `userspace` governor | | `/proc/slabinfo` | at about 6 Hz | on change, or heartbeat | `budget` | | `/proc/buddyinfo` | every tick | on change, or heartbeat | `rate` | | MTD ECC counters | every 10 s | on change, or heartbeat | `budget` | A slower cadence is a whole number of ticks: the sampler rate divided by the floor, rounded to the nearest integer and never less than one. So the reported rate is what actually happens, not the nominal floor. **Temperature.** On the reference RB5009 both zones declare a `polling_delay` of 1 000 ms (`polling-delay-passive` 250 ms, read 2026-09-14), so the kernel itself re-reads the sensor at 1 Hz; at 10 Hz the agent reads every 10th tick. The sensor quantises to about 0.42 °C steps and the raw reading dithers across a boundary tens of times a second. Storing on change would store that dither as signal; sampling and holding at the declared cadence captures the real curve and drops the dither. On a board that declares no cadence the zones are read every tick. **CPU frequency.** A frequency step is a real DVFS transition, not noise, so the frequency is read every tick and stored when any core moves. On a device with a pinned clock that stores nothing after the first reading but the heartbeat; on a scaling device it catches every move. The reference RB5009's governor read `userspace` on 2026-09-14, so by the rule above its reason reads `policy` there. **`/proc/slabinfo`** is the expensive one: 13 833 bytes and 129 lines per read on the reference device (2026-09-14), the largest per-tick parse by an order of magnitude. That cost is the reason it is slowed at all; the 6 Hz it is slowed to is the rate its fastest cache, `nf_conntrack`, was measured changing. It is stored on change. At 10 Hz that is every 2nd tick (5 Hz); at 100 Hz, every 17th (about 5.9 Hz). How that amortisation shows in the per-sample cost is on [the rate ceiling](/mikroscope/cost/rate-ceiling/). **`/proc/buddyinfo`** is about 100 bytes, among the cheapest files the agent reads, and has no floor because none has been measured. The free lists churn with every allocation, so on a busy router it will be stored on most ticks, and that is the measurement rather than noise. **MTD ECC counters** are read every 10 seconds. That is not a measured floor: the counters move on the scale of a device's life (all zero on the reference board after years), and each read is six small sysfs files per partition, so ten seconds is an arbitrary but generous choice. The code still labels that cadence `budget`, although no parse cost was measured for it. They need `privileged=yes`. ## The heartbeat, and why a gauge does not vanish Every source stored on change is also re-emitted about once every 60 seconds (on the first read due after 60 s), so a value that sits still for an hour still has a recent row in any store. The container's own `memory.max`, a constant, is re-emitted in the samples once per heartbeat (every tick under `FLOOR_HZ`); the `limits` on `/capabilities` and the [device-info stream](/mikroscope/sinks/device-info/) carry it from start. On `/metrics`, a floored gauge between emissions holds the last reading, because a level's value between readings is the last one read, not nothing. `mikroscope_source_age_seconds{source}` says how old that held reading is. The change filter is re-armed once the sampler has taken its baseline reading. Measured against the fixture tree on 2026-09-15, the three floored families and `mikroscope_slab_limit_objects` were present in 6 of 6 scrapes from 5 s after start. ## Where the floors come from, and what they are not One floor comes from a 10.5 h capture at 50 Hz on the reference RB5009 that measured how often each source actually changes: the 6 Hz that `/proc/slabinfo` is slowed to. That capture is one board, one idle night, its clock pinned at the owner's setting. "cpufreq never changed" means it did not change _that night_. A router with a scaling clock, a busy dirty-page workload, or a different board has different floors. The floors are named constants in `internal/agent/source.go` rather than numbers buried in the logic. > **True of this device, not of yours** > > Every cadence on this page is the reference RB5009's, on RouterOS 7.24.2. Before trusting any of > them on another device or another workload, re-measure with `FLOOR_HZ` equal to the sampler rate. ## `FLOOR_HZ`: every floor off at once Every floor is overridable at once, without a rebuild: ```sh mikroscope install --rate 10 --floor-hz 10 # writes FLOOR_HZ=10 into the agent's envlist ``` `FLOOR_HZ` is the agent's environment variable; `--floor-hz` is the deployment flag (`install`, `upgrade`, and `plan` for the listing) that writes it into the container's envlist, only when it is above zero. It is one global setting in hertz, from 0 to 1000. - `0`, the default, keeps the per-source floors above. - Any `N > 0` puts the thermal zones, `/proc/slabinfo` and the MTD counters on one cadence of N Hz, and turns off the store-on-change filter for every level source, so nothing is held back and a capture sees every read. Every level source then reports the reason `override`. - `N` equal to or above `--rate` reads and emits everything every tick. That is the configuration the floors were measured from, and the one to re-run before trusting any number on this page on your device. Two things `FLOOR_HZ` does not change. The per-device row filter on `/proc/yaffs` and `/proc/diskstats` stays: a device whose counters did not move still has no row, which loses nothing because the delta was zero. And `scaling_cur_freq` and `/proc/buddyinfo` are read every tick with or without it; with `N` below `--rate` they are emitted every tick, but their reported cadence is the override's, not the sampler rate. What it costs to read everything every tick on the RB5009, at 50 and 100 Hz, is two of the runs on [the rate ceiling](/mikroscope/cost/rate-ceiling/). ## See also - [The rate ceiling](/mikroscope/cost/rate-ceiling/): the runs with and without `FLOOR_HZ`, and why per-sample cost falls as the rate rises. - [The device-info stream](/mikroscope/sinks/device-info/): where each source's cadence and reason reach a store. - [Environment variables](/mikroscope/reference/environment/): `FLOOR_HZ` beside the agent's other settings. - [The resolution floor is the kernel's](/mikroscope/limits/): the floor no setting can move. --- # What runs where Which piece of mikroscope runs on the router and which on your host, what each one reaches, and where every credential lives. Source: https://jmrplens.github.io/mikroscope/security/ This page answers the question to ask before putting anything on a production router: what mikroscope runs on it, what it runs on your own machine, what each piece can reach, and which credential sits where. The short form is that the router holds an agent that listens and never connects out, and every credential that opens something other than the agent stays on your host. ## What runs where | Piece | Runs | Reaches | Credentials | | ---------------------------------------------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | | `mikroscope-agent` | in a scratch container on the router | serves HTTP on the veth address only; **no outbound connection** | none it presents; an optional bearer token it _requires_ | | `mikroscope doctor`, `install`, `upgrade`, `uninstall`, `status` | your host, when you run them | the router over your own admin ssh; `install`, `upgrade` and `status` also probe the agent's `/healthz` | your ssh key | | `mikroscope plan`, `install --dry-run` | your host | nothing: they build the image and print the listing without connecting to the router | none, unless `plan --rsc` is given a token (below) | | `mikroscope record`, `forward` | your host | the agent over HTTP; the router's binary API for `forward`'s API tier, the relay transport and `record --log-markers` | the agent's token if one is set; a dedicated read-only API user | | `mikroscope mark` | your host | the recording's local files; the router's binary API only with `--log-markers`; never the agent | the API user, only with `--log-markers` | | sinks | your host, inside `forward` | serve `/metrics` for your Prometheus; push to InfluxDB and the other destinations you name | their tokens, in your environment | | `mikroscope dashboards import`, `check` | your host | your Grafana | `GRAFANA_TOKEN`, in your environment | `plan` and `install --dry-run` never reach the router, so they do not run the ownership checks either; those run only on a real `install`. Nothing about the tool is reported anywhere. There is no update check. In the code, the agent's only network code is its HTTP server. The outbound connections all live in the CLI — the sinks (HTTP, TCP or UDP), the Grafana client, the RouterOS API client, the direct transport and the `/healthz` probe — and each goes only to an address you gave it or, for the agent, to the `.2` of `--subnet` (`172.30.10.2` unless you change it). The collector's own Prometheus exposition, `forward --prom
`, listens on the address you pass and serves `/metrics` with no authentication. Bind it to an address only your Prometheus reaches. ## Credentials stay off the router **Over the binary API, `/container/print` returned every property of every container, `cmd` and `envlist` included, to a user with only `read,api`** (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11; only the property names were printed). Reading the envlist entries' values, in `/container/envs`, as such a user was not checked separately. The design assumes they are readable too: whatever is in an envlist is treated as readable by every `read` user on that router, not only by administrators. That is why the agent has no push sink: a sink token on the router would be readable by every `read` user. If push ever comes, it will be agent → collector with the same NDJSON, never agent → InfluxDB. What `install` does put in the envlist `-env` is configuration, and nothing that opens anything else: The entries install writes into the agent's envlist: | Key | Written | From | Holds | | --- | --- | --- | --- | | `MIKROSCOPE_TAG` | always | `--name` | the ownership marker `mikroscope: (managed by mikroscope)`, written first and removed last; the agent ignores it | | `RATE_HZ` | always | `--rate`, default `10`, 1–100 | the sampler rate, in Hz | | `BUFFER_S` | always | `--buffer`, default `300`, 10–3600 | the ring's length, in seconds | | `PORT` | always | `--port`, default `9123`, 1–65535 | the agent's HTTP port | | `ADDR` | always | `--subnet` | the agent's address, the `.2` of the /30; the agent binds only there | | `MEM_LIMIT_MB` | always | `--mem-limit-mb`, default `40`, 8–1024 | the agent's Go soft memory limit, in MiB | | `FLOOR_HZ` | only when above 0 | `--floor-hz`, default `0`, 0–1000 | one cadence for every level source, in Hz | | `CAPTURE_MB` | always | `--capture-mb`, default `4`, 0–256 | the triggered-capture budget, in MiB; `0` turns captures off | | `TRIGGERS` | only when set | `--triggers` | the trigger conditions; unset, the agent uses its default set | | `TOKEN` | only when set | `--token` | the bearer token the agent requires, from `--token` or `MIKROSCOPE_TOKEN`, with or without `--expose` | The last row is the one secret that does live on the router. It is written whenever `--token` or `MIKROSCOPE_TOKEN` is set, with or without `--expose`, and like the rest of the envlist it is treated as readable by every `read` user. It opens the agent's own HTTP paths and nothing else. `plan` and `--dry-run` print it as `value="(token)"`, the options line as `token=(set)`, and the agent's start line in the router log as `token=true`. `plan --rsc` is the exception to that masking. It writes the install as a RouterOS script to run on the router itself, so the envlist line has to carry the real token; the script says so in its own header. A generated `.rsc` with a token in it is a credential — [what the installer refuses](/mikroscope/security/installer/) has how to handle it. ## Where the collector's credentials live On your host, the credentials that open something other than the agent are read from the environment only, never from a flag. The code gives the reason: a flag is visible in `ps` and in a shell history. - The API user's password: `MIKROSCOPE_API_PASSWORD`. The address and user come from `--api` and `--api-user`, or `MIKROSCOPE_API_ADDR` and `MIKROSCOPE_API_USER`. - Sink credentials: `MIKROSCOPE_INFLUX_TOKEN`, `MIKROSCOPE_LOKI_TOKEN`, `MIKROSCOPE_OTLP_TOKEN`, `MIKROSCOPE_ELASTIC_AUTH`, `MIKROSCOPE_TELEGRAF_TOKEN`. - Grafana: `GRAFANA_TOKEN`. The agent's token is the exception: it has a `--token` flag as well as `MIKROSCOPE_TOKEN`. The same reasoning applies to it, so prefer the variable. The CLI does not read `.env` itself. Export the variables into the shell that runs it, for example with `set -a; . ./.env; set +a`. ## Where the agent image comes from The container runs an image, and which route puts it there decides what you are trusting. - `install` with a Go toolchain and a checkout builds the image on your host from the source in front of you and uploads it over your own ssh session. You trust your own tree. - `install --agent-tar ` uploads the tar the release publishes, over the same ssh session. The CLI checks that the tar is a mikroscope agent image of the architecture `--arch` names before it sends it; checking that it is the file the release published is yours to do, against `checksums.txt`. - `install --remote-image ` uploads nothing: the router itself fetches the image from the registry its global `/container/config registry-url` names. You trust that registry and the router's path to it, and mikroscope verifies nothing about what arrives. - `plan --rsc` writes the same commands as a RouterOS script for you to paste or `/import`; the image still has to come from one of the two routes above that need no upload from the CLI. ## The container `install` creates one container with these settings, all printed by `plan` before anything is written: - **`privileged=yes` by default**, `--privileged=false` to opt out. The setting needs RouterOS 7.24 or later. It drops the container's user namespace (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-12), which is what makes the kernel log, `/proc/slabinfo`, `/proc/pagetypeinfo` and the MTD ECC counters readable. It does **not** drop the network or PID namespace: no interface counters, no router conntrack table, no view of RouterOS's processes. [What privileged buys](/mikroscope/limits/privileged/) has the measurements. - **`memory-max=64M`** (`--memory-max`), enforced as the container's cgroup limit, with the agent's Go soft limit at 40 MiB (`--mem-limit-mb`) inside it. - **`restart-policy=on-failure`**, bounded to five retries ten seconds apart, so a broken image cannot loop at boot. - **`start-on-boot=yes`**, or `no` with `--ephemeral`, whose root lives on the tmpfs disk and does not survive a reboot. - **`logging=yes`**, so the agent's lifecycle lines reach the router log. - **`ignore-remote-image-change=yes`**: with the default, RouterOS watches the image and, once the tar is removed, stops and removes the container and re-extracts it minutes later (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). `install` removes the tar right after extraction, which is why it sets this. - Root and image on the disk you chose with `--disk`: the internal flash by default. - **No bind mount.** `install` mounts no host path into the container. The agent reads `/proc`, `/sys` (`/sys/fs/cgroup` for its own accounting, `/sys/class/thermal` and `/sys/class/mtd` for the device), and — privileged — `/dev/kmsg` and the hardware performance counters. It writes nothing to its root at runtime: the ring and the triggered captures are held in memory. It catches SIGTERM, because RouterOS kills a container that does not, at once. The last setting in the list is deliberate. On 2026-09-15 a privileged container on the reference RB5009 was given host `/proc`, `/sys` and `/` as bind mounts, to see whether a mount buys more access. Host `/proc` mounted but read zero PIDs, and host `/sys` had no `class/net`: the namespaces held. Host `/` did work, and it exposes the RouterOS flash filesystem — configuration and files, including secrets. mikroscope does not do this, and a container you build yourself should not either. > **Untested** > > Reboot survival of the persistent install is untested: the reference router is production and is > not rebooted for tests. The container settings above were verified on one RB5009 running RouterOS > 7.24.2; no other board or RouterOS version was tried. ## See also - [The API user](/mikroscope/security/api-user/): the RouterOS user the collector needs, and the policy it gets. - [What --expose opens](/mikroscope/security/expose/): the two firewall rules, and why the token becomes mandatory. - [What the installer refuses](/mikroscope/security/installer/): the objects it will not build on, and the values it will not put in a command. - [What privileged buys](/mikroscope/limits/privileged/): what the default privilege grant reads, and what it does not. --- # The API user The dedicated RouterOS user the collector logs in as, the policy it needs for each command, and what that policy lets it read. Source: https://jmrplens.github.io/mikroscope/security/api-user/ The agent needs no RouterOS account. Three things on your host do: the collector's API tier, the relay transport, and the router-log markers. This page answers which of them needs which policy, how to create a user that has that and nothing more, and what such a user can still read. ## Which commands need it | Used by | Runs over the binary API | Policy | | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | | `forward`'s API tier | `/interface/monitor-traffic`, `/interface/print` (`name`, `default-name`, `type`, `comment`, `actual-mtu`), `/interface/list/member/print` (`list`, `interface`), `/interface/bridge/port/print` (`interface`, `bridge`), `/interface/print stats-detail`, `/interface/ethernet/print stats`, `/system/resource/print`, `/system/resource/cpu/print`, `/system/health/print`, `/ip/firewall/connection/print count-only` | `read,api` | | `record --log-markers`, `mark --log-markers` | `/log/print` (`time`, `topics`, `message`) | `read,api` | | the relay transport (`--transport relay`, or `auto` when direct does not answer) | `/tool/fetch output=user` against the agent's address | `read,api,test` | `/tool fetch` and `/tool profile` both require the `test` policy (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11). mikroscope uses `/tool fetch` for the relay and does not use `/tool profile`. `read,api` is enough for the API tier and for `--log-markers`. `test` is needed only when the relay is used — `--transport relay`, or `auto` when the direct transport does not answer — whatever `--api-mode` is. With `--api-mode off` the tier needs no user at all. Not every call in the first row runs every time. `/system/health/print` is skipped in `slow` and with `--no-health`; the conntrack `count-only` runs only when `--conntrack-every` is above 0, and its default is 0 even in `full`; `monitor-traffic` runs only when `--interfaces` is set, while the three configuration reads behind the interface inventory run whenever the tier is on, once before the first kernel pull and again every `--labels-every`; the two stats reads follow `--counters-every`, 10 s by default. Which calls run at which cadence is on [the RouterOS API tier](/mikroscope/sinks/api-tier/). Without an address and a user, each command says so differently: - `forward` runs the kernel tier alone and logs `api tier disabled: …`. - `mark --log-markers` and `--transport relay` fail with `the RouterOS API needs --api, --api-user and MIKROSCOPE_API_PASSWORD`. - `record --log-markers` keeps the recording, prints `log markers: the RouterOS API needs …` on stderr and exits 0. - `--transport auto`, when the direct transport does not answer, fails with ``direct transport did not answer and the relay is not configured: the RouterOS API needs … (or `install --expose`)``. That check looks only at `--api` and `--api-user`. An empty `MIKROSCOPE_API_PASSWORD` is not caught there; it shows up as a failed login, `api
: …`. ## The group and the user A group that grants `read`, `api` and `test` and denies every other policy by name, and a user in it restricted to the collector's address: ```text /user/group/add name=mikroscope policy=read,api,test,!write,!ftp,!local,!telnet,!ssh,!reboot,!policy,!winbox,!password,!web,!sniff,!sensitive,!romon,!rest-api /user/add name=mikroscope group=mikroscope password= address=/32 ``` Drop `test` from the group if you will never use the relay. Restrict `address=` to the collector host, and restrict `/ip/service` for `api` to your LAN. Both are RouterOS-side limits on where the password is accepted from, and neither depends on mikroscope. Make this user for mikroscope rather than reusing one made for another tool. A least-privilege API user built for something else typically belongs to a group that denies `test`, so it cannot run the relay, and widening that group widens it for the other tool as well. ## What `read` still reads `read` is not narrow. **Over the binary API, `/container/print` returned every property of every container, `cmd` and `envlist` included, to a user with only `read,api`** (the user `mikroscope`, RB5009UG+S+, RouterOS 7.24.2, 2026-09-11; only the property names were printed). Reading the envlist entries' values in `/container/envs` was not checked separately; the design assumes this user can, and so treats the agent's `TOKEN` entry, when one is set, as readable by it. The collector narrows what it asks for where it reads configuration. The three interface-inventory reads name their properties — `name`, `default-name`, `type`, `comment` and `actual-mtu`; `list` and `interface`; `interface` and `bridge` — and no other field, so none of them brings back a field that could carry a secret; `/system/resource`, `/system/resource/cpu`, `/system/health` and `/log/print` carry a `.proplist` as well. The counter reads — `monitor-traffic`, `stats`, `stats-detail` — and the conntrack `count-only` do not. None of this limits what the user is allowed to request: it can still ask for every container's properties. ## How the CLI passes it | Setting | Flag | Environment variable | | -------- | ------------ | ------------------------- | | address | `--api` | `MIKROSCOPE_API_ADDR` | | user | `--api-user` | `MIKROSCOPE_API_USER` | | password | none | `MIKROSCOPE_API_PASSWORD` | The password has no flag on purpose: a flag is visible in `ps` and in a shell history. `.env.example` gives the address as `192.168.88.1:8728`. > **The API connection is not encrypted** > > `record`, `mark` and `forward` connect to the plain binary API. The vendored client can dial TLS, > but no flag uses it, so the login and every reply cross the network between the collector and the > router unencrypted. `address=` limits where the login is accepted from; it does not change that. ## See also - [What runs where](/mikroscope/security/): every piece, what it reaches and which credential it holds. - [The RouterOS API tier](/mikroscope/sinks/api-tier/): what the collector reads over this user, and the `off`, `slow` and `full` presets. - [Reaching the agent](/mikroscope/install/reaching-the-agent/): the direct and relay transports, and when the relay needs `test`. --- # What --expose opens The two tagged firewall rules install --expose adds, who can reach the agent through them, and why the token becomes mandatory. Source: https://jmrplens.github.io/mikroscope/security/expose/ `--expose` is the one install option that changes the router's firewall beyond the two list memberships every install adds. This page answers exactly what it writes, who can then reach the agent, what the token protects and what it does not, and how the rules come off again. **What `install --expose` adds** - two firewall rules, tagged - a token becomes mandatory - `uninstall` and `status` see the two rules only when given `--expose` again Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. ## Without it The agent binds to the container's address, the `.2` of `--subnet` (`172.30.10.2` by default), on `--port` (`9123`). Without `--expose` it is reachable only from hosts the router routes to that veth /30. On the reference RB5009 the interface-list and address-list memberships every install adds were enough for a LAN host to reach it directly; [the two firewall traps](/mikroscope/install/firewall/) explains why those two are needed. ## The two rules `install --expose --lan-address --token …` adds, after the list memberships and before the container, a dst-nat from the router's LAN address on the agent port to the veth: ```text /ip/firewall/nat/add chain=dstnat dst-address= protocol=tcp dst-port=9123 action=dst-nat to-addresses=172.30.10.2 to-ports=9123 comment="mikroscope:mikroscope (managed by mikroscope)" ``` and a forward accept for that flow, placed before the first `chain=forward action=drop` rule, or appended when the forward chain has no drop: ```text /ip/firewall/filter/add chain=forward dst-address=172.30.10.2 protocol=tcp dst-port=9123 connection-nat-state=dstnat action=accept comment="mikroscope:mikroscope (managed by mikroscope)" place-before= ``` The addresses and port above are the defaults, and `` stands for the lookup the real command does on the router before it adds the rule; `plan` prints both commands exactly as they will run, with your values. Both carry the tag, and both were verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11: the LAN reached the agent through the router's own address, and both rules were removable by tag. `--lan-address` must be an IPv4 address; `install` refuses `--expose` without one (`--expose needs the router's IPv4 LAN address`). An existing rule matching the same chain, destination address, port and protocol that does not carry the tag stops `install`, as any foreign object does — see [what the installer refuses](/mikroscope/security/installer/). ## Who reaches the agent through them Every LAN host can then reach the agent at `:9123`. Neither rule restricts the source: the dst-nat has no `in-interface` and no `src-address`, and the accept matches only the destination, the port and `connection-nat-state=dstnat`. Which hosts get through is decided by which hosts can send a packet to the router's LAN address, and by whatever your other rules do before these. > **Untested** > > Only a LAN host reaching the router's LAN address was tested. Whether anything outside the LAN can > reach that address on your router depends on the rest of your firewall, which mikroscope neither > reads nor changes, and no such path was tried. ## The token Because the agent is no longer reachable only through the veth, the token is mandatory: `install` refuses `--expose` without one (`--expose makes the agent reachable from the LAN: a token is mandatory`). With a token set, every endpoint the agent serves except `/healthz` returns `401 token required`, with `WWW-Authenticate: Bearer`, unless the request carries `Authorization: Bearer `: `/capabilities`, `/snapshot`, `/stream`, `/metrics`, `/captures`, `/captures/{id}` (including `DELETE`) and `POST /capture`. A path the agent does not serve gets `404`, and a wrong method `405`, token or not. The agent strips an optional `"Bearer "` prefix before comparing, so a header holding the bare token is accepted too. `/healthz` stays open. It returns the agent's version, rate, sequence numbers, uptime, slip count, capabilities hash, its wall and monotonic clocks, and the board's device-tree model — the model is there on purpose, because it is what an operator is asked to send when their board has no kernel-to-RouterOS port map yet. What the token is, and is not: - It may contain letters, digits, `_`, `.` and `-`, up to 128 characters; anything else is refused before the first command. - It is stored in the envlist as `TOKEN`. `/container/print` returned the `envlist` property to a `read,api` user (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11); reading the entries' values was not checked separately, and the design assumes a `read` user can. Treat it as guarding the agent's HTTP paths from the LAN, not from the router's own `read` users. - A token set without `--expose` is still written and still required. - The agent compares it as a plain string, over plain HTTP: the dst-nat carries no TLS, so the header crosses the LAN unencrypted. ## What uses the exposed path, and what does not The rules serve a client that addresses `:` — a Prometheus job, a browser, a `curl` with the header. mikroscope's own commands do not use that address: - `record` and `forward` build the agent's URL from `--subnet` and `--port`, so they always dial the container's address. There is no flag that points them at the LAN address. - `install`, `upgrade` and `status` probe `/healthz` at the container's address too. - The relay transport cannot carry the token: `/tool fetch` on the router sends no `Authorization` header. Against an agent with a token, the relay's `/healthz` answers and every sample request is refused. A token needs the direct transport, or a deployment without a token. ## Removing them `uninstall` removes both rules, selecting each by the tag together with the chain, destination address, port and protocol, then asks the router whether anything tagged is still there and fails naming the step if it is. Those `find` selectors quote the address and the port: unquoted, RouterOS parses them as typed values and matches nothing — an unquoted `dst-port=9123` found no rule on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11, and an uninstall built that way would have reported success with the rule still in place. `upgrade` does not touch either rule. It removes the container step — the container, the envlist `-env` and the image — and creates it again, writing the envlist from the flags given to `upgrade`, not from the ones the install used. Its only check is that every step of the plan built from its own flags exists, and a plan built without `--expose` has no rule steps to miss. > **Give upgrade the flags install had** > > Pass `upgrade` the same `--token` (or `MIKROSCOPE_TOKEN`), `--expose` and `--lan-address`, and the > same tuning flags (`--rate`, `--mem-limit-mb`, `--privileged` and the rest), as the install. An > upgrade without the token passes its check, leaves both rules in place and writes an envlist with > no `TOKEN`: the agent is then reachable from the LAN with no token. A tuning flag left out comes > back at its default. ## See also - [Reaching the agent](/mikroscope/install/reaching-the-agent/): direct, relay and `--expose` compared. - [What runs where](/mikroscope/security/): where the token sits among the other credentials. - [What the installer refuses](/mikroscope/security/installer/): the ownership checks every rule goes through. - [The agent's HTTP endpoints](/mikroscope/reference/http/): every path the token guards. --- # What the installer refuses The objects install will not build on, the values it will not put in a RouterOS command, and how uninstall proves it left nothing behind. Source: https://jmrplens.github.io/mikroscope/security/installer/ The installer writes to a router it did not configure, over the operator's own admin ssh session, so there is no privilege boundary between a mistake and the router. What stands in for one is a set of refusals. This page lists them: what `install` stops on before it writes, what it will not touch, what it treats as a failure, and how `uninstall` shows that it is done rather than saying so. ## Nothing is written before it is listed `install` builds the image, prints every command it would run with its exact RouterOS text, and then, in this order: 1. runs `doctor`, the read-only preflight, unless `--no-doctor` is given; a missing prerequisite stops it with `N prerequisite(s) missing; nothing was written`; 2. asks `write the objects above to the router? [y/N]`, unless `--yes` is given; anything but `y` or `Y` stops it with `not confirmed; nothing written`; 3. only then writes. `plan`, and `install --dry-run`, stop after the listing. The listing masks the token as `value="(token)"`. `upgrade` does not get this guarantee. It builds the image, refuses a router where any step of the plan built from its own flags is missing (`nothing to upgrade: run install first`), and asks the same `write the objects above to the router? [y/N]` — but it prints no listing and runs no `doctor` first, so there are no objects above. On `y` it removes the container, the envlist and the image and writes them again, the envlist from the flags given to `upgrade`; [what --expose opens](/mikroscope/security/expose/) has what that means for the token. **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. ## Objects it does not own Before writing, `install` asks the router three questions about every step in one ssh connect: is our object there, does something with the same effect exist, and does it carry our tag. An object that exists but does not carry mikroscope's tag stops `install`, naming the step: ```text veth interface veth-mikroscope exists on the router and was not created by mikroscope (no ownership tag); pick another --name/--veth/--subnet, or remove it by hand if it is yours ``` What counts as "the same effect" is the object's identity, not its name alone: | Step | Collides with any existing | Ours when it carries | | ------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | | veth | `/interface/veth` with the same name | the tag in `comment` | | router address | `/ip/address` on that veth | the tag in `comment` | | interface-list membership | member with that interface in that list | the tag in `comment` | | address-list membership | entry with the /30 in that list | the tag in `comment` | | expose dst-nat | `dstnat` rule with that destination address, port and protocol | the tag in `comment` | | expose forward accept | `forward` rule with the container's address, that port and protocol | the tag in `comment` | | container | a container using the same image file, an envlist named `-env`, or a file at the image path | a container with the tag; an envlist holding the marker | A step that is already ours is skipped, so running `install` twice creates nothing the second time. The refusal happens when `install` reaches the colliding step. Steps before it that were absent have already been created; they carry the tag, and `uninstall` removes them. `uninstall` never touches the foreign object. ### The envlist and the image carry no comment Neither `/container/envs` nor `/file` has a comment field, so the container step signs them another way: the first entry written to the envlist is `MIKROSCOPE_TAG` holding the exact tag, and the image file counts as ours only while that marker exists. A foreign envlist with the same name, a foreign file at the image path, or another container's envlist is therefore foreign, and stops `install`. Leftovers of an earlier mikroscope install — an envlist under our marker with no container — are ours to replace, and `install` clears them before writing new ones. ## Selectors are exact Every object `install` creates carries the comment `mikroscope: (managed by mikroscope)`, verbatim. Every network object is removed by that exact comment together with the identity its check used — `comment="…"`, never a pattern match. The container is removed by the comment alone, and the envlist and the image, which carry no comment, by `list="-env"` and the exact image file name, only while the marker entry holding the exact tag exists. So `uninstall` cannot reach a hand-made setup, or anything else whose comment or name shares a substring with the container name. Every `find` quotes address and port attributes. Unquoted, RouterOS parses them as typed values and the comparison with the stored one comes back empty; verified for both on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11. ## A write that prints is a failure A RouterOS write prints nothing on success, and over ssh it reports errors as text with exit status 0, abandoning the rest of a `;`-joined line. So `install` treats any output from a write as a failure (`create : router said "…"`) and stops there. The image tar is uploaded with scp before the container step runs, and before the marker exists. If that step then fails, `install` removes the uploaded file (`undo removed the uploaded …`); otherwise it would count as a foreign file on every later attempt. `uninstall` reads output the same way in the other direction: a removal that printed something is reported as `skip`, not `gone`. ## Values it will not put in a command Every flag that reaches a RouterOS command is interpolated into it verbatim. There is no privilege to escalate — the command runs as your admin — but a quote or a semicolon would turn a clear error into a confusing RouterOS syntax error, or a selector into something wider than intended. So every value is checked before the first connect, and one outside these bounds stops the command with the rule it broke: | Flag | Accepted | | --------------------------------------- | ------------------------------------------------------------------------------------------------- | | `--name` | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$` | | `--veth`, `--iface-list`, `--addr-list` | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$` | | `--disk` | `^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$`, or empty for the internal flash; `--ephemeral` forces `tmpfs` | | `--arch` | `^[a-z0-9]{1,16}$` | | `--token` | `^[A-Za-z0-9_.-]{0,128}$` | | `--subnet` | an IPv4 /30, written as its network address | | `--port` | 1–65535 | | `--rate` | 1–100 Hz | | `--buffer` | 10–3600 s | | `--memory-max` | `^\d{1,6}[KMG]?$` | | `--mem-limit-mb` | 8–1024 | | `--floor-hz` | 0–1000 | | `--capture-mb` | 0–256 | | `--expose` | needs `--lan-address` as an IPv4 address, and a non-empty token | | `--triggers` | the agent's own condition list, parsed by `agent.ParseTriggers` | | `--remote-image` | a registry reference: `owner/name:1.0.0`, with or without a host, no quote, space or semicolon | There is no exception. `--triggers` is checked by the same gate as the rest: `Finish` hands it to `agent.ParseTriggers`, the agent's own parser, which is the authority on what a condition means. An unknown condition, a malformed threshold, a quote or a semicolon fails the verb with exit status 2 before the first connect, and nothing is written. The agent parses `TRIGGERS` again when it starts, because the envlist can be edited on the router by hand. On a value it cannot parse it exits non-zero with a `mikroscope-agent: bad configuration: …` line in the router log; under the on-failure policy RouterOS may retry it up to five times. No run with a bad `TRIGGERS` on the router is recorded. ## What it refuses to send to the router Two of the four install routes hand the router something the CLI did not build, and each gets a check of its own before anything is written. - **`--agent-tar `**, the image tar the release publishes, is read and inspected on your host first. It has to be a docker-save tar of exactly one image with one layer whose entrypoint is `/mikroscope-agent`, and its architecture has to match `--arch`; otherwise the verb stops, and for a mismatch it names the asset to download instead (`--agent-tar … is a linux/arm64 image and --arch says arm: download the mikroscope-agent-arm.tar asset instead`). That check says the tar is a mikroscope agent image of the right architecture. It does not say the tar is the one the release published — verify it against `checksums.txt` from the release, and its cosign signature if you use one, before you pass it. - **`--remote-image `** makes the router pull the image itself, so nothing is uploaded and no tar lands on the device. The reference is matched against a registry-reference pattern before it reaches the command line, because RouterOS takes it inside a quoted string on a `;`-joined line. The router then needs to reach that registry over its own network, and it takes the registry host from `/container/config registry-url`, a setting global to the device and shared with every other container on it and ships as `https://registry-1.docker.io`. **mikroscope never writes that setting.** `doctor` reads it, and when the reference names a host the setting does not match it prints the one command to run (`/container/config/set registry-url=https://ghcr.io`, for the GHCR copy of the image) or says to use `--agent-tar` instead. The Docker Hub reference `jmrplens/mikroscope-agent:1.0.0` carries no host and leaves the setting as the router has it. Trust in the image is trust in that registry: nothing in the CLI verifies what the router pulls. ## A generated .rsc script is a credential `plan --rsc` writes the install as a RouterOS script for a router you reach only through WinBox or WebFig. It carries the same commands `install` runs, in the same order, with the same tags — and, when `--token` or `MIKROSCOPE_TOKEN` is set, the envlist line carries the token in clear, because the router needs it. The script says so in its own header. Treat the file the way you treat the token: do not commit it, do not paste it where it is logged, and delete it from the router's Files after `/import`. Without a token it holds no secret, only the plan. `plan` and `install --dry-run` mask the token in what they print to the terminal (`value="(token)"`); `--rsc` cannot, since the script has to run. ## How uninstall proves it is done `uninstall` runs every removal newest first, ignoring what is already gone. Then it asks the router, in one connect, how many objects each step created are still there, prints one line per step with the count, and fails naming every step whose count is not zero (`uninstall left objects behind: …`). A removal that printed nothing is not evidence; the count is. `status` runs the same count on its own. The container step is the slow one, and the order inside it is what keeps the count honest: - the container is stopped (guarded, because stopping a stopped container is an error) and removed; - `/container/remove` returns before the container is gone, and a `/file/remove` of the image issued meanwhile did nothing, silently (RB5009UG+S+, RouterOS 7.24.2, 2026-09-11) — so it waits up to 20 s for the container to vanish, then retries the file removal for up to 15 s; - the rest of the envlist goes, and the marker goes last, only once the file is gone. If the file removal does not take, the marker stays, the count keeps including the envlist and the file, and `uninstall` says so instead of reporting clean. A doctor → install → status → upgrade → uninstall round trip (`make roundtrip`) left the router's `/export` byte-identical, compared by hash (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-12). > **True of this device, not of yours** > > The byte-identical `/export` after that one round trip, and every RouterOS behaviour quoted on > this page — the silent `/file/remove`, the quoted `find` selectors, errors printed with exit > status 0 — were observed on one RB5009 running RouterOS 7.24.2. The refusal logic itself is > covered by tests against a fake router, not by a run on another board or RouterOS version. ## See also - [Installing the agent](/mikroscope/install/): the commands, and what `doctor` checks before any of this. - [Where things go](/mikroscope/install/layout/): every object `install` creates, and the flag that moves it. - [What --expose opens](/mikroscope/security/expose/): the two optional firewall rules and their selectors. - [Commands and flags](/mikroscope/reference/cli/): every flag with its default and variable. --- # Commands and flags Every verb of the mikroscope CLI, every flag with its default, its range and the MIKROSCOPE_* variable that sets it, read from cmd/mikroscope. Source: https://jmrplens.github.io/mikroscope/reference/cli/ This page answers one question for each flag of `mikroscope`: what it does, what it defaults to, what range is accepted, and whether an environment variable can set it. It is read from `cmd/mikroscope/*.go` and `internal/router/options.go`, not from the help text, and where the two differ the page says so. ## Usage and exit status ```sh mikroscope [flags] ``` The verbs fall into five groups, each with its own flag set: the deployment verbs (`doctor`, `plan`, `install`, `upgrade`, `uninstall`, `status`, `image`), the recording verbs (`record`, `mark`, `plot`), the collector (`forward`), `dashboards`, and `version`. Flags are Go `flag` flags: `-rate 50` and `--rate 50` are the same, and a boolean is turned off with `-privileged=false`. | Status | When | | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | the verb finished | | `1` | the verb ran and failed (a missing prerequisite, a router error, a sink that could not be built, a panel with no data), the verb is unknown, or a flag of `record`, `mark`, `plot`, `forward` or `dashboards` failed to parse or validate | | `2` | no verb was given, or a flag of a deployment verb failed to parse or validate; nothing was sent to the router | Every flag is checked before the first connection, by `Finish` in `internal/router/options.go`, which runs for the deployment verbs and for `record`, `mark`, `plot` and `forward` alike. `--triggers` goes through the agent's own `ParseTriggers`, so an unknown condition, a bad threshold, a quote or a semicolon fails the verb with exit 2 and nothing is sent. The agent parses `TRIGGERS` again when it starts, which is what catches an envlist edited by hand on the router: a bad value there shows up as a container that exits with status `2` after a `mikroscope-agent: bad configuration:` line in the router log, which `restart-policy=on-failure` then restarts. ### Where a default comes from A flag that names a variable in the tables below reads its default from `MIKROSCOPE_`; a variable set to the empty string counts as unset, and a flag on the command line always wins. **Only the flags that name a variable have one.** `--rate`, `--buffer`, `--port`, `--memory-max`, `--mem-limit-mb`, `--capture-mb`, `--triggers`, `--floor-hz`, `--privileged`, `--ephemeral` and `--expose` are flags only. The CLI does not read `.env` itself; export it first with `set -a; . ./.env; set +a`. [Environment variables](/mikroscope/reference/environment/) lists every variable, including the credentials that have no flag at all. ## The deployment verbs | Verb | What it does | Writes to the router | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `doctor` | Read-only preflight in one ssh connect; each failing check names its fix. Exits 1 if any prerequisite is missing. | no | | `plan` | Prints every object `install` would create, then stops. Same as `install --dry-run`; with `--rsc` it writes a RouterOS script instead. | no | | `install` | Gets the image, prints the listing, runs `doctor`, asks for confirmation, writes, then probes the agent from this host. | yes | | `upgrade` | Gets a new image, checks that every install step is present, asks, removes and re-creates the container step, then probes. | yes | | `uninstall` | Removes every step newest first, then verifies by ownership counts and fails naming anything that remains. | yes | | `status` | Prints the ownership count of every step; if anything is installed, probes the agent and prints its health and board. | no | | `image` | Builds the agent image tar and writes it to `--out`, for side-loading by hand. | no | `doctor` runs these checks, in this order: The checks doctor runs: | Check, as printed | Passes when | The fix it names | | --- | --- | --- | | registry-url is https:// | with `--remote-image`, `/container/config registry-url` names the reference's registry host. Without `--remote-image` doctor does not ask: the setting is global to the device and mikroscope never writes it | `/container/config/set registry-url=https://` on the router, which applies to every container on it, or install from a tar with `--agent-tar` | | container package installed and enabled | a `container` package exists with `disabled=no` | download, upload, reboot; then `/system/package/enable container` | | device-mode container=yes | `/system/device-mode` reports `container=yes` | `/system/device-mode/update container=yes`, then the reset or mode button, or a power cycle, within 5 minutes | | architecture matches --arch | the router's `architecture-name` is the one `--arch` maps to (`arm64`, `arm`, `x86_64`) | re-run with the `--arch` it names | | free memory ≥ <--memory-max> | `free-memory` is at least what `--memory-max` asks for, 64 MiB by default | free memory on the router, or ask for less with `--memory-max` | | free flash ≥ (image tar + extracted root) | without `--disk`: `free-hdd-space` is at least twice the image plus 4 MiB | free flash, or install with `--disk tmpfs` or `--ephemeral` where a tmpfs disk exists | | disk exists | with `--disk` or `--ephemeral`: a disk with that slot exists; its free space is not checked | `/disk/add type=tmpfs tmpfs-max-size=64M slot=tmpfs` for a RAM disk, or name an existing disk with `--disk` | | interface list exists (raw rule trap) | the `--iface-list` list (default `LAN`) exists | `/interface/list/add name=…`, or pass the list your `in-interface-list=!…` drop rule uses | | address list has entries (raw rule trap) | the `--addr-list` list (default `LANs`) has at least one entry | pass the list your `drop local if not from default IP range` rule uses; an empty list is fine only if there is no such rule | | veth name is free or ours | always reported `ok`, with the count found | none: a collision is caught by `install` itself | > **Three ways to get the agent image, one flag apart** > > `plan`, `install`, `upgrade` and `image` need an agent image, and `buildImage` in > `cmd/mikroscope/main.go` gets it by whichever of three routes the flags ask for. With neither > flag they run `go build -trimpath ./cmd/mikroscope-agent` with `CGO_ENABLED=0`, `GOOS=linux` and > the `--arch` as `GOARCH` (`internal/image/image.go`), which needs a checkout of the repository > and a Go toolchain on `PATH`. With `--agent-tar` they read the published tar instead, check that > it is a mikroscope agent image for `--arch`, and upload that. With `--remote-image` nothing is > built and nothing is uploaded: the router pulls the image itself, and `image` refuses, because > there is no tar to write. Without a Go toolchain and with neither flag the build fails naming > both flags and the Go version, before anything is listed. The fourth install route gets no image > on this machine at all: `plan --rsc` writes a script that runs on the router, so it needs > `--remote-image` or a tar already on the device. [Four ways to > install](/mikroscope/install/routes/) sets the four side by side, and [Installing the > agent](/mikroscope/install/) walks what `install` does. ### Flags of the deployment verbs | Flag | Default | Variable | Accepted | Meaning | | ---------------- | ------------------------------- | ------------------------- | ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--router` | none, required | `MIKROSCOPE_ROUTER` | `user@host` or an ssh config alias | ssh target; every verb but `plan`, `install --dry-run` and `image` fails without it | | `--ssh-port` | empty (ssh config) | `MIKROSCOPE_SSH_PORT` | | ssh port | | `--ssh-key` | empty (agent or ssh config) | `MIKROSCOPE_SSH_KEY` | | ssh identity file | | `--name` | `mikroscope` | `MIKROSCOPE_NAME` | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$` | container name; tags every object as `mikroscope: (managed by mikroscope)` | | `--veth` | `veth-mikroscope` | `MIKROSCOPE_VETH` | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$` | veth interface name on the router | | `--subnet` | `172.30.10.0/30` | `MIKROSCOPE_SUBNET` | an IPv4 /30 at its network address | the router takes `.1`, the agent `.2` | | `--iface-list` | `LAN` | `MIKROSCOPE_IFACE_LIST` | same pattern as `--veth` | interface list the veth joins | | `--addr-list` | `LANs` | `MIKROSCOPE_ADDR_LIST` | same pattern as `--veth` | address list the /30 joins | | `--disk` | empty (internal flash) | `MIKROSCOPE_DISK` | `^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$` | RouterOS disk for the image tar and the root: `tmpfs`, `disk1`, `usb1` … | | `--ephemeral` | `false` | none | | forces `--disk tmpfs` and `start-on-boot=no`: nothing written to flash, nothing survives a reboot | | `--arch` | `arm64` | `MIKROSCOPE_ARCH` | `^[a-z0-9]{1,16}$`; `doctor` knows `arm64`, `arm`, `amd64` | device architecture, used as `GOARCH` and in the image manifest | | `--goarm` | `5` | none | | `GOARM` level, used only with `--arch arm`: 5 runs on every 32-bit ARM MikroTik ships, 7 does not run on EN7562CT boards (hEX Refresh) | | `--agent-tar` | empty (build the agent here) | `MIKROSCOPE_AGENT_TAR` | a path to an agent image tar | `plan`, `install`, `upgrade`, `image`: upload this tar instead of building one, so neither a Go toolchain nor a checkout is needed. The tar is checked first: one that is not a mikroscope agent image, or is built for an architecture other than `--arch`, fails the verb naming the asset to download | | `--remote-image` | empty (upload a tar) | `MIKROSCOPE_REMOTE_IMAGE` | a registry reference, `owner/name:tag` or `host/owner/name:tag` | `plan`, `install`, `upgrade`: the router pulls the image itself, so nothing is built and nothing is uploaded, and no tar lands on the device. RouterOS takes the registry host from the global `/container/config registry-url`, which ships as `https://registry-1.docker.io` and which mikroscope never writes; `doctor` checks that setting against a reference that names a host of its own, such as the GHCR one, and names the command to run | | `--rsc` | `false` | none | | `plan`: write a RouterOS script that installs from the router itself, instead of the listing | | `--port` | `9123` | none | 1–65535 | agent HTTP port on the veth | | `--rate` | `10` | none | 1–100 | sampler rate in Hz (envlist `RATE_HZ`); 10, 50 and 100 Hz measured lossless on the RB5009 ([rate ceiling](/mikroscope/cost/rate-ceiling/)) | | `--buffer` | `300` | none | 10–3600 | ring buffer in seconds (envlist `BUFFER_S`) | | `--memory-max` | `64M` | none | `^\d{1,6}[KMG]?$` | container cgroup `memory-max`, RouterOS syntax | | `--mem-limit-mb` | `40` | none | 8–1024 | agent Go soft memory limit in MiB (envlist `MEM_LIMIT_MB`); must fit the ring, rate × buffer × about 2.4 kB, with room for the garbage collector | | `--capture-mb` | `4` | none | 0–256 | triggered-capture budget in MiB (envlist `CAPTURE_MB`); `0` turns captures off | | `--triggers` | empty (the agent's default set) | none | see [triggered capture](/mikroscope/record/triggers/) | trigger conditions, comma-separated (envlist `TRIGGERS`) | | `--floor-hz` | `0` | none | 0–1000 | one cadence for every level source, in Hz (envlist `FLOOR_HZ`); `0` keeps the per-source floors; equal to `--rate` reads and emits every source every tick | | `--privileged` | `true` | none | | runs the container `privileged=yes`; `-privileged=false` opts out | | `--token` | empty | `MIKROSCOPE_TOKEN` | `^[A-Za-z0-9_.-]{0,128}$` | bearer token the agent requires (envlist `TOKEN`); mandatory with `--expose` | | `--expose` | `false` | none | needs `--lan-address` and `--token` | dst-nat the agent port on the router's LAN address; adds two tagged firewall rules | | `--lan-address` | empty | `MIKROSCOPE_LAN_ADDRESS` | an IPv4 address | the router's LAN address for `--expose` | | `--dry-run` | `false` | none | | `install`: print the listing and write nothing | | `--yes` | `false` | none | | `install`, `upgrade`: do not ask before writing | | `--no-doctor` | `false` | none | | `install`: skip the preflight checks | | `--out` | `mikroscope-agent-.tar` | none | | `image`: output path of the tar. `plan --rsc`: where the script is written; empty writes it to standard output | Three settings of the container are not flags: `logging=yes`, `restart-policy=on-failure` with `restart-max-count=5` and `restart-interval=10s`, and `ignore-remote-image-change=yes`. The restart values are the ones `Defaults()` sets; all three are what `install` writes (`internal/router/steps.go`). `start-on-boot` has no flag of its own either: it is `no` with `--ephemeral` and `yes` without. ### Pass the same shape to status, upgrade and uninstall `status`, `upgrade` and `uninstall` do not read what is on the router to learn how it was installed. Each one rebuilds the install plan from the flags of its own invocation and selects objects by that plan's names, paths and tag. So: - An install made with `--expose` is only removed, and only verified, by an `uninstall` that is also given `--expose --lan-address … --token …`. Without them the two firewall rules are not in the plan, and the verification does not look for them. - `upgrade` removes the envlist with the container and writes it again from its own flags. An `upgrade` without the `--rate`, `--buffer`, `--mem-limit-mb`, `--capture-mb`, `--triggers`, `--floor-hz` or `--token` you installed with writes the defaults in their place, and `--memory-max` and `--privileged` likewise go back to theirs. - `--name`, `--veth`, `--subnet`, `--iface-list`, `--addr-list`, `--port` and `--disk` or `--ephemeral` decide what the selectors match; change one and the verb is looking for different objects. > **Untested** > > The three consequences above are read from `internal/router/deploy.go`, `probe.go` and `steps.go`. > None of them was exercised against the router: an `upgrade` that drops the token of an exposed > install, in particular, has not been run. ### What the writing verbs write **What `install` writes to your router** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. **What `install --expose` adds** - two firewall rules, tagged - a token becomes mandatory - `uninstall` and `status` see the two rules only when given `--expose` again Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. **What `upgrade` replaces** - a new image and the container - the envlist, rewritten from the flags `upgrade` is given - network objects stay Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. **What `uninstall` removes** - a veth - one address - one interface-list membership - one address-list entry - an envlist - the image tar, unless `--remote-image` has the router pull the image - the container Every object carries the comment `mikroscope: (managed by mikroscope)` `mikroscope plan` prints every command before anything is written. `uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains. ## record, mark and plot The three verbs share one flag set with `forward`, so a flag that means something to a sibling is accepted and ignored: `plot --for 5m` parses and does nothing. The table marks which verb reads each flag. | Flag | Default | Variable | Read by | Meaning | | --------------- | ---------------------------- | --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--out` | `capture-` | none | `record`, `mark` | output prefix: `.jsonl`, `.csv`, `.markers.csv`, `.meta.json`. `mark` needs the prefix of an existing recording | | `--for` | `0` (until Ctrl-C) | none | `record`, `forward` | run this long, then stop | | `--from-start` | `false` | none | `record` | backfill everything the agent's ring holds before going live | | `--poll` | `500ms` | none | `record`, `forward` | how often the agent's ring is pulled | | `--batch` | `0` | none | `record`, `forward` | samples per pull; `0` is twice what one `--poll` produces at the agent's rate, never under 20. The relay caps a pull at 18. A pull repeats until it comes back short | | `--transport` | `auto` | none | `record`, `forward` | `auto` (direct, then relay), `direct` (HTTP to the veth) or `relay` (`/tool fetch` over the RouterOS API; each call returns at most 64 512 B and took either ~3 ms or ~1 s on the RB5009, RouterOS 7.24.2, 2026-09-11; see [reaching the agent](/mikroscope/install/reaching-the-agent/)) | | `--log-markers` | `false` | none | `record`, `mark` | `record`: after recording, pull the router log over the API and append the matching lines to `.markers.csv`. `mark`: add the log lines of the recording's window | | `--topics` | `system,interface,container` | none | `record`, `mark` | log topics kept as markers with `--log-markers` | | `--router-tz` | `Local` | none | `record`, `mark` | IANA zone the router's clock shows; RouterOS log times carry no zone | | `--in` | none, required | none | `plot` | recording prefix, or its `.jsonl` path | | `--svg` | `.svg` | none | `plot` | output SVG | | `--title` | the prefix | none | `plot` | chart title | | `--api` | empty | `MIKROSCOPE_API_ADDR` | `record`, `mark`, `forward` | RouterOS API `host:port`, for the relay, `--log-markers` and the API tier | | `--api-user` | empty | `MIKROSCOPE_API_USER` | `record`, `mark`, `forward` | API user; its password comes only from `MIKROSCOPE_API_PASSWORD` | | `--token` | empty | `MIKROSCOPE_TOKEN` | `record`, `forward` | bearer token the direct transport sends to the agent | | `--port` | `9123` | none | `record`, `forward` | agent HTTP port | | `--subnet` | `172.30.10.0/30` | `MIKROSCOPE_SUBNET` | `record`, `forward` | the agent's /30; its address is `.2` | `mark` takes the marker's text as its remaining arguments, `mikroscope mark --out cap "queue tree applied"`, or `--log-markers` instead of text. `record` turns every line typed on a terminal into a marker; when standard input is not a terminal it does not read it. `auto` tries `/healthz` over the direct transport first. If that does not answer it needs `--api`, `--api-user` and `MIKROSCOPE_API_PASSWORD` to try the relay. It fails naming `install --expose` when the direct transport does not answer and the relay is not configured; a configured relay that fails reports `relay transport:` and its error. The relay does not carry the token: `/tool fetch` on the router sends no `Authorization` header, so an agent with a token set can only be pulled over the direct transport. ## forward `forward` runs the collector: it pulls the kernel tier from the agent, polls the RouterOS API tier, and writes both to every sink named. It reads `--for`, `--poll`, `--batch`, `--transport`, `--api`, `--api-user`, `--token`, `--port` and `--subnet` from the table above, plus its own flags below. It refuses to start without at least one sink. ### API tier | Flag | Default | Variable | Meaning | | ------------------- | ------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--api-mode` | `full` | none | preset. `off` sets `--api-every 0` unless you set it yourself, so the collector opens no API-tier session; the relay transport still uses the API. `slow` runs the tier every 10 s with no `/system/health` and no conntrack count; `/system/resource`, `/system/resource/cpu`, `monitor-traffic` on `--interfaces` and the `--counters-every` port counters are still read. `full` (the flag defaults) reads everything. An explicit flag below wins over it | | `--api-every` | `1s` | none | API-tier cadence; `0` disables the tier. `off` sets it to `0`, `slow` to `10s` | | `--interfaces` | empty | `MIKROSCOPE_INTERFACES` | comma-separated interfaces for `monitor-traffic`, one call for all | | `--conntrack-every` | `0` | none | ask the conntrack count this often; `0` never, because it is a table scan. One scan took 1.3 ms at 6 212 entries on the RB5009 (date not recorded). `slow` sets it to `0` unless given explicitly | | `--counters-every` | `10s` | none | read every port's cumulative counters (typed errors, fast-path split, link-downs, frame sizes) this often; `0` never | | `--labels-every` | `5m` | none | re-read what each interface is (label from its comment, type, interface lists, bridge, MTU); read once before the first kernel pull and again this often; `0` takes the 5 min default | | `--no-health` | `false` | none | skip `/system/health`. `slow` sets it | Without `--api` and `--api-user`, `forward` logs `api tier disabled` and runs the kernel tier alone; that is a warning, not a failure. ### Sinks | Flag | Default | Variable | Meaning | | ------------------- | --------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------ | | `--file` | empty | none | write the merged timeline as JSONL to this path (truncated at start) | | `--prom` | empty | none | serve Prometheus `/metrics` on this address, for example `:9124` | | `--influx` | empty | `MIKROSCOPE_INFLUX_URL` | InfluxDB 3 write URL, for example `http://host:8181/api/v3/write_lp?db=mikroscope&precision=nanosecond` | | `--loki` | empty | `MIKROSCOPE_LOKI_URL` | Loki push URL; carries events (kernel log, gaps, API errors, detections, triggers, the device record), not samples | | `--loki-tenant` | empty | `MIKROSCOPE_LOKI_TENANT` | `X-Scope-OrgID` for a multi-tenant Loki | | `--otlp` | empty | `MIKROSCOPE_OTLP_URL` | OTLP/HTTP metrics endpoint, for example `http://host:4318/v1/metrics` | | `--graphite` | empty | `MIKROSCOPE_GRAPHITE_ADDR` | Graphite carbon plaintext listener, `host:port` | | `--graphite-prefix` | `mikroscope` | none | first node of every Graphite metric path | | `--elastic` | empty | `MIKROSCOPE_ELASTIC_URL` | Elasticsearch or OpenSearch base URL for `_bulk` | | `--elastic-index` | `mikroscope-%Y.%m.%d` | none | index name; `%Y`, `%m`, `%d` expand to the event's date | | `--sql` | empty | none | write PostgreSQL/TimescaleDB statements to this path, or to standard output with `-`; there is no database driver | | `--sql-hypertable` | `false` | none | also emit TimescaleDB `create_hypertable` calls in the SQL header | | `--telegraf` | empty | `MIKROSCOPE_TELEGRAF_URL` | Telegraf listener: `http://host:8186/telegraf`, `tcp://host:8094` or `udp://host:8094` | | `--stdout` | empty | none | write to standard output as `lp` (InfluxDB line protocol) or `json` (NDJSON); any other value is refused | | `--host-tag` | `router` | `MIKROSCOPE_HOST_TAG` | host tag or label on every point, in every sink | | `--queue-seconds` | `60` | none | seconds of data each queued sink may hold before it drops the oldest batch; `0` or less takes 60 | No sink credential is a flag, because a flag is visible in `ps` and in shell history: the tokens come from `MIKROSCOPE_INFLUX_TOKEN`, `MIKROSCOPE_LOKI_TOKEN`, `MIKROSCOPE_OTLP_TOKEN`, `MIKROSCOPE_ELASTIC_AUTH` and `MIKROSCOPE_TELEGRAF_TOKEN`. A sink that was asked for and cannot be built stops `forward` before it pulls anything. On exit `forward` prints the number of kernel samples, API samples, gaps and skew jumps, and for each sink how many events it wrote, dropped and failed on. While it runs it logs a line on standard error once a minute with the kernel, API, gap, trigger and detection counts, the last sequence number, and each sink's written, dropped and error counts. The collector's Prometheus sink sizes its busy-tick histogram for 10 Hz whatever rate the agent runs at (`promHistogramRateHz` in `cmd/mikroscope/sinkflags.go`), so the bucket layout does not change when it reconnects to an agent configured differently. [Prometheus metric families](/mikroscope/reference/metrics/) says what else follows from that constant. ## dashboards ```sh mikroscope dashboards gen mikroscope dashboards import --store influxdb --datasource-uid mikroscope dashboards check --store influxdb --datasource-uid --window 1h --end 2026-09-13T08:30:00Z ``` | Flag | Default | Variable | Read by | Meaning | | ------------------ | --------------- | ------------- | ----------------- | ------------------------------------------------------------------------------------------------ | | `--out` | `dashboards` | none | `gen` | output directory for `mikroscope-.json` and `mikroscope-alerts-.yaml`, both stores | | `--store` | `influxdb` | none | `import`, `check` | `influxdb` or `prometheus` | | `--grafana` | empty | `GRAFANA_URL` | `import`, `check` | Grafana base URL; the token comes only from `GRAFANA_TOKEN` | | `--datasource-uid` | empty, required | none | `import`, `check` | the datasource UID bound to `DS_MIKROSCOPE` | | `--no-probe` | `false` | none | `import`, `check` | do not ask the datasource which measurements it holds; use the compiled defaults | | `--window` | `15m` | none | `check` | length of the query window | | `--end` | now | none | `check` | RFC 3339 instant the window ends at, to check against a capture that has already finished | `import` and `check` refuse to run without `--grafana` (or `GRAFANA_URL`), `GRAFANA_TOKEN` and `--datasource-uid`. Unless `--no-probe` is given, they first ask the datasource which measurements it holds; if that question fails they warn and continue with the compiled defaults. `check` prints one line per panel (`ok`, `none` for a known-empty panel, `FAIL`) and exits 1 if any panel fails. The variables are `GRAFANA_URL` and `GRAFANA_TOKEN`, without the `MIKROSCOPE_` prefix. ## version `mikroscope version` prints `mikroscope` and the build identity from `internal/version`. It takes no flags. ## See also - [Environment variables](/mikroscope/reference/environment/): every `MIKROSCOPE_*` variable, the credentials with no flag, and the agent's own envlist. - [Installing the agent](/mikroscope/install/): the deployment verbs in the order they are used. - [Record, mark, plot](/mikroscope/record/): what a recording is and how markers get into it. - [The collector](/mikroscope/sinks/): what `forward` does with the flags above. --- # Environment variables The MIKROSCOPE_* variables the CLI reads, the credentials that exist only as variables, and the envlist variables the agent reads on the router. Source: https://jmrplens.github.io/mikroscope/reference/environment/ Two programs read environment variables, and they read different ones. The **CLI** on your machine reads `MIKROSCOPE_*` variables as flag defaults, plus a handful of credentials that have no flag, plus `GRAFANA_URL` and `GRAFANA_TOKEN`. The **agent** on the router reads unprefixed variables (`RATE_HZ`, `TOKEN` …) from its container envlist, which `install` writes. This page lists both, from `cmd/mikroscope/*.go` and `internal/agent/config.go`. ## How the CLI reads them - A variable sets a flag's **default**; the flag on the command line wins. - A variable set to the empty string is treated as unset. - The CLI never reads a `.env` file. Export it into the shell first: ```sh set -a; . ./.env; set +a ``` - Quote a URL that holds `&` when it lives in a file you `source`: unquoted, `&` is a shell operator. `.env.example` quotes `MIKROSCOPE_INFLUX_URL` for that reason. Most deployment flags have **no** variable. `--rate`, `--buffer`, `--port`, `--memory-max`, `--mem-limit-mb`, `--capture-mb`, `--triggers`, `--floor-hz`, `--privileged`, `--ephemeral` and `--expose` are set on the command line or not at all. [Commands and flags](/mikroscope/reference/cli/) has every flag. ## Router and deployment | Variable | Flag | Read by | Default in the code | Value in `.env.example` | | ------------------------- | ---------------- | ----------------------------------------------------- | ---------------------------- | ----------------------- | | `MIKROSCOPE_ROUTER` | `--router` | `doctor`, `install`, `upgrade`, `uninstall`, `status` | none (required) | empty | | `MIKROSCOPE_SSH_PORT` | `--ssh-port` | the same | ssh config | `22` | | `MIKROSCOPE_SSH_KEY` | `--ssh-key` | the same | ssh agent or config | empty | | `MIKROSCOPE_ARCH` | `--arch` | the deployment verbs | `arm64` | `arm64` | | `MIKROSCOPE_AGENT_TAR` | `--agent-tar` | `plan`, `install`, `upgrade`, `image` | empty (build the agent here) | not in the file | | `MIKROSCOPE_REMOTE_IMAGE` | `--remote-image` | `plan`, `install`, `upgrade` | empty (upload a tar) | not in the file | | `MIKROSCOPE_NAME` | `--name` | the deployment verbs | `mikroscope` | commented out | | `MIKROSCOPE_VETH` | `--veth` | the deployment verbs | `veth-mikroscope` | `veth-mikroscope` | | `MIKROSCOPE_SUBNET` | `--subnet` | the deployment verbs, `record`, `forward` | `172.30.10.0/30` | `172.30.10.0/30` | | `MIKROSCOPE_IFACE_LIST` | `--iface-list` | the deployment verbs | `LAN` | `LAN` | | `MIKROSCOPE_ADDR_LIST` | `--addr-list` | the deployment verbs | `LANs` | `LANs` | | `MIKROSCOPE_DISK` | `--disk` | the deployment verbs | empty (internal flash) | commented out, `tmpfs` | | `MIKROSCOPE_LAN_ADDRESS` | `--lan-address` | the deployment verbs | empty | commented out | | `MIKROSCOPE_TOKEN` | `--token` | the deployment verbs, `record`, `forward` | empty | commented out | `MIKROSCOPE_TOKEN` is two things at once. For `install` it is the token written into the agent's envlist, which the agent then requires. For `record` and `forward` it is the token the direct transport sends. The envlist is not a secret store: any RouterOS user with the `read` policy can list every container's envlist over the API (verified on RB5009UG+S+, RouterOS 7.24.2, 2026-09-11), which is why the token is the only credential that goes there. ## Reaching the agent and the RouterOS API | Variable | Flag | Read by | Meaning | | ------------------------- | -------------- | --------------------------- | -------------------------------------------------------------------------------------------------- | | `MIKROSCOPE_API_ADDR` | `--api` | `record`, `mark`, `forward` | RouterOS binary API `host:port`; `.env.example` has `192.168.88.1:8728` | | `MIKROSCOPE_API_USER` | `--api-user` | `record`, `mark`, `forward` | the dedicated API user | | `MIKROSCOPE_API_PASSWORD` | none | `record`, `mark`, `forward` | that user's password; there is deliberately no flag | | `MIKROSCOPE_INTERFACES` | `--interfaces` | `forward` | comma-separated interfaces for `monitor-traffic`; `.env.example` has `bridge,ether1` commented out | The relay, `--log-markers` and the API tier fail, or in `forward`'s case are disabled with a warning, unless `--api`, `--api-user` and `MIKROSCOPE_API_PASSWORD` are all set. [The API user](/mikroscope/security/api-user/) has the policy that user needs. ## Sinks | Variable | Flag | Default in the code | Meaning | | -------------------------- | --------------- | ------------------- | -------------------------------------------------------------------------------- | | `MIKROSCOPE_INFLUX_URL` | `--influx` | empty | InfluxDB 3 write URL | | `MIKROSCOPE_LOKI_URL` | `--loki` | empty | Loki push URL | | `MIKROSCOPE_LOKI_TENANT` | `--loki-tenant` | empty | `X-Scope-OrgID` | | `MIKROSCOPE_OTLP_URL` | `--otlp` | empty | OTLP/HTTP metrics endpoint | | `MIKROSCOPE_GRAPHITE_ADDR` | `--graphite` | empty | carbon plaintext `host:port` | | `MIKROSCOPE_ELASTIC_URL` | `--elastic` | empty | Elasticsearch or OpenSearch base URL | | `MIKROSCOPE_TELEGRAF_URL` | `--telegraf` | empty | Telegraf listener URL (`http://`, `tcp://` or `udp://`) | | `MIKROSCOPE_HOST_TAG` | `--host-tag` | `router` | host tag on every point in every sink; `.env.example` has `rb5009` commented out | Setting a sink's URL variable is enough to turn that sink on: `forward` builds every sink whose flag ends up non-empty, from the command line or from the variable. Every one of these names carries the `MIKROSCOPE_` prefix: a bare `LOKI_URL` is read by nothing, and `.env.example` says so beside `MIKROSCOPE_LOKI_URL`. ## Credentials that have no flag A flag is visible in `ps` and in a shell history, so every secret the CLI uses is read from the environment only. | Variable | Used for | | --------------------------- | ------------------------------------------------- | | `MIKROSCOPE_API_PASSWORD` | the RouterOS API user | | `MIKROSCOPE_INFLUX_TOKEN` | InfluxDB, sent as `Authorization: Bearer ` | | `MIKROSCOPE_LOKI_TOKEN` | Loki | | `MIKROSCOPE_OTLP_TOKEN` | the OTLP endpoint | | `MIKROSCOPE_ELASTIC_AUTH` | Elasticsearch or OpenSearch | | `MIKROSCOPE_TELEGRAF_TOKEN` | the Telegraf HTTP listener | | `GRAFANA_TOKEN` | `dashboards import` and `dashboards check` | ## Grafana `dashboards import` and `dashboards check` read `GRAFANA_URL` (the default of `--grafana`) and `GRAFANA_TOKEN`. Neither has the `MIKROSCOPE_` prefix. `.env.example` sets `GRAFANA_URL` twice, once empty in its LAN-services block and once commented out in the Grafana block, and says the token never goes in the file. ## The agent's envlist The agent reads these from its container's environment when it starts. An invalid value makes it print one line starting `mikroscope-agent: bad configuration:`, which RouterOS puts in its log under the `container` topic, and exit with status 2. | Variable | Agent default | Accepted | Meaning | | ---------------------- | -------------------------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `RATE_HZ` | `10` | 1–100 | sampler rate | | `BUFFER_S` | `300` | 10–3600 | ring length in seconds | | `PORT` | `9123` | 1–65535 | HTTP port | | `ADDR` | empty | an address | bind address. Empty binds `:PORT`, every address in the container's network namespace; `install` always sets it to the agent's veth address | | `TOKEN` | empty | | when set, every path but `/healthz` requires `Authorization: Bearer ` | | `IRQ_TOP_K` | `8` | 0–64 | interrupt lines kept per sample, busiest first; `0` ships no interrupt lines and no `irq_total` | | `MEM_LIMIT_MB` | `14` | 8–1024 | Go soft memory limit in MiB | | `FLOOR_HZ` | `0` | 0–1000 | `0` keeps the per-source floors; N puts every level source on N Hz and turns off emit-on-change | | `SOURCES` | empty (every source detected) | comma-separated names | restrict the optional sources to this list; `/proc/stat` is always read | | `TRIGGERS` | `softnet-drop,oom,kmsg<=3,reset,irq-err,flash-bad` | conditions, comma-separated | capture trigger conditions | | `CAPTURE_MB` | `4` | 0–256 | pinned-bytes budget for captures; `0` turns them off | | `CAPTURE_PRE_S` | `5` | 1–60 | seconds kept before the sample a trigger fires on | | `CAPTURE_POST_S` | `5` | 1–60 | seconds kept after it | | `CAPTURE_POLICY` | `first` | `first` or `last` | on a full budget: `first` refuses the new capture, `last` evicts the oldest | | `TRIGGER_REFRACTORY_S` | `10` | 0–3600 | quiet time per condition after it fires | | `PROC_ROOT` | `/proc` | a path | where `/proc` is read from; the kernel log is `/dev/kmsg` only when this is `/proc`, otherwise `dev/kmsg` beside the given directory | | `SYS_ROOT` | `/sys` | a path | where `/sys` is read from | The names `SOURCES` accepts are the optional sources `/capabilities` reports: `meminfo`, `loadavg`, `softnet`, `softirqs`, `interrupts`, `vmstat`, `psi`, `schedstat`, `self`, `buddyinfo`, `yaffs`, `diskstats`, `slabinfo`, `kmsg`, `thermal`, `cpufreq`, `mtd` and `perf`. The trigger conditions and what each compares are on [triggered capture](/mikroscope/record/triggers/). Before it listens, the agent checks the ring against the container's own `memory.max`: `RATE_HZ × BUFFER_S × 2 560 B` plus `CAPTURE_MB` must fit, or it refuses to start and names the three variables and `--memory-max`. If that figure is more than half of `MEM_LIMIT_MB`, it starts and logs a warning with the limit to raise to. 2 560 B is not itself a measurement: it is the mean line measured on the RB5009 with every source of that date (2 439 B, 4 cores, `IRQ_TOP_K` 8, RouterOS 7.24.2, 2026-09-12) rounded up. A board with more cores or more interrupt lines writes longer lines, so the check errs open. When the agent cannot read its `memory.max`, the refusal check does not run. ### What install writes into it `install` and `upgrade` write the envlist `-env` with these entries, in this order, and nothing else: The entries install writes into the agent's envlist: | Key | Written | From | Holds | | --- | --- | --- | --- | | `MIKROSCOPE_TAG` | always | `--name` | the ownership marker `mikroscope: (managed by mikroscope)`, written first and removed last; the agent ignores it | | `RATE_HZ` | always | `--rate`, default `10`, 1–100 | the sampler rate, in Hz | | `BUFFER_S` | always | `--buffer`, default `300`, 10–3600 | the ring's length, in seconds | | `PORT` | always | `--port`, default `9123`, 1–65535 | the agent's HTTP port | | `ADDR` | always | `--subnet` | the agent's address, the `.2` of the /30; the agent binds only there | | `MEM_LIMIT_MB` | always | `--mem-limit-mb`, default `40`, 8–1024 | the agent's Go soft memory limit, in MiB | | `FLOOR_HZ` | only when above 0 | `--floor-hz`, default `0`, 0–1000 | one cadence for every level source, in Hz | | `CAPTURE_MB` | always | `--capture-mb`, default `4`, 0–256 | the triggered-capture budget, in MiB; `0` turns captures off | | `TRIGGERS` | only when set | `--triggers` | the trigger conditions; unset, the agent uses its default set | | `TOKEN` | only when set | `--token` | the bearer token the agent requires, from `--token` or `MIKROSCOPE_TOKEN`, with or without `--expose` | > **The agent's defaults and install's defaults are not the same** > > An agent started with no envlist entry uses a 14 MiB soft memory limit. `install` always writes > `MEM_LIMIT_MB=40`, and the container's `memory-max` is `64M`, because a 300 s ring at 10 Hz holds > about 7.3 MB and a 14 MiB limit kept the garbage collector running continuously on the RB5009: 9.38 % of one core against 1.39 % with room, both with the ring full, measured > on RouterOS 7.24.2 on 2026-09-12. `IRQ_TOP_K`, `CAPTURE_PRE_S`, `CAPTURE_POST_S`, `CAPTURE_POLICY`, > `TRIGGER_REFRACTORY_S`, `SOURCES`, `PROC_ROOT` and `SYS_ROOT` have no flag, so an installed agent > runs with their defaults. ## In .env.example but not read by the code `.env.example` is also the development template for the project's own devices, so it carries names that no binary reads. Setting them changes nothing: - `MIKROSCOPE_ROUTEROS`, the RouterOS version of the reference device. - `OPERATOR_HOST_IP`, the operator host for push-sink and probe tests. - The `HEXS_*` block (`HEXS_ROUTER`, `HEXS_SSH_PORT`, `HEXS_SSH_KEY`, `HEXS_ARCH`, `HEXS_API_ADDR`, `HEXS_API_USER`, `HEXS_API_PASSWORD`), kept for a hEX S that has not arrived. ## See also - [Commands and flags](/mikroscope/reference/cli/): every flag, including the ones no variable sets. - [The API user](/mikroscope/security/api-user/): the RouterOS user behind `MIKROSCOPE_API_USER`. - [Each source at its own floor](/mikroscope/limits/source-floors/): what `FLOOR_HZ` overrides. - [The cost of the observer](/mikroscope/cost/): why the memory limit is sized to the ring. --- # The agent’s HTTP endpoints Every path the agent serves on the veth, its query parameters, what it returns, how the token guards it, and the line formats of /snapshot and /stream. Source: https://jmrplens.github.io/mikroscope/reference/http/ The agent is an HTTP server and nothing else: it has no outbound connection. This page answers what each path returns, which query parameters it takes, which status codes it answers with, and what one line of its NDJSON looks like. It is read from `internal/agent/http.go`, `capture.go`, `source.go` and `internal/sample/sample.go`. ## Where it listens The agent binds `ADDR:PORT`. `install` writes the agent's own veth address into `ADDR` and `9123` into `PORT`, so the default is `http://172.30.10.2:9123`. An agent started with `ADDR` unset binds `:PORT`, every address inside its container's network namespace. How a host reaches that address, directly, through the relay or through `--expose`, is on [reaching the agent](/mikroscope/install/reaching-the-agent/). The server gives a client 5 s to send its request headers and 30 s to take a response (`ReadHeaderTimeout` and `WriteTimeout` in `internal/agent/agent.go`). ## Authentication With `TOKEN` unset, every path is open. With it set, **every path but `/healthz`** needs: ```http Authorization: Bearer ``` The header's value is compared with the token after an optional `"Bearer "` prefix is removed, so a header holding the bare token is accepted too. A request without it, or with a different token, gets `401`, the body `token required` and `WWW-Authenticate: Bearer`. `/healthz` stays open on purpose: it carries the board string an operator is asked to send when their board has no port map, and a token should not be needed for that. The relay transport cannot present a token. `/tool fetch` runs on the router and sends no `Authorization` header (`internal/transport/transport.go`), so the relay reaches `/healthz` and nothing else on an agent that has one. `--expose` makes the token mandatory; [what --expose opens](/mikroscope/security/expose/) says why. ## The paths | Method | Path | Token | Returns | | -------- | ---------------- | ----- | --------------------------------------------------------------------------------------------- | | `GET` | `/healthz` | no | JSON: liveness, sequence numbers, clocks, rate, slipped ticks, capability hash, board | | `GET` | `/capabilities` | yes | JSON: the kernel, the board, the sources, the device's own ceilings and each source's cadence | | `GET` | `/snapshot` | yes | NDJSON, then closes: the last N seconds, or up to N samples after a sequence number | | `GET` | `/stream` | yes | NDJSON, chunked and open: backfill from a sequence number, then live | | `GET` | `/metrics` | yes | Prometheus text exposition, `text/plain; version=0.0.4` | | `GET` | `/captures` | yes | JSON: the capture index | | `GET` | `/captures/{id}` | yes | NDJSON: one capture header line, then the sample lines verbatim | | `DELETE` | `/captures/{id}` | yes | `204`, and the capture's bytes return to the budget | | `POST` | `/capture` | yes | JSON: arms a capture now | Any other path is `404`; a known path with another method is refused by Go's router with `405`. ## `GET /healthz` | Field | Type | Meaning | | ------------------- | ------- | ------------------------------------------------------------------------------ | | `ok` | bool | always `true` when the agent answers | | `seq` | integer | sequence number of the newest sample in the ring; 0 when the ring is empty | | `oldest_seq` | integer | oldest sequence number still held; 0 when empty | | `wall_ns` | integer | the router's wall clock at the reply, ns since the epoch | | `mono_ns` | integer | the agent's monotonic clock at the reply, ns | | `uptime_s` | float | seconds since the HTTP server was set up | | `rate_hz` | integer | configured sampler rate | | `slipped` | integer | ticks whose read finished after the next tick was due | | `capabilities_hash` | string | eight hex digits over the kernel, the core count and the enabled sources | | `version` | string | the agent's build identity | | `board` | string | the device tree's model, for example `RB5009`; omitted when the board has none | The CLI uses `wall_ns` against its own clock to measure skew, `seq` and `oldest_seq` to plan a backfill, and `capabilities_hash` to notice that the kernel or the source set under it changed; no CLI code reads `mono_ns`. The probe after `install` prints this reply as `direct transport ok: agent , Hz, seq , slipped, round trip`, where `` is the CLI's own version, which it stamps into the agent it builds (`dev` when the CLI itself was built without one). The round trip recorded on the RB5009 on 2026-09-12 has two probes, 7 ms after `install` and 5 ms after `upgrade`; the first printed `direct transport ok: agent 4857d0a-dirty, 10 Hz, seq 29, 0 slipped, 7ms round trip`. ## `GET /capabilities` What the agent established at start about the kernel and the board, with no RouterOS API. [The device-info stream](/mikroscope/sinks/device-info/) is this object as the collector hands it to every sink. | Field | Meaning | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kernel` | the kernel version string from `/proc/version` | | `board` | the device tree's model; omitted when absent | | `ports` | kernel interface name to RouterOS default name, for a board in the port table; omitted otherwise | | `ports_from` | how that map was established; omitted with it | | `cores` | core count, from the first read of `/proc/stat` | | `user_hz` | `USER_HZ`, the tick counters' unit | | `sources` | source name to `true` when the file opened and the source is enabled: `stat`, `meminfo`, `loadavg`, `softnet`, `softirqs`, `interrupts`, `vmstat`, `psi`, `schedstat`, `self`, `yaffs`, `diskstats`, `slabinfo`, `kmsg`, `thermal`, `cpufreq`, `perf`, `buddyinfo`, `mtd` | | `namespaced` | what the agent deliberately does not read as router data: `net/dev`, `net/snmp`, `net/netstat`, `sys/net/netfilter/nf_conntrack_count` | | `cgroup` | `true` when cgroup2 `cpu.stat` is readable, so the self-cost is exact | | `privileged` | `true` when both `/proc/slabinfo` and `/dev/kmsg` opened | | `limits` | the device's own ceilings, read once at start (below) | | `cadences` | source name to `{"hz", "reason"}` for each level source (below) | | `hash` | the same value as `/healthz`'s `capabilities_hash` | `limits` has no JSON tags in the code, so its keys are the Go field names: `ThermalCriticalMilliC` (zone to the lowest critical trip, m°C), `ThermalPollingMS` (zone to its `polling_delay`), `CPUFreqMinKHz`, `CPUFreqMaxKHz`, `CPUFreqStepsKHz` (core to the ladder), `CPUFreqGovernor`, `CPUFreqRelated` (core to the cores that change frequency with it), `CgroupMemoryMaxBytes` and `ConntrackMax`. A ceiling the device does not publish is empty or 0. A cadence's `reason` is one of `rate` (read at the sampler rate), `declared` (the device publishes its own refresh cadence), `policy` (a `userspace` cpufreq governor means the clock cannot move without a write), `budget` (a measured parse cost), `change` (read every tick, stored on change) or `override` (`FLOOR_HZ`). Counters never appear here, because counters are never floored. [Each source at its own floor](/mikroscope/limits/source-floors/) has the measurements behind the floors. ## `GET /snapshot` Writes NDJSON and closes. Two forms, chosen by whether `since` is present. | Parameter | Default | Accepted | Meaning | | --------- | ------- | ----------------- | -------------------------------------------------------------------------------------------------------- | | `seconds` | `1` | 1–3600 | without `since`: the newest `seconds × rate` samples the ring holds, oldest first | | `since` | absent | a sequence number | up to `max` samples with a sequence number above it, oldest first; `since=0` starts from the oldest held | | `max` | `20` | 1–10000 | with `since`: the most samples one reply carries | A value outside its range is `400` with a one-line reason. The `since` form is what `record` and `forward` pull, over both transports. It adds two line kinds the `seconds` form does not: a gap line first when `since` is older than the ring, and a trigger line before each sample a capture condition fired on. `max` exists for the relay, because `/tool fetch output=user` returns at most 64 512 B on RouterOS 7.24.2 and truncates the rest silently; the relay asks for at most 18 lines. > **A large /snapshot is not a free way to read the cost** > > A 60 s `/snapshot` at 10 Hz makes the agent hand over about 600 lines, around 1.5 MB, and the > `self.cpu_us` inside those samples includes the cost of serving them. Read the agent's cost from > `/metrics` instead: [the cost of the observer](/mikroscope/cost/) has the procedure. ## `GET /stream` Chunked NDJSON with `Cache-Control: no-store`, held open until the client leaves. - `since` absent or `0`: starts live. The first sample written is the next one produced after the request; the newest sample already in the ring is not written. - `since=N`: backfills every held sample after N first, then follows. A `since` older than the ring writes one gap line first. - The ring is checked every half period and every new sample is written as it arrives, with trigger lines before the samples they fired on. - Every 5 s a comment line `# heartbeat seq=` is written. A consumer skips lines that start with `#`. A `since` that is not a number is `400`. > **Untested** > > The server's 30 s `WriteTimeout` applies to every response, and Go's HTTP server does not lift it > for a handler that keeps writing, so a `/stream` connection is expected to end about 30 s after it > opens. That is read from `internal/agent/agent.go` and `http.go`, not measured. The CLI does not > depend on `/stream`: `record` and `forward` pull `/snapshot?since=`. ## The line kinds Apart from `/stream`'s heartbeat comment, every line `/snapshot`, `/stream` and `/captures/{id}` write is one JSON object and a newline. Four kinds exist, told apart by their first key: | Line | Where | Meaning | | ---------------------------- | ------------------------------ | ---------------------------------------------------- | | a sample, starting `{"seq":` | all three | one tick, below | | `{"gap":{"from":F,"to":T}}` | `/snapshot?since=`, `/stream` | samples F to T are no longer in the ring | | `{"trigger":{…}}` | `/snapshot?since=`, `/stream` | a capture condition fired on the sample that follows | | `{"capture":{…}}` | first line of `/captures/{id}` | the capture's header | A consumer that does not know a kind skips it, so a line kind it has never seen costs it nothing. ### A sample line Every numeric field is a **delta since the previous sample** unless the table says it is a level. A source the kernel does not have, or that the deployment cannot read, is **omitted**, never written as zero. The busy ticks of a core are `u + n + s + q + sq + st`; idle and iowait are not busy. | Field | Kind | Content | | ------------------ | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seq` | — | sequence number, from 1 | | `mono_ns` | — | the agent's monotonic clock at the read | | `wall_ns` | — | the router's wall clock at the read, ns since the epoch | | `dt_ns` | — | the real interval since the previous sample; every delta is over this, not over the nominal period | | `cpu` | delta | one object per core: `u` user, `n` nice, `s` system, `i` idle, `w` iowait, `q` irq, `sq` softirq, `st` steal, in `USER_HZ` ticks | | `cpu_total` | delta | the same keys, from `/proc/stat`'s own `cpu` summary line | | `ctxt`, `intr` | delta | context switches and interrupts from `/proc/stat` | | `forks` | delta | the `processes` line of `/proc/stat` | | `procs_blocked` | level | tasks in uninterruptible sleep | | `psi` | delta | `cpu_some`, `mem_some`, `mem_full`, `io_some`, `io_full`, µs; only on a kernel with PSI | | `sched` | delta | per CPU `run_ns`, `wait_ns`; only on a kernel with `/proc/schedstat` | | `softnet` | delta | per CPU `p` processed, `d` dropped, `ts` time squeeze | | `softirq` | delta | softirq kind to an array of per-CPU counts | | `irq` | delta | the `IRQ_TOP_K` busiest interrupt lines this tick (8 by default): `id`, `name`, `cpu` array | | `irq_total` | delta | every interrupt line summed, the top-K's denominator | | `irq_err` | delta | the `Err` row of `/proc/interrupts`; omitted when zero | | `mem` | level | `/proc/meminfo` in kB, keyed by Go field name: `MemTotal`, `MemFree`, `MemAvailable`, `Buffers`, `Cached`, `Dirty`, `Shmem`, `Slab`, `SReclaimable`, `CommittedAS`, `Writeback`, `SUnreclaim`, `AnonPages`, `Mapped`, `KernelStack`, `PageTables`, `CommitLimit`, `Active`, `Inactive` | | `load` | level | `/proc/loadavg`: `Load1`, `Load5`, `Load15`, `Running`, `Total`, `LastPID` | | `vm` | delta | `/proc/vmstat` events: `pgfault`, `pgmajfault`, and when non-zero `pgscan_kswapd`, `pgscan_direct`, `pgsteal_kswapd`, `pgsteal_direct`, `pgalloc`, `pgfree`, `allocstall`, `compact_stall`, `oom_kill`, `pswpin`, `pswpout` | | `vmg` | level | `nr_free_pages`, `nr_dirty`, `nr_writeback`, `nr_slab_reclaimable`, `nr_slab_unreclaimable`, in pages | | `self` | mixed | `cpu_us` (delta, µs), `rss` (level, bytes), `cg_mem` (level, omitted when zero), `cg` (`true` when cgroup2 was read), and `throttled`, `throttled_us`, `oom_kill` (deltas); the last four are omitted when zero or false | | `thermal` | level | per zone `type`, `mc` (m°C) and `Celsius`; at the zone's declared cadence, every tick when no zone declares one, or at `FLOOR_HZ` | | `freq_khz` | level | per core, kHz; on change or on the 60 s heartbeat | | `thermal_critical` | level | zone to critical trip in m°C, on the rows that carry `thermal` | | `freq_max_khz` | level | core to cpufreq ceiling in kHz, on the rows that carry `freq_khz` | | `cgroup_mem_max` | level | the container's `memory.max` in bytes, on the first tick and once per heartbeat | | `flash` | mixed | per YAFFS device `dev`, `pw`, `pr`, `er`, `gcc`, `gc` (deltas) and `bad`, `free` (levels); a device with no operation and unchanged free chunks is omitted | | `disk` | mixed | per block device `name`, `r`, `rs`, `w`, `ws`, `io_ms` (deltas) and `inflight` (level); an idle device is omitted | | `slab` | level | cache to active objects; [needs `privileged=yes`](/mikroscope/limits/privileged/); on change, at a budget floor | | `slab_limit` | level | cache to its published ceiling, today only `nf_conntrack` | | `perf` | delta | per hardware counter `name`, `cpu` array, and `enabled_ns`, `running_ns`; [needs `privileged=yes`](/mikroscope/limits/privileged/), and only counters the CPU implements | | `buddy` | level | per zone `node`, `zone`, `free` array indexed by order; on change or heartbeat | | `mtd` | level | per partition `dev`, `name`, `corr`, `fail`, `bad`, `bbt`, `bitflip_threshold`, `ecc_strength`; [needs `privileged=yes`](/mikroscope/limits/privileged/); kernel's since-boot counts | | `events` | — | kernel-log records this tick: `prio`, `lvl` (0 emerg … 7 debug), `fac`, `seq`, `us` (µs since boot), `msg`, and, when the text names a port, `iface`, `ros_iface` and `kind` (`link-up`, `link-down`, `stp-`, `own-address` or `other`) | | `events_dropped` | — | kernel-log loss events this tick, not records: one if the tick hit the 64-record cap, one per kernel ring overrun (which can stand for many records); while non-zero, `events` is a lower bound; omitted when zero | | `resets` | — | monotonic counters that went backwards without a 32-bit wrap this tick; omitted when zero | The perf counter names the agent tries to open are `cycles`, `instructions`, `cache-references`, `cache-misses`, `branch-instructions`, `branch-misses` and `bus-cycles`. The slab caches it keeps are `nf_conntrack`, `skbuff_head_cache`, `skbuff_fclone_cache`, `TCP`, `UDP`, `TCPv6`, `UDPv6`, `sock_inode_cache`, `dst_cache`, `ip_dst_cache`, `kmalloc-1k` and `kmalloc-2k`, where the kernel has them. ## `GET /metrics` The Prometheus text exposition, built from cumulative counters that nothing resets on a scrape, so `rate()` over any range is correct and two scrapers see the same values. The text is built in memory and written after the counters' lock is released, so a slow scraper cannot hold up the sampler. [Prometheus metric families](/mikroscope/reference/metrics/) lists every family. ## Captures All four capture paths answer `404` with `captures disabled (CAPTURE_MB=0)` when the budget is 0. ### `GET /captures` | Field | Meaning | | -------------- | ----------------------------------------------------------------------------- | | `policy` | `first` or `last` | | `budget_bytes` | the pinned-bytes budget, `CAPTURE_MB` in bytes | | `bytes` | bytes the held captures pin | | `pending` | the capture still collecting its post-fire window; omitted when there is none | | `captures` | the held captures' headers, oldest first | | `triggers` | the configured conditions, each `name` and `threshold` | ### `GET /captures/{id}` One `{"capture":{…}}` header line, then the sample lines exactly as `/snapshot` would have written them, so no new parser is needed. The bytes served are counted in `mikroscope_capture_bytes_served_total`: a download runs on the same core as the sampler. An `id` that is not a number is `400`; an unknown one is `404` with `no such capture`. | Field | Meaning | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | `id` | capture number, from 1 | | `cause` | the condition's name as configured, or `manual` | | `condition` | the condition as configured, for example `busy>=0.95`; empty for a manual capture | | `field` | what was compared, for example `cpu[2].busy_ratio`; for a manual capture, the `reason` | | `value` | the value that tripped it | | `threshold` | the configured threshold | | `fire_seq` | the sample it fired on | | `fire_mono_ns`, `fire_wall_ns` | the two clocks at the fire | | `first_seq`, `last_seq` | the window held | | `samples`, `bytes` | its size | | `complete` | `false` when it holds fewer than `pre + post + 1` samples: the ring did not reach back far enough, or the agent stopped first | The trigger line in `/snapshot` and `/stream` carries `id`, `cause`, `field` (omitted when empty), `value`, `threshold`, `seq` and `wall_ns`. The agent keeps the last 64 of them. ### `DELETE /captures/{id}` Frees the capture's bytes and answers `204` with no body. Unknown or non-numeric ids answer as for `GET`. ### `POST /capture` Arms a capture at the newest sample, as if a condition had fired: ```sh curl -X POST "http://172.30.10.2:9123/capture?reason=queue-tree-applied" ``` `reason` defaults to `operator`. The reply is `{"id":N,"armed":true}`. If another capture is still collecting its window, or a manual capture fired within the refractory window, the reply is `409` and nothing is armed. The conditions, the budget and what a capture cannot show are on [triggered capture](/mikroscope/record/triggers/). ## See also - [Reaching the agent](/mikroscope/install/reaching-the-agent/): the three ways a host gets to these paths. - [Prometheus metric families](/mikroscope/reference/metrics/): what `/metrics` carries. - [Triggered capture](/mikroscope/record/triggers/): the conditions behind `/captures`. - [What --expose opens](/mikroscope/security/expose/): when the token becomes mandatory. --- # Prometheus metric families Every family on the agent’s /metrics and on the collector’s --prom exposition, grouped by source, with its type, its labels and when it is absent. Source: https://jmrplens.github.io/mikroscope/reference/metrics/ This page lists every metric family mikroscope exposes in Prometheus text format: its name, its type, its labels, what it counts, and when it is not there. It is read from `internal/agent/metrics.go`, `internal/agent/capture.go` and `internal/sinks/prometheus.go`. A family is listed under the source it comes from, because that is what decides whether a given board has it. ## Two expositions, one renderer The agent serves `/metrics` on the router, and `forward --prom :9124` serves one on the collector host. Both are written by the same `Totals` code: the collector feeds it the samples it pulled, so a deployment reached only through the relay still gets scrape-independent families. They are not identical. | Families | Agent `:9123/metrics` | Collector `--prom` | | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | --------------------------------------------------- | | everything recomputed from samples: CPU, windows, runs, receive path, memory, PMU, sensors, flash, disk, kernel log, observer counters | yes | yes | | device facts (`mikroscope_device_info`, ceilings, cadences) | yes | yes, once the collector has fetched `/capabilities` | | the sampler's timing histograms and `mikroscope_slipped_total` | yes | no | | trigger and capture families | yes, while `CAPTURE_MB` is above 0 | no | | `mikroscope_collector_*`, `mikroscope_derived_*` | no | yes | | `mikroscope_api_*` | no | yes, once the API tier has delivered a sample | The collector leaves `mikroscope_slipped_total` out rather than writing 0, because a 0 there would be a claim about a sampler it never ran. To have both sets in one Prometheus, scrape the collector for everything and the agent only for what the collector cannot produce. Scraping the agent without the keep list doubles every counter the collector also exposes: ```yaml - job_name: "mikroscope" scrape_interval: 5s static_configs: [{ targets: [":9124"] }] - job_name: "mikroscope-agent" scrape_interval: 5s static_configs: [{ targets: ["172.30.10.2:9123"] }] metric_relabel_configs: - source_labels: [__name__] regex: "mikroscope_(tick_.*|trigger_.*|capture.*|captures_held|slipped_total)" action: keep ``` ### What the collector's copy does differently The collector's sink is built for a nominal 10 Hz whatever rate the agent runs at (`promHistogramRateHz` in `cmd/mikroscope/sinkflags.go`), so that its bucket layout does not change when it reconnects to a differently configured agent. Five things follow from that constant, read from the code: - `mikroscope_info{rate_hz}` on the collector reads `10`, not the agent's rate. `/healthz` has the agent's. - `mikroscope_cpu_busy_ticks` has the buckets `le="0"` to `le="11"`, sized for a 100 ms sample. An agent below 10 Hz puts its busier samples in `+Inf`. - The ring behind the trailing windows holds 60 s of the connected agent's samples: the collector reads the agent's rate from its health check and sizes the ring from it, so `window="60s"` is 60 s at any rate. - The trailing mean behind `mikroscope_softnet_burst_samples_total` has a weight of 1/600, which is a 60 s memory at 10 Hz and shorter above it. - An interrupt line leaves `mikroscope_irq_total` after 36 000 samples out of every top-K, which is one hour only at 10 Hz: 12 min at 50 Hz, 6 min at 100 Hz. The check runs every 1 000 samples, so a line can stay up to that much longer. The counters on the collector count from the collector's start, and `mikroscope_uptime_seconds` and `mikroscope_info{version}` describe the collector. `mikroscope_self_*` still describe the agent: they come from the agent's samples. > **Untested** > > The consequences listed above for an agent running at a rate other than 10 Hz are arithmetic from > `internal/sinks/prometheus.go`. None of them was compared against the agent's own exposition at 50 > or 100 Hz. ## Conventions - **Counters count since the exporter started and never reset on a scrape.** The ring carries deltas; `/metrics` accumulates them. `rate()` over any range is then correct, and two scrapers see the same values. - **Gauges are the newest sample.** A floored source (thermal, cpufreq, slab, buddyinfo, MTD) is absent from most samples by design, so its gauge holds the last reading between emissions, and `mikroscope_source_age_seconds` says how old that reading is. - **Absent is not zero, with five exceptions.** Most families for a source the kernel does not have, or the deployment cannot read, are not rendered at all. The exceptions are `mikroscope_meminfo_kbytes`, `mikroscope_load`, `mikroscope_threads`, `mikroscope_self_rss_bytes` and `mikroscope_irq_errors_total`: they are written from the first sample and read 0 when their file cannot be read or `SOURCES` leaves it out. `mikroscope_irq_errors_total` also reads 0 on a kernel whose `/proc/interrupts` has no `Err` row. - **One dimension, one name.** A processor is the label `cpu` everywhere. Which cores, zones, interrupt lines, caches and counters exist is a property of the board; read the label, never assume a set. - **No ratio in the samples.** The busy-ratio gauges are window statistics computed at scrape, and the collector's `mikroscope_derived_*` gauges are divisions it made beside their inputs. Every other ratio is yours to divide. ## CPU ticks and `/proc/stat` | Family | Type | Labels | Meaning | | -------------------------------------- | --------- | ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_cpu_ticks_total` | counter | `cpu`, `mode`: `user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal` | `USER_HZ` ticks per core and mode | | `mikroscope_cpu_aggregate_ticks_total` | counter | `mode` | the same, from `/proc/stat`'s own `cpu` summary line; a separate name so summing the per-core series cannot double-count it | | `mikroscope_cpu_busy_ticks` | histogram | `cpu`, `le` | busy ticks per sample, one bucket per achievable integer, 0 up to one more than a period holds, plus `+Inf`. `_sum` is the exact busy total. Recovers time above a threshold to one sample, not contiguity | | `mikroscope_context_switches_total` | counter | none | `ctxt` | | `mikroscope_interrupts_total` | counter | none | `intr`, every source | | `mikroscope_forks_total` | counter | none | the `processes` line: RouterOS spawning scripts, fetches and containers, although the PID namespace hides the processes | | `mikroscope_procs_blocked` | gauge | none | tasks in uninterruptible sleep; on a kernel without PSI the only direct stall signal | A tick is busy when it is `user`, `nice`, `system`, `irq`, `softirq` or `steal`. On the RB5009 with RouterOS 7.24.2 the `irq` column is always 0 (measured 2026-09-11), so hard-IRQ time is inside `system`: read the `irq` mode as absent there, not as a router with no interrupt load. ## Trailing windows and busy runs Computed at scrape from the ring over fixed wall-clock windows, so a scraper at any interval up to 60 s sees a transient's peak whoever scraped last. | Family | Type | Labels | Meaning | | -------------------------------------- | --------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `mikroscope_cpu_busy_ratio_window` | gauge | `cpu`, `window`: `1s`, `10s`, `60s`; `stat`: `max`, `min`, `p95` | busy ratio statistics over the trailing window | | `mikroscope_sample_interval_seconds` | gauge | `window`, `stat` | the sampler's measured interval over the trailing window; its `max` says whether the rate was delivered | | `mikroscope_cpu_busy_run_seconds` | histogram | `cpu`, `threshold`: `0.5`, `0.9`; `le`: 0.1, 0.2, 0.5, 1, 2, 5, 10, 30, 60, `+Inf` | length of each run of consecutive samples at or above the busy ratio, in seconds of the samples' own intervals, observed when the run ends | | `mikroscope_cpu_busy_run_open_seconds` | gauge | `cpu`, `threshold` | how long the run in progress has lasted; 0 below the threshold | A 2 s plateau is one observation of 2 in `mikroscope_cpu_busy_run_seconds` and twenty scattered 100 ms spikes are twenty of 0.1; the busy-tick histogram cannot tell those apart. ## The sampler's own timing Agent only. A sample is read over a stretch of time, not at an instant, and these families measure that stretch. | Family | Type | Labels | Meaning | | -------------------------------------- | --------- | ------ | -------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_tick_interval_seconds` | histogram | `le` | measured interval between samples; buckets at 0.5, 0.9, 0.95, 0.99, 1.01, 1.05, 1.1, 1.25, 1.5, 2 and 5 times the nominal period | | `mikroscope_tick_wake_latency_seconds` | histogram | `le` | how late the loop ran after its ticker fired; buckets 0.1 ms, 0.25, 0.5, 1, 2, 5, 10, 20, 50, 100 ms | | `mikroscope_tick_read_seconds` | histogram | `le` | how long reading every due source took; same buckets | | `mikroscope_slipped_total` | counter | none | ticks whose read finished after the next tick was due | A slipped tick is not a lost sample: the sample is still produced with its real `dt_ns`. If `mikroscope_slipped_total` moves, distrust the sampler's own accounting before the router's. ## Receive path and interrupts `/proc/net/softnet_stat`, `/proc/softirqs` and `/proc/interrupts` are global inside the container, unlike `/proc/net/dev`. | Family | Type | Labels | Meaning | | ------------------------------------------- | ------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_softnet_total` | counter | `cpu`, `kind`: `processed`, `dropped`, `time_squeeze` | per-CPU softnet counters | | `mikroscope_softnet_squeezed_samples_total` | counter | `cpu` | samples in which that CPU had any squeeze or drop; its rate is a duty cycle | | `mikroscope_softnet_burst_samples_total` | counter | `cpu` | the squeezed samples whose packet count was at or below that CPU's trailing mean (EWMA with a one-minute memory); evidence of bursts shorter than a sample, not a count of them | | `mikroscope_softirq_total` | counter | `cpu`, `kind` | softirq counts; which kinds exist is the kernel's list. `NET_RX` carries a router's forwarding load | | `mikroscope_irq_total` | counter | `irq`, `name`, `cpu` | lines that appeared in any sample's top-K. A line out of every top-K for an hour leaves the family, and restarts from 0 if it returns; on the collector the hour holds only at 10 Hz (above) | | `mikroscope_irq_delivered_total` | counter | none | every interrupt line summed, including those outside the top-K: the denominator for `mikroscope_irq_total` | | `mikroscope_irq_errors_total` | counter | none | the `Err` row of `/proc/interrupts`, which the top-K never shows while it sits at zero | Match an interrupt on its `name` label or on its rate, never on a hardcoded name: which lines a NIC raises is a property of the board and its driver. ## Scheduler, load and PSI | Family | Type | Labels | Absent when | Meaning | | ------------------------------------- | ------- | -------------------------------------------------------------------------------------------- | ----------------------------------- | --------------------------------------- | | `mikroscope_load` | gauge | `period`: `1m`, `5m`, `15m` | no sample yet | load averages | | `mikroscope_threads` | gauge | `state`: `running`, `total` | no sample yet | from `/proc/loadavg` | | `mikroscope_psi_stall_usec_total` | counter | `resource`, `kind`: `cpu`/`some`, `memory`/`some`, `memory`/`full`, `io`/`some`, `io`/`full` | the kernel has no `/proc/pressure` | PSI stall microseconds | | `mikroscope_sched_run_seconds_total` | counter | `cpu` | the kernel has no `/proc/schedstat` | time tasks spent on each CPU | | `mikroscope_sched_wait_seconds_total` | counter | `cpu` | the same | time runnable tasks waited for each CPU | Both PSI and schedstat families are absent on the RB5009, whose RouterOS 7.24.2 kernel has neither file: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-11 · `/proc/pressure` and `/proc/schedstat` absent ## Memory | Family | Type | Labels | Meaning | | ------------------------------ | ------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_meminfo_kbytes` | gauge | `field` | `/proc/meminfo` in kB: `MemTotal`, `MemFree`, `MemAvailable`, `Buffers`, `Cached`, `Dirty`, `Writeback`, `Shmem`, `Slab`, `SReclaimable`, `SUnreclaim`, `AnonPages`, `Mapped`, `KernelStack`, `PageTables`, `Active`, `Inactive`, `Committed_AS`, `CommitLimit` | | `mikroscope_vm_events_total` | counter | `event` | `/proc/vmstat` events: `pgfault`, `pgmajfault`, `pgscan_kswapd`, `pgscan_direct`, `pgsteal_kswapd`, `pgsteal_direct`, `pgalloc`, `pgfree`, `allocstall`, `compact_stall`, `oom_kill`, `pswpin`, `pswpout`. `pgalloc` and `allocstall` are summed over zones | | `mikroscope_vm_pages` | gauge | `field` | `/proc/vmstat` levels in pages: `nr_free_pages`, `nr_dirty`, `nr_writeback`, `nr_slab_reclaimable`, `nr_slab_unreclaimable` | | `mikroscope_buddy_free_blocks` | gauge | `node`, `zone`, `order` | free blocks of 2^order pages from `/proc/buddyinfo`: fragmentation that `MemFree` cannot show | The events and the levels are separate families on purpose: `nr_dirty` falling is pages being written back, not a negative event count. `MemFree` in kB divided by `nr_free_pages` gives the page size from the data rather than from an assumption. The vmstat families are absent until a sample shows a fault or a free-page level, because an unreadable `/proc/vmstat` and a quiet tick both arrive as zeros. ## PMU and CPU frequency | Family | Type | Labels | Absent when | Meaning | | -------------------------------------------- | ------- | ---------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_perf_events_total` | counter | `counter`, `cpu` | not privileged, or no reachable PMU | hardware counter events from `perf_event_open`; a counter the CPU does not implement is absent | | `mikroscope_perf_time_enabled_seconds_total` | counter | `counter`, `cpu` | the same | seconds each counter was enabled | | `mikroscope_perf_time_running_seconds_total` | counter | `counter`, `cpu` | the same | seconds it was actually counting; below `enabled` the PMU is multiplexed and the counts are scaled down by running over enabled | | `mikroscope_cpu_clock_cycles_total` | counter | `cpu` | no cpufreq | the governor's frequency integrated over each sample's interval: nominal cycles offered, not cycles retired. `rate()` of it is the mean frequency over any window | | `mikroscope_cpu_frequency_hertz` | gauge | `cpu` | no cpufreq | the governor's frequency at the newest emission | Instructions per cycle is `rate(mikroscope_perf_events_total{counter="instructions"})` over `rate(…{counter="cycles"})`; the agent ships the counts and never the ratio. ## Temperature and slab caches | Family | Type | Labels | Absent when | Meaning | | -------------------------------- | ----- | ------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `mikroscope_thermal_celsius` | gauge | `zone` | no thermal zone | temperature under the kernel's own zone name, for example `cpu-thermal`, `soc-thermal` on the RB5009 | | `mikroscope_slab_active_objects` | gauge | `cache` | not privileged | active objects per slab cache; `nf_conntrack` is the router's real connection count, although the container's namespace reports zero | | `mikroscope_slab_limit_objects` | gauge | `cache` | not privileged, or no ceiling published | the ceiling for caches that have one, today `nf_conntrack` from `nf_conntrack_max`; read once at agent start | Connection-table occupancy is `mikroscope_slab_active_objects{cache="nf_conntrack"} / ignoring(cache) mikroscope_slab_limit_objects{cache="nf_conntrack"}`. A change to `nf_conntrack_max` shows after the agent restarts. ## Flash and block devices | Family | Type | Labels | Absent when | Meaning | | ----------------------------------------- | ------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | `mikroscope_flash_operations_total` | counter | `device`, `kind`: `page_writes`, `page_reads`, `erasures`, `gc_copies`, `gcs` | no `/proc/yaffs` | YAFFS NAND operations; `erasures` maps to flash lifetime, `gc_copies` over `page_writes` is write amplification | | `mikroscope_flash_bad_blocks` | gauge | `device` | no `/proc/yaffs`, and on any scrape whose newest sample carried no flash row (no operation, free chunks unchanged); not held between emissions | blocks the NAND has retired; a rise is the flash wearing out | | `mikroscope_flash_free_chunks` | gauge | `device` | the same as `mikroscope_flash_bad_blocks` | chunks still free | | `mikroscope_mtd_ecc_corrected_bits_total` | counter | `device`, `partition` | not privileged | bits the ECC corrected since boot, the kernel's own count; climbs before a block is retired | | `mikroscope_mtd_ecc_failures_total` | counter | `device`, `partition` | not privileged | reads the ECC could not correct since boot: data loss | | `mikroscope_mtd_blocks` | gauge | `device`, `partition`, `kind`: `bad`, `bbt` | not privileged | bad blocks, and blocks the bad-block table occupies | | `mikroscope_mtd_bitflip_threshold` | gauge | `device`, `partition` | not published | corrected bits per ECC step at which the kernel moves a block's data | | `mikroscope_mtd_ecc_strength` | gauge | `device`, `partition` | not published | most bits per ECC step the code can correct | | `mikroscope_disk_operations_total` | counter | `device`, `op`: `read`, `write` | no device has done I/O | requests completed | | `mikroscope_disk_sectors_total` | counter | `device`, `op` | the same | sectors transferred; the conversion to bytes is left to the reader | | `mikroscope_disk_io_seconds_total` | counter | `device` | the same | time the device had I/O in flight | | `mikroscope_disk_inflight` | gauge | `device` | no I/O on that device in the newest sample; not held between emissions | requests in flight | A block device that did nothing is not in the sample and so not here either; a comment in `internal/agent/metrics.go`, undated and with no RouterOS version, says the RB5009 lists sixteen idle `nbd` devices. Unlike the floored sources in the conventions above, the flash levels and `mikroscope_disk_inflight` are not carried forward, so they are missing from most scrapes of a quiet board. ## Kernel log `/dev/kmsg` is root-only, so these families need a privileged container. The record text is never a label. | Family | Type | Labels | Meaning | | ------------------------------------ | ------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_kmsg_records_total` | counter | `level`: `emerg`, `alert`, `crit`, `err`, `warn`, `notice`, `info`, `debug` | records per syslog severity | | `mikroscope_kmsg_dropped_total` | counter | none | loss events, not records: one per tick that hit the cap of 64 records, one per kernel ring overrun (which can stand for many records); while it moves, the per-level counts above are a lower bound | | `mikroscope_kmsg_port_records_total` | counter | `port`, `kind`, `level` | records whose text named a port: a subset of the family above. `kind` is `link-up`, `link-down`, `stp-` (`blocking`, `listening`, `learning`, `forwarding`, `disabled`), `own-address` — the bridge received a frame carrying its own MAC as source address, the layer-2 loop signature — or `other`; only non-zero triples are written | Both `mikroscope_kmsg_records_total` and `mikroscope_kmsg_dropped_total` are rendered from the start, at 0, on both expositions when the capabilities list `kmsg` as a source, so a quiet router reads as silent and not as unreadable. Without that, they appear with the first record. The port family appears with the first record that names a port. On the agent, `port` is the RouterOS default name on a board in the port table and the kernel name otherwise. On the collector's copy the API tier's interface inventory puts the port's current RouterOS name there, so a port renamed from `ether5` to `WAN` is counted under `WAN`; without an API tier the record keeps the board's default name. The collector classifies any record that reaches it without a `kind` of its own, so its exposition carries the label even when the agent's does not — which is how the reference RB5009 stands on 2026-09-16, with an agent whose `/metrics` has no `kind`. A port coming up writes four records on a bridge, not four faults: `link-up`, then `stp-blocking`, `stp-learning` and `stp-forwarding` on its bridge port. `own-address` is the one kind that is a fault on its own. ## The observer itself | Family | Type | Labels | Meaning | | ----------------------------------------- | ------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_self_cpu_usec_total` | counter | none | CPU microseconds the agent's container used: from cgroup2 when mounted, else `/proc/self/stat` ticks | | `mikroscope_self_rss_bytes` | gauge | none | the agent's resident set | | `mikroscope_self_cgroup_memory_bytes` | gauge | none | the container cgroup's `memory.current`, RSS plus page cache charged to it; absent without cgroup2 | | `mikroscope_self_throttled_periods_total` | counter | none | periods the container's `cpu.max` quota stopped it; absent without cgroup2 | | `mikroscope_self_throttled_seconds_total` | counter | none | seconds it spent stopped; absent without cgroup2 | | `mikroscope_self_oom_kills_total` | counter | none | processes OOM-killed inside the agent's own cgroup; not the router's, which is `mikroscope_vm_events_total{event="oom_kill"}` | | `mikroscope_samples_total` | counter | none | samples folded into these counters | | `mikroscope_sample_seq_total` | counter | none | sequence number of the newest sample; `increase()` of it against `increase(mikroscope_samples_total)` is exactly the ticks produced and not folded in | | `mikroscope_sampled_seconds_total` | counter | none | the samples' own intervals summed; `rate()` of it is the wall-clock time covered per second, 1 while no tick slipped. The denominator for derived ratios | | `mikroscope_counter_resets_total` | counter | none | monotonic counters that went backwards without a 32-bit wrap; a rate across such a tick is a lower bound | | `mikroscope_info` | gauge | `version`, `rate_hz` | always 1 | | `mikroscope_uptime_seconds` | gauge | none | seconds since this exporter started | The agent's cost is two reads of `mikroscope_self_cpu_usec_total` 60 s apart at steady state, divided by 60 000 000. [The cost of the observer](/mikroscope/cost/) has the procedure and the measured figures. ## Device facts What the agent established about the board at start, with no RouterOS API. Each family is absent when the device does not publish that fact. | Family | Type | Labels | Meaning | | ----------------------------------------- | ----- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_device_info` | gauge | `board`, `kernel`, `cores`, `privileged`, `cgroup`, `ports_from`, `hash` | always 1; `hash` changes when the source set does | | `mikroscope_thermal_critical_celsius` | gauge | `zone` | the lowest critical trip point the zone declares; passive and active trips are not included | | `mikroscope_thermal_polling_seconds` | gauge | `zone` | the zone's `polling_delay`: reads faster than this see the same value | | `mikroscope_cpu_frequency_limit_hertz` | gauge | `cpu`, `bound`: `min`, `max` | the hardware clock range | | `mikroscope_cpu_frequency_step_hertz` | gauge | `cpu`, `step` | every frequency the driver uses; `step` counts from the slowest | | `mikroscope_cpu_frequency_governor_info` | gauge | `cpu`, `governor` | always 1; `userspace` is a pinned clock, `ondemand` or `schedutil` one that scales | | `mikroscope_cpu_frequency_cluster` | gauge | `cpu` | the lowest-numbered core that changes frequency with this one; `{0,1}` and `{2,3}` on the RB5009 | | `mikroscope_self_cgroup_memory_max_bytes` | gauge | none | the container's own `memory.max`, as the operator set it | | `mikroscope_source_cadence_hz` | gauge | `source`, `reason` | the rate each level source is read and stored at, and why: `declared`, `policy`, `budget`, `change`, `override` or `rate` | | `mikroscope_source_age_seconds` | gauge | `source` | seconds since each floored source (`thermal`, `cpufreq`, `slabinfo`, `buddyinfo`, `mtd`) was last actually read | All but the last are read once at agent start; a ceiling changed while the agent runs shows after it restarts. ## Triggers and captures Agent only, and only while `CAPTURE_MB` is above 0. Every condition-and-reason pair is written from the start at 0. | Family | Type | Labels | Meaning | | --------------------------------------- | ------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | | `mikroscope_trigger_fired_total` | counter | `condition` | times each configured condition armed a capture; `condition="manual"` appears after the first `POST /capture` | | `mikroscope_trigger_suppressed_total` | counter | `condition`, `reason`: `refractory`, `pending` | times a condition was true and nothing was armed: how much of a burst was not seen | | `mikroscope_capture_refused_total` | counter | `reason`: `budget`, `empty` | captures collected and not kept: the budget was full, or the ring no longer held the window | | `mikroscope_captures_held` | gauge | none | captures retained | | `mikroscope_capture_bytes` | gauge | none | ring bytes the retained captures pin | | `mikroscope_capture_budget_bytes` | gauge | none | the budget | | `mikroscope_capture_bytes_served_total` | counter | none | bytes handed out over `/captures/{id}`, which runs on the sampler's core | ## Collector only: the derive stage | Family | Type | Labels | Meaning | | -------------------------------------------- | ------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_collector_gaps_total` | counter | none | ring gaps the collector saw: samples lost between pulls | | `mikroscope_collector_triggers_total` | counter | `cause` | capture triggers the collector saw the agent fire; absent until the first. The windows stay on the agent under `/captures` | | `mikroscope_collector_detections_total` | counter | `rule` | detection events per rule, every rule at 0 from the first scrape: `counter-reset`, `agent-restart`, `agent-oom`, `microburst`, `reboot`, `link-flap`, `conntrack-cliff`, `conntrack-high`, `thermal-high`, `thermal-rising`, `ipc-collapse` | | `mikroscope_collector_bursts_total` | counter | none | samples flagged as a sub-sample burst | | `mikroscope_derived_memory_pressure` | gauge | none | the allocator's ladder at the newest sample: 0 none, 1 kswapd scanned, 2 direct reclaim, 3 an allocation stalled or a page swapped out, 4 the OOM killer ran | | `mikroscope_derived_cycles_per_packet` | gauge | none | PMU cycles per packet, summed over cores; absent without a PMU, in a sample with no packets, and in one with a counter reset | | `mikroscope_derived_instructions_per_packet` | gauge | none | the same, instructions | | `mikroscope_derived_cache_misses_per_packet` | gauge | none | the same, cache misses | | `mikroscope_derived_packets_per_interrupt` | gauge | none | packets per device interrupt, the NAPI coalescing depth; absent when the timer row was not in the sample's top-K | | `mikroscope_derived_fastpath_share` | gauge | `interface`, `direction`: `rx`, `tx` | the fast-path share of the traffic the interface hands the CPU, between the last two counter polls: `fp-rx-byte` over `driver-rx-byte` on a switch port, over `rx-byte` on a software interface. Absent for a direction that moved no bytes | The fast-path share is not a share of the wire: a frame the switch chip forwards in hardware is in neither of its two numbers. On a switch port the two are close together — measured 2026-09-16 on the reference RB5009, `fp-rx-byte` equals `driver-rx-byte` within a few kB, so every port reads about 100 %. The software interfaces are where the number moves: the bridge fast-pathed 211.9 GB of the 663.0 GB it took to the CPU since boot (32 %), `PPPoE_DIGI` 99.97 %. `fp-tx-byte` reads 0 on every interface of that router after hundreds of GB transmitted, so `direction="tx"` is withheld while the cumulative `fp-tx-byte` is 0 rather than published as a fabricated 0 %. The rules and what each may not claim are on [detections](/mikroscope/sinks/detections/); the derived values on [what the collector derives](/mikroscope/sinks/derive/). ## Collector only: the RouterOS API tier Absent until the API tier has delivered a sample, and absent entirely with `--api-mode off` unless `--api-every` is also given explicitly. | Family | Type | Labels | Meaning | | ---------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `mikroscope_api_up` | gauge | none | 1 once the API tier has delivered a sample. It does not go back to 0 if the tier stops later, although its HELP line says "while" | | `mikroscope_api_cpu_load` | gauge | none | `cpu-load` as `/system/resource` reports it, a one-second average | | `mikroscope_api_memory_bytes` | gauge | `kind`: `free`, `total` | from `/system/resource` | | `mikroscope_api_uptime_seconds` | gauge | none | the router's uptime | | `mikroscope_api_core_percent` | gauge | `cpu`, `kind`: `load`, `irq`, `disk` | `/system/resource/cpu` | | `mikroscope_api_health` | gauge | `name` | `/system/health` readings, by RouterOS's name | | `mikroscope_api_interface` | gauge | `interface`, `kind`: `rx_bps`, `tx_bps`, `rx_pps`, `tx_pps`, and `rx_drops`, `tx_drops`, `tx_queue_drops`, `rx_errors`, `tx_errors` only when the router returned them | instantaneous rates from `monitor-traffic` | | `mikroscope_api_interface_info` | gauge | `interface`, `label`, `type`, `role`, `bridge`, `default_name` | always 1, one series per interface: `label` is its RouterOS comment, `type` RouterOS's own interface type, `role` its interface lists, `bridge` the bridge it is a port of, `default_name` the factory name of a physical port. An empty value is how the exposition spells "none" | | `mikroscope_api_interface_counter_total` | counter | `interface`, `counter` | every numeric per-port counter RouterOS keeps, under its own name (`rx-overflow`, `fp-rx-byte`, `link-downs` …), for every interface | | `mikroscope_api_conntrack_entries` | gauge | none | `/ip/firewall/connection` count, held between polls; only with `--conntrack-every` | What each interface is — comment, type, interface lists, bridge — is an info family and not a set of labels on every rate and counter, because a human edits those and a changed label starts a new series. The collector reads the configuration once at start and again every `--labels-every` (5 minutes by default), and the family holds one series per interface, with or without a comment. Join it in a query: ```text mikroscope_api_interface_counter_total * on(interface) group_left(label, type, role) mikroscope_api_interface_info ``` The join is worth making because RouterOS counts different things on different types, and `type` is what says which: an `ether` port in a bridge counts its wire, including the frames the switch chip forwarded in hardware, while the `bridge` counts its CPU side. Measured 2026-09-16 on the reference RB5009, `ether1` had received 255.8 GB on the wire and 29.7 GB at the driver. The two are different planes and neither is a subset of the other: never sum a port and its bridge. Sizes and configuration that happen to parse as integers — `mtu`, `actual-mtu`, `l2mtu`, `max-l2mtu` and `sfp-shutdown-temperature` — are not in `mikroscope_api_interface_counter_total`, because a `rate()` of an MTU counts nothing. The MTU travels with the interface inventory instead, which the row sinks write as a field. On the RB5009 with RouterOS 7.24.2, `monitor-traffic` returned `rx-drops`, `tx-drops` and `tx-queue-drops` per second and no error keys at all (2026-09-15), so the `rx_errors` and `tx_errors` kinds are absent there; `mikroscope_api_interface_counter_total` carries the port's typed errors instead. > **Help text that says otherwise** > > The HELP line of `mikroscope_kmsg_records_total` still says the family appears only once a record > has been seen. The code also renders it at 0 from the start when the capabilities list `kmsg`; > this page follows the code. ## See also - [Prometheus](/mikroscope/sinks/prometheus/): running the collector's exposition and scraping it. - [InfluxDB and SQL measurements](/mikroscope/reference/measurements/): the same data as rows. - [The agent's HTTP endpoints](/mikroscope/reference/http/): `/metrics` and the paths beside it. - [What the numbers do not say](/mikroscope/cost/limits/): what these families can and cannot recover. --- # InfluxDB and SQL measurements Every measurement the InfluxDB line-protocol encoder writes and every table the SQL sink creates, with tags, fields, columns, keys, units and whether each value is a delta or a level. Source: https://jmrplens.github.io/mikroscope/reference/measurements/ This page answers what a row in the store holds: which measurement or table, which tags or key columns identify it, which fields it carries, in what unit, and whether a value is a delta over the sample's interval or a level. It is read from `internal/sinks/influx.go`, `device.go` and `sql.go`. The two stores do not hold the same set; [where they differ](#where-the-two-stores-differ) is the last section. ## Which sinks write these - `forward --influx` posts the InfluxDB line protocol below. - `forward --stdout lp` and `forward --telegraf` render with the same encoder, so they write the same measurements and lines. The order of the `mikroscope_api_health`, `mikroscope_softirq` and `mikroscope_slab` lines within a batch is not stable: the encoder walks Go maps for them, and `internal/sinks/telegraf.go` records 12 renders of an 8-name map giving 7 distinct orders (2026-09-12). - `forward --sql` writes its own PostgreSQL tables, listed further down. Loki, OTLP, Graphite and Elasticsearch shape the same timeline differently; they are on [the file and the other sinks](/mikroscope/sinks/other/). ## Conventions - **Every row carries `host`**, the `--host-tag` value (`router` by default), as a tag in InfluxDB and as a `host` column in SQL. - **Timestamps are the agent's wall clock**, in ns for InfluxDB and as `TIMESTAMPTZ` for SQL. An API-tier row is stamped with the collector's clock corrected by the measured skew, so both tiers share the agent's time. Gaps and the device-info rows have no clock of their own and carry the collector's at the moment it handled them. - **Counters are deltas since the previous sample**, not running totals, and levels are the value as read. The tables below say which is which; never sum a level. - **Absent is absent, for the per-source measurements.** `psi`, `thermal`, `slab`, `flash`, `mtd`, `disk`, `perf`, `kmsg`, `buddy` and `irq` write no row for a source the kernel does not have or the deployment cannot read. A ceiling the device does not publish is a missing field in InfluxDB and `NULL` in SQL, never 0. The exception: InfluxDB's `mikroscope_mem`, `mikroscope_load`, `mikroscope_vm`, `mikroscope_vm_level`, `mikroscope_stat`, `mikroscope_sample` and `mikroscope_self`, and SQL's `mikroscope_mem`, `mikroscope_load`, `mikroscope_stat` and `mikroscope_self`, are written on every sample and read 0 for a source that could not be read. - **One dimension, one name.** A processor is `cpu` in every tag and column. - **The unit is in the field name**: `_kb`, `_khz`, `_ns`, `_us`, `_s`, `_ms`, `_bps`, `_pps`. Temperatures are `celsius` beside `critical_celsius`; block-device busy time is `io_s`, converted once from the kernel's milliseconds. In line protocol a trailing `u` is an unsigned integer, `i` a signed one, a bare number a float, `true`/`false` a boolean, and a quoted value a string. ## InfluxDB: the kernel tier One set of rows per kernel sample, all at that sample's `wall_ns`. ### CPU, interrupts and the sample itself | Measurement | Tags | Fields | Kind | | -------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `mikroscope_cpu` | `cpu` | `user`, `nice`, `system`, `idle`, `iowait`, `irq`, `softirq`, `steal` (u, `USER_HZ` ticks); `busy_ratio` (float, 4 decimals); `dt_ns` (i) | ticks are deltas; `busy_ratio` is busy ticks over `dt_ns`, capped at 1 | | `mikroscope_cpufreq` | `cpu` | `khz` (u); `max_khz` (u) where the core publishes a ceiling | level, on the samples that carry it: on change or the 60 s heartbeat | | `mikroscope_stat` | none | `ctxt`, `intr`, `forks`, `irq_total`, `irq_err` (u) | deltas | | `mikroscope_softnet` | `cpu` | `processed`, `dropped`, `time_squeeze` (u) | deltas | | `mikroscope_softirq` | `kind`, `cpu` | `count` (u) | delta; only non-zero (kind, cpu) pairs | | `mikroscope_irq` | `irq`, `name` | `count` (u), summed over CPUs | delta; the sample's top-K lines only | | `mikroscope_irq_cpu` | `irq`, `name`, `cpu` | `count` (u) | delta; only non-zero CPUs | | `mikroscope_sample` | none | `seq` (u), `dt_ns` (i), `mono_ns` (i) | one row per sample | | `mikroscope_psi` | none | `cpu_some_us`, `mem_some_us`, `mem_full_us`, `io_some_us`, `io_full_us` (u) | deltas; only on a kernel with PSI | `busy_ratio` is the one ratio in the kernel tier, and it is computed by the sink, not shipped by the agent. The ticks beside it are what to divide when the window matters: sum `user + nice + system + irq + softirq + steal` and divide by `dt_ns / 1e9 × 100`. ### Memory and load | Measurement | Tags | Fields | Kind | | --------------------- | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | | `mikroscope_mem` | none | `total_kb`, `free_kb`, `available_kb`, `cached_kb`, `buffers_kb`, `slab_kb`, `sreclaimable_kb`, `sunreclaim_kb`, `anon_kb`, `mapped_kb`, `dirty_kb`, `writeback_kb`, `kernel_stack_kb`, `page_tables_kb`, `committed_kb`, `commit_limit_kb`, `shmem_kb`, `active_kb`, `inactive_kb` (u) | levels | | `mikroscope_load` | none | `load1`, `load5`, `load15` (float, 2 decimals); `running`, `threads`, `procs_blocked` (u) | levels | | `mikroscope_vm` | none | `pgfault`, `pgmajfault`, `pgscan_kswapd`, `pgscan_direct`, `pgsteal_kswapd`, `pgsteal_direct`, `pgalloc`, `pgfree`, `allocstall`, `compact_stall`, `oom_kill`, `pswpin`, `pswpout` (u) | deltas | | `mikroscope_vm_level` | none | `nr_free_pages`, `nr_dirty`, `nr_writeback`, `nr_slab_reclaimable`, `nr_slab_unreclaimable` (u, pages) | levels | | `mikroscope_buddy` | `node`, `zone` | `free_pages` (u, the sum over orders in pages); `order_0` … `order_N` (u, free blocks of 2^N pages) | levels, on change or heartbeat | `mikroscope_vm` and `mikroscope_vm_level` are separate measurements because a delta of `pgscan` is an event rate and `nr_dirty` is a depth. ### Sensors, slab caches and flash | Measurement | Tags | Fields | Kind | | -------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | | `mikroscope_thermal` | `zone` | `celsius` (float, 3 decimals); `critical_celsius` (float) where the zone declares a critical trip | level, at the zone's declared cadence | | `mikroscope_slab` | `cache` | `active` (u, objects); `limit` (u) for caches with a published ceiling, today `nf_conntrack` | level; [needs `privileged=yes`](/mikroscope/limits/privileged/); stored on change at a budget floor | | `mikroscope_flash` | `device` | `page_writes`, `page_reads`, `erasures`, `gc_copies`, `gcs` (u); `bad_blocks`, `free_chunks` (u) | first five deltas, last two levels; only when something changed | | `mikroscope_mtd` | `device`, `partition` | `corrected_bits`, `ecc_failures`, `bad_blocks`, `bbt_blocks` (u); `bitflip_threshold`, `ecc_strength` (u) where published | levels: the kernel's since-boot counts as read, never differenced; [needs `privileged=yes`](/mikroscope/limits/privileged/) | | `mikroscope_disk` | `device` | `reads`, `read_sectors`, `writes`, `write_sectors` (u); `io_s` (float, 3 decimals); `inflight` (u) | `inflight` is a level, the rest deltas; an idle device writes no row | `limit` is an SQL keyword, so an InfluxDB 3 SQL query double-quotes it, as the shipped connection-table panel does: `max("limit")`. The shipped CPU-frequency panel double-quotes the `cluster` field of `mikroscope_device_cpufreq` the same way (`internal/dashboards/panels_p5.go`); no source in the repository says why. > **The generated InfluxDB alert asks for the SQL column name** > > `mikroscope dashboards gen` writes a conntrack alert into `mikroscope-alerts-influxdb.yaml` whose > query reads `limit_objs` from `mikroscope_slab`. That is the SQL sink's column; InfluxDB's field > is `limit`. Read from `internal/dashboards/alerts.go` and `internal/sinks/influx.go`; the rule was > not run against a store for this page. ### Observer, PMU and kernel log | Measurement | Tags | Fields | Kind | | ----------------- | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_self` | none | `cpu_us` (u); `rss`, `cgroup_mem` (u, bytes); `cgroup_mem_max` (u) on the samples that carry it; `throttled`, `throttled_us`, `oom_kill` (u) with cgroup2; `resets`, `kmsg_dropped`, `seq` (u) | `cpu_us`, `throttled`, `throttled_us`, `oom_kill`, `resets`, `kmsg_dropped` are deltas; `rss`, `cgroup_mem`, `cgroup_mem_max` levels | | `mikroscope_perf` | `counter`, `cpu` | `count` (u); `enabled_ns`, `running_ns` (u) when the kernel reported them | deltas; [needs `privileged=yes`](/mikroscope/limits/privileged/); `running_ns` below `enabled_ns` means the count is multiplexed | | `mikroscope_kmsg` | `level`, `port`, `kind`, `label`, `role` | `count` (u), records in this sample | per (level, port, kind); a record that names no port carries none of `port`, `kind`, `label` and `role`; `label` and `role` only where the inventory knows them; only non-zero combinations | `resets` counts monotonic counters that went backwards this tick without a 32-bit wrap, and `kmsg_dropped` kernel-log loss events, not records: one for a tick that hit the 64-record cap, one per kernel ring overrun, which can stand for many records. Both are usually 0; either one non-zero means the tick is not to be trusted as a rate. The text of kernel-log records is not in InfluxDB; `port` is the interface's current RouterOS name where the collector holds the API tier's inventory, the board's default name otherwise, and the kernel's own name on a board with no port table. `kind` says what happened to that port: `link-up`, `link-down`, `stp-blocking`, `stp-listening`, `stp-learning`, `stp-forwarding`, `stp-disabled`, `own-address` — the bridge received a frame carrying its own MAC as source address, the layer-2 loop signature — or `other`. A link coming up writes four records, not four faults: the link-up and the three STP transitions of its bridge port. `label` and `role` are the port's RouterOS comment and its interface lists, tagged only where the inventory has them. Because a query naming a column the store has never received fails at planning time, `kind` can be queried only after the first port record has been written with it. ## InfluxDB: the RouterOS API tier One set per API poll, stamped in the agent's time. | Measurement | Tags | Fields | Kind | | --------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_api_system` | none | `cpu_load` (u, percent); `free_memory`, `total_memory`, `free_hdd` (u, bytes); `uptime_s` (u) | RouterOS's one-second view, as reported | | `mikroscope_api_core` | `cpu` | `load`, `irq`, `disk` (u, percent) | as `/system/resource/cpu` reports | | `mikroscope_api_health` | `name` | `value` (float) | as `/system/health` reports | | `mikroscope_api_iface` | `interface`, `label`, `type`, `role`, `bridge` | `rx_bps`, `tx_bps`, `rx_pps`, `tx_pps` (u); `rx_drops`, `tx_drops`, `tx_queue_drops`, `rx_errors`, `tx_errors` (u) only when the router returned them | instantaneous rates from `monitor-traffic`; the four inventory tags are omitted one by one when empty | | `mikroscope_api_ifcounters` | `interface`, `label`, `type`, `role`, `bridge` | one field per numeric counter RouterOS returned, its name with `-` folded to `_` (`rx_overflow`, `fp_rx_byte`, `link_downs` …) (u) | cumulative since boot or the port's last reset; every interface, on the `--counters-every` polls | | `mikroscope_api_ifinfo` | `interface`, `label`, `type`, `role`, `bridge` | `default_name` (string, the factory name of a physical port, `""` for an interface that has none); `mtu` (u) only above 0 | what each interface is; written once before the first kernel pull and again on every `--labels-every` re-read, never per poll | | `mikroscope_api_conntrack` | none | `entries` (u) | only on the polls that asked, every `--conntrack-every` | The fields of `mikroscope_api_ifcounters` differ by port and by board: a counter a port does not report is not a field on its row. `mtu`, `actual-mtu`, `l2mtu`, `max-l2mtu` and `sfp-shutdown-temperature` are integers that count nothing, so they are not fields there; the MTU is the `mtu` field of `mikroscope_api_ifinfo`. The inventory behind `label`, `type`, `role` and `bridge` is three configuration reads — `/interface/print`, `/interface/list/member/print` and `/interface/bridge/port/print`. `label` is the RouterOS comment, `type` RouterOS's own (`ether`, `bridge`, `vlan`, `pppoe-out`, `wg`, `veth`, `loopback`), `role` the interface lists an interface belongs to, sorted and comma-joined (`WAN`, `LAN,VPN`), where a bridge member in no list of its own takes its bridge's lists, and `bridge` the bridge it is a port of. A failed `/interface` read keeps the inventory already held and writes an error record instead; the list and bridge reads are best effort. `type` is what keeps two rows from being added together, because RouterOS counts different things on different interfaces. A switch port counts its wire, including the frames the switch chip forwarded in hardware; a bridge counts its CPU side; a VLAN or a PPPoE counts what the CPU sent and received. On the reference RB5009 (RouterOS 7.24.2, 2026-09-16) ether1 received 255.8 GB on the wire and 29.7 GB of that reached the CPU: the `ether` row and the `bridge` row are different planes, neither a subset of the other, and summing them counts nothing that exists. ## InfluxDB: derived values, annotations and device facts | Measurement | Tags | Fields | Stamped at | | --------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- | | `mikroscope_derived` | none | `mem_pressure` (i, 0–4); `burst`, `suspect` (bool); `cycles_per_packet`, `instructions_per_packet`, `cache_misses_per_packet`, `packets_per_irq` (float, 3 decimals) when computable | the kernel sample's time | | `mikroscope_derived_iface` | `interface` | `rx_bytes`, `fp_rx_bytes`, `tx_bytes`, `fp_tx_bytes` (u, deltas since the previous counter poll); `fp_rx_share`, `fp_tx_share` (float, 4 decimals) when bytes moved | the API sample's time | | `mikroscope_detection` | `rule`, `key` | `value`, `threshold` (float); `seq` (u); `message` (string) | the detection's time; `key` omitted when the rule has none | | `mikroscope_trigger` | `cause` | `id`, `seq` (u); `value`, `threshold` (float); `field` (string) | the agent's wall clock at the fire | | `mikroscope_gap` | none | `from`, `to` (u, sequence numbers no longer in the ring) | the collector's clock | | `mikroscope_device` | `board`, `kernel` | `cores` (i); `privileged`, `cgroup` (bool); `sources`, `hash` (string); `conntrack_max`, `cgroup_mem_max` (u) where published; `ports_from` (string) where known | the collector's clock | | `mikroscope_device_thermal` | `zone` | `critical_celsius` (float); `polling_ms` (i) | the collector's clock | | `mikroscope_device_cpufreq` | `cpu` | `cluster` (i); `min_khz`, `max_khz` (u); `governor` (string); `steps` (string, space-separated kHz) | the collector's clock | | `mikroscope_device_cadence` | `source`, `reason` | `hz` (float) | the collector's clock | `fp_rx_share` is the fast-path share of the traffic an interface hands the CPU, not a share of the wire: `fp-rx-byte` over `driver-rx-byte` on a switch port, whose `rx-byte` is the wire total, and over `rx-byte` on a software interface, which has no driver counters. Hardware-switched frames are in neither number. `rx_bytes` and `tx_bytes` are the share's denominators, not wire totals. Measured on the reference RB5009 (2026-09-16): switch ports read ~100 % (`fp-rx-byte` equals `driver-rx-byte` within a few kB), the bridge fast-pathed 211.9 GB of 663.0 GB since boot, PPPoE_DIGI 99.97 %. `fp-tx-byte` stayed 0 on every interface after hundreds of GB transmitted, so the tx share and its deltas are withheld while the cumulative `fp-tx-byte` is 0: `fp_tx_share` is absent and `tx_bytes`, `fp_tx_bytes` are 0. A derived value is written beside its inputs and never instead of them, so the store can recompute it. `suspect` marks a sample with a counter reset, where a per-packet figure would be a lower bound, so the per-packet fields are left out. The `board` and `kernel` tags read `unknown` when the agent could not establish them. The four `mikroscope_device*` measurements are written once when `forward` starts and again within a minute of the agent's capability hash changing, because the collector re-reads `/healthz` once a minute. A transport that cannot fetch `/capabilities` writes no device rows at all. What each value and rule means is on [what the collector derives](/mikroscope/sinks/derive/), [detections](/mikroscope/sinks/detections/) and [the device-info stream](/mikroscope/sinks/device-info/). ## What InfluxDB 3 Core requires - A node holds at most five databases. A write to a sixth fails with `422`; the sink backs off and keeps trying. - Every query must be time-bounded. - A column's type is immutable once written. - A query naming a field the store has never received fails at planning time, exactly as a missing table does: `No field named limit. Valid fields are …` (verified through the Grafana datasource proxy, 2026-09-14). - The write URL holds `&`; quote it when it lives in a file you `source`. The sink sends one batch per second and queues up to `--queue-seconds` × 64 KiB of batches, a budget sized for about 1.2 KiB per 10 Hz sample, then drops the oldest. A batch larger than 64 KiB, at 50 or 100 Hz or with the privileged sources, means fewer than `--queue-seconds` batches fit; [InfluxDB 3](/mikroscope/sinks/influxdb/) covers delivery. ## SQL: the file `--sql out.sql` writes PostgreSQL text: a header, then one `INSERT` per record. There is no database driver; `psql` owns the connection. ```sh mikroscope forward --sql out.sql --for 10m && psql -f out.sql ``` - The header is `SET standard_conforming_strings = on;` and one `CREATE TABLE IF NOT EXISTS` per table, so every file re-states the schema harmlessly. `--sql-hypertable` adds one `SELECT create_hypertable('', 'time', if_not_exists => TRUE);` per table. - Every table's primary key starts with `time, host`, and every `INSERT` ends in `ON CONFLICT DO NOTHING`: applying the same file twice is a no-op, because a row is one immutable instant, never a total a later sweep revises. - `TIMESTAMPTZ` stores microseconds, so the last three digits of the agent's nanoseconds are rounded away; two samples closer than 1 µs would collide on the key. - A float that is NaN or infinite is written `NULL`. A NUL byte in a string is dropped, and bytes that are not valid UTF-8 become U+FFFD, because PostgreSQL refuses both. - `--sql -` writes to standard output. With no queue in this sink, a `psql` that falls behind blocks the collector's pull loop instead of dropping. - The sink counts events it wrote to the file, not rows the server stored. `dt_ns` is on `mikroscope_cpu` only. A rate over any other delta table joins `mikroscope_cpu` on `(time, host)` for the real interval rather than assuming the nominal period. ## SQL: kernel-tier tables | Table | Primary key | Columns after `time` | | -------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_cpu` | `time, host, cpu` | `host`, `cpu`, `user_ticks`, `nice_ticks`, `system_ticks`, `idle_ticks`, `iowait_ticks`, `irq_ticks`, `softirq_ticks`, `steal_ticks` (deltas), `busy_ratio`, `dt_ns` | | `mikroscope_softnet` | `time, host, cpu` | `host`, `cpu`, `processed`, `dropped`, `time_squeeze` (deltas) | | `mikroscope_irq` | `time, host, irq` | `host`, `irq`, `name`, `count` (delta, summed over CPUs; top-K lines) | | `mikroscope_mem` | `time, host` | `host`, `free_kb`, `available_kb`, `cached_kb`, `slab_kb`, `sunreclaim_kb` (levels) | | `mikroscope_load` | `time, host` | `host`, `load1`, `load5`, `load15`, `running`, `threads`, `procs_blocked` (levels) | | `mikroscope_stat` | `time, host` | `host`, `ctxt`, `intr`, `forks`, `irq_total`, `irq_err`, `pgfault`, `pgmajfault` (deltas) | | `mikroscope_self` | `time, host` | `host`, `cpu_us` (delta), `rss`, `cgroup_mem` (levels), `throttled`, `throttled_us`, `oom_kill` (deltas, `NULL` without cgroup2), `seq` | | `mikroscope_buddy` | `time, host, node, zone, block_order` | `host`, `node`, `zone`, `block_order`, `free_blocks` (level; one row per zone and order, and `order` is reserved) | | `mikroscope_mtd` | `time, host, device` | `host`, `device`, `partition`, `corrected_bits`, `ecc_failures`, `bad_blocks`, `bbt_blocks`, `bitflip_threshold`, `ecc_strength` (levels; thresholds `NULL` unpublished) | | `mikroscope_psi` | `time, host` | `host`, `cpu_some_us`, `mem_some_us`, `mem_full_us`, `io_some_us`, `io_full_us` (deltas) | | `mikroscope_thermal` | `time, host, zone` | `host`, `zone`, `celsius`, `critical_celsius` (`NULL` unpublished) | | `mikroscope_slab` | `time, host, cache` | `host`, `cache`, `active_objs`, `limit_objs` (`NULL` for caches with no published ceiling) | | `mikroscope_disk` | `time, host, device` | `host`, `device`, `reads`, `read_sectors`, `writes`, `write_sectors`, `io_s` (deltas), `inflight` (level) | | `mikroscope_flash` | `time, host, device` | `host`, `device`, `page_writes`, `page_reads`, `erasures`, `gc_copies`, `gcs` (deltas), `bad_blocks`, `free_chunks` (levels) | | `mikroscope_event` | `time, host, kernel_seq` | `host`, `level`, `facility`, `kernel_seq`, `time_usec` (µs since boot, the kernel's monotonic clock, not `time`'s), `message`, `port`, `kind` (both `NULL` for a record that names no port) | The column names avoid quoting in PostgreSQL: the tick columns are `user_ticks` and friends because `user` is reserved. The counter columns are `BIGINT`, since PostgreSQL has no unsigned 64-bit type and no delta a router produces in one sample comes near 2^63. ## SQL: API-tier, collector and device tables | Table | Primary key | Columns after `time` | | --------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `mikroscope_api_system` | `time, host` | `host`, `cpu_load`, `free_memory`, `total_memory`, `free_hdd`, `uptime_s`, `version` | | `mikroscope_api_core` | `time, host, cpu` | `host`, `cpu`, `load`, `irq`, `disk` | | `mikroscope_api_health` | `time, host, name` | `host`, `name`, `value` | | `mikroscope_api_iface` | `time, host, interface` | `host`, `interface`, `label`, `rx_bps`, `tx_bps`, `rx_pps`, `tx_pps`, `rx_drops`, `tx_drops`, `tx_queue_drops`, `rx_errors`, `tx_errors` (loss columns `NULL` when the router did not return them) | | `mikroscope_api_conntrack` | `time, host` | `host`, `entries`: the last count, written on every API poll once one has been read | | `mikroscope_api_ifinfo` | `time, host, interface` | `host`, `interface`, `default_name`, `type`, `role`, `bridge`, `label`, `mtu` (`NULL` where the router gives none): what each interface is, one row per interface per inventory read | | `mikroscope_api_ifcounter` | `time, host, interface, counter` | `host`, `interface`, `counter` (RouterOS's own name, with its dashes), `value`: one row per counter | | `mikroscope_api_error` | `time, host, message` | `host`, `message`: which API command failed on that poll and why | | `mikroscope_gap` | `time, host, seq_from, seq_to` | `host`, `seq_from`, `seq_to` | | `mikroscope_trigger` | `time, host, id` | `host`, `id`, `cause`, `field`, `value`, `threshold`, `seq` | | `mikroscope_derived` | `time, host` | `host`, `seq`, `mem_pressure`, `burst`, `suspect`, `cycles_per_packet`, `instructions_per_packet`, `cache_misses_per_packet`, `packets_per_irq` (`NULL` where not computable) | | `mikroscope_derived_iface` | `time, host, interface` | `host`, `interface`, `rx_bytes`, `fp_rx_bytes`, `tx_bytes`, `fp_tx_bytes`, `fp_rx_share`, `fp_tx_share` | | `mikroscope_detection` | `time, host, rule, key` | `host`, `rule`, `key` (empty string when the rule has none), `seq`, `value`, `threshold`, `message` | | `mikroscope_device` | `time, host` | `host`, `board`, `kernel`, `cores`, `privileged`, `cgroup`, `sources`, `conntrack_max`, `cgroup_mem_max`, `ports_from`, `hash` | | `mikroscope_device_thermal` | `time, host, zone` | `host`, `zone`, `critical_celsius`, `polling_ms` | | `mikroscope_device_cpufreq` | `time, host, cpu` | `host`, `cpu`, `cluster`, `min_khz`, `max_khz`, `governor`, `steps` | | `mikroscope_device_cadence` | `time, host, source` | `host`, `source`, `reason`, `hz` | ## Where the two stores differ | Data | InfluxDB | SQL | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | CPU frequency | `mikroscope_cpufreq` | not written | | per-CPU interrupts and softirqs | `mikroscope_irq_cpu`, `mikroscope_softirq` | not written; `mikroscope_irq` has the sum only | | vmstat events and levels | `mikroscope_vm`, `mikroscope_vm_level` | only `pgfault` and `pgmajfault`, on `mikroscope_stat` | | PMU counters | `mikroscope_perf` | not written | | sample sequence and clocks | `mikroscope_sample` | `seq` on `mikroscope_self`; `dt_ns` on `mikroscope_cpu` | | `/proc/meminfo` | nineteen fields on `mikroscope_mem` | five columns on `mikroscope_mem` | | kernel log | counts per level, port and kind, `mikroscope_kmsg`; no text | every record with its text, `port` and `kind`, `mikroscope_event`; no counts | | observer extras | `cgroup_mem_max`, `resets`, `kmsg_dropped` on `mikroscope_self` | not written | | slab ceiling | field `limit` | column `limit_objs`; population `active` against `active_objs` | | free lists | one row per zone, a field per order | one row per zone and order | | what an interface is | `label`, `type`, `role`, `bridge` as tags on `mikroscope_api_iface` and `mikroscope_api_ifcounters`, beside `mikroscope_api_ifinfo` | `label` alone on `mikroscope_api_iface`; the rest through `mikroscope_api_ifinfo` | | port counters | `mikroscope_api_ifcounters`, one row per port, `_` in names | `mikroscope_api_ifcounter`, one row per counter, RouterOS's names | | conntrack count | only on the polls that asked | the last count, repeated on every poll after the first | | API command failures | not written | `mikroscope_api_error` | | gap bounds | `from`, `to` | `seq_from`, `seq_to` | | `mikroscope_api_system.version` | not written | `version` | > **Not measured, so not claimed** > > The SQL sink has never been run against a live TimescaleDB in this repository: the > `create_hypertable` calls follow TimescaleDB 2.x's documented signature and are not verified. Its > size was measured on 2026-09-12 on the package's own test fixture (2 cores, one softnet queue, one > interrupt line, no privileged sources), not on a router: a kernel event renders to 1 375 B of SQL > against 716 B of line protocol, an API event to 1 138 B against 608 B, and 10 Hz plus the 1 Hz API > tier writes about 14 KiB/s after a 5.6 KiB header; with the privileged sources the same kernel > event grows to 2 749 B. The figures do not cover fourteen of the thirty-two tables the sink > writes — `load`, `stat`, `buddy`, `mtd`, `api_ifcounter`, `api_ifinfo`, `trigger`, `derived`, > `derived_iface`, `detection` and the four `device` tables — and have not been re-measured, so > they predate the `port` and `kind` columns of `mikroscope_event`. The header for all thirty-two, > computed from the schema strings in `internal/sinks/sql.go` rather than measured, is 7 757 B, > about 7.6 KiB. How `--sql - | psql` behaves when `psql` falls behind a 10 Hz agent is not > measured. ## See also - [InfluxDB 3](/mikroscope/sinks/influxdb/): pointing `forward` at InfluxDB and what delivery guarantees it has. - [Prometheus metric families](/mikroscope/reference/metrics/): the same data as a scrape. - [The file and the other sinks](/mikroscope/sinks/other/): JSONL, SQL in practice, Loki, OTLP, Graphite, Elasticsearch, Telegraf. - [Import and check](/mikroscope/dashboards/import-and-check/): the dashboards that query these measurements. --- # RouterOS ports and kernel names The kernel log says eth5 where RouterOS says ether6 — how to measure the mapping on a dead port in one safe step, what the agent ships for the RB5009, and how much of that table was measured. Source: https://jmrplens.github.io/mikroscope/reference/port-names/ The kernel log names netdevs (`eth0`, `eth5`), RouterOS names interfaces (`ether1`, `sfp-sfpplus1`), and **they do not agree**. This page answers which cable a kernel-log record is about. During [the loop case study](/mikroscope/playbooks/loop/) this cost real time: `eth1` in the log looks like it should be `ether1`, and it is not. Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · Linux 5.6.3 · 2026-09-12 · agent at 10 Hz in an ephemeral privileged container Later observations are dated where they appear. ## Why it has to be measured RouterOS exposes no mapping, and the container cannot read one. Network devices are namespaced, so `/sys/class/net` inside the container shows only `lo` and the veth; `/sys/class/mdio_bus` holds only `fixed-0` and `/sys/class/phy` is empty. `privileged=yes` does not change that. RouterOS's names live in RouterOS's configuration, not in the kernel. ## Measure it on a dead port Pick a port that is definitely carrying nothing, toggle it, and read the name the kernel prints. Find a genuinely dead port first — zero packets in both directions, for its whole life: ```text /interface/print stats where name="ether6" or name="ether7" ``` Then, with the agent running: ```text /interface/ethernet/disable [find name="ether6"] :delay 4s /interface/ethernet/enable [find name="ether6"] ``` The kernel said: ```text [6] br0: port 7(eth5) entered blocking state [4] eth5: set isolation from 0 to 1 [4] eth5: set isolation from 1 to 0 ``` So RouterOS `ether6` is kernel `eth5`. Repeating on `ether7` gave `eth6`: a **−1 offset**, seen at two points. ## The RB5009 table The mapping is a shift by one — RouterOS numbers ports from 1 and the kernel from 0, the switch chip included (`switch0` is the `switch=switch1` every port reports): | RouterOS | kernel | how known | | ------------------- | --------------- | ---------------------------------------------- | | `ether1` | `eth0` | inferred | | `ether2` | `eth1` | **measured** — the case-study loop, 2026-09-13 | | `ether3` … `ether5` | `eth2` … `eth4` | inferred | | `ether6` | `eth5` | **measured** — flapped 2026-09-15 | | `ether7` | `eth6` | **measured** — flapped 2026-09-15 | | `ether8` | `eth7` | inferred | | `sfp-sfpplus1` | `eth8` | inferred, last in the enumeration | | `switch1` | `switch0` | inferred | The `ether6` and `ether7` pairs were measured on **2026-09-15**. Both ports are commented `Unused` and neither was `RUNNING` — no cable, no link — so each was disabled and enabled again over the API while the agent read `/dev/kmsg`. The kernel logged `br0: port 7(eth5) entered disabled state` inside `ether6`'s window and `port 7(eth6)` inside `ether7`'s, nine seconds apart, which is what rules out reading one flap twice. No traffic was interrupted and both ports were back within four seconds. That makes three measured pairs, all on the same shift by one, which is what the six inferred rows rest on. The inferred pairs rest on two observations from a read-only `/interface/ethernet/print` (2026-09-13, the date the table's own evidence string carries): the nine ports carry consecutive MAC addresses, `…:55` for `ether1` through `…:5D` for `sfp-sfpplus1`, in RouterOS's own enumeration order; and all nine report `switch=switch1` against the kernel's one `switch0`. ## Two cautions - **The name is reliable; the bridge port number is not.** Both flaps reported `port 7`, because the kernel reuses port slots when a port leaves and rejoins the bridge. Match on `eth5`, never on `port 7`. - **The offset is a property of this model's driver, not a rule.** Re-measure on a different device rather than assuming, and be careful with the intuition that the SFP+ must be `eth0` — here it is last, not first. The device tree does not help either: it shows the SoC's `ethernet@0` with three MACs, only `eth0` enabled (the 10 G uplink) and `eth1`/`eth2` disabled, while the nine front-panel ports are netdevs the switch driver creates at runtime (the device tree was parsed on 2026-09-14). The kernel-log names are the runtime ones. ## What the agent does with the table The agent reads the board model from the device tree at start — `/proc/device-tree/model` reads `RB5009` even unprivileged — and, on a board in its table, names ports without asking RouterOS: - every kernel-log record whose text names a port carries both names and what happened to that port: `iface`, `ros_iface` and `kind` on the event, `port` and `kind` tags on the InfluxDB rows, and `iface=eth1 ros_iface=ether2 port_event=own-address` on a Loki line; - `/metrics` gains `mikroscope_kmsg_port_records_total{port,kind,level}`, with the RouterOS name as `port` on a board in the table, the kernel name on a board that is not, and no port series at all on a device whose device tree reports no model. It is a subset of `mikroscope_kmsg_records_total`, not a partition of it: records naming no port are absent from it; - the collector's `link-flap` detection is keyed by the port name and reads the same classification: a flap is `link-up` and `link-down` records on one port, counted, not the text parsed a second time; - `mikroscope status` prints the board and whether it has a map, for example `eth1 (ether2)`; `/healthz` and `/capabilities` carry the board, and `/capabilities` and `mikroscope_device_info` carry the table's evidence string as `ports_from`, so nobody has to take the mapping on trust. **An unknown board gets no port names, not guessed ones.** The shift by one is not applied to a board nobody has measured, because a confidently wrong port name sends someone to the wrong cable. On such a board `status` says so and asks for the pair: bring one port down, see which `ethN` the log names, and send that pair with the board string. ## What kind of event it was A port name alone does not say what happened to the port, so every record that names one is classified as well, by `procfs.KmsgKind` over the record's text. The kinds, and the record shapes the RouterOS kernel prints (RB5009, kernel 5.6.3, records seen 2026-09-12 … 2026-09-15): | `kind` | the record | | ------------- | ----------------------------------------------------------------------------------- | | `link-up` | `eth8: link up, 1Gbps, full-duplex`, `eth1: Link is Up - 1Gbps/Full`, `eth1: phy link up` | | `link-down` | `eth1: link down` | | `stp-` | `br0: port 2(eth1) entered blocking state` — `blocking`, `listening`, `learning`, `forwarding`, `disabled` | | `own-address` | `br0: received packet on eth1 with own address as source address (addr:…, vlan:0)` — the layer-2 loop signature | | `other` | anything else that names a port, including `eth1: link becomes ready`, which is IPv6 address configuration noticing the carrier and not a transition of its own | The agent classifies at read time and ships `kind` in the record, and a collector in front of an older agent whose records carry none classifies them itself with the same function. That is the case on the reference router today: the agent running there has not been redeployed with this build, so its `/metrics` carries no `kind` label yet and the collector does the work. **Four records are not four faults.** A normal link-up is followed by `stp-blocking`, `stp-learning` and `stp-forwarding` as the bridge walks the port back into service. ## From the default name to the current one The table maps to RouterOS's **default** names. A port renamed on the router (`ether5` → `WAN`) has a name the table cannot know, so the collector asks the API tier instead. Its interface inventory — read once before the first kernel pull and again every `--labels-every`, five minutes by default — holds each interface's factory default name, its current name, its comment and its interface lists, and the collector uses it on every record that names a port to: - replace the default name with the **current** one, so an operator who renamed `ether5` to `WAN` reads `WAN` on the event, on the InfluxDB `port` tag, on the Loki line and in the `link-flap` key; - attach `label`, the port's comment, and `role`, its interface lists — so the `eth5` flap arrives as `ether6`, label `Unused`, rather than as a netdev number somebody has to look up. Without an API tier the record keeps the board's default name and gets no label, which is what it can support. ## Where a kernel message becomes a cable Two panels in the dashboards' **Kernel log** row are where the name, the kind and the comment meet: - **Port events from the kernel log, per port and kind** — bars per bin, one series per port and kind, from `sum by (port, kind) (increase(mikroscope_kmsg_port_records_total[$__interval]))` on Prometheus and from the `port`/`kind` tags on InfluxDB; - **Port events in the window, per port** — a table with one row per port that the log named: the port, its `label` and `role`, and a column per kind (link down, link up, own address (loop), STP blocking, STP disabled, STP learning, STP forwarding, other). Both are known-empty panels — a quiet set of ports is the healthy state — and the Prometheus form of each reads the collector's `/metrics`, whose copy of the family carries a `kind` on every port record however the agent shipped it. In an InfluxDB store the `kind` column exists only once a first port record classified by kind has been written, so both queries were validated on 2026-09-16 against a synthetic table in the same InfluxDB 3, the live store having no such column yet. Two alert rules ship beside them, in both provisioning files. Both read `/dev/kmsg` through the agent and ask RouterOS nothing: - `mikroscope-l2-loop`, critical: any `own-address` record in five minutes. On the reference RB5009 that signature ran at 1.49 records/s for hours on 2026-09-12 while every RouterOS counter looked healthy. - `mikroscope-port-link-down`, warning: any `link-down` record in five minutes — the single event, where `link-flap` covers the repeats. The InfluxDB form of each needs a store that has already held one port record classified by kind; before that the query fails at planning time. > **Untested** > > Neither port-event rule has been seen firing on a real event: they were written against the > record shapes measured on the reference RB5009 and have not been put in front of a live loop or a > live link-down. The row-by-row headless render walk of 2026-09-15 covered 168 InfluxDB panels and > 130 Prometheus ones and has not been repeated for these two. The port table all of this rests on carries its own limit. > **True of this device, not of yours** > > The shift by one, the position of the SFP+ cage and the reuse of `port 7` were observed on one > RB5009UG+S+ running RouterOS 7.24.2. They are not claimed for any other board, and the agent does > not apply them to one. ## See also - [A loop only the kernel could see](/mikroscope/playbooks/loop/): the fault that made this mapping worth an hour. - [The device-info stream](/mikroscope/sinks/device-info/): where the board and `ports_from` travel. - [Detections](/mikroscope/sinks/detections/): the `link-flap` rule, keyed by these names. - [The RouterOS API tier](/mikroscope/sinks/api-tier/): the interface inventory that turns a default name into the current one, with its comment and its lists. - [Alert rules](/mikroscope/dashboards/alerts/): the loop and link-down rules these records fire. --- # When something does not work The symptoms this project produces, in the words you actually see — a RouterOS error, a container that exits, an empty panel, a sink that drops — with what each one means and the page that explains it. Source: https://jmrplens.github.io/mikroscope/reference/troubleshooting/ Every page here explains one thing properly. This one is the index you reach for when something is already broken: find the line you are looking at, and it says what it means and where the explanation lives. ## Installing | You see | It means | | --------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `doctor`: `device-mode container=no` | [The one step nobody can do remotely](#device-mode-containeryes) | | `doctor`: `container package installed and enabled … found=0` | [The package is not on the router](#no-container-package) | | `doctor`: `architecture matches --arch … router=arm` | Re-run with the `--arch` it names | | RouterOS: `unknown parameter privileged` | [RouterOS older than 7.24](#unknown-parameter-privileged) | | Container log: `exec format error` | [The wrong image for the board](#exec-format-error) | | `no Go toolchain on PATH` | Install with `--remote-image` or `--agent-tar` instead | | `--agent-tar …: this is not a mikroscope agent image` | The wrong asset — see [which tar](/mikroscope/install/routes/#which-tar) | | `doctor`: `registry-url is https://ghcr.io … registry-url=…` | [The registry host is global](#the-registry-host) | | `doctor`: `free flash ≥ …` fails | `--disk tmpfs` or `--ephemeral`, or free space on the flash | ### device-mode container=yes MikroTik gates containers behind a switch that cannot be flipped over the network. `/system/device-mode/update container=yes` starts it, and then the console asks for a physical confirmation — the reset button, or a power cycle — within five minutes. No flag, no script and no version of this tool can do that step for you. It is the first thing to arrange, because everything else waits on it: [What the router needs](/mikroscope/install/prerequisites/). ### No container package The `container` package is a separate download from mikrotik.com, per architecture and per RouterOS version. Upload it, reboot, then `/system/package/enable container`. `doctor` counts it as present only when it is installed **and** not disabled. ### unknown parameter privileged RouterOS 7.24 added `privileged=`, and the container step writes it, so an earlier 7.x fails there — after the tar has been uploaded, which is why the install then takes it back with it. Either upgrade RouterOS, or install with `--privileged=false` and read [what privileged buys](/mikroscope/limits/privileged/) first: without it the agent cannot read `/dev/kmsg`, and the kernel log is where several of this project's playbooks start. ### exec format error The container starts and dies immediately, and the log says `exec format error`. The image is for another architecture than the board — and on 32-bit ARM, "arm" is not one architecture. MikroTik's container documentation says devices with the EN7562CT CPU, the hEX Refresh line, "support only arm32v5 container images"; its other 32-bit ARM boards run an ARMv7 userland. An ARMv5 image runs on both, an ARMv7 image does not run on the first. So: - With `--remote-image`, this cannot happen: the published index carries all four platforms and the router matches its own. - With `--agent-tar`, take `mikroscope-agent-armv5.tar` when the board is 32-bit ARM and you are not certain which kind it is. - Building from a checkout, `--goarm 5` is the default for the same reason. [Which tar](/mikroscope/install/routes/#which-tar) is the table. ### The registry host `/container/config registry-url` is one global RouterOS setting, shared with every other container on the device, and mikroscope reads it and never writes it. The Docker Hub reference works out of the box because that is what RouterOS ships pointing at; the GHCR one needs the setting changed first, which changes it for everyone else on that router too. ## The agent is installed and nothing answers ```sh mikroscope status ``` That prints the ownership counts and, if it can reach the agent, its health. If the counts are there and the health is not, the container is running and something between you and it is not: - **The firewall.** Two rules commonly eat this traffic, and neither is obvious: [The two firewall traps](/mikroscope/install/firewall/) is that page, and `doctor` checks the two list memberships that avoid them. - **The route.** The agent answers on its `/30`, on the router's LAN side. A collector elsewhere reaches it the ways [Reaching the agent](/mikroscope/install/reaching-the-agent/) lists. - **The container never started.** `/container/print detail` on the router, and `/log/print where topics~"container"`. ## Data is arriving and something is empty | You see | It means | | ----------------------------------------------------- | -------------------------------------------------------------------------- | | No `events` at all, ever | `privileged=yes` is what `/dev/kmsg` needs | | No PMU panels, no cycles or instructions | `perf_event_open` is unavailable on that kernel or board | | A panel says **No data** and the others are fine | That measurement is not produced on this device; the dashboard has a row for it | | A panel shows a red error badge | The query failed — the datasource, not the data | | `forward` prints `… dropped` for a sink | [The sink could not keep up](#a-sink-is-dropping) | | Loki accepted everything and a query returns nothing | A push is not queryable until the chunk flushes | | Numbers stop at a round moment and resume | A gap: the ring wrapped before the collector pulled it | | The kernel tier stops dead and the API tier carries on | [The agent restarted](#the-agent-restarted-and-the-kernel-tier-stopped) | The dashboards carry a row named **"Not available on this device"** for exactly this: panels whose measurement the kernel or the board does not produce are moved into it rather than left to draw an empty graph among the others. `mikroscope dashboards check` asks the datasource which measurements it really holds and does that sorting for your store: [Import and check](/mikroscope/dashboards/import-and-check/). ### The agent restarted and the kernel tier stopped The agent numbers its samples from 1 at every start, so an agent that restarts — an upgrade, a container restart, a reboot — has a newest sequence number far below the collector's cursor. The collector notices on its next health read, which is once a minute, logs ```text agent restarted: its newest sample is 571 and the cursor was 1737212; resuming from 1 ``` and resumes from the new ring's oldest sample, so what the agent took while nobody was collecting is picked up rather than skipped. The minute's worth of samples between the restart and the health read is lost with the container, not by the collector. Before 1.0.4 it did not notice: the cursor stayed where it was, the agent's ring answered an empty batch to every pull, and the kernel tier stopped for good while the API tier kept counting and the sinks kept being written — so the run looked healthy. If you are on an earlier version, restart the collector after restarting the agent; it takes its cursor from the health read at start. ### A sink is dropping Every network sink is queued, and the queue is bounded — `--queue-seconds`, 60 by default. A destination that cannot keep up loses the oldest batch rather than stalling the pull loop, and the count is printed at the end of the run and exported as a metric. That is a deliberate choice, and [the collector](/mikroscope/sinks/) explains it: the agent's ring is what protects the data, and a collector waiting on a slow store would lose more than the store does. ## Reading what it shows Once the data is arriving, the question changes from "why is this broken" to "what is this telling me". That is a different set of pages: [How to read what it shows](/mikroscope/playbooks/) — the shape of an idle router first, then seven faults read against it. > **Ask the tools before reading further** > > `mikroscope doctor` names the fix for anything missing on the router, `mikroscope status` says > what is installed and whether it answers, and `mikroscope dashboards check` runs every panel's > query and prints which ones came back with nothing. Between them they answer most of this page > for your own device rather than in general. ## See also - [What the router needs](/mikroscope/install/prerequisites/): the three prerequisites, and what `doctor` checks. - [The two firewall traps](/mikroscope/install/firewall/): why the agent can be running and unreachable. - [What privileged buys](/mikroscope/limits/privileged/): what is lost without it, source by source. - [How the project tests itself](/mikroscope/reference/testing/): what has been proven and how, if you are wondering whether it is you or the project. --- # How the project tests itself The three layers of test — the bytes each sink puts on the wire, whether a real store accepts them, and whether the dashboards' own queries answer — what each one proves, what none of them prove, and the command for each. Source: https://jmrplens.github.io/mikroscope/reference/testing/ A sink can be wrong in three places, and each one needs a different test. It can encode the wrong bytes. It can encode bytes a store refuses — which a capture server never notices, because a capture server says 204 to everything. And it can write something a store keeps but no dashboard can read back. The project therefore has three layers, and they are three commands. | Layer | Command | Docker | What it proves | | ------------------- | ---------------------- | ------ | ----------------------------------------------------------------- | | 1, the contract | `make test-e2e` | no | the exact bytes each sink puts on the wire | | 2, the stores | `make test-e2e-docker` | yes | a real store accepts those bytes, and hands them back unchanged | | 3, the dashboards | the same target | yes | every panel's own query answers against what the sinks just wrote | None of the three touches a router. Layers 1 and 2 drive the collector against a fake agent that serves canned samples, and layer 1 also drives the real agent binary against `testdata/proc/rb5009`, a captured `/proc` and `/sys` tree of the reference device. That is what makes a run reproducible on any machine, and what keeps the router out of the loop. ## Layer 1 — the contract `make test-e2e` builds both binaries and points every sink at a receiver inside the test binary: HTTP capture servers for InfluxDB, Loki, OTLP, Elasticsearch and Telegraf, TCP and UDP listeners for Graphite and Telegraf's socket modes, real files for the file and SQL sinks, the process's own stdout, and a scrape of the collector's own `/metrics` for Prometheus. Each receiver asserts what arrived, byte for byte. It needs no router, no Grafana, no database, no container and no network: every address it binds or dials is on loopback. `make test-e2e-offline` proves that rather than asserting it, by running the whole suite inside a network namespace that has nothing but `lo`. ## Layer 2 — the stores `make test-e2e-docker` starts nine stores with docker compose, runs the same collector against the same fake agent with every sink pointed at them, and then asks each store its own question with its own API. The file sink's JSONL is the oracle each store is compared against — value by value, not only by count. | Store | The question it is asked | | ----------------------- | --------------------------------------------------------------------- | | InfluxDB 3 Core | SQL over HTTP: the tables, a row count per table, every `ctxt` value | | PostgreSQL 18 | the SQL sink's script through `psql`, then counts and the `ctxt` range | | Elasticsearch 9 | `_search` with an aggregation by kind, and one whole document | | Loki 3 | `query_range` for this run's labels, and the text of each record | | Graphite | `metrics/find` for the tree, `render` for the points and their order | | OpenTelemetry Collector | what it decoded, written back out as OTLP/JSON | | Telegraf 1.39 | the line protocol it parsed: measurements, tags, field types, timestamps | | Prometheus 3 | a scrape of the exporter, against the exposition it served | Each of those is there because that product refuses something a capture server accepts: - **InfluxDB 3** fixes a column as a tag or a field the first time it sees the table and refuses a later write that disagrees. - **Carbon** answers nothing at all: a point older than its longest archive, or a name whisper cannot make a path of, is dropped in silence. - **Loki** answers 204 for a push that is not queryable until the chunk flushes, and rejects out-of-order entries per stream, in the body. - **Elasticsearch** infers a mapping from the first document it sees for a field and then rejects a later one that does not fit it — per document, inside a bulk request that still answers 200. - **Telegraf** is a real line-protocol parser: an unescaped space in a tag value, or a field with no type, is dropped and the batch is still 204. - **PostgreSQL** is the only thing that can say whether the SQL sink's script is valid SQL, whether the types it chose hold the values it emits, and whether its primary keys collide on a real run. ## Layer 3 — the dashboards The same target then imports both dashboards into a real Grafana, pointed at the store the run just filled, and runs every panel's query through Grafana's own `/api/ds/query`. That is where a panel fails for reasons no unit test reaches: a type an aggregate returns that the datasource plugin cannot decode, a macro the plugin escapes, a column the store does not have. The first full run, on 2026-09-16, found one: the port-event table names `label` and `role`, which the sink writes onto a kernel-log row only from the API tier's inventory. A column that is not in the table is not an empty column on InfluxDB 3 — it is `Schema error: No field named label` and a panel that cannot render at all. ## Running it ```sh make test-e2e-docker # the whole thing; the stack comes up and goes down with it make e2e-docker-up # keep the stack up, for a targeted run go test -v -tags dockere2e -run TestLoki ./test/e2e/docker/ make e2e-docker-down ``` The package is behind the `dockere2e` build tag, so `make test` and every default CI job compile none of it — the tag is how you ask for nine containers. `make lint` type-checks it so it cannot rot, and the weekly E2E workflow and the release gate run it. Measured on the development machine on 2026-09-16: the stack comes up in 55 to 81 s with the images already pulled, and the whole suite takes 75 to 100 s from nothing. > **Linux, for two of the nine** > > Prometheus and Grafana run on the host's network stack, because Prometheus is > the one sink that is scraped rather than pushed to and it has to reach the > collector's exporter on the host. A container on a bridge network reaches the > host through the bridge's gateway, and that path goes through the host's > INPUT chain, which a default-deny firewall drops. The other seven services > are ordinary bridge-network containers. ## What the three layers still leave to a human > **What none of these prove** > > That a sink works from the router. The samples in all three layers come from a > fake agent or a captured `/proc` tree, never from a live device, and the > container stack runs on the development machine rather than across the > router's veth. File, Prometheus and InfluxDB 3 have carried real RB5009 > samples end to end; the other seven have not. ## See also - [Where the project stands](/mikroscope/about/status/): what has run against the reference device and what has not. - [The collector and its sinks](/mikroscope/sinks/): what each sink writes. - [Importing and checking the dashboards](/mikroscope/dashboards/import-and-check/): the same `dashboards check` the third layer runs, against your own store. --- # Where the project stands What works end to end, what it costs on the one device it has run on, what was found and not fixed, and what the release publishes. Source: https://jmrplens.github.io/mikroscope/about/status/ This page answers what a reader deciding whether to try mikroscope needs first: which parts work end to end, on what hardware that was shown, what the observer costs today, which defects are known and still open, and whether there is anything to download. It is checked against the code as of 2026-09-16. ## What works end to end Everything below except the test suites and the seven sinks marked as such has run against the reference RB5009 (RouterOS 7.24.2), not only against fakes: - **The agent** reads the shared kernel's `/proc`, `/sys`, `/dev/kmsg` and `perf_event_open` counters on a fixed ticker at 1 to 100 Hz (10 Hz by default; 10, 50 and 100 Hz measured), keeps the samples in a ring, and serves them: `/healthz`, `/capabilities`, `/snapshot`, `/stream`, `/metrics`, and the triggered-capture endpoints `/captures` and `/capture`. - **The deployment CLI** installs, upgrades and removes it — `doctor`, `plan`, `install`, `status`, `upgrade`, `uninstall` — with every write listed before it happens and every removal verified by ownership counts. On 2026-09-12, `doctor` → `install` → `status` → `upgrade` → `uninstall` left the router's `/export` byte-identical, compared by hash in memory and never written to disk. The agent answered 3 s after install, at a 5–7 ms round trip. On 2026-09-17 all four install routes were run against the same device, one after another and each removed before the next: the checkout build, the published agent tar, the router's own pull from Docker Hub, and the `plan --rsc` script imported on the router with no CLI in the install itself. The agent answered in every one, and the `/export` after the four was byte-identical to the one before them. - **`record`, `mark` and `plot`** work end to end. A 60 s recording at 10 Hz on 2026-09-12 yielded exactly 600 samples, 0 gaps and −7 ms of clock skew. - **`forward`**, the collector, merges the kernel tier with the RouterOS API tier and writes to ten sinks. File, Prometheus and InfluxDB 3 have run from the RB5009; Loki, OTLP, Graphite, Elasticsearch, SQL, Telegraf and stdout have not yet had router samples pushed through them, but every one of them now writes into the real product — InfluxDB 3, PostgreSQL, Elasticsearch, Graphite, Loki, an OpenTelemetry Collector, Telegraf and Prometheus in containers — and the suite reads each one back through its own API (2026-09-16). Its derive stage adds derived values and detection events beside the raw rows, and it emits a device-info stream once per capability hash. - **Five Grafana dashboards**, one per store — InfluxDB 3, Prometheus, PostgreSQL, Graphite and Elasticsearch — generated from one panel list, with alert rules for the three whose query language the rules are written in. The two SQL stores are asked the same question in two dialects: the PostgreSQL panels are the InfluxDB ones rewritten, and every one of its 209 queries is planned by a real PostgreSQL in the container suite. Graphite and Elasticsearch carry fewer panels on purpose (41 and 30 against 171): Graphite has no labels and Elasticsearch no nested documents, so what they cannot express is absent rather than wrong. All five are imported into a real Grafana over the stores the suite filled, and every panel is asked: on 2026-09-17, 57, 98, 136, 35 and 28 panels returned data and none failed. On 2026-09-16, in Grafana 13.2.1, against the RB5009 with the default triggers and a 30-minute `forward` into both stores, `dashboards check` passed 171 InfluxDB panels (0 failing, 10 known-empty tolerated) and 133 Prometheus panels (0 failing, 9 known-empty tolerated). The headless row-by-row render walk of both dashboards — 0 error badges, 0 "No data" — is from 2026-09-15 and covers 168 and 130 panels; it has not been repeated for the two port-event panels and the interface-inventory table it does not cover. A walk on 2026-09-14, against a 10.5 h capture from the same device, rendered 140 InfluxDB panels with the same result. - **An end-to-end suite** builds both binaries and drives them against a captured `/proc` tree of the reference device and against a fake agent, with one receiver per sink protocol asserting the bytes. It needs no router, no Grafana and no network, and runs in CI. - **A second suite against the stores themselves** (`make test-e2e-docker`) starts nine of them with docker compose, runs the same collector against the same fake agent with every sink pointed at them, and then asks each store its own question with its own API: SQL against InfluxDB 3, `psql` against the script the SQL sink wrote, a search against Elasticsearch, `query_range` against Loki, `render` against Graphite, the decoded OTLP the collector received, the line protocol Telegraf parsed, and a scrape of the exporter stored in Prometheus. It then imports both dashboards into Grafana and runs every panel's query through Grafana's own API. It needs Docker and no router: the samples are the same canned ones, so the run is reproducible anywhere. The first full run, on 2026-09-16, found a panel that named two columns the store only has when the API tier ran. ## What the agent costs today At the install default — 10 Hz, default per-source floors, a 300 s ring — the agent costs **2.85 % of one core and 31.3 MiB RSS**, read from its own cgroup at steady state with the ring full: Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · 2026-09-15 · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once The measured runs: | rate | floors | CPU of one core | µs/sample | RSS | slipped ticks | gaps / drops | | --- | --- | --- | --- | --- | --- | --- | | 10 Hz (default) | default | **2.85 %** | 2 856 | 31.3 MiB | **0** | 0 / 0 | That is above the budget of 2 % of one core and 16 MiB RSS. The budget is guidance rather than a contract: cost scales with the device, the source set and the ring size. The image budget is 8 MiB; the one image size on record is 6.1 MiB, and it carries no date. Two rules keep the cost figure honest: wait out the ring (`BUFFER_S`) before quoting a steady-state figure — on the RB5009, at a 14 MiB soft memory limit, a reading taken in the first minute after install came back at 1.47 % of one core against a 9.38 % steady state — and read cost from `/metrics` rather than from a large `/snapshot`, whose ~1.5 MB response the agent must serialise. ### Cost at 10 Hz, by configuration Every row was measured on the same RB5009 at 10 Hz, and each is one window with no spread recorded. They are the settings a reader can choose, so the table says what each one costs — not what the number was at some earlier point. | Configuration | CPU of one core | RSS | Measured | Note | | ----------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------ | ---------- | -------------------------------------------------------------------------------------------------------- | | Every source read every tick | 2.43 % | not recorded | 2026-09-12 | above the budget | | Ring full, `MEM_LIMIT_MB` 14 | 9.38 % (9 374 µs/sample) | not recorded | 2026-09-12 | a 300 s ring of lines of about 2.4 kB holds ~7.3 MB; the Go GC runs without pause | | Ring full, `--mem-limit-mb 40`, `--memory-max 64M` | 1.39 % (1 388 µs/sample) | 25.13 MiB | 2026-09-12 | 0 slipped ticks | | PMU counters on, a live `forward` writing to InfluxDB | 1.72 % | not recorded | 2026-09-12 | 0 slipped ticks; 2 400 samples forwarded, 0 gaps, 0 drops | | The install default | 2.85 % | 31.3 MiB | 2026-09-15 | the figure above | ## One device, one RouterOS version Every on-device figure in this documentation comes from one RB5009UG+S+ on RouterOS 7.24.2, the owner's production router. There is no lab device. > **Untested** > > A second board of any kind. The hEX S (2025) — 32-bit RouterOS on an ARM64 chip, which is what the > agent's `linux/arm` build is for — has not arrived, so the 32-bit counter-wrap path and the > `linux/arm` image have not run on hardware. No x86_64 RouterOS host. No RouterOS version other > than 7.24.2. Nothing that needs a reboot, which waits for a maintenance window. The end-to-end > suite has run on Linux only; it has not been run on macOS or Windows. ## Found and not fixed Checked against the code as of this page: - **`forward`'s end-of-run summary goes to stdout**, the same stream the `--stdout` sink writes records to, so `forward --stdout=lp | telegraf` ends every run with lines the consumer cannot parse. - **A wrong `--token` does not fail the run.** Each pull is logged as `401 Unauthorized`, and the run then ends at its `--for` deadline with exit status 0 and `forwarded 0 kernel samples`. The health check `forward` makes before it starts pulling reads `/healthz`, which needs no token, so it passes. The message is right; the exit status tells a unit file nothing. - **Duplicate sequence numbers in the 2026-09-13 overnight run.** The 50 Hz run wrote repeated `seq` values into InfluxDB; the dashboards name those rows "duplicate sample (seq repeated)". The leading explanation, which is unverified, is Go's HTTP client replaying a POST on a dead pooled connection after the server had committed it. The InfluxDB, Loki, OTLP, Elasticsearch and Telegraf sinks refuse that replay, so a dead connection is an error the sink retries and counts. No run since that night shows whether the duplicates are gone. - **Graphite names thermal zones by index.** The index is deliberate — a zone's `type` string is not unique across zones — but the path listing at the top of `internal/sinks/graphite.go` reads `thermal.`. ### Found on the device and not collected Two readable files on the reference device, read there on 2026-09-15, are not collected: `/proc/cmdline`, which carries `board=5009 ver=7.24.1` — a second device identity without the API — and the hardware watchdog at `/sys/class/watchdog/watchdog0`. ## What the release publishes The current release is **v1.0.0** — the version in `VERSION`, compiled into both binaries and reported by `mikroscope version`. A `v*` tag runs the GoReleaser configuration, which publishes: - **CLI archives** for linux, darwin, windows and freebsd on amd64, arm64 and arm: `.tar.gz`, and `.zip` on Windows, each carrying `LICENSE` and `README.md`. - **Agent archives** for linux on the same three architectures, for a host that wants the bare binary rather than an image. - **Side-loadable agent image tars**, one per architecture — `mikroscope-agent-arm64.tar`, `mikroscope-agent-arm.tar`, `mikroscope-agent-amd64.tar` — which `install --agent-tar` uploads to the router. - **The agent image in two registries**, `jmrplens/mikroscope-agent:1.0.0` on Docker Hub and `ghcr.io/jmrplens/mikroscope-agent:1.0.0` on GHCR, each one manifest over `linux/amd64`, `linux/arm64` and `linux/arm/v7`, which `install --remote-image` makes the router pull. RouterOS takes the registry host from the global `/container/config registry-url`, which ships as `https://registry-1.docker.io`, so the Docker Hub reference needs nothing set on the device and the GHCR one needs that setting changed first. - **`checksums.txt`**, covering every archive and every image tar, a keyless cosign signature over it, and an SPDX SBOM per archive, signed in its own right. Installing from a release needs no Go toolchain and no checkout. A checkout and Go 1.27 install an agent built from your own tree instead: `make build` for the CLI, `make build-agent` for the agent, which `install` and `image` also compile themselves. The four ways to get the agent onto a router, and what each one needs, are on [Installing the agent](/mikroscope/install/). What the release publishes for `linux/arm` and `linux/amd64` is cross-built and checked in CI. Of the three architectures, only `arm64` has run on hardware — on the one RB5009 above. ## See also - [The cost of the observer](/mikroscope/cost/): the budget, the current figure and how to measure it on your own device. - [The rate ceiling](/mikroscope/cost/rate-ceiling/): the five measured runs at 10, 50 and 100 Hz. - [The file and the other sinks](/mikroscope/sinks/other/): the ten sinks `forward` writes to. - [Lineage and licence](/mikroscope/about/lineage/): where the deployment code and the API client came from. --- # The mark Nine sample bars and the line of their own mean, why none of it is drawn at partial opacity, and the contrast each tone measures against its page. Source: https://jmrplens.github.io/mikroscope/about/brand/ This page answers what the mikroscope mark says, how it is made, and why its colours are the ones they are — with the contrast of every tone measured against the background it is drawn on, and the cost of that choice stated beside it. ![The mikroscope mark: nine sample bars, three of them standing above the flat line of their own mean](https://raw.githubusercontent.com/jmrplens/mikroscope/main/site/src/assets/mark-inline.svg) ## A spike standing over its own average The mark is geometry, not a drawing: nine sample bars whose envelope is a burst, crossed by the flat line of their own mean. That is the whole claim of the project in one shape — a one-second average reports the line, and sub-second sampling is what resolves the spike standing over it. The burst is asymmetric the way a real one is: three quiet samples, a fast rise, the peak, a slower fall, three quiet again. The peak sits in the middle of an odd number of bars, so the mark balances on its own centre. ## The line is the mean, and it is computed The generator draws the line at the arithmetic mean of the heights it was given, not at a number chosen to look right, and a bar takes the loud tone exactly when it stands above that mean. So the sentence the mark makes is one the code enforces: three of the nine samples are above their own average, and they are the three the eye goes to. `TestTheLineSitsAtTheMeanOfTheBars` fails the build if the drawing and the arithmetic drift apart. ## Four tones, two per theme, and no opacity anywhere Measured against the background each is drawn on, as WCAG 2.2 defines contrast: | Theme | Background | Above the mean | At or below it, and the line | Apart | | ----- | ---------- | --------------------- | ---------------------------- | ------ | | Dark | `#0e1316` | `#fbbf24` **11.20:1** | `#c2740a` **5.16:1** | 2.17:1 | | Light | `#ffffff` | `#633009` **10.73:1** | `#b45309` **5.02:1** | 2.14:1 | One palette per theme is not a refinement; it is the only way the mark is legible in both. An amber light enough to read on the near-black page reads 1.67:1 on white (`#fbbf24`), and one dark enough for white disappears into the dark page. On white, the sample standing above the mean is the darker tone, so the mark reads the same way round in both themes. `TestEveryToneClearsAAInItsOwnTheme` recomputes the ratios from the hex on every run and fails the build if any tone reads under 4.5:1 on its own background, or if a theme's two tones are less than 2.0:1 apart. A tone edited without checking it fails the build rather than shipping. `TestNothingInTheMarkDependsOnOpacity` guards the other half of the decision. ## Why the quiet bars are not opacity A quiet bar drawn as the loud colour at a low opacity reads well and fails. Composited over the page, 0.28 of `#f59e0b` gives **1.73:1** on the dark background and 0.42 of `#a16207` gives **1.80:1** on white, where AA asks 4.5:1. Reaching 4.5 by raising the alpha needs 0.68 in the dark theme and 0.96 in the light one — at which point a quiet bar is a loud bar and the one distinction the mark exists to make is gone. So the split is two solid tones. That costs punch. Solid against solid gives about **2.1:1** between the two groups; drawing the quiet bar at 0.28 (0.42 on white) separates them by about 5:1 in the dark theme and 2.7:1 in the light one, at alphas that read 1.73:1 and 1.80:1 against their own page and so fail AA. What it buys is that nothing in the mark composites against a background this repository does not control — a README on GitHub, an `og:image` in a chat client, a favicon over browser chrome — so every number in the table is the number the reader actually gets. ## A house rule, stricter than the standard WCAG 1.4.11 asks **3:1** of a graphical object, and 1.4.3 exempts logotypes from any minimum at all. 4.5:1 on every bar is a house rule stricter than the standard, chosen by the owner on 2026-09-15 over the smaller change of lifting the alphas to 3:1. ## Amber also means threshold Amber also means _threshold_ on this project's dashboards, where a panel turns orange before it turns red. The two do not collide literally — the panels use Grafana's own named colours and never these hexes — but a reader who has learnt "amber means look at this" on a dashboard is being asked to read the same hue as the project's own mark. That is the cost of the choice, made knowingly (owner, 2026-09-15). ## The bars are proportions, not packets The bar heights are not a real capture. The burst measured on the reference RB5009 on 2026-09-15 peaked at 736 packets in one 20 ms sample against a median of 28, and 26× is a range no single square renders: the quiet samples collapse to dots, or a log scale flattens the very spike the mark exists to show. The heights in the mark are proportions of the drawn height. ## The favicon is a different drawing Nine bars at sixteen pixels is mush, so the favicon drops to five and keeps the spike standing over the line, which is the part that carries the meaning. The site's `favicon.svg` is the only icon with no ground of its own. It carries both palettes and switches on the reader's own `prefers-color-scheme`, the signal browser chrome itself follows, so it draws `#fbbf24`/`#c2740a` over a dark chrome and `#633009`/`#b45309` over a light one. Every raster brings its own `#0e1316` ground instead, because an `.ico` has no way to ask. ## One generator, three families of files The mark lives as a generator, `cmd/gen_brand`, rather than as a folder of hand-drawn files, because changing the palette or the bar count is then one edit instead of nine in each of a dozen files. It is a build-time tool and is not one of the two released binaries. From the root of the repository: ```sh go run ./cmd/gen_brand mark -out brand # the mark and the favicon, per theme go run ./cmd/gen_brand compose -out brand # the banner, the social image and the og:image go run ./cmd/gen_brand icons -out site/public # the favicon and the touch icons ``` `mark` is pure text. `compose` reads the three `bg-*.png` backgrounds from the directory it writes to and shells out to `rsvg-convert` for the PNGs that ship. `icons` shells out to `rsvg-convert` and to ImageMagick for the `.ico`, and is the only one that writes outside `brand/`. Every coordinate is written with two decimals, rounded half to even, so the files are meant to reproduce byte for byte on any machine; `TestMarkIsByteForByteReproducible` checks that two runs on the same host give the same four mark and favicon files. | File | Where | Subcommand | What it is | | ------------------------------------------------------ | -------------- | ---------- | ----------------------------------------------------------------------------------------------------- | | `mark-dark.svg`, `mark-light.svg` | `brand/` | `mark` | The mark, one per theme | | `favicon-dark.svg`, `favicon-light.svg` | `brand/` | `mark` | The five-bar variant, one per theme | | `mark-inline.svg` | `brand/` | `mark` | The mark for a page that inlines it: the loud tone is `currentColor`, the quiet one `--ms-mark-quiet` | | `banner.svg` and `.png` | `brand/` | `compose` | 1280×320, for the README | | `social.svg` and `.png` | `brand/` | `compose` | 1280×640, the repository social preview | | `og.svg` and `.png` | `brand/` | `compose` | 1200×630, the documentation `og:image` | | `background.png` | `brand/` | none | The generated field the three compositions crop from | | `bg-banner.png`, `bg-social.png`, `bg-og.png` | `brand/` | none | Those crops, which `compose` reads | | `favicon.svg` | `site/public/` | `icons` | Both palettes, switching on `prefers-color-scheme` | | `favicon-32x32.png` | `site/public/` | `icons` | 32 px, on its own ground | | `favicon.ico` | `site/public/` | `icons` | Three drawings, at 16, 32 and 48 px, rather than one scaled three ways | | `apple-touch-icon.png`, `icon-192.png`, `icon-512.png` | `site/public/` | `icons` | 180, 192 and 512 px, each drawn at its own size | | `icon-maskable-512.png` | `site/public/` | `icons` | Inset further, to sit inside the middle 80 % a launcher may crop to | The last four rows are written for a web app manifest. This site declares none and links only `favicon.svg`, `favicon.ico` and `apple-touch-icon.png`, so `favicon-32x32.png`, `icon-192.png`, `icon-512.png` and `icon-maskable-512.png` ship without a consumer: no launcher reads the maskable inset. The mark in this site's header is `mark-inline.svg`, painted by the site's own palette, so the drawing in the chrome is the drawing in `brand/`. ## The background Not written by the generator: `background.png` was generated once with inference.sh (`openai/gpt-image-2`, 1536×1024, `quality: high`, $0.16) and is kept as a raster. Everything drawn over it is vector, so the type stays crisp at whatever size the raster is produced. The prompt asked for a near-black field of faint vertical sample bars growing denser and warmer toward the right, and for the left third to stay empty — which is where the mark and the type sit, so the composition never fights its own background. The three crops keep the whole left-to-right gradient rather than taking a window out of the middle of it: ```sh magick background.png -resize 1280x -gravity center -crop 1280x320+0+0 +repage bg-banner.png magick background.png -resize 1280x -gravity center -crop 1280x640+0+0 +repage bg-social.png magick background.png -resize 1200x -gravity center -crop 1200x630+0+0 +repage bg-og.png ``` The compositions are light on dark throughout, so they take the dark theme's two tones and need no theme pair. Measured over the field's own darkest ground (`#020608`, sampled from the left edge), the heading `#f6f3ee` reads 18.38:1, the tagline `#cfc6b8` 12.04:1, and the mark's two tones 12.19:1 and 5.62:1 — higher than on the page, the field being darker than it. The tagline is "Sub-second kernel telemetry from inside the router". ## Setting the social preview Manual: Settings, then Social preview, then upload `social.png`. GitHub offers no API for it. ## See also - [Five dashboards, one panel list](/mikroscope/dashboards/): where amber means a threshold. - [What it is](/mikroscope/start/): the claim the mark draws, in words. - [Lineage and licence](/mikroscope/about/lineage/): where the rest of the code came from. --- # Lineage and licence Which parts of mikroscope came from cs-routeros-bouncer and from go-routeros, what changed on the way, and the MIT licence both arrive under. Source: https://jmrplens.github.io/mikroscope/about/lineage/ This page answers where mikroscope's code came from: which packages started as someone else's, what was changed when they arrived, and under what licence. The attribution is also in the code, so it survives being read without this site: in the package doc comment of `internal/router`, `internal/image`, `internal/chart` and `internal/dashboards`, in `internal/rosapi`'s `README.md` and `LICENSE`, and in the header of `.golangci.yml`. ## Where the sampler came from The sampler that proved this works is `cmd/perfmon` in [cs-routeros-bouncer](https://github.com/jmrplens/cs-routeros-bouncer), built for one benchmark and kept there as a developer instrument. The deployment steps, the dockerless image builder, the deterministic chart and the vendored RouterOS API client come from that repository, under its MIT licence. ## What was taken, package by package | mikroscope | Taken from | What changed on the way | | ----------------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `internal/router` | cs-routeros-bouncer `cmd/perfmon`, the deployment steps after PR #123 | Extended with quoted ports in every `find`, the optional `--expose` firewall pair, mikroscope's own container settings, and a listing of every write before it happens | | `internal/router` tests | cs-routeros-bouncer `cmd/perfmon`, the containment tests | Adapted to mikroscope's option names and to two rules of its own: quoted ports, and the `--expose` pair as part of the plan | | `internal/image` | cs-routeros-bouncer `cmd/perfmon/image.go` | Reworked for the agent's name, the ARM variant (arm images declare `v7`) and a build goreleaser can reuse | | `internal/chart` | cs-routeros-bouncer `cmd/perfmon/chart.go` | The original drew its palette from a docs theme; this one carries its own, validated for the light surface (adjacent-pair CVD ΔE 9.1, normal-vision 22.9) | | `internal/rosapi` | cs-routeros-bouncer `internal/rosapi`, taken 2026-09-11 | Only the import path | | `.golangci.yml` | cs-routeros-bouncer, itself based on `maratori/golangci-lint-config` | The module path | What PR #123 brought, and what mikroscope's installer keeps from it, is the containment rule: every object created carries one exact comment tag, every removal selects by that tag plus the object's identity — never by pattern — and `uninstall` verifies by ownership counts before it reports success. Nothing is written without being listed first. The chart's output is deterministic: the same input always yields the same bytes. ## What was modelled, not taken `internal/dashboards` takes its Grafana export shape from `cmd/gen_dashboards` in the owner's ghchronicle — "the shape, not the code", as its package comment says — and `dashboards check` checks every panel on a real Grafana the way ghchronicle does. Two more pieces are modelled the same way, after a reference project this documentation does not name: the dashboards' named sections follow that project's `sections.go`, and the end-to-end suite under `test/e2e/` follows its suite. ## The RouterOS API client `internal/rosapi` has two generations of lineage. mikroscope's copy is cs-routeros-bouncer's, with nothing modified for mikroscope; mikroscope uses it for the API tier's reads, for `mark --log-markers`, and for the `/tool fetch` relay. The bouncer's copy is itself a pruned vendoring of [`github.com/go-routeros/routeros/v3`](https://github.com/go-routeros/routeros) at **v3.0.1** (upstream commit of 2025-02-16), MIT, copyright 2016 André Luiz dos Santos, whose `LICENSE` file stays in the package. ### Why it is vendored Upstream is effectively unmaintained: its last commit predates the copy by eighteen months, and the pull requests and issues filed against the async mode in mid-2026 (#31–#34) have had no maintainer response. No maintained alternative exists — `swoga/go-routeros` is a copy that only receives dependabot bumps for its GitHub Actions, and `jda/routeros-api-go` stopped in 2016. Vendoring keeps the code buildable and lets the copy fix and prune what it uses. ### What differs from upstream v3.0.1 These changes were made in cs-routeros-bouncer and arrive unchanged: 1. **The async/listen mode is removed**, with its tests and the async branch of `RunArgsContext`. Nothing in the bouncer called `Async()`, `Listen()` or the context-cancelling run variants. The sentence-kind constants moved to `reply.go`, which consumes them. 2. **The protocol reader and writer read and write directly**, instead of dispatching every call to a fresh goroutine with a channel and a copy buffer; only the removed async mode ever cancelled an in-flight read. At the bouncer's production scale — 22k address-list entries fetched every cycle — that layer cost ~294 000 goroutine spawns, ~62 MB of garbage and ~76 % of all allocations per reconcile cycle. The rewrite was checked against pristine upstream with a differential test: the SHA-256 fingerprint over all 22 037 parsed sentences identical, error shapes identical (`io.ErrUnexpectedEOF` on truncation, `*DeviceError` on `!trap`), and upstream's suite green under `go test -race`. `proto/reader_shape_test.go` pins that behaviour. Those four figures are cs-routeros-bouncer's own, measured on its own host against its own workload, not on a router and not by this project; `internal/rosapi/README.md`, where they come from, records no date and no host, so none can be given here. 3. **The pre-6.43 MD5 challenge login is removed.** That login exists only before RouterOS 6.43 (2018), and answering it means hashing the password with MD5. A router that sends a `ret` challenge gets `ErrLegacyLoginUnsupported` instead. 4. **The async and listen-mode tests are removed** with the mode; the pre-6.43 login tests assert the rejection instead of the handshake. The rest of upstream's suite is kept and passing. Compare against upstream's `v3.0.1` tag, not its master branch. ## Licence mikroscope is MIT-licensed, copyright 2026 jmrplens; the full text is `LICENSE` at the root of the repository. The code taken from cs-routeros-bouncer arrives under that repository's MIT licence, and `internal/rosapi` keeps go-routeros's MIT `LICENSE` beside the code it covers. ## See also - [What the installer refuses](/mikroscope/security/installer/): the containment rules the deployment code inherited, as they stand now. - [The RouterOS API tier](/mikroscope/sinks/api-tier/): what the vendored client reads. - [Record, mark, plot](/mikroscope/record/): the deterministic chart `internal/chart` draws. - [Where the project stands](/mikroscope/about/status/): what works today, and what the release publishes.