Skip to content

The cost of the observer

An observer that costs 20 % of the thing it observes is not measuring the router, it is measuring itself. So this number is a first-class result here, not a footnote: it is reported by the agent on every sample and exposed on /metrics, and the image-size budget is asserted in CI.

The budget is ≤ 2 % of one core, ≤ 16 MiB RSS, ≤ 8 MiB image. The image is 6.1 MiB. The other two depend on the rate and on how much you ask it to read, and the honest answer is a table rather than a number.

At the install default — 10 Hz, default per-source floors, a 300 s ring — the agent costs 2.85 % of one core and 31.3 MiB RSS, from its own cgroup:

Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · · 60 s windows at steady state (ring full), full source set, collector forwarding to a file, a Prometheus exposition and InfluxDB 3 at once

That is above the 2 % the budget asks for, with every source read — the perf timings, buddyinfo, the MTD ECC counters, the cgroup events and the port counters among them.

The budget says what the agent is allowed to cost. The other comparison, the one a reader usually wants, is against doing it the obvious way: a busybox shell loop reading the same file set at the same rate. On the same router that costs 2.4 % of one core, while the reads themselves are about 0.77 ms per sample.

Measured on RB5009UG+S+ · 4 × 1.4 GHz Cortex-A72 · RouterOS 7.24.2 · · a busybox shell loop reading the full file set at 10 Hz, one fork per iteration, in a container on the router

So most of the shell loop’s cost is not the reading. It pays a fork per iteration and the agent pays none: one process starts, opens its files once and keeps them open. That is the whole of the difference, and it is why the agent is a binary rather than a script.

The single most expensive mistake available here is giving the process less memory than its ring needs. The Go garbage collector responds to a tight soft limit by running more often, and the CPU cost jumps for a reason that has nothing to do with the sampling rate.

The ring holds pre-encoded lines rather than structs for the same reason. If you raise --buffer or the rate, raise --mem-limit-mb and the container’s --memory-max with it; the numbers used for each row of the measurements are listed beside them on the rate ceiling.

The parse is not where the time goes. Parsing the seven global /proc files plus one delta took 27 µs and 239 allocations per sample on the amd64 development host (Go 1.27.1, three runs, 26.7–27.5 µs, 2026-09-11). On the RB5009’s Cortex-A72 a whole tick — timers, JSON and the garbage collector included — costs 2 856 µs at 10 Hz with the default floors, and the five runs give it at every rate. The parse was not measured on the A72 on its own.

Nothing here transfers to a board that is not an RB5009 — a different core count, a different clock, a different kernel and a different flash all move it. The procedure is three commands and takes a minute:

Terminal window
curl -s http://172.30.10.2:9123/metrics | grep -E 'self_cpu_usec_total|self_rss_bytes'
sleep 60
curl -s http://172.30.10.2:9123/metrics | grep -E 'self_cpu_usec_total|self_rss_bytes'

The difference in self_cpu_usec_total divided by 60 000 000 is the share of one core. Take it at steady state, with the ring full: a freshly started agent is still filling it and will read low.