Skip to content

When something does not work

Every page here explains one thing properly. This one is the index you reach for when something is already broken: find the line you are looking at, and it says what it means and where the explanation lives.

You see It means
doctor: device-mode container=no The one step nobody can do remotely
doctor: container package installed and enabled … found=0 The package is not on the router
doctor: architecture matches --arch … router=arm Re-run with the --arch it names
RouterOS: unknown parameter privileged RouterOS older than 7.24
Container log: exec format error The wrong image for the board
no Go toolchain on PATH Install with --remote-image or --agent-tar instead
--agent-tar …: this is not a mikroscope agent image The wrong asset — see which tar
doctor: registry-url is https://ghcr.io … registry-url=… The registry host is global
doctor: free flash ≥ … fails --disk tmpfs or --ephemeral, or free space on the flash

MikroTik gates containers behind a switch that cannot be flipped over the network. /system/device-mode/update container=yes starts it, and then the console asks for a physical confirmation — the reset button, or a power cycle — within five minutes. No flag, no script and no version of this tool can do that step for you. It is the first thing to arrange, because everything else waits on it: What the router needs.

The container package is a separate download from mikrotik.com, per architecture and per RouterOS version. Upload it, reboot, then /system/package/enable container. doctor counts it as present only when it is installed and not disabled.

RouterOS 7.24 added privileged=, and the container step writes it, so an earlier 7.x fails there — after the tar has been uploaded, which is why the install then takes it back with it. Either upgrade RouterOS, or install with --privileged=false and read what privileged buys first: without it the agent cannot read /dev/kmsg, and the kernel log is where several of this project’s playbooks start.

The container starts and dies immediately, and the log says exec format error. The image is for another architecture than the board — and on 32-bit ARM, “arm” is not one architecture.

MikroTik’s container documentation says devices with the EN7562CT CPU, the hEX Refresh line, “support only arm32v5 container images”; its other 32-bit ARM boards run an ARMv7 userland. An ARMv5 image runs on both, an ARMv7 image does not run on the first. So:

  • With --remote-image, this cannot happen: the published index carries all four platforms and the router matches its own.
  • With --agent-tar, take mikroscope-agent-armv5.tar when the board is 32-bit ARM and you are not certain which kind it is.
  • Building from a checkout, --goarm 5 is the default for the same reason.

Which tar is the table.

/container/config registry-url is one global RouterOS setting, shared with every other container on the device, and mikroscope reads it and never writes it. The Docker Hub reference works out of the box because that is what RouterOS ships pointing at; the GHCR one needs the setting changed first, which changes it for everyone else on that router too.

The agent is installed and nothing answers

Section titled “The agent is installed and nothing answers”
Terminal window
mikroscope status

That prints the ownership counts and, if it can reach the agent, its health. If the counts are there and the health is not, the container is running and something between you and it is not:

  • The firewall. Two rules commonly eat this traffic, and neither is obvious: The two firewall traps is that page, and doctor checks the two list memberships that avoid them.
  • The route. The agent answers on its /30, on the router’s LAN side. A collector elsewhere reaches it the ways Reaching the agent lists.
  • The container never started. /container/print detail on the router, and /log/print where topics~"container".
You see It means
No events at all, ever privileged=yes is what /dev/kmsg needs
No PMU panels, no cycles or instructions perf_event_open is unavailable on that kernel or board
A panel says No data and the others are fine That measurement is not produced on this device; the dashboard has a row for it
A panel shows a red error badge The query failed — the datasource, not the data
forward prints … dropped for a sink The sink could not keep up
Loki accepted everything and a query returns nothing A push is not queryable until the chunk flushes
Numbers stop at a round moment and resume A gap: the ring wrapped before the collector pulled it
The kernel tier stops dead and the API tier carries on The agent restarted

The dashboards carry a row named “Not available on this device” for exactly this: panels whose measurement the kernel or the board does not produce are moved into it rather than left to draw an empty graph among the others. mikroscope dashboards check asks the datasource which measurements it really holds and does that sorting for your store: Import and check.

The agent restarted and the kernel tier stopped

Section titled “The agent restarted and the kernel tier stopped”

The agent numbers its samples from 1 at every start, so an agent that restarts — an upgrade, a container restart, a reboot — has a newest sequence number far below the collector’s cursor. The collector notices on its next health read, which is once a minute, logs

agent restarted: its newest sample is 571 and the cursor was 1737212; resuming from 1

and resumes from the new ring’s oldest sample, so what the agent took while nobody was collecting is picked up rather than skipped. The minute’s worth of samples between the restart and the health read is lost with the container, not by the collector.

Before 1.0.4 it did not notice: the cursor stayed where it was, the agent’s ring answered an empty batch to every pull, and the kernel tier stopped for good while the API tier kept counting and the sinks kept being written — so the run looked healthy. If you are on an earlier version, restart the collector after restarting the agent; it takes its cursor from the health read at start.

Every network sink is queued, and the queue is bounded — --queue-seconds, 60 by default. A destination that cannot keep up loses the oldest batch rather than stalling the pull loop, and the count is printed at the end of the run and exported as a metric. That is a deliberate choice, and the collector explains it: the agent’s ring is what protects the data, and a collector waiting on a slow store would lose more than the store does.

Once the data is arriving, the question changes from “why is this broken” to “what is this telling me”. That is a different set of pages: How to read what it shows — the shape of an idle router first, then seven faults read against it.