Test suites

Four suites test mikroscope, and none of them needs a router. Run the one that covers your change before you open a pull request. CI runs the first three on every pull request. The lab runs weekly and on dispatch, so run it yourself before a pull request that touches what installs the agent.

Suite	Command	Needs	What it proves
Unit	go test ./cmd/... ./internal/...	Go	each package, against captured /proc trees, fakes and golden files
End to end	make test-e2e	Go	both binaries as separate processes, and the exact bytes each sink sends
Stores	make test-e2e-docker	Linux, Docker	a real store accepts those bytes and hands them back; every dashboard panel’s query answers
Virtual RouterOS lab	make lab-up, then make test-lab	Linux, Docker, /dev/kvm for x86_64	the deploy verbs against a real RouterOS, with the router’s /export back to its start after each one

make test runs the unit and end-to-end suites together. The stores and lab suites sit behind the build tags dockere2e and labe2e, so make test compiles neither, and make lint type-checks both so that they cannot rot.

When each suite first ran, how long it takes and what it found are on Tested on.

Choose a suite

You changed	Run
Any Go code	make test
The sampler, the ring, the stream or a sink	make test-race as well
A test that might have grown a dependency on the network	make test-e2e-offline (Linux)
A sink, its encoding or its schema, or a dashboard	make test-e2e-docker
internal/router, a deploy verb, the agent image or how the agent starts	make lab-up, then make test-lab
The lab’s driver (cmd/mikroscope-lab, internal/lab)	go test ./internal/lab/... ./cmd/mikroscope-lab/, then make test-lab

The static checks a pull request owes as well (make analyze, the agent’s size budget, the site’s gates) are in CONTRIBUTING.md.

Unit suite

go test ./cmd/... ./internal/...   # the unit suite alone
make test                          # every untagged package, the end-to-end suite included
make test-race                     # every suite under the race detector
make cover-check                   # fails below COVERAGE_MIN over cmd/ and internal/

Each package tests itself against fixtures, never against a router:

• The /proc and /sys parsers read a captured tree of a real router, testdata/proc/rb5009/.
• plan, the install script and the listings are compared with golden files in internal/router/testdata/, one set per case of cases.json. make check-generated fails when the site’s copy of those scripts is stale.
• The lab’s driver runs against a fake Docker and a fake router: provisioning, the lock, the downloads and their checksums, the snapshot chain, the firewall rules, QEMU’s command line and the CLI’s refusals.

Job	What it runs	When
Cross-platform, Linux, macOS, Windows	the unit suite; on macOS and Windows the end-to-end suite too	every pull request
Coverage	make cover-check	every pull request
Race detector	make test-race	weekly, and before a release

End-to-end suite

make test-e2e builds both binaries and drives them as separate processes. The agent reads the captured tree. The collector reads that agent, or a fake one that serves canned samples, and writes every sink to a receiver inside the test binary, which asserts what arrived byte for byte.

Sink	Receiver
InfluxDB, Loki, OTLP, Elasticsearch, Telegraf	an HTTP capture server
Graphite, Telegraf’s socket modes	TCP and UDP listeners
File, SQL	real files
stdout	the process’s own standard output
Prometheus	a scrape of the collector’s /metrics

It needs no router, no Grafana, no database, no container and no network: every address it binds or dials is on loopback. make test-e2e-offline proves that on Linux by running the suite in a network namespace with nothing but lo. It needs unshare and setpriv, and runs unshare under sudo unless it already has the capability.

CI runs the suite and its offline form on every pull request, weekly and before a release, and runs the suite on macOS and Windows as well, where the executable’s name, the files the file and SQL sinks write and how a child process is stopped can differ.

Stores suite

A capture server answers 204 to anything; a store does not. make test-e2e-docker starts eight stores and a Grafana with docker compose, runs the same collector against the same fake agent with every sink pointed at them, and asks each store its own question through its own API. The file sink’s JSONL is the oracle: each store is compared with it value by value, not only by count.

Store	The question it is asked
InfluxDB 3 Core	SQL over HTTP: the tables, a row count per table, every ctxt value
PostgreSQL 18	the SQL sink’s script through psql, and the connecting PostgreSQL sink into a second database: counts, the ctxt range, and whether both hold the same rows
Elasticsearch 9	_search with an aggregation by kind, and one whole document
Loki 3	query_range for this run’s labels, and the text of each record
Graphite	metrics/find for the tree, render for the points and their order
OpenTelemetry Collector	what it decoded, written back out as OTLP/JSON
Telegraf 1.39	the line protocol it parsed: measurements, tags, field types, timestamps
Prometheus 3	a scrape of the exporter, against the exposition it served

The suite also publishes the five datasources with forward --grafana, and empties every store again with uninstall --targets data.

Failures only a store shows

• InfluxDB 3 fixes a column as a tag or a field the first time it sees the table, and refuses a later write that disagrees.
• Carbon answers nothing at all: a point older than its longest archive, or a name whisper cannot make a path of, is dropped in silence.
• Loki answers 204 to a push that is not queryable until the chunk flushes, and rejects out-of-order entries per stream, in the body.
• Elasticsearch infers a mapping from the first document it sees for a field, then rejects a later one that does not fit it, per document, inside a bulk request that still answers 200.
• Telegraf is a real line-protocol parser: an unescaped space in a tag value, or a field with no type, is dropped and the batch is still 204.
• PostgreSQL is the only thing that can say whether the SQL sink’s script is valid SQL, whether the types it chose hold the values it emits, and whether its primary keys collide on a real run.

Dashboards

The same run imports all five dashboards into Grafana, each pointed at the store the run just filled, and runs every panel’s query through Grafana’s own /api/ds/query: the dashboards check you can run against your own store. A panel fails there for reasons no unit test reaches: a type an aggregate returns that the datasource plugin cannot decode, a macro the plugin escapes, a column the store does not have. On InfluxDB 3 a missing column is not an empty one: the query fails with Schema error: No field named <column>, and the panel cannot render at all (found this way).

Run the stores suite

make test-e2e-docker        # the whole suite; the stack comes up and goes down with it
make e2e-docker-up          # keep the stack up between runs
go test -v -tags dockere2e -run TestLoki ./test/e2e/docker/
make e2e-docker-down        # stop the stack and delete its volumes
make test-e2e-docker-race   # the same suite under the race detector

The harness reuses a stack it did not start, and leaves it up afterwards. Why each store is a real one and not a fake is in test/e2e/docker/README.md. CI runs the suite on every pull request, weekly, on dispatch and before a release.

Linux, for two of the nine

Prometheus and Grafana run on the host’s network stack, because Prometheus is the one sink that is scraped rather than pushed to, and it has to reach the collector’s exporter on the host. A container on a bridge network reaches the host through the bridge’s gateway, and that path goes through the host’s INPUT chain, which a default-deny firewall drops. The other seven services are ordinary bridge-network containers.

Virtual RouterOS lab

doctor, plan, install, upgrade, status and uninstall need a RouterOS to be tested at all. The lab is MikroTik’s Cloud Hosted Router (CHR), a real RouterOS, under QEMU in a Docker container. It is provisioned once into a clean snapshot, with the container package installed and device-mode container=yes confirmed, and put back to that snapshot in seconds. The full account, how it is wired and what CHR taught the installer, is test/lab/README.md.

Lab	Runs as	Use it for
x86_64	CHR under KVM	every scenario, fast; CI runs it weekly and on dispatch
arm64	CHR emulated by QEMU (TCG), CPU model cortex-a72	the arm64 container package and agent image, and RouterOS choosing the image for arm64; slow

A third lab, RouterOS x86 installed from MikroTik’s ISO (make lab-up LAB_KIND=iso, x86_64 only), is an opt-in recipe that CI never runs. It adds the PC install path, an x86 board name and the x86 licence (a 24-hour trial, then a Level 1 registration or a paid licence per device), and nothing the agent reads that CHR x86_64 does not.

The CLI runs inside the lab’s LAN. mikroscope-lab cli starts it in the lab container’s network namespace, where 172.30.0.0/16 routes to the lab router, so the agent’s default address, 172.30.10.2, reaches the lab’s agent and nothing else. From your own shell the same address leaves by your default route, toward whatever network that is. So every deploy verb, by hand (make lab-cli) or in the suite, goes through mikroscope-lab cli, which refuses --router and a --subnet outside the lab’s routes. The lab’s namespace has its own firewall: it opens no new connection to a private address outside the lab, and takes none from another container.

Lab suite

make test-lab builds the CLI and the agent image tars, then runs the suite in test/e2e/lab/ against the running lab, holding the lab’s lock for the whole run. Every scenario starts from a reset and the profile it names, takes the router’s /export there as its baseline, and ends by comparing the two.

Scenario	What it does	What it asserts
S1	doctor on a stock CHR, with the lists none and with the defaults	nothing missing with none; with the defaults only the interface list LAN, whose fix offers --iface-list none
S2	install by pulling the image from Docker Hub, status, upgrade, uninstall	the agent answers /healthz and /capabilities; the export is back to its baseline
S3	install and upgrade from the branch’s own image tar	the agent reports the branch build’s version, commit and date
S4	plan --rsc for both image routes, run with /import	status recognises the script’s objects; uninstall leaves the baseline
S5	every golden script the lab can run, imported as the site hands it over	the agent answers on its own /30; uninstall with the case’s flags leaves the baseline
S6	--ephemeral on a tmpfs disk, then a power cut	the container is left stopped with its root gone; uninstall --ephemeral leaves nothing at all
S7	a persistent install, 45 s for it to reach the disk, then a power cut	the agent answers again within 90 s: start-on-boot works
S8	--expose with a token	/capabilities answers 401 without the token and 200 with it; both firewall rules go with uninstall
S9	uninstall while a client reads /stream, ten times	every first attempt verifies the router clean
S10	the raw rules of MikroTik’s advanced firewall, in list and range form	doctor names the rule that drops the agent’s replies, and the list memberships that pass
S11	a foreign veth, a routed /30, a foreign envlist	doctor names each; install writes nothing; the plan --rsc script stops at its guard for the veth and the envlist
S12	two installs side by side	removing one leaves the other running and untouched
S13, S14	installs with other lists, and with --expose, uninstalled with no flag	everything they created is removed; a contradicting flag is refused
S17	doctor on a lab below the minimum RouterOS	RouterOS 7.24 or later is missing, and every other check still reads
S18	a pull from GHCR with no registry credential	the agent answers
F4	every install route, an install made by a released CLI, objects the user made first	/export equals the one before the install and /file holds no mikroscope path; the user’s objects stay
repeat	tar installs and uninstalls in a row, ten on x86_64 and three on arm64	no install fails

The scenarios assert the fixed behaviour: none runs a second uninstall, and none allows a mikroscope path left on /file. Four more tests hold what the CLI must keep doing: the agent token on no command line of the host, RouterOS’s words in every error, the probe reading the running flag, and doctor’s warning for start-on-boot with the root on a tmpfs disk.

S7 waits 45 s between the install and the power cut. A cut made as soon as the agent answers can bring back a container that cannot start (seen in the lab), which is a question of its own; S7 asks only whether start-on-boot works.

Only the scenarios that test a pull make the router pull. Docker Hub limits anonymous pulls per address, and a CI runner shares its address with whatever else ran from it, so every other scenario installs the branch’s own tar from make agent-tars, which also tests the branch’s agent. The pulls are anonymous unless the lab is given an account (Pull as an account), and the scenarios about a router with no credential, S1, S18 and S5’s Docker Hub and GHCR scripts, boot without it in any case. The suite drops every MIKROSCOPE_* variable before it runs anything, so a shell set up for a real router cannot steer it.

Run the lab

make lab-up                   # start the lab; the first run downloads RouterOS and provisions it
make lab-status               # the container, who holds the lock, what the router reports
make test-lab                 # the whole suite, x86_64
make test-lab LAB_RUN='S09'   # the tests a go test -run pattern matches
make lab-cli ARGS='doctor --arch amd64'
make lab-reset                # back to the clean snapshot
make roundtrip                # doctor, install, status, upgrade, uninstall; /export compared
make lab-down                 # stop it; the disks keep their state
make lab-up LAB_ARCH=arm64    # the emulated lab: every target takes LAB_ARCH=arm64

S17 needs a lab below the minimum RouterOS, which is a lab of its own:

make lab-up LAB_ROS=7.23.7
make test-lab LAB_ROS=7.23.7 LAB_RUN=S17

The lab needs:

• Linux, and Docker allowed to give a container NET_ADMIN and /dev/net/tun.
• Go, for the lab’s driver and for the CLI and agent under test.
• About 910 MB of disk for both architectures, 810 MB for one.
• The loopback ports 220N, 800N, 870N and 910N free, with N = 1 for x86_64 and 2 for arm64, plus an instance’s offset.
• Network access: Docker Hub and Debian’s mirrors for the lab image the first time, download.mikrotik.com once per RouterOS version, and Docker Hub for what the router pulls.
• /dev/kvm for x86_64. Without it, LAB_KVM=auto falls back to emulation, many times slower, and LAB_KVM=require stops instead. arm64 is emulated on an x86 host whatever you set.

Without a running lab every test of the suite skips, and MIKROSCOPE_LAB_REQUIRED=1 makes that a failure. The lab’s admin password and agent token are generated into test/lab/.env on first use, gitignored, and never printed.

Every target runs the lab’s driver, bin/mikroscope-lab, which make lab-tool builds from cmd/mikroscope-lab/ and internal/lab/: a build-time tool that nothing ships. bin/mikroscope-lab help lists its verbs, and test/lab/lab.sh only builds and starts it, for a command written for the old script. The same static binary is the lab container’s first process, which sets up its network and runs QEMU until the router powers off.

One driver drives a lab at a time: every verb takes the lab’s lock, and bin/mikroscope-lab lock <command> holds it for a whole session, as make test-lab and make roundtrip do. The two labs run side by side, and so can their suites, but not as two make test-lab in one checkout, because each rebuilds the agent tars the other may be reading. Build once, and start each suite under its own lab’s lock:

make build agent-tars lab-tool
LAB_ARCH=x86_64 bin/mikroscope-lab lock go test -tags labe2e -count=1 -timeout 150m ./test/e2e/lab/ &
LAB_ARCH=arm64 bin/mikroscope-lab lock go test -tags labe2e -count=1 -timeout 150m ./test/e2e/lab/

Pull as an account

The lab router pulls the agent image itself, about a dozen times per run of the suite, and Docker Hub limits anonymous pulls per address. Given a Docker Hub account, the router pulls as that account instead:

# in a file of your own, mode 0600, outside the repository
LAB_REGISTRY_USER=<the Docker Hub account>
LAB_REGISTRY_TOKEN=<a personal access token of it, read-only>

set -a; . ~/.config/mikroscope/lab-registry.env; set +a
make lab-reset                  # every up and reset gives the router the credential

The lab only pulls, so on your own machine a read-only token is enough. At every up and reset the lab’s driver writes /container/config/set registry-url=… username=… password=… to a file, copies it to the router, runs it with /import and deletes it, so the credential is on no command line and the cached snapshot never has it. Exported from a file, as above, it stays off make’s command line too, which the process table would show. Without the two variables the router pulls anonymously; one without the other is refused. The suite prints the user and the token as <LAB_REGISTRY_USER> and <LAB_REGISTRY_TOKEN>, and one of its tests reads the host’s process table every 2 ms through a reset to hold the driver to all of it.

registry-url is registry-1.docker.io, with no scheme (LAB_REGISTRY_URL names another registry’s host). RouterOS presents the username and password for a reference whose host is registry-url as written, and mikroscope writes the host into every reference; with https:// in front, the same pull goes out anonymously (verified).

Lab settings

Each setting goes on the make line or in the environment.

Setting	Default	What it does
LAB_ARCH	x86_64	which lab: x86_64 or arm64
LAB_ROS	the Makefile’s	the RouterOS version to download and run; a new version is a new provision
LAB_KIND	chr	iso is RouterOS x86 from MikroTik’s ISO, x86_64 only
LAB_KVM	auto	require stops without /dev/kvm, off never uses it, auto uses it when it is there
LAB_RUN	every test	a go test -run pattern for make test-lab, such as S09 or F4
LAB_INSTALL_REPEAT	10 on x86_64, 3 on arm64	how many installs the repeat test makes
LAB_S5_CASES	every case the lab can run	S5’s cases, by id, separated by commas
LAB_REMOTE_IMAGE	the last release on Docker Hub	the image the pull scenarios pull
LAB_GHCR_IMAGE	the same image on GHCR	the image S18 pulls
LAB_STATE_DIR	this checkout’s test/lab	where the lab’s .cache/ and .env live: point a second checkout at a lab another one runs
LAB_INSTANCE	none	a second lab of an architecture, with its own container, ports, lock and disks
LAB_LOCK_WAIT	as long as it takes	seconds to wait for another driver’s lock before failing and naming the holder
MIKROSCOPE_LAB_REQUIRED	unset	1 turns a skip for want of a lab into a failure
LAB_REGISTRY_USER, LAB_REGISTRY_TOKEN	unset: the router pulls anonymously	a registry account the lab router pulls with, given to it at every up and reset, never in the snapshot
LAB_REGISTRY_URL	registry-1.docker.io	the registry that account belongs to, written as the host the references name

Lab in CI

The workflow .github/workflows/lab.yml runs the same make lab-up and make test-lab, with MIKROSCOPE_LAB_REQUIRED=1, for both architectures, weekly on main and on dispatch. It never runs on a pull request or before a release: a run takes half an hour or more. Run make test-lab yourself before a pull request that touches what installs the agent: internal/router, internal/image, internal/agent, cmd/mikroscope, cmd/mikroscope-agent, Dockerfile.agent, the Makefile, the agent tar and round-trip scripts, the RouterOS scripts the site renders, and the lab, its driver and its suite. A dispatch takes ref, arch (both, x86_64 or arm64) and ros.

arm64 runs under emulation, which is slow, and depends on MikroTik’s download server and on Docker Hub being up.

The Actions cache keeps MikroTik’s downloads per architecture and RouterOS version, checked against the SHA-256 sums the repository pins in test/lab/SHA256SUMS, and the provisioned router, keyed on the driver, the lab image and those sums. The snapshot carries no credential: admin has CHR’s empty password and no key until the run’s make lab-up gives it that run’s own, so the .env and the ssh key never leave the runner. A failed or timed-out run uploads the console log, the container log, the lab’s status, what an install left on the router and the test log, each lab credential replaced by its name, and never the .env or the ssh key.

The router pulls as an account when the repository has two secrets, DOCKERHUB_USERNAME and DOCKERHUB_TOKEN, the token the release also pushes images with; the lab only pulls with it. They become LAB_REGISTRY_USER and LAB_REGISTRY_TOKEN for the steps that bring the lab up, run the suite and redact the failure report, and no other; ci.yml and release.yml pass the two by name. A pull request from a fork gets no secrets, and a repository without DOCKERHUB_TOKEN gets neither variable: both run anonymously. So does a dispatch that checks out another ref, which may be a fork’s merge commit: the lab builds and runs that code, and the token stays out of its environment.

Lab limits

• Virtual, not hardware. No RouterBOARD, no flash, no sensors, no switch chip and no device tree. cpufreq, mtd, psi, schedstat and thermal are absent from the agent’s /capabilities, and status cannot map kernel port names to RouterOS ones. Anything about a board still needs a real one.
• The free CHR licence caps what the router sends at 1 Mbit/s per interface. A rate test above 10 Hz in the lab measures the licence rather than the agent (the arithmetic).
• Emulated arm64 figures are not costs. Under emulation the guest’s clock follows the host’s, so every duration, every CPU figure (cpu-load, self.cpu_us, read_ns, wake_ns, the dt_ns spread, slipped), every interrupt, softirq and context-switch rate and every PMU count from the arm64 lab measures the host’s emulation, not the core it emulates.
• No 32-bit ARM. MikroTik publishes CHR for x86_64 and arm64 only, so the armv5 and armv7 agent meets RouterOS only on hardware. make agent-smoke starts its image under QEMU user-mode, which is not RouterOS.

Manual checks

No suite shows:

• A sink fed from a router. The samples in every suite come from a fake agent, a captured tree or the lab’s CHR, and the stores run on the test host, not across a router’s veth. Which sinks have carried a router’s samples is under Feature status.
• Anything about a board: its flash, sensors, switch chip and CPU frequency, and what the agent costs on it. Agent cost says how to measure it on yours.

See also

• Tested on: when each suite first ran, how long it takes, what the lab found, and what has not been tested.
• Run the collector: what each sink writes.
• Set up in Grafana: the same dashboards check the stores suite runs, against your own store.