Other sinks

Besides Prometheus and InfluxDB 3, forward writes to nine more destinations. The queue, backoff and counters they share are on Run the collector.

Sink coverage

Sink	Kernel samples	Kernel-log text	API tier	Derived and detections	Triggers, gaps, device
file	verbatim agent lines	inside the sample lines	{"api":…} lines	{"derived":…}, {"detection":…}	all three, as lines
stdout json	verbatim agent lines	inside the sample lines	{"api":…} lines	as the file	all three, as lines
stdout lp	the InfluxDB measurements	counts per level, port and kind	the InfluxDB measurements	the InfluxDB measurements	all three
SQL (--sql, --postgres)	one table per source	mikroscope_event rows	tables, plus mikroscope_api_error	mikroscope_derived, mikroscope_detection	all three
Loki	no	one line per record	per-command errors only	detections only	all three, as lines
OTLP	sums and gauges	no	gauges, and an error count	gauges, a detection sum	all three
Graphite	one path per value	no	paths	paths	numeric parts only
Elasticsearch	one document per tick	one document per record	one document per read, no errors	on the kernel document, a document per detection	all three
Telegraf	the InfluxDB measurements	counts per level, port and kind	the InfluxDB measurements	the InfluxDB measurements	all three

Kernel-log records exist only when the container runs with privileged=yes; see Privileged mode.

File

mikroscope forward --file timeline.jsonl

The file is opened and truncated, with mode 0600. Every line is one JSON object, and its first key says what it is:

• a kernel sample — the agent’s own line, byte for byte, as /snapshot served it;
• {"trigger":…} — the agent’s capture marker, also verbatim;
• {"derived":…} — the derive stage’s values, on the line after the sample they belong to, so a reader that wants only raw samples skips that kind;
• {"detection":…}, {"device":…}, {"api":…}, {"sampler":…} and {"gap":…}.

Writes are synchronous through a 64 KiB buffer, and the counters are in events. A write the filesystem refuses counts one error and one drop.

Standard output

mikroscope forward --stdout=lp | telegraf --config …
mikroscope forward --stdout=json | jq

--stdout lp renders InfluxDB line protocol with the InfluxDB sink’s own encoder, so a pipe shows exactly what --influx would post. --stdout json writes the same line kinds as the file. Any other value is refused before the run starts.

Standard output can stall — a slow reader fills the pipe buffer and the write then blocks with nothing to bound it — so this sink is queued: one batch per second, each batch written in a single call of whole lines so a reader never sees a partial record, 64 KiB × --queue-seconds of budget, oldest dropped first. A batch that a stalled reader lets grow past the whole budget is closed early. That budget was sized for a line-protocol sample of about 1.2 KiB. For json, whose lines carry sources line protocol leaves out and are larger, it is not measured.

• A broken pipe does not reach the counters: Go leaves SIGPIPE on standard output uncaught, so the process exits.
• On shutdown the sink waits at most 3 s for the reader to accept the last batch, so a reader that stopped reading cannot hang forward.
• forward prints its exit summary on standard output as well, so a pipe into telegraf ends every run with lines the consumer cannot parse.

SQL script (--sql)

mikroscope forward --sql out.sql --for 10m && psql -f out.sql
mikroscope forward --sql - | psql        # see the caution below

--sql writes PostgreSQL text — a DDL header, then one INSERT per row — to a file, or to standard output with -. It is driverless on purpose: speaking the PostgreSQL wire protocol needs a third-party driver, so the SQL text is the interface and psql owns the connection. The cost is that the sink cannot know whether a row was stored; it counts the events it wrote.

The header is applyable on its own and idempotent:

• SET standard_conforming_strings = on; — so a backslash in a kernel message can never turn into an escape and swallow the statements after it;
• CREATE TABLE IF NOT EXISTS for every table, each with a primary key that starts with time, host;
• with --sql-hypertable, a TimescaleDB create_hypertable call on time for each table, with if_not_exists => TRUE.

Every INSERT ends in ON CONFLICT DO NOTHING, so applying the same file twice is a no-op rather than a duplicate-key abort. A row is one immutable instant of a counter delta, never a running total a later file revises.

PostgreSQL (--postgres)

mikroscope forward --postgres 'postgres://user@host/mikroscope?sslmode=require'

--postgres writes to a PostgreSQL that is running, instead of to a file somebody replays later. --sql and --postgres are the same sink twice: the connecting one sends the statements the file one produces, from one renderer. Not two that agree today — the dashboards this project generates for the postgres store have to be true of a deployment that used either, and a second renderer would pass a row count and still drift a column.

The file is still what you want when the database is somewhere the collector cannot reach, when the load is meant to happen later or under review, or when the reader is not PostgreSQL at all: the statements are ordinary SQL and another engine can take them. Both can run at once, into different databases or the same one — ON CONFLICT DO NOTHING makes the overlap a no-op.

What the connection can do that a file cannot:

• It knows whether a row was stored. A batch goes inside a transaction and lands whole or not at all, so a retry after a failure cannot leave half an event behind.
• It checks the assumption the header can only ask for. SET standard_conforming_strings = on is advice in a file the operator may run without it. Over a connection the setting can be read back, so it is — and a server that answers off is refused with the reason rather than written to, because a kernel message ending in a backslash would escape its own closing quote and everything after it would be parsed as string content.
• It can describe its own Grafana datasource, which the file sink can never do: see Datasource per sink.

Unlike --sql, --postgres is queued like the remote sinks: one batch a second, 64 KiB × --queue-seconds of budget with the oldest batch dropped first, each batch one transaction with a 30 s timeout, and the counters in batches. A slow server therefore never blocks the pull loop, which the pipe into psql can.

What it gives up: nothing about the schema, and one dependency. --postgres links pgx into the collector binary. The agent does not link it — the agent links procfs, sample, agent and the standard library, so its image carries nothing new, and CI’s agent-size job still holds it under the 8 MiB budget.

The DSN is a flag rather than an environment-only secret because libpq’s own conventions are the point: pgx reads PGPASSWORD, ~/.pgpass and the service file the way psql does, so a DSN with no password in it works the same way every other PostgreSQL client does. MIKROSCOPE_POSTGRES_DSN sets the default.

A password in a DSN is visible in the process list

A connection string passed on the command line is readable by any process on the machine. Put the password in ~/.pgpass or PGPASSWORD, or in MIKROSCOPE_POSTGRES_DSN in an environment file, rather than in the flag.

The docker end-to-end suite runs both sinks at once, the script into one database and the connection into another, and checks that the two hold the same rows (Test suites).

Table	Key after time, host	Holds
mikroscope_cpu	cpu	user_ticks … steal_ticks, busy_ratio, dt_ns
mikroscope_softnet	cpu	processed, dropped, time_squeeze
mikroscope_irq	irq	name, count summed over CPUs
mikroscope_mem	—	levels: every /proc/meminfo field the agent reads, in _kb
mikroscope_load	—	levels: load averages, running, threads, procs_blocked
mikroscope_stat	—	deltas: ctxt, intr, forks, irq_total, irq_err, pgfault, pgmajfault
mikroscope_self	—	cpu_us delta; rss, cgroup_mem levels; cgroup events throttled, throttled_us, oom_kill, NULL without cgroup2; resets, kmsg_dropped, seq, wake_ns, read_ns
mikroscope_buddy	node, zone, block_order	free_blocks, one row per zone and order
mikroscope_mtd	device	partition, ECC counters as read, thresholds NULL where unpublished
mikroscope_psi	—	stall microseconds; no row where the kernel has no PSI
mikroscope_sample	—	seq, dt_ns, mono_ns: one row per tick
mikroscope_softirq	kind, cpu	count, per vector and core
mikroscope_perf	counter, cpu	PMU count, enabled_ns, running_ns; no row without privileged=yes and a PMU
mikroscope_vm	—	/proc/vmstat deltas: faults, allocs and frees, reclaim scans and steals, stalls, oom_kill, swap
mikroscope_vm_level	—	/proc/vmstat levels: nr_free_pages, nr_dirty, nr_writeback, slab pages
mikroscope_cpufreq	cpu	khz, and max_khz where the board publishes one
mikroscope_irq_cpu	irq, cpu	name, count per core
mikroscope_thermal	zone	celsius, critical_celsius
mikroscope_slab	cache	active_objs, limit_objs (NULL for every cache but nf_conntrack)
mikroscope_disk	device	read and write deltas, io_s; inflight is a level
mikroscope_flash	device	wear deltas; bad_blocks and free_chunks are levels
mikroscope_event	kernel_seq	one kernel-log record: level, facility, time_usec, message, and port, kind
mikroscope_api_system	—	cpu_load, memory, free_hdd, uptime_s, version
mikroscope_api_core	cpu	RouterOS’s load, irq, disk percent
mikroscope_api_health	name	value
mikroscope_api_iface	interface	label, rates, and five loss columns that are NULL where the router returned no key
mikroscope_api_conntrack	—	entries, the last value repeated at the API cadence
mikroscope_api_ifinfo	interface	what each interface is: default_name, type, role, bridge, label, mtu
mikroscope_api_ifcounter	interface, counter	value, long form, under RouterOS’s own counter name
mikroscope_api_error	message	one per API command that failed
mikroscope_gap	seq_from, seq_to	the lost range, at the collector’s clock
mikroscope_trigger	id	cause, field, value, threshold, seq
mikroscope_derived	—	seq, mem_pressure, burst, suspect, per-packet values NULL where not computed
mikroscope_derived_iface	interface	the four byte deltas and the two fast-path shares
mikroscope_detection	rule, key	seq, value, threshold, message
mikroscope_sampler	—	ticks, slipped, captures_held, capture_bytes, capture_budget_bytes, capture_served_bytes
mikroscope_trigger_count	condition	fired
mikroscope_trigger_suppressed	condition, reason	count
mikroscope_capture_refused	reason	count
mikroscope_device, mikroscope_device_thermal, mikroscope_device_cpufreq, mikroscope_device_cadence	—, zone, cpu, source	the device-info stream

Columns never need quoting: the tick columns are user_ticks and friends because user is reserved, and block_order because order is. dt_ns rides on mikroscope_cpu and mikroscope_sample only, so a rate over any other delta table joins mikroscope_sample on (time, host) for the real interval rather than assuming the nominal period. The last four tables are read from the agent’s GET /sampler on the collector’s health cadence, not produced by a tick.

mikroscope_api_ifinfo holds one row per interface, written at collector start and on every --labels-every re-read (5 minutes by default), so a query joins it on interface to give any interface series a type, a role and the port’s comment. default_name is the factory name of a physical port and empty for a bridge, VLAN or tunnel; mtu is RouterOS’s actual-mtu, NULL where the router reports none. In mikroscope_event, port is the port the record names — its current RouterOS name where the API tier’s inventory supplies one, the board’s default name otherwise — and kind is what happened to it: link-up, link-down, the stp-* states, own-address (the bridge received a frame carrying its own MAC as source, the layer-2 loop signature) or other. Both are NULL for a record that names no port.

A source the deployment cannot read emits no row. A value that was not measured is NULL, never 0: a router can return no interface error keys at all (verified), and a 0 there would claim a measurement that was never made. Text is coerced where PostgreSQL would refuse it — a NUL byte is dropped, invalid UTF-8 becomes U+FFFD — and a NaN or infinite float becomes NULL. TIMESTAMPTZ resolves to 1 µs, so two samples closer than that would collide on the primary key; at 10 Hz they are 100 ms apart.

Compared with InfluxDB the SQL sink is short of exactly one thing: the kernel-log count table. It carries two InfluxDB does not: the kernel-log text in mikroscope_event, and the API tier’s errors.

A pipe into psql can block the collector

The SQL sink is synchronous, with no queue. With --sql - feeding psql, a psql that falls behind — each INSERT is its own transaction and commit — fills the pipe, and the next write blocks the collector’s pull loop instead of dropping (not measured). Write a file and apply it afterwards, or wrap the file in BEGIN/COMMIT by hand.

Sizes from the sink’s test fixture — two cores, one softnet queue, one interrupt, no privileged sources — not from a router:

• A kernel event renders to 1 375 B of SQL and an API event to 1 138 B, so 10 Hz plus the 1 Hz API tier is about 14 KiB/s of file after the fixture’s 5.6 KiB header.
• The same two events in line protocol are 716 B and 608 B, about 1.9× smaller, though part of that is content the SQL rows carry and line protocol did not.
• With the privileged sources present the kernel event grows to 2 749 B.
• The header for all forty-three tables the sink declares, rendered by the sink’s own header(), is 10 482 B, about 10.2 KiB — 13 966 B with the TimescaleDB hypertable statements.

Loki

export MIKROSCOPE_LOKI_URL=http://host:3100/loki/api/v1/push
mikroscope forward --loki "$MIKROSCOPE_LOKI_URL" --loki-tenant team-a

Loki receives the timeline’s events, not its samples. A sample is a measurement and belongs in a metrics store; 10 Hz of numbers in a log store is a slower, larger copy of it. What arrives is what happened once, at a known moment, posted as JSON to Loki’s push endpoint, /loki/api/v1/push. A bearer token comes from MIKROSCOPE_LOKI_TOKEN, and --loki-tenant (MIKROSCOPE_LOKI_TENANT) sets X-Scope-OrgID, the header a multi-tenant Loki reads the tenant from.

Streams carry three labels and no more — host, source and level — because Loki indexes labels and cardinality is a cost:

source	level	One line per
kmsg	the record’s own: emerg … debug	kernel-log record, stamped at its tick
gap	warn	lost sequence range, stamped when the collector noticed it
api	err	API-tier command that failed
detection	warn	detection, at the sample that raised it
trigger	info	capture trigger, at the fire; the capture stays on the agent
device	info	device-info record: board, kernel, cores, privileged, cgroup, sources, hash

A kernel record’s line is the message followed by logfmt pairs — level, facility, prio, kseq, us (the kernel’s own microseconds since boot), seq (the sample), and, when the record names a port, iface (the kernel’s name for it), ros_iface (RouterOS’s name for it), port_event (what happened: link-up, link-down, an stp-* state, own-address or other), label (the port’s comment, quoted) and role (its interface lists). ros_iface is the port’s current RouterOS name where the API tier’s inventory supplies one and the board’s default name without it; label and role come from that inventory too, so a run with no API tier has neither. The port rides in the line, not in a label, because a stream per port multiplies the stream count for a field LogQL extracts on demand:

{source="kmsg"} | logfmt | ros_iface="ether2"
{source="kmsg"} | logfmt | port_event="own-address"

Each record is stamped with its tick’s wall clock, never with its own since-boot stamp, which would date it 1970 plus uptime and be refused by Loki. Records of one tick are 1 ns apart, counting up, because a Loki stream is ordered by timestamp alone and a “blocking state” then “learning state” pair can arrive inside one 100 ms tick at the same level: their order is the signal. The offset is at most 63 ns, bounded by the agent’s cap of 64 records per tick.

One push per second, 64 KiB × --queue-seconds of budget. A rendered line is about 150 B with its JSON envelope, so at the 1.49 /s of kernel-log records a live layer-2 loop produced, one second of budget holds hours of that traffic. Delivery during a kernel-log storm is not measured.

OTLP

mikroscope forward --otlp http://collector:4318/v1/metrics

--otlp posts OpenTelemetry metrics to an OTLP/HTTP receiver in the JSON encoding, one request per second. A bearer token comes from MIKROSCOPE_OTLP_TOKEN. JSON rather than protobuf is a dependency decision: protobuf would add a code generator and a runtime. The OTLP specification says a server SHOULD accept JSON-encoded payloads on the same port as binary protobuf — a recommendation, not a requirement, so a receiver that takes protobuf only is within the letter of it. The OpenTelemetry Collector takes JSON (tested). The resource carries host.name (the host tag) and service.name=mikroscope; the scope carries the collector’s version.

The mapping is the reason this sink is cheap to consume. mikroscope ships raw deltas, and OTLP has an exact home for them: every counter is a Sum with AGGREGATION_TEMPORALITY_DELTA and isMonotonic=true. For the kernel-sample counters startTimeUnixNano = the sample’s wall clock minus its real interval, and timeUnixNano = its wall clock — so the receiver is told the interval each delta covers instead of guessing the nominal one. The other sums do not carry an interval: mikroscope.api.errors, mikroscope.collector.gaps and mikroscope.collector.gap.samples have no start time, and mikroscope.trigger.fired and mikroscope.detection have a start time equal to their time. Every level is a Gauge. Nothing from the kernel tier is pre-divided except mikroscope.cpu.busy_ratio; the mikroscope.derived.* gauges are the collector’s derive stage.

Kind	Metric	Attributes
Sum	mikroscope.cpu.ticks	cpu, mode
Sum	mikroscope.context_switches, mikroscope.interrupts, mikroscope.forks, mikroscope.self.cpu.time	—
Sum	mikroscope.self.throttled_periods, mikroscope.self.throttled_time, mikroscope.self.oom_kills (with cgroup2 only)	—
Sum	mikroscope.irq.count; mikroscope.irq.total, mikroscope.irq.errors	irq, name; —
Sum	mikroscope.softnet; mikroscope.sched	cpu, kind
Sum	mikroscope.softirq; mikroscope.vm.events	kind
Sum	mikroscope.psi.stalled	resource, scope
Sum	mikroscope.flash, mikroscope.disk; mikroscope.disk.io_time (ms)	device, kind; device
Sum	mikroscope.api.errors; mikroscope.collector.gaps, mikroscope.collector.gap.samples	tier; —
Sum	mikroscope.trigger.fired; mikroscope.detection	cause; rule
Gauge	mikroscope.cpu.busy_ratio, mikroscope.cpu.frequency	cpu
Gauge	mikroscope.sample.dt, mikroscope.sample.seq, mikroscope.threads, mikroscope.procs_blocked	—
Gauge	mikroscope.memory (KiB), mikroscope.self.memory, mikroscope.vm.pages	kind
Gauge	mikroscope.load	window
Gauge	mikroscope.thermal.temperature	zone
Gauge	mikroscope.slab.objects, mikroscope.slab.limit	cache
Gauge	mikroscope.memory.buddy_free_blocks	node, zone, order
Gauge	mikroscope.mtd.ecc; mikroscope.mtd.bitflip_threshold, mikroscope.mtd.ecc_strength	device, partition, kind; device, partition
Gauge	mikroscope.flash.blocks; mikroscope.disk.io_in_progress	device, kind; device
Gauge	mikroscope.api.cpu_load, mikroscope.api.uptime; mikroscope.api.memory	—; kind
Gauge	mikroscope.api.core; mikroscope.api.health	cpu, kind; name
Gauge	mikroscope.api.interface	interface, kind, and label, type, role where the inventory has them
Gauge	mikroscope.api.interface.counter; mikroscope.api.conntrack.entries	interface, counter, and label, type, role where the inventory has them; —
Gauge	mikroscope.derived.memory_pressure, mikroscope.derived.cycles_per_packet, ….instructions_per_packet, ….cache_misses_per_packet, ….packets_per_irq	—
Gauge	mikroscope.derived.fastpath_share	interface, direction
Gauge	mikroscope.device.cores; mikroscope.device.thermal.critical, mikroscope.device.thermal.polling (s)	board, kernel, hash; zone
Gauge	mikroscope.device.cpu.frequency_max, mikroscope.device.cpu.frequency_min; mikroscope.device.source_cadence	cpu; source, reason
Gauge	mikroscope.sampler.{ticks,slipped,trigger_fired,trigger_suppressed}; mikroscope.capture.{held,bytes,budget_bytes,served_bytes,refused}	condition, reason

The reclaim and swap members of mikroscope.vm.events are omitted at zero: for a delta Sum an absent point and a zero point mean the same. The per-port counters are gauges of a running total, not Sums, because the sink has no start time for a counter RouterOS keeps since boot. A gauge whose value is NaN or infinite is dropped. Kernel-log records are not emitted: their home is OTLP logs on /v1/logs, which this sink does not implement.

An OTLP partial success — a 2xx whose body rejects some data points — counts as written and is logged, not retried, which is what the OTLP specification requires of a client. The rejection is deterministic (a Prometheus OTLP receiver refusing a point older than its window is the common case), so the same batch would be rejected identically.

Sizes from a two-core test fixture, not from a router: one kernel sample renders to 8 132 B of OTLP JSON against 716 B of line protocol, about 11×, the price of repeating attribute keys and quoting every 64-bit integer. One second of 10 Hz samples plus one API sample is 68 868 B, so the 64 KiB-per-second budget holds about one second of backlog per second and the default 60 s about 57 batches. A board with more cores and a real interrupt top-K renders larger (not measured).

Graphite

mikroscope forward --graphite carbon:2003 --graphite-prefix mikroscope

--graphite (MIKROSCOPE_GRAPHITE_ADDR) writes carbon’s plaintext protocol — path value timestamp, one line per value — over one persistent TCP connection. Graphite has no labels, so every dimension is a path node under <prefix>.<host>.; --graphite-prefix defaults to mikroscope. In a node only ASCII letters, digits, _, - and : survive, every other byte becomes _, and an empty value becomes none, so a path’s depth never changes.

Paths	From
sample.{seq,dt_ns}, stat.{ctxt,intr,forks,procs_blocked,irq_total,irq_err}	the sample
cpu.<n>.{user,nice,system,idle,iowait,irq,softirq,steal,busy_ratio,freq_khz}	/proc/stat, cpufreq
softnet.<n>.{processed,dropped,time_squeeze}, irq.<id>.<name>.count, softirq.<kind>.count	softnet, interrupts, softirqs
mem.<field>_kb for every /proc/meminfo level the agent reads (total, free, available, cached, slab, sunreclaim, sreclaimable, dirty, writeback, anon, buffers, active, inactive, shmem, mapped, kernel_stack, page_tables, committed, commit_limit), load.{load1,load5,load15,running,threads}	meminfo, loadavg
vm.{pgfault,pgmajfault,pgscan_kswapd,pgscan_direct,pgsteal_kswapd,pgsteal_direct,allocstall,oom_kill}, vmg.{nr_free_pages,nr_dirty,nr_writeback}	vmstat
self.{cpu_us,rss_bytes,cgroup_mem,throttled,throttled_us,oom_kill}	the agent’s own cost
psi.*_us, sched.<n>.{run_ns,wait_ns}	only where the kernel has them
thermal.<index>.celsius, slab.<cache>.{active_objs,limit_objs}, buddy.<node>.<zone>.order_<n>	thermal, slab, buddyinfo
mtd.<dev>.<counter>, flash.<dev>.<counter>, disk.<dev>.{reads,read_sectors,writes,write_sectors,io_s,inflight}	flash and block devices
api.system.<field>, api.core.<n>.{load,irq,disk}, api.health.<name>, api.conntrack.entries	the API tier
api.iface.<if>.{rx_bps,tx_bps,rx_pps,tx_pps,<loss>,fp_rx_share,fp_tx_share}, api.ifcounter.<if>.<counter>	the API tier and the derive stage
derived.{mem_pressure,cycles_per_packet,instructions_per_packet,cache_misses_per_packet,packets_per_irq}	the derive stage
trigger.<cause>, detection.<rule> — the value 1 at each event	triggers and detections
device.{cores,conntrack_max,cgroup_mem_max}, device.thermal.<zone>.*, device.cpufreq.<n>.*, device.cadence.<source>.hz	the device-info stream
sampler.{ticks,slipped,captures_held,capture_bytes,capture_budget_bytes,capture_served_bytes}, sampler.trigger_fired.<condition>, sampler.trigger_suppressed.<condition>.<reason>, sampler.capture_refused.<reason>	the agent’s own counters, on the health cadence
collector.gap.{samples,from,to}	gaps, at the collector’s clock

What the protocol cannot promise, and what each limit changes on a Graphite panel:

• Whole-second timestamps. Whisper’s finest retention is one second, so at 10 Hz nine of every ten samples land on a slot that already holds a value and carbon keeps the last one written. That is a valid reading for a level (mem, load, thermal, freq_khz, slab, vmg) and an understatement for a delta: a sum over cpu.0.user sees about a tenth of the ticks the agent shipped. Summing each second’s deltas in the sink would fix the delta paths and not the level ones; that trade has not been made. A consumer that needs every tick has the InfluxDB or file sink.
• No reply. written counts batches handed to the socket, not points carbon stored. A write into a socket carbon has already closed succeeds once and fails on the next, so the sink retries once on a fresh connection; the batch that went into the closed socket is lost, one per carbon restart.
• Thermal zones by index, not by name, because a zone’s type string is not unique; the name is kept in the file and InfluxDB sinks.
• Less detail: per-CPU interrupt counts are dropped, softirqs are summed over CPUs, cpu.total is not emitted (sumSeries over cpu.*.user gives it), pgalloc, pgfree and the swap counters are omitted, and kernel-log records are dropped because Graphite stores numbers only. Board, kernel and governor strings have no Graphite form.

The byte budget is 256 KiB per queued second, four times the others: a four-core tick with every source present renders to 6 411 B in 131 lines in the sink’s test fixture, not on a router, so 10 Hz is about 63 KiB/s. Above 10 Hz it is not measured.

The Graphite dashboard reads these paths. Grafana queries Graphite’s web API, not the carbon port the sink writes to, so --grafana needs that address in --grafana-datasource-url, or a Graphite datasource you already have in --grafana-datasource-uid (Graphite datasource).

Elasticsearch and OpenSearch

export MIKROSCOPE_ELASTIC_AUTH=elastic:…   # or an API key
mikroscope forward --elastic http://opensearch:9200 --elastic-index 'mikroscope-%Y.%m.%d'

--elastic (MIKROSCOPE_ELASTIC_URL) writes through the bulk API both products share: Elasticsearch’s and OpenSearch’s take the same newline-delimited JSON. /_bulk is appended to a cluster root, keeping any query string. MIKROSCOPE_ELASTIC_AUTH holding user:password is sent as basic auth, anything else as Authorization: ApiKey; credentials embedded in the URL are redacted from the sink’s printed name.

One document per event, with kind telling them apart:

• kernel — one agent tick: per-CPU ticks with busy and busy_ratio, stat, mem, load, vm, self, each optional source only when it was read (no key, never a zero), and the derive stage’s values under derived;
• event — one kernel-log record: priority, level, facility, seq, time_usec, message, and, when the record names a port, iface, ros_iface, port_event (what happened to the port), label and role;
• api — one API read: system, cores, health, ifaces, iface_counters, the last conntrack count, the fast-path shares under fastpath, and inventory on the rounds that read what each interface is. Each ifaces entry carries label, type, role and bridge, and each iface_counters entry comment, type, role and bridge, where the inventory has them. The API tier’s per-command errors are not written;
• gap (from, to, lost), device, detection, trigger and sampler.

Every document carries @timestamp from the agent’s clock (the collector’s for gaps and device records) and host. The index name expands %Y, %m and %d — only those — against that timestamp, so a batch queued across midnight lands in the day it was sampled, and is lower-cased because the cluster refuses an upper-case index name for the whole request.

The action is index with an _id built from kind, host, the document’s timestamp in nanoseconds and, where one exists, the sequence number that tells documents at the same instant apart, so a batch the cluster applied but whose response was lost is re-sent without duplicating anything. The timestamp is part of the identity because the agent’s sequence starts again from 1 on every launch: without it a restarted agent’s samples would overwrite the day’s earlier ones, each with a 201.

A bulk request answers 200 even when every item in it was refused. The sink reads the per-item verdicts, adds one to dropped for each refused document, and logs the first reason once a minute — separately from delivery failures, because a mapping conflict and an unreachable cluster call for different actions. Batches close once a second or at 1 MiB, whichever comes first.

Sizes from the sink’s two-core fixture, not from a router: 1 077 B of NDJSON for a kernel sample with no optional source, 1 952 B across two documents with the optional sources and one kernel-log record. A four-core sample is not rendered in this format.

--grafana builds a Grafana Elasticsearch datasource (type elasticsearch) from --elastic: @timestamp as its time field, an index pattern that covers every index --elastic-index makes, and MIKROSCOPE_ELASTIC_AUTH sent the way the sink sends it (Elasticsearch datasource). Against OpenSearch that datasource is not tried.

Telegraf

mikroscope forward --telegraf http://host:8186/telegraf
mikroscope forward --telegraf tcp://host:8094

--telegraf (MIKROSCOPE_TELEGRAF_URL) ships the same line-protocol records as the InfluxDB sink — it calls that encoder rather than copying it — so Telegraf’s own outputs can fan them out to destinations this repository has no sink for. Telegraf passes a timestamp through unchanged.

The endpoint’s scheme picks the transport:

• http:// or https:// — post to an http_listener_v2 or influxdb_v2_listener input. A bare host:port is read as HTTP, and an HTTP endpoint with no path is given /telegraf, the path http_listener_v2’s sample configuration lists: a listener answers 404 on / and the body does not say why. MIKROSCOPE_TELEGRAF_TOKEN holding user:password is sent as basic auth, anything else as Authorization: Token ….
• tcp:// — newline-delimited records to a socket_listener, a fresh connection per batch.
• udp:// — datagrams of at most 1 432 bytes, cut at record boundaries so no datagram carries half a line. There is no acknowledgement: written counts batches the local kernel accepted, a datagram lost in flight is invisible, and a retry after a mid-batch failure can deliver some records twice. Prefer http:// or tcp:// for anything that matters.

The budget is the InfluxDB sink’s 64 KiB per queued second: at about 1.2 KiB per 10 Hz sample (measured), about 5 minutes of backlog at the default 60 s. The shared encoder writes the /system/health readings in Go map order, so records within a batch are not ordered stably — 12 renders of an 8-name map gave 7 orders. Every record carries its own timestamp, so nothing is lost or mis-timed.

Every sink on this page is tested against a local receiver that asserts the bytes its protocol accepts. Test suites lists the real stores each sink also runs against, and Not tested what has not been run against a store or fed from a router.

See also

• Run the collector: the queue, backoff and counters these sinks share.
• InfluxDB 3: the measurements --stdout lp and --telegraf carry.
• Detection rules: what the detection lines, rows and documents say.
• Environment variables: every MIKROSCOPE_* a sink reads.