Choosing a store

Eleven sinks, and running more than one is the normal arrangement. Every one of them pushes: the tool is meant to run wherever it is convenient and reach its stores from there, not to be scraped. The Prometheus exporter is the one exception, and it exists because Prometheus insists.

The comparison

Store	Keeps	Good for	Config
InfluxDB	the dated history	“how fast were we merging in July”	url, token, org, bucket
PostgreSQL	the dated history, in a database it connects to	a Grafana user with a Postgres and no InfluxDB	dsn, batch
PostgreSQL, to a file	the same tables, as SQL you pipe into psql	a load that happens later, or elsewhere	dialect, path
Graphite	the dated history	a Graphite that is already there	addr, prefix
Elasticsearch	the dated history, as documents	search across everything collected	url, prefix, api_key
Prometheus	the current value	alerting, and a number on a wall	listen, path
OpenTelemetry	either, depending on the backend	an existing collector pipeline	endpoint, raw
Loki	the events, as log lines	“what happened, in order”	url, labels, max_age
Telegraf	whatever its outputs keep	reaching anything Telegraf can reach	url
File and stdout	line protocol or JSON	a shipper you already run, and a buffer	path, format

There is no sink for a specific hosted vendor, and that is deliberate. A managed backend is reached through one of the two sinks that exist to route onward: Telegraf, whose own outputs cover Datadog, New Relic, Wavefront, Azure Monitor and a hundred more, or OpenTelemetry, which most of them now accept directly. A sink per vendor is a key to rotate, an API to track and a test that needs a paid account, for a hop those two already make.

The thing that decides everything

A point carries the date the thing happened. A star is dated when it was given, a workflow run when it finished, a traffic day at that day’s own date.

InfluxDB keys a point by measurement, tag set and timestamp, so writing the same fourteen-day traffic window every six hours converges on the right answer rather than accumulating copies. That is what makes the whole backfill design work, and it is why InfluxDB is the sink that keeps history.

Prometheus cannot do that. It stamps a sample at scrape time and rejects anything meaningfully older: measured against Prometheus 3.14 with the OTLP receiver enabled and a thirty-minute out-of-order window, a sample dated two days back comes back as HTTP 400. So the exporter reduces the per-item rows to current values before serving them.

The full argument, and what the reduction does to each measurement, is on dating a point.

Only what changed is written

A sweep offers the same history every time: the fourteen day traffic window, every open pull request, the contribution calendar. Writing it all again is harmless to what the store holds, since a row is keyed by series and timestamp and simply overwrites, and it is how a collector that was down for a day repairs itself.

It is not harmless to the store’s files. InfluxDB 3 Core writes one Parquet file per partition per write request and never compacts them, and it refuses any query that would open more than its file limit. Measured before the fix below and with the limit then at ten thousand: gh_notification held barely more than one row per Parquet file, and a query over fourteen days came back with “Query would scan 10000 Parquet files, exceeding the file limit”.

Asking for a coarser interval does not help, because the limit counts the files the planner opens, before any aggregation. So the tool keeps a small ledger of what it has already written and sends only the points whose values have moved:

sinks:
  dedupe_file: /var/lib/ghchronicle/state-written.bin # default: beside state_file
  dedupe_horizon: 720h # forget a point nothing offers any more
  influxdb:
    dedupe: true # the default, here and for telegraf, graphite, sql and elasticsearch

The ledger stores two 64 bit hashes and a day per point, so a large account costs a few megabytes. It is keyed by sink, so a store that was unreachable still receives everything on its next write. Losing it, or setting dedupe_file: off, costs one sweep of rewriting and nothing else, which is exactly what a store that has been wiped and needs filling again wants. The jobs of the workflow runs the cache file remembers come with it: a start that finds no ledger lists them again. It is the second file worth putting on a persistent path, next to the state file.

A run that ends when its sweep does never opens the ledger at all. -once, -backfill and a card render write once and exit, so there is nothing to save and nothing to prune, and each of them offers the whole history again. That is what a backfill is for; it is also why a scheduled -once job, which is the shape the Action runs in, writes every point every time. Where that matters, run the loop instead.

The sweep log says what this saved:

level=INFO msg=written sink=influxdb family=events points=0 unchanged=300

If the files have already accumulated, the writer fix stops the growth but does not remove them: raise --query-file-limit on the server, rewrite the affected tables, or move to InfluxDB 3 Enterprise, which compacts on its own and is free for home use.

What the written line counts

points is what that sink took, not what it was handed. Every sink here drops something of its own inside the write: InfluxDB skips the measurements exclude names, Loki keeps only the measurements it has an event rendering for and drops the rest, the SQL, Graphite, Telegraf, file and stdout sinks skip a point that renders no row or line, and the two exporters skip a measurement they have no rule for. A third key appears when that happened:

level=INFO msg=written sink=loki family=repos points=0 unchanged=0 filtered=42

filtered is what the sink did not write, and each sweep ends with one line per sink carrying the total since the run started, the way the ledger’s own total beside it reads. Lines a store refused to parse are not counted as written either; they get a warning line of their own.

Until this was counted the line said what the runner had handed over, so a sweep read points=440 for a measurement InfluxDB excludes by default and has never held a row of, and reading Loki’s lines told you nothing about what Loki stored.

Start here

You want the historyInfluxDB is the reference implementation and the dashboard is built against it. PostgreSQL, Graphite and Elasticsearch keep the same facts in their own shapes.

You want alertsPrometheus serves the current value of everything that has one. Run it alongside a history store rather than instead of one.

You want to read what happenedLoki turns twenty-two of the measurements into log lines that read as sentences and carry every tag after them in logfmt.

You already have a pipelineTelegraf and OpenTelemetry hand the points to something that already knows where they should go.

Running several

Normal, and cheap: the collection happens once and the points are handed to every configured sink. The usual arrangement is one history store plus one of the current-value ones.

sinks:
  influxdb:
    url: http://localhost:8181
    token: ${INFLUX_TOKEN}
    org: default
    bucket: github
  prometheus:
    listen: 127.0.0.1:9605
    path: /metrics

One combination to avoid

sinks.sql.path: "-" and sinks.stdout: true both write to standard output, and interleaving SQL statements with line protocol produces a stream that neither psql nor Telegraf can read. Choose one.

What every sink does with a failure

A sink that fails is logged and the sweep continues; a database being down does not stop collection, and with the file sink configured the data is still on disk when it comes back. Two failures are reported specially rather than as errors, because they are partial successes:

• rejected lines, where everything parseable was written and the refused lines are logged individually with the store’s own reason.
• dropped entries, which is Loki’s age horizon leaving out what its out-of-order window would have refused, rather than losing the whole push.