Graphite

The Graphite sink sends ghchronicle’s numeric fields to Graphite over TCP, each at the date the thing happened.

sinks:
  graphite:
    addr: graphite:2003
    prefix: github
    batch: 1000

The plaintext protocol over TCP: path value timestamp, one line per numeric field. String fields are skipped, because Graphite has no way to hold one.

Graphite keeps a point at the time it was given, so the raw dated points go out as they are and the traffic window lands on its own days. Writing the same point twice fills the same slot of the same whisper file, which is exactly what a rewrite of the window wants.

The connection is reopened on a failed write, once, before the write is reported as failed.

The path is the dashboard’s contract

<prefix>.<measurement without gh_>.<tag values, in tag key order>.<field>

• Every tag is one node, holding its value. The nodes are ordered by the tag’s key, alphabetically, never by the order a collector set them.
• An empty tag value is written as none, so a measurement’s depth never changes from one point to the next and github.repo.*.*.*.*.*.*.*.*.*.stars keeps matching.
• A node keeps ASCII letters, digits, _, - and :. Everything else becomes _: the dot, the space, the comma, and the slash in owner/repo. A dot would split the node and a slash would nest a directory.

So a gh_repo point tagged archived=false default_branch=main fork=false full_name=acme/edge-cache language=Go license=MIT owner=acme repo=edge-cache visibility=public with a stars field becomes:

github.repo.false.main.false.acme_edge-cache.Go.MIT.acme.edge-cache.public.stars 37 1757280000

The dashboard names a series or a row by the node it is grouped under, so it shows a name as the path holds it: the (ghost) the collector writes for a deleted account reads _ghost_, the Actions Linux SKU Actions_Linux, the Q&A category Q_A, another/project another_project, and a cache key holding go-1.27.1 holds go-1_27_1. Nothing turns them back on the way out, since an underscore in a node may have been one in the name as well, so the Graphite dashboard leaves them as they are, and the other four stores show each name as the collector wrote it.

A field that shares its name with a tag is skipped, the same rule the line protocol applies: the tag wins, because it is the one that can be grouped by.

A new tag changes the path depth

Adding a tag to a collector inserts a node into every path of that measurement, so every existing dashboard target for it stops matching. This is the cost of a store with no schema, and it is why TAGS in the dashboard specification mirrors the tag set of each measurement.

What it cannot answer

Graphite keeps the dated points but has no rows. A series is a path and a number, so:

• A table that needs several fields of one row cannot be built. The shipped dashboard reduces each series to the one or two numbers a row of it can carry, and its panel description names those and every column it drops.
• A title or any other string is not a metric there at all, and the sink drops it; the panels that would show one say so. A boolean is kept as 1 or 0 like any number.
• A panel cannot join two measurements, since each lives under its own path. Work elsewhere has no Stars column, and the community profile keeps the API’s issue template flag rather than the count of templates, which lives under another measurement.
• A target cannot ask about one window while it reads another. The Overview and Every repository, ever count an archived repository the default filter sets aside from its points of the last seven days, where the SQL stores ask whether the collector still writes it, so a range that ended more than a week ago leaves those repositories out.

Everything time-shaped works normally, which is most of the dashboard.

The dashboard

dashboards/ghchronicle-graphite.json has the same 154 panels as the InfluxDB one, written against these paths with the default prefix. It needs Graphite 1.1 or later for the functions it uses, and any panel a series cannot carry says so in its description.

Change prefix and the dashboard targets have to change with it, since the prefix is the first node of every path.

Every chart over time adds its points up into buckets the width the range calls for, the width the SQL dashboards bin by: the range over a hundred, rounded the way Grafana rounds an interval and never under the chart’s floor of a day, an hour or five minutes. The dashboard computes them in three hidden variables, bucket_1d, bucket_1h and bucket_5m, and each chart asks for 5,000 points, more than it has buckets, so graphite-web hands the buckets back as they are. Asked for fewer points than a series holds, graphite-web fits the series into bands and moves each point one step later as it does, and before 2.6.2 that put the newest hour of a chart past its right edge in the last hour before each band boundary. Measured against graphiteapp/graphite-statsd:1.1.10-5 with one hour a step at 15:35 UTC, the star count a sweep had written in that hour came back stamped 16:00 at a hundred points, after the end of the range, and a chart whose one point was that one read “Data outside time range”; summed into the day’s bucket it came back at 00:00, and the artifact storage in its six-hour bucket at 12:00. A bucket follows the dashboard’s range, so a chart pinned to its own range, as two of the Code panels are, sums into days whatever the page is set to.

Two charts sum into a fixed day instead, because their rows are already a day or a week apart and each belongs at its own date: the contribution calendar, a row a day, and the weekly commits, a row a week stamped at the Sunday GitHub starts the week on, with the days between the weeks left out. Seven days is not a bucket there: graphite-web counts one from the epoch, a Thursday, and every weekly bar stood three days early, the first week of a thirty-day range before the range began.

Every other panel asks for 5,000 points as well. A table, a bar chart or a stat is not summarized: it reduces the points of the whole range to a number, and the fitting drops the first of them, one fewer than the steps it moves the first band’s start by. Grafana asks a panel that names no number for as many points as it is wide, so before 2.6.2 a panel narrower than a range’s points lost the first hour or hours of the range, depending on where the range began. Measured on the same Graphite, a point at 20:00 UTC read from 19:05 over thirty days summed to nothing at 500 points and to 1 at 5,000, and “Languages starred” drew a star given at that hour as 0 in a window 640 pixels wide. Such a panel holds a point per storage step, so 5,000 covers the 2,880 hours of the hundred and twenty days the containerised suite’s schema keeps at an hour a step and the 4,380 days of the twelve years it keeps at a day. A retention that keeps more than 5,000 steps of a range, an hour a step for longer than a hundred and twenty days or anything finer, is fitted into bands again.

A chart or a bar chart that names its busiest series and folds the rest into one called other does the same here as in the SQL dashboards: the busiest over the whole range are named, and other is every series less those, point by point, drawn only where something is left over.

What a migration does here

Graphite cannot be cleared from here. Nothing on carbon’s ingest port deletes a path, and graphite-web deletes none that has no tags: measured against graphite-statsd 1.1.10-5, /metrics/delete answers 404 and /tags/delSeries answers true and leaves the series where it was. What a release left in another shape is decided by the state file’s record of the release that first wrote the store, and applying the change is -migrate -yes printing, once, the commands for the Graphite host, and recording the change as applied, since whether they were run is something only whoever runs them knows:

find <storage>/whisper/github/discussion_comment -mindepth 11 -name '*.wsp' -delete
find <storage>/whisper/github/discussion_comment -type d -empty -delete

<storage> is carbon’s storage directory, /opt/graphite/storage in the official image, and github is the prefix. A path is a node per tag and then the field, so the old shape of a measurement that lost a tag sits one level deeper than the new one, and -mindepth reaches that level and nothing above it. Measured in that image: the command took the two files of the old shape and left the new shape’s and every other measurement’s. The files are the host’s and the order does not matter, since the two shapes are different files. A start never applies this on its own.

Where to go next

• Choosing a store compares Graphite with the others, and holds the write ledger every one of them shares.
• The dashboards says which of the five is drawn against which store, and what a panel a store cannot answer becomes.