# The test layers Three layers, only one of them free: the contract test of the bytes, the containerised stores that accept them, and the dashboards' own queries. Source: https://jmrplens.github.io/ghchronicle/reference/testing/ The tests come in three layers, and they are worth telling apart because only the first one is free. Two of them start nine containers, and anyone deciding whether to wait for that deserves to know what it buys. | Layer | Command | Docker | Time | What it proves | | ------------------ | ---------------------- | ------ | -------------- | -------------------------------------------------------------------- | | L1, the contract | `make test` | no | about 5 s | the exact bytes each sink puts on the wire | | L2, the stores | `make test-e2e-docker` | yes | 53 s warm | a real store accepts those bytes, and keeps the date of the event | | L3, the dashboards | the same target | yes | included above | the five dashboards' own queries answer against what the sinks wrote | L1 runs on every push. L2 and L3 are one suite behind the `dockere2e` build tag, so `go test ./...` never starts a container; in CI they run weekly, on demand, and as a release gate. ## L1: the bytes on the wire `test/e2e` builds the real binary, runs it against a fake GitHub whose fixtures live in `test/e2e/testdata` and whose route table is `test/e2e/fakegh`, and points every sink at an `httptest` capture server. Then it asserts the bytes: the line protocol, the `_bulk` envelope, the SQL statements, the Graphite path, the OTLP payload. The fake also prices its answers the way api.github.com does: one ETag per REST fixture and none on a GraphQL answer, a 304 charged nothing for a request that presents it, one for every other answer on the API, nothing for the object storage a job log redirects to, and the budget block on every GraphQL answer. That is what lets `TestTheSecondSweepIsPricedByTheCache` run two sweeps in a single process and hold that every URL answered 200 the first time came back 304 the second, that the second sweep charged fewer than half the core requests of the first, and that `own_cost` on the `gh_rate_limit` row is the number of queries the process made rather than zero. That is the right test for a format, and it is fast enough to run while a sink is being changed. What it cannot catch is anything the receiver has an opinion about. A capture server answers 204 to everything. It has no column types, no mapping, no query planner and no schema. ## L2: the stores themselves `test/e2e/docker` starts the real stores in containers, runs one sweep from the same fake GitHub into all of them, then reads each store back and asserts the value, the tags and above all the timestamp. The dating rule is the product of this tool: a star is stamped when it was given, a workflow run when it finished, a traffic day at that day's own date. No capture test can prove a store kept the date of the event rather than the date of the sweep, because storing it is the store's job. These are the defects this layer exists for, each of them real: **InfluxDB fixes a column's type on first sight.** InfluxDB 3 decides that a column is a tag or a field the first time it sees it and refuses every later write that disagrees: `400 invalid column type for column 'owner', expected iox::column_type::tag`. A capture server answers 204 and notices nothing. This has already cost a database wipe, and reproducing it was the first thing the containerised stack was used for. **Elasticsearch's dynamic mapping decides whether the dashboards can aggregate.** One panel pulls a url through a `top_metrics`, which normally needs a keyword field rather than a text one. Two separate audits recorded that as unverifiable for want of a real Elasticsearch. This layer answers it, by indexing through `_bulk` and reading back the mapping the cluster built for itself. **PostgreSQL has to accept the DDL.** The SQL sink emits statements rather than speaking the wire protocol, so until now nothing ever had them parsed. The suite pipes them through `psql`, inserts, and plans the panels' queries against the schema the sink created rather than against one transcribed from InfluxDB. **Graphite paths have to have the depth the dashboards index.** The dashboards address path nodes by position, and the agreement between those positions and what the sink writes was kept by a hand-maintained table that nothing checked. Here the sink writes to carbon and the render API is asked for the path back. ## L3: the dashboards' own queries With the stores loaded, the five generated dashboards are run through Grafana's `/api/ds/query`, which is the path `cmd/check_dashboards` takes against a live Grafana. Every datasource is provisioned at boot with a fixed uid and the harness mints a service account token, so a panel query goes through Grafana exactly as it would for a person looking at the dashboard. That is what turns three manual checkers into something CI runs, and it is what settles the Elasticsearch question above: a panel that cannot aggregate returns no frame. ## What none of them catch Every layer runs against the fake GitHub, so nothing here notices GitHub changing a payload, retiring an endpoint or throttling differently. That is what `ghchronicle -once` against a real token is for. They also prove nothing about a store the suite does not start. The answer covers InfluxDB 3 Core, PostgreSQL 18, Elasticsearch 9, Graphite 1.1, Prometheus 3, Loki 3, the OpenTelemetry collector and Telegraf, at the pinned versions. OpenSearch, TimescaleDB and anything behind the Telegraf or OTLP hop are still an inference from the format. There are five more things, and none of them is a layer. Each is switched on by an environment variable and skipped when it is absent, so an ordinary `go test ./...` stays offline. **The store you actually run.** `test/live` pushes a handful of points at a Loki or an OpenTelemetry collector named in `GHC_LIVE_LOKI` or `GHC_LIVE_OTLP`. It answers the one question containers cannot: whether your instance accepts them. ```sh GHC_LIVE_LOKI=http://localhost:3100 go test ./test/live/ ``` **The real API, end to end.** `GHC_E2E_LIVE=1` runs `TestLiveAPI` against GitHub itself rather than the fake, with a real `GITHUB_TOKEN`, sweeping the account named in `GHC_E2E_USER`. ```sh GHC_E2E_LIVE=1 GHC_E2E_USER=octocat GITHUB_TOKEN=ghp_... go test ./test/e2e/ -run TestLiveAPI ``` **What a sweep costs in cache.** `GHC_LIVE_CONFIG` points `TestLiveSweepCacheFootprint` at a configuration file and sweeps the account it names, reporting the entries and the bytes the conditional-request cache holds after each sweep. Those are the figures the 256 MB bound and the [cost of a sweep](/ghchronicle/api/cost/) rest on, and this is how to reproduce them for your own account. `GHC_LIVE_DUMP=1` adds the per-URL list to standard output. ```sh GHC_LIVE_CONFIG=config.yaml go test ./internal/ghapi/ -run TestLiveSweepCacheFootprint -v ``` **One repository, one family.** `cmd/probe` runs the collectors against a single repository and prints the line they would write, writing nothing anywhere. `GHC_DUMP=` prints every point of that family in full, which is the fastest way to see what a collector actually produces. ```sh go run ./cmd/probe owner/name GHC_DUMP=actions go run ./cmd/probe owner/name ``` **The pictures of the card.** `GHC_CARD_GALLERY` names an existing directory and `TestCardGallery` renders one card per layout into it, from the fake GitHub rather than from anybody's account. That is where the pictures on [the layouts page](/ghchronicle/card/layouts/) come from, and a layout that changes shape is one command away from a set that agrees with it. The account is the base fixtures with `test/e2e/testdata/gallery/` laid over them: a year of contributions, GitHub's whole fourteen days of traffic, five repositories to rank and one of them in six languages, which the smaller account every other suite asserts on cannot give a picture. A fixture named `~` there answers for that one repository, and any other repository borrows hello-world's. Each layout comes out of one sweep under `-card-theme both` as two files, `card-.svg` in the light palette and `card-_dark.svg` in the dark one, which is what the site's `ThemeImage` and the README's `` read. The two layouts that loop come out a second time under `-card-motion loop`, as `card--loop.svg` and its `_dark` twin. Only those two: on every other layout `loop` draws the same card as `once`, so a looping picture of one would be a second copy of the first under a name that promises something else. Which layouts they are is the registry's `Loops`, and the gallery reads it rather than keeping its own list. ```sh mkdir -p /tmp/cards GHC_CARD_GALLERY=/tmp/cards go test ./test/e2e/ -run TestCardGallery ``` `make check-gallery` renders the gallery into a scratch directory and fails, naming every difference, if the committed set no longer matches it byte for byte; `make gallery` regenerates it in place. CI's "Generated artifacts" job runs the check on every pull request. ## Running the stack Docker with the compose plugin, and room for the images. Then: ```sh make test-e2e-docker ``` Up, run, down on every path including a failing assertion, and then a check that `docker ps` shows nothing of the project left. A suite that leaves nine containers behind on a failure is a suite nobody runs twice. > **Nothing of this listens outside loopback** > > Every port is published on `127.0.0.1`, on a free port Docker picks out of > 49200 to 49299, never a store's default: 9200, 8086, 5432, 2003, 9090, 3100, > 3000 and 4318 belong to whatever else is on the machine. The harness reads the > chosen ports back with `docker compose port`, and the project is named > `ghchronicle-e2e` in the compose file as well as on every command, so nothing > here can reach a container it did not start. Boot, measured cold with the images already pulled: Elasticsearch 29 s, Loki 21 s, Grafana 13 s, the Graphite render API 10 s, InfluxDB 8 s, PostgreSQL 6 s, the rest 6 s. The stack is ready in 30 s; the target end to end, teardown included, is 53 s. ## Debugging with the stack up The reason to fail an assertion is to go and look at the store, and a suite that tore the store down first cannot be looked at. So the two halves are separate targets, and the harness reuses a stack it finds already running and leaves it running. 1. Start the stores and leave them up. The command prints the port each service ended up on. ```sh make e2e-docker-up ``` 2. Run the suite, or one test of it, as many times as it takes. ```sh go test -count=1 -tags dockere2e -timeout 30m -v ./test/e2e/docker/ ``` `GHCHRONICLE_E2E_KEEP=1` also stops the test binary tearing down a stack it started itself, which is what you want when a single `-run` is failing. 3. Ask the store what it thinks, then tear it down. ```sh make e2e-docker-logs SERVICE=influxdb make e2e-docker-down ``` With the ports from step 1: ```sh # What InfluxDB thinks each column is. This is the answer to a 400 on write. curl -s "http://127.0.0.1:/api/v3/query_sql?db=ghchronicle" \ --data-urlencode "q=SELECT * FROM information_schema.columns WHERE table_name = 'gh_repo'" # The mapping Elasticsearch built for itself. curl -s "http://127.0.0.1:/ghchronicle-*/_mapping?pretty" # What the SQL sink actually created. psql "postgres://ghchronicle:ghchronicle@127.0.0.1:/ghchronicle" -c '\d+ gh_repo' # The Graphite path, node by node. curl -s "http://127.0.0.1:/metrics/find?query=github.repo.*" ``` Grafana is at the port it published, with `admin` and `admin`, and every datasource is already provisioned, so a panel query can be pasted into Explore and run by hand. ## Three things the stack had to be told Each of these silently produced a wrong answer before it was found, and each is in the compose file or its configuration with the measurement beside it: - **Carbon drops a point older than its longest archive without saying so.** A star dated 2020 vanished under a six year retention and the write was reported as accepted. The retention is `1d:12y` for that reason. - **Carbon's default `MAX_CREATES_PER_MINUTE` is 50**, fewer paths than one sweep creates, so most of a first sweep would be dropped. - **Loki answers a push with 204 and will not serve it until the chunk is flushed.** The test polls rather than asking once, and `chunk_idle_period` is 5 s. > **A firewalled machine needs one rule to scrape the exporter** > > The Prometheus exporter is the one sink that is scraped rather than pushed to, > so the scrape has to cross from the container back out to the host. Where the > default INPUT policy is deny, no container on any bridge reaches any host > port. One narrow rule is enough: allow TCP 49300-49399, the range > `scrapePortLow` and `scrapePortHigh` bound in `sweep_push_stores_test.go`, > from the container address space and from nothing else. Under ufw that is > > ```sh > ufw allow proto tcp from 172.16.0.0/12 to any port 49300:49399 > ``` > > Without it the harness reports `ErrExporterUnreachable` and the Prometheus > assertions skip with a reason instead of failing, which means they have never > run on that machine. A CI runner needs no rule and neither does an ordinary > workstation. A machine that adds the rule runs those assertions for the first time, which is when `promNeedsHistory` in `dashboards_test.go` starts to matter: the exporter is alive only for the length of the test, so every timeseries panel and every panel built on `increase()` is held to nothing and only the instant panels are asserted. ## In CI `.github/workflows/e2e.yml` runs `make test-e2e-docker` and has three ways in: manual dispatch with an optional ref, a weekly schedule on main, and `workflow_call`, so a release pipeline gates a tag on it with one line rather than a copy of the job that drifts from the original. It is not a required check on a pull request: nine containers and around 10 GB of image is too much for every push, and L1 is what covers every push. The weekly run is the point of the schedule. Nothing else in the repository ever starts a container, so without it the suite would run only when somebody remembered it, which is how a suite ends up broken for weeks with nobody knowing. The same suite also runs under the race detector, in `.github/workflows/race.yml`: weekly, at every release beside the E2E gate, and by hand. The harness builds the collector with `-race` and starts it with `GORACE=halt_on_error=1`, so a race inside the collector fails the test that started it, with the report. Locally it is `make test-e2e-docker-race`.