# ghchronicle Collects every metric GitHub exposes about an account and keeps it with the date it happened. Source: https://jmrplens.github.io/ghchronicle/ Every one of them carries the date the thing happened, which is what makes a question about last July still have an answer. ## What it collects - [91 measurements](/ghchronicle/collectors/measurements/) - [34 collectors](/ghchronicle/collectors/) - [10 outputs](/ghchronicle/sinks/) - [152 dashboard panels](/ghchronicle/dashboards/) ## What it is GitHub answers most questions about the present and almost none about the past. The traffic API serves fourteen days and forgets. The activity feed keeps the last three hundred events, whatever their dates. Read notifications disappear. Job logs are deleted after ninety days. **ghchronicle** sweeps those surfaces on a schedule and writes every observation as a dated point, into whichever database you already run. One Go binary, no dependencies beyond a YAML parser. ## Who it is for - Anyone who already runs a time series database and a Grafana - Maintainers who want traffic and stars kept past GitHub's window - Teams who want merge times, review load and CI cost as history rather than as a number that resets - Anyone who wants their own data out of GitHub before GitHub drops it ## What it is not - Not a hosted service: it runs on your machine, with your token - Not a replacement for GitHub Insights, which answers about now - Not able to recover what GitHub has already dropped: it starts from the day you run it - Not a badge generator, though it can draw one ## What it looks like in use Three steps, and the first sweep lands in your database. ### Install A single binary, or a release, or the container image. ```sh go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` ### Configure The smallest configuration that does something. Every ${VAR} is read from the environment, so this file holds no secrets. ```yaml github: token: ${GITHUB_TOKEN} targets: user: your-login sinks: influxdb: url: http://localhost:8181 bucket: github ``` ### Collect What lands in the database. Note the timestamps: the star is dated 2024, not today, because that is when it was given. ```text gh_star,repo=parser,user=someone starred=1i 1731590400000000000 gh_traffic,repo=parser,kind=views count=142i,uniques=61i 1757376000000000000 gh_pull_request,repo=parser,number=318,state=MERGED churn=214i,seconds_to_merge=5820i 1757462400000000000 ``` ## How it is shaped Four ideas, and the second is the one everything else follows from. 1. **Sweep**: Each family of metrics has its own cadence, because they move at very different speeds: workflow runs every quarter of an hour, the contribution calendar every twelve. 2. **Date**: A point carries the moment the thing happened, not the moment it was collected. That is what makes re-collection converge instead of accumulating copies. 3. **Push**: Nothing here is scraped. It pushes to InfluxDB, PostgreSQL, Graphite, Elasticsearch, Prometheus, OpenTelemetry, Loki, Telegraf, a file or anything Telegraf can reach, so it runs wherever it can reach them. 4. **Draw**: One dashboard specification, rendered once per store. The same panels whichever database you chose, and where a store cannot answer one honestly, the panel says so. ## Where to start - [Quickstart](/ghchronicle/start/quickstart/): From nothing to a first sweep. - [The token](/ghchronicle/start/token/): Which scopes, and why the automatic one is not enough. - [Dating a point](/ghchronicle/how/dating/): The idea the rest of the design follows from. - [Choosing a store](/ghchronicle/sinks/): What each of the 10 can and cannot answer. --- # What it is What ghchronicle collects, and the problem it exists to solve. Source: https://jmrplens.github.io/ghchronicle/start/ GitHub answers most questions about the present and almost none about the past. The traffic API serves fourteen days and forgets. The activity feed keeps the last three hundred events, whatever their dates. Read notifications disappear within weeks. Job logs are deleted after ninety days. The star list will tell you when each star was given, but only if you ask before the list gets long enough to be expensive to walk. None of it is archived anywhere unless you archive it. `ghchronicle` sweeps those surfaces on a schedule and writes every observation as a point stamped with the date the thing actually happened, so a year from now the question "how fast were we merging in July" still has an answer. ```sh ghchronicle -config config.yaml ``` It is one Go binary with no dependencies beyond a YAML parser, and it pushes to every store it supports, so it runs wherever it can reach them: a server, a container, a scheduled workflow. > **The card is not the point** > > It can also draw an SVG summary card for a profile README. That is a side > feature. The reason the project exists is the ingestion. ## The six words the rest of this uses | Word | What it means here | | --------------- | ------------------------------------------------------------------------------------------------------ | | **family** | One collector, named in the configuration: `actions`, `stars`, `issues`. There are 34 | | **group** | A named set of families, for switching a whole area on or off: `ci`, `security`, `audience`. There are 8 | | **measurement** | One kind of row in the store, named `gh_*`: `gh_star`, `gh_workflow_run`. There are 91 | | **point** | One row: a measurement, its tags, its fields and the date the thing happened | | **sweep** | One pass over the families that are due, which is what the process does on a loop | | **backfill** | A run with `-backfill`, which walks the history instead of the increment | `ghchronicle -groups` prints the groups with their families, and `ghchronicle -config config.yaml -list` prints the repositories a sweep would cover. ## Where to go next - [Quickstart](/ghchronicle/start/quickstart/): from nothing to a first sweep. - [The token](/ghchronicle/start/token/): which scope buys which family. - [Dating a point](/ghchronicle/how/dating/): the design idea everything else follows from. --- # Quickstart From nothing to a first sweep, and what that first sweep does that later ones do not. Source: https://jmrplens.github.io/ghchronicle/start/quickstart/ Six steps and a configuration file. The only decision worth thinking about before you start is which store keeps the history, and you can defer that by printing the points to the terminal first. ## Zero to a first sweep 1. **Install the binary.** - **Go** ```sh go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` - **Release** Take the archive for your platform from the [releases page](https://github.com/jmrplens/ghchronicle/releases) and put `ghchronicle` on your `PATH`. - **Container** ```sh docker pull ghcr.io/jmrplens/ghchronicle ``` 2. **Create a token** at `https://github.com/settings/tokens` and export it. ```sh export GITHUB_TOKEN=github_pat_... ``` A classic token with `repo`, `read:packages`, `read:user`, `read:org`, `security_events`, `read:public_key` and `read:gpg_key` sees everything this collects. The [token page](/ghchronicle/start/token/) explains which scope buys which family, and why the automatic `GITHUB_TOKEN` of an Action is not enough. 3. **Write the configuration.** Two decisions, and this is the whole file: ```yaml # config.yaml github: token: ${GITHUB_TOKEN} targets: user: your-login sinks: stdout: true # swap for influxdb once you have somewhere to put it ``` Every `${VAR}` is read from the environment at start-up, so the file itself holds no secrets and can be committed. Everything else has a default. The documented version, which comments every option there is, lives in the repository rather than in the install, so take it from there when you want to read the rest: ```sh curl -O https://raw.githubusercontent.com/jmrplens/ghchronicle/main/config.example.yaml ``` 4. **See what would be collected**, before spending any quota on it. ```sh ghchronicle -config config.yaml -list ``` That prints the repositories in scope. Forks and archived repositories are excluded by default. If something you expected is missing, this is the command that tells you. 5. **Run one sweep.** ```sh ghchronicle -config config.yaml -once ``` With `sinks.stdout: true` the points go to the terminal as line protocol instead of to a database, which is the cheapest way to see the shape of what you are about to store. 6. **Leave it running.** ```sh ghchronicle -config config.yaml ``` Each family then runs on its own cadence: workflow runs every fifteen minutes, the contribution calendar every twelve hours. ## What the first sweep does that later ones do not Three things happen once, and they are why the first run is the expensive one. - The whole stargazer list is walked, page by page, so every star carries the date it was given. After that, the newest hundred of every repository ride in one GraphQL query per ten of them. - A month of workflow runs, so a fresh install does not chart a CI history that begins fifteen minutes ago. After that, twice the cadence, and never less than two hours. - Every year's contribution calendar, if `every.history` is set, back to the day the account was created. After that, only the year in progress, rewritten at its cadence. Expect a few thousand points from a first sweep of twenty repositories, and a few hundred from each one after. None of that is the history. The first sweep is a wider increment, and a dashboard at ninety days or two years then begins on the day you installed the collector: measured after a day of sweeps, about a fifth of the pull requests the repositories report, commits from the last thirty days only, and jobs for a tenth of the workflow runs. Run `ghchronicle -config config.yaml -backfill` once, before the service or right after it; [the backfill page](/ghchronicle/how/backfill/) says what it reaches and what it costs. > **Keep the state file** > > `state_file` is what remembers where each family got to, and > [six things live in it](/ghchronicle/configuration/#state_file). Delete it and > the next sweep re-collects everything, which costs quota and nothing else for > five of the six; the sixth is the commit each dependency diff starts from, and > the changes in the gap are not collected again. ## Where to go next - [Ways to install](/ghchronicle/install/) is what turns the command above into something that keeps running: systemd, Docker or a scheduled Action. - [Dating a point](/ghchronicle/how/dating/) is the design idea everything else follows from. - [Choosing a store](/ghchronicle/sinks/) decides which questions you will be able to ask later. - [Cost of a sweep](/ghchronicle/api/cost/) is the measured price in API calls. --- # The token Which scopes buy which families, and why the automatic GITHUB_TOKEN is not enough. Source: https://jmrplens.github.io/ghchronicle/start/token/ Everything here is read access. The collector never writes to GitHub. What varies is how much of the account a given token is allowed to see, and a handful of scopes are worth understanding rather than just granting. ## Creating one 1. Go to `https://github.com/settings/tokens`. 2. Choose the kind of token and give it the scopes below. - **Classic** `repo`, `read:packages`, `read:user`, `read:org`, `security_events`, `read:public_key` and `read:gpg_key` covers everything this collects. - **Fine-grained** Read access to the repositories, plus the account permissions for followers, gists, packages, plan, Git SSH keys and GPG keys. 3. Put it where the process can read it, and nowhere else. ```sh export GITHUB_TOKEN=github_pat_... ``` The configuration file refers to it as `${GITHUB_TOKEN}`, which is expanded from the environment at start-up. That is what lets the file be committed while the token stays out of it. ## What each scope buys | Scope | Without it | | ----------------------------- | ------------------------------------------------------------------------------------------------ | | Push access to the repository | Traffic is a 403. GitHub only shows views and clones to someone who could push | | `security_events` | Dependabot and code scanning alerts look exactly like a repository with the feature switched off | | `read:packages` | The container registry is invisible. Package versions cost one call per package | | `read:user` | Followers, contributions, gists and social accounts are missing | | `read:org` | Repositories owned by an organisation are not discovered | | `read:public_key`, `read:gpg_key` | The `keys` family writes nothing, and says nothing. No other scope implies either | Traffic is the one that surprises people. It is not a read scope at all: GitHub decides who may see views and clones by asking whether the caller could push, so a read-only token gets a 403 for every repository. The collector records that as "unavailable" and moves on, which is why the symptom is an empty traffic panel rather than a failed sweep. > **Unavailable is not an error** > > Anything the token cannot see is recorded as unavailable and skipped. A > repository with a feature switched off must not stop the sweep for the other > forty, so the log says `not available (403)` and the sweep continues. ## Why the automatic GITHUB_TOKEN is not enough A workflow gets a `GITHUB_TOKEN` for free. It is not enough for this, and the reason is worth being specific about rather than letting someone discover it as an empty dashboard. - **It is scoped to one repository.** Traffic needs push access to _every_ repository being collected. The automatic token has it for the repository the workflow is running in, and nothing else. - **It has no `security_events`.** Dependabot and code scanning alerts are invisible to it. - **It has no `read:packages`.** The container registry is invisible to it. - **It is not a user.** Everything account-wide (followers, the contribution calendar, billing, notifications, the stars you gave) is about the person, not about a repository, and the automatic token is an installation, not a person. So a workflow needs a personal access token stored as a repository secret and passed in as the `token` input. See [GitHub Actions](/ghchronicle/install/actions/). ## Running without a token Possible, and honest about what it costs. A card of public numbers can be drawn from unauthenticated calls, but the traffic and alert panels will be empty and the log will say `not available (403)` for each of them. That is the collector reporting a permission, not a failure. --- # Ways to install A page per operating system, four ways to get the binary running, and what each one is good for. Source: https://jmrplens.github.io/ghchronicle/install/ One static binary with no runtime dependencies. How it gets onto the machine is the only decision, and it follows from where you want it to run. ## Your system, from download to service Releases carry binaries for Linux, macOS and Windows, on amd64 and arm64. The three pages below are the same walk for each system, taken all the way: which archive, how to check it, where the binary goes, how to run it once, and how to keep it running once you are done watching it. - [Linux](/ghchronicle/install/linux/): A tar.gz, /usr/local/bin, and a hardened systemd unit. - [macOS](/ghchronicle/install/macos/): The darwin archive, the quarantine attribute, and a launchd agent or daemon. - [Windows](/ghchronicle/install/windows/): A zip, PowerShell and cmd, a scheduled task, and what is genuinely different there. ## Getting the binary - **Go** ```sh go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` Needs a Go toolchain, and builds from source at whatever the newest tag is. - **Release** Take the archive for your platform from the [releases page](https://github.com/jmrplens/ghchronicle/releases). Archives are built for Linux, macOS and Windows, on amd64 and arm64, and ship as `tar.gz` (`zip` on Windows). ```sh tar -xzf ghchronicle_1.0.0_linux_amd64.tar.gz sudo install -m 755 ghchronicle /usr/local/bin/ ``` - **Container** ```sh docker run -v $PWD/config.yaml:/config.yaml:ro -e GITHUB_TOKEN \ ghcr.io/jmrplens/ghchronicle -config /config.yaml ``` Distroless, static, and running as uid 65532. - **Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml ``` A composite Action that downloads a release binary, so the runner needs no Go toolchain. ## Choosing where it runs The collector pushes to every store it supports, so it does not need to be reachable from anywhere. It needs to reach GitHub and to reach its databases. That is the only constraint, and it is what makes all four of these viable. Three of the four want a host you own: systemd, Docker and cron. GitHub Actions is the one that does not. The system pages above are where the scheduler lives, one per system: systemd or cron on Linux, launchd on macOS, a scheduled task on Windows. - [systemd](/ghchronicle/install/systemd/): A long-running service on a host you own. The default choice: the state file persists, the schedule is the tool's own, and the unit can be locked down hard. - [Docker](/ghchronicle/install/docker/): The same thing in a container. Mount a writable volume for the state file and any file sink. - [GitHub Actions](/ghchronicle/install/actions/): No host at all. Good for the card and for a scheduled sweep; the state file does not survive between runs unless you cache it. - [cron](/ghchronicle/install/systemd/#cron-instead-of-a-service): `-once` runs a single sweep and exits, which is all a scheduler needs. Keep the state file on a persistent path. ## The three run modes | Command | Does | | ------------------------------------------- | ---------------------------------------------------------- | | `ghchronicle -config config.yaml` | Runs forever, each family on its own cadence | | `ghchronicle -config config.yaml -once` | One sweep, then exits | | `ghchronicle -config config.yaml -backfill` | Walks every surface to its end, waiting for the rate limit | Plus two that write nothing: `-list` prints the repositories in scope, and `-card ... -card-only` renders the SVG without touching a database. > **The state file is the one thing that must persist** > > Whichever way it runs, keep `state_file` on a path that survives a restart. It > is what stops the full stargazer walk and the year-by-year contribution > backfill happening again on every run. --- # Linux The whole path on Linux: the right archive, the signature, the PATH, a service, and building it yourself. Source: https://jmrplens.github.io/ghchronicle/install/linux/ One static binary, `CGO_ENABLED=0`, so there is no C library to install and no distribution to match. A release archive and a `PATH` entry is the whole install; everything below that is about doing it deliberately. ## Pick the archive Release archives are named `ghchronicle__linux_.tar.gz`, with `` either `amd64` or `arm64`. `uname -m` answers which. | `uname -m` reports | The archive to take | | ------------------ | ------------------- | | `x86_64` | `linux_amd64` | | `aarch64` | `linux_arm64` | ```sh VERSION=1.0.0 arch=$(uname -m); case "$arch" in x86_64) arch=amd64 ;; aarch64) arch=arm64 ;; esac base=https://github.com/jmrplens/ghchronicle/releases/download/v$VERSION curl -fsSLO "$base/ghchronicle_${VERSION}_linux_${arch}.tar.gz" ``` > **Name the version rather than asking for the newest** > > The repository publishes a moving `v1` tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > The `v1` release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded Two files are published beside the archives: `checksums.txt`, which holds the SHA-256 of every archive, and `checksums.txt.sigstore.json`, which is a signature over that file. So the chain is one signature and a digest: verify the file, then verify the archive against the file. 1. Take the checksum file and its signature. ```sh curl -fsSLO "$base/checksums.txt" curl -fsSLO "$base/checksums.txt.sigstore.json" ``` 2. Check the archive against it. `--ignore-missing` is what lets one line of a twelve-line file be checked without the other eleven archives being present. ```sh sha256sum --ignore-missing -c checksums.txt ``` ```text ghchronicle_1.0.0_linux_amd64.tar.gz: OK ``` 3. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/). ```sh cosign verify-blob \ --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --bundle checksums.txt.sigstore.json \ checksums.txt ``` ```text Verified OK ``` The signing is keyless: there is no public key to fetch and no private key for anyone to lose, because the identity being verified is the workflow that ran, recorded in a public transparency log. That is what the two `--certificate` flags say, and why they are not optional: without them cosign would confirm that somebody signed the file, which is not the question. ## Put it on the PATH The archive holds three files and no directory, so extract it somewhere you meant to. - ghchronicle_1.0.0_linux_amd64.tar.gz - ghchronicle the binary - LICENSE - README.md - **For everyone** ```sh tar -xzf ghchronicle_1.0.0_linux_amd64.tar.gz ghchronicle sudo install -m 755 ghchronicle /usr/local/bin/ ghchronicle -version ``` - **For one account** ```sh mkdir -p ~/.local/bin tar -xzf ghchronicle_1.0.0_linux_amd64.tar.gz -C ~/.local/bin ghchronicle chmod 755 ~/.local/bin/ghchronicle ghchronicle -version ``` `~/.local/bin` is on the `PATH` of most distributions already. If `ghchronicle -version` answers "command not found", it is not on yours. ```text ghchronicle 1.0.0 (commit 4e5dfc2, built 2026-09-14T23:04:02Z) ``` ## Run it once The binary needs a configuration file and a token, and [the quickstart](/ghchronicle/start/quickstart/) writes both in six steps. With those in place: ```sh ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` ## Keep it running [systemd](/ghchronicle/install/systemd/) is the arrangement this documentation treats as the default on Linux, and the unit there is hardened rather than minimal, because this is very likely the only process on the host holding a GitHub token with read access to every repository of an account. A scheduler works too: [cron with `-once`](/ghchronicle/install/systemd/#cron-instead-of-a-service) is a single line, at the cost of giving every family the same cadence. ## Build it from source Go 1.27.1 or newer is what the module declares. Nothing else is needed: the build sets `CGO_ENABLED=0`, so there is no compiler and no header package to find. - **go install** ```sh go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)/bin`, which is `~/go/bin` unless you moved it, and that directory has to be on your `PATH`. A binary built this way reports its version but not its commit or its build date: ```text ghchronicle 1.0.0 (commit unknown, built unknown) ``` The version comes from the `VERSION` file the module embeds; the other two are stamped by the release build and by nothing else, and a module downloaded through the proxy carries no checkout to read them from. - **make** ```sh git clone https://github.com/jmrplens/ghchronicle cd ghchronicle make build # into bin/ghchronicle make install # into GOBIN, stamped like a release build ``` `make build` stamps the version, the short commit and the commit date, so `-version` says exactly which tree it came from. `make` with no target lists every target there is. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. So the path is a decision you make once and then pass on every invocation. What the rest of this documentation assumes: | File | Path | Mode | | ------------------ | ---------------------------------- | --------------------------- | | Configuration | `/etc/ghchronicle/config.yaml` | world readable, no secrets | | Tokens | `/etc/ghchronicle/ghchronicle.env` | `600` | | State and ledger | `/var/lib/ghchronicle/` | written by the service user | `state_file` has a default of its own, `ghchronicle-state.json` in the working directory, with the write ledger beside it as `ghchronicle-state-written.bin`. That default is fine for a first run in a directory you made, and wrong for a service, whose working directory is not something to rely on. Set it. --- # macOS The whole path on macOS: the darwin archive, quarantine, a launchd agent or daemon, and building it yourself. Source: https://jmrplens.github.io/ghchronicle/install/macos/ The same static binary as everywhere else, built for `darwin` on both Apple silicon and Intel. macOS is not a build target that is merely compiled and hoped for: every change runs the whole unit suite and the end-to-end suite on a macOS runner, beside Linux and Windows. ## Pick the archive Release archives say `darwin`, which is the name of the system the Go toolchain uses; macOS is the name Apple uses for the same thing. `uname -m` answers which architecture. | `uname -m` reports | The machine | The archive to take | | ------------------ | ------------------ | ------------------- | | `arm64` | Apple silicon | `darwin_arm64` | | `x86_64` | Intel | `darwin_amd64` | ```sh VERSION=1.0.0 arch=$(uname -m); case "$arch" in x86_64) arch=amd64 ;; esac base=https://github.com/jmrplens/ghchronicle/releases/download/v$VERSION curl -fsSLO "$base/ghchronicle_${VERSION}_darwin_${arch}.tar.gz" ``` > **Name the version rather than asking for the newest** > > The repository publishes a moving `v1` tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > The `v1` release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded Two files are published beside the archives: `checksums.txt`, which holds the SHA-256 of every archive, and `checksums.txt.sigstore.json`, which is a signature over that file. 1. Take the checksum file and its signature. ```sh curl -fsSLO "$base/checksums.txt" curl -fsSLO "$base/checksums.txt.sigstore.json" ``` 2. Check the archive against it. macOS ships `shasum` rather than the `sha256sum` of a Linux box, so the line for your archive is selected and piped in, which works whatever version of `shasum` the system has. ```sh grep "darwin_${arch}.tar.gz$" checksums.txt | shasum -a 256 -c - ``` ```text ghchronicle_1.0.0_darwin_arm64.tar.gz: OK ``` 3. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/). ```sh cosign verify-blob \ --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --bundle checksums.txt.sigstore.json \ checksums.txt ``` ```text Verified OK ``` The signing is keyless: the identity being verified is the workflow that ran, recorded in a public transparency log, which is why the two `--certificate` flags are not optional. Without them cosign would confirm that somebody signed the file, which is not the question. ## Put it on the PATH The archive holds three files and no directory. - ghchronicle_1.0.0_darwin_arm64.tar.gz - ghchronicle the binary - LICENSE - README.md - **For everyone** ```sh tar -xzf ghchronicle_1.0.0_darwin_arm64.tar.gz ghchronicle sudo install -m 755 ghchronicle /usr/local/bin/ ghchronicle -version ``` `/usr/local/bin` is on the default `PATH` of every macOS install, on both architectures, because `/etc/paths` lists it first. - **For one account** ```sh mkdir -p ~/bin tar -xzf ghchronicle_1.0.0_darwin_arm64.tar.gz -C ~/bin ghchronicle chmod 755 ~/bin/ghchronicle echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zprofile ``` `~/bin` is not on the `PATH` by default, hence the last line. `zsh` is the login shell on every supported macOS. > **If macOS refuses to run it** > > The binary carries no Developer ID signature and is not notarized, so a copy > that arrives with the quarantine attribute is refused with "cannot be opened > because the developer cannot be verified". The attribute is not carried > inside the archive: it is set by whatever process writes a file, and > inherited by the processes that one starts. A browser writes the `.tar.gz` it > downloads with the mark on it, `curl` does not, and `tar -xzf` run from > Terminal writes an unmarked binary either way. The case that bites is opening > the archive in Finder, because Archive Utility passes the mark on to what it > extracts. So look before you clear anything: > > ```sh > xattr -l ghchronicle_1.0.0_darwin_arm64.tar.gz # what a browser marked > xattr -l ghchronicle # the extracted binary > xattr -c ghchronicle # clear it > ``` > > `xattr -c` clears every extended attribute and succeeds when there are none. > `xattr -d com.apple.quarantine` does not: on a binary extracted from Terminal > it stops with `No such xattr: com.apple.quarantine`, which looks like a > broken instruction and is only the attribute never having been there. ## Run it once The binary needs a configuration file and a token, and [the quickstart](/ghchronicle/start/quickstart/) writes both in six steps. With those in place: ```sh ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` ## Keep it running with launchd launchd is what macOS has instead of systemd, and the choice it asks you to make first is agent or daemon. | | A LaunchAgent | A LaunchDaemon | | ---------- | ------------------------- | -------------------------- | | Lives in | `~/Library/LaunchAgents/` | `/Library/LaunchDaemons/` | | Runs as | you | `root`, or the `UserName` you give it | | Runs when | you are logged in | the machine is up, from boot | | Good for | a laptop you use | a Mac that stays on | The agent is the one to start with. It needs no `sudo`, and a collector that stops while the laptop's owner is logged out loses nothing that the next sweep does not pick up. ```xml title="~/Library/LaunchAgents/io.jmrp.ghchronicle.plist" Label io.jmrp.ghchronicle ProgramArguments /usr/local/bin/ghchronicle -config /Users/you/Library/Application Support/ghchronicle/config.yaml EnvironmentVariables GITHUB_TOKEN github_pat_... RunAtLoad KeepAlive StandardOutPath /Users/you/Library/Logs/ghchronicle.log StandardErrorPath /Users/you/Library/Logs/ghchronicle.log ``` 1. Protect the file before it holds a token, then load it. ```sh chmod 600 ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist ``` 2. See that it is up, and read what it says. ```sh launchctl print gui/$(id -u)/io.jmrp.ghchronicle tail -f ~/Library/Logs/ghchronicle.log ``` 3. After editing the file, unload it and load it again. launchd reads the property list once, when it is bootstrapped. ```sh launchctl bootout gui/$(id -u)/io.jmrp.ghchronicle launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist ``` > **The token sits in the property list** > > A launchd job does not read your shell profile, so `EnvironmentVariables` is > where the token has to be. That file is then the macOS equivalent of the > systemd environment file, and it deserves the same treatment: mode `600`, > never in a repository, and remembered when the token is rotated. `launchctl > setenv` is the alternative, and it is worse: it puts the token in every > process of the session. ### Or one sweep on a timer `StartInterval` is launchd's cron, in seconds, and `-once` is the mode that suits it. Replace `KeepAlive` with it and add `-once` to `ProgramArguments`: ```xml StartInterval 3600 ``` Keep `state_file` on a path that survives, in this mode above all: it is what stops the whole stargazer walk and the year by year contribution backfill happening again on every run. ## Build it from source Go 1.27.1 or newer is what the module declares. The build sets `CGO_ENABLED=0`, so the Xcode command line tools are not needed for it. - **go install** ```sh go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)/bin`, which is `~/go/bin` unless you moved it, and that directory has to be on your `PATH`. A binary built this way reports its version but not its commit or its build date, because those two are stamped by the release build and a module downloaded through the proxy carries no checkout to read them from. - **make** ```sh git clone https://github.com/jmrplens/ghchronicle cd ghchronicle make build # into bin/ghchronicle make install # into GOBIN, stamped like a release build ``` The Makefile is written for GNU Make 3.81, which is the version macOS ships, so the stock `make` runs it. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. macOS has no `/etc/ghchronicle` habit, so these are the conventional places rather than ones the tool knows: | File | For an agent | For a daemon | | ---------------- | --------------------------------------------------- | -------------------------------- | | Configuration | `~/Library/Application Support/ghchronicle/` | `/usr/local/etc/ghchronicle/` | | State and ledger | `~/Library/Application Support/ghchronicle/` | `/usr/local/var/ghchronicle/` | | Log | `~/Library/Logs/ghchronicle.log` | `/usr/local/var/log/` | `state_file` has a default of its own, `ghchronicle-state.json` in the working directory, with the write ledger beside it as `ghchronicle-state-written.bin`. A launchd job's working directory is not something to rely on. Set it. --- # Windows The whole path on Windows: the zip, PowerShell and cmd, a scheduled task, and what is genuinely different there. Source: https://jmrplens.github.io/ghchronicle/install/windows/ Windows is a released platform, not an afterthought: every change runs the whole unit suite and the end-to-end suite on a Windows runner, and the source carries Windows-only code where the system behaves differently. What follows is written for PowerShell, with the `cmd` form beside it wherever the two differ. ## What is different here Read this first. Four of the five lines below are the reason a command copied from a Linux page does not work. - **The program** is `ghchronicle.exe`. Typing `ghchronicle` finds it, because `PATHEXT` lists `.EXE`. - **Runtime dependencies**: none. The binary is built `CGO_ENABLED=0`, so there is no C runtime to install. - **Stopping it** is Ctrl+C in its console. There is no signal to send it, and `taskkill /F` ends it without closing its sinks. - **Paths in the configuration**: a backslash inside a double-quoted YAML scalar is an escape. Use forward slashes. - **Running unattended** is a scheduled task. The binary is not a Service Control Manager service. ## Pick the archive Windows archives are `zip` rather than `tar.gz`, and they are named `ghchronicle__windows_.zip`, with `` either `amd64` or `arm64`. | `$env:PROCESSOR_ARCHITECTURE` | The archive to take | | ----------------------------- | ------------------- | | `AMD64` | `windows_amd64` | | `ARM64` | `windows_arm64` | That variable describes the **process**, not the machine. A 32-bit PowerShell on a 64-bit machine reports `x86` and leaves the machine's own architecture in `$env:PROCESSOR_ARCHITEW6432`; an x64 PowerShell running under emulation on an ARM64 machine reports `AMD64` and sets nothing else, which is the case that quietly hands you the wrong archive. If either could be you, ask the machine rather than the shell: ```powershell (Get-CimInstance Win32_Processor).Architecture # 9 is x64, 12 is ARM64 ``` ```powershell $version = "1.0.0" $arch = if ($env:PROCESSOR_ARCHITECTURE -eq "ARM64") { "arm64" } else { "amd64" } $base = "https://github.com/jmrplens/ghchronicle/releases/download/v$version" $zip = "ghchronicle_${version}_windows_${arch}.zip" Invoke-WebRequest -UseBasicParsing -Uri "$base/$zip" -OutFile $zip ``` `-UseBasicParsing` is not decoration. On Windows PowerShell 5.1 `Invoke-WebRequest` builds its result through the Internet Explorer engine unless told not to, and fails outright on a host where Internet Explorer was removed or never went through its first-run configuration, which covers Server Core and most hardened images. On PowerShell 7 the switch is accepted and does nothing. > **Name the version rather than asking for the newest** > > The repository publishes a moving `v1` tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > The `v1` release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded `checksums.txt` holds the SHA-256 of every archive, and `checksums.txt.sigstore.json` is a signature over that file. 1. Take the checksum file and compare the one line that is yours. ```powershell Invoke-WebRequest -UseBasicParsing -Uri "$base/checksums.txt" -OutFile checksums.txt $expected = (Select-String -Path checksums.txt -Pattern ([regex]::Escape($zip) + '$')).Line.Split(" ")[0] $actual = (Get-FileHash -Algorithm SHA256 -Path $zip).Hash if ($actual -eq $expected) { "OK" } else { "MISMATCH" } ``` `Get-FileHash` returns the digest in upper case and `checksums.txt` holds it in lower case. They still compare equal because PowerShell's `-eq` on two strings ignores case, which is the one place here where that default is convenient rather than a trap. 2. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/). The command is the same one the release notes print, and the same one a Linux or macOS reader runs. ```powershell cosign verify-blob ` --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' ` --certificate-oidc-issuer https://token.actions.githubusercontent.com ` --bundle checksums.txt.sigstore.json ` checksums.txt ``` The backtick is PowerShell's line continuation, where a shell script uses a backslash. ## Put it somewhere and on the PATH The archive holds three files and no directory, so unpack it into a directory you made. - ghchronicle_1.0.0_windows_amd64.zip - ghchronicle.exe the binary - LICENSE - README.md - **For everyone** An elevated PowerShell, because both the directory and the machine `Path` need administrator rights. ```powershell $dir = "C:\Program Files\ghchronicle" Expand-Archive -Path $zip -DestinationPath $dir -Force $machine = [Environment]::GetEnvironmentVariable("Path", "Machine") [Environment]::SetEnvironmentVariable("Path", "$machine;$dir", "Machine") ``` - **For one account** No elevation needed, and nothing outside your profile is touched. ```powershell $dir = "$env:LOCALAPPDATA\Programs\ghchronicle" Expand-Archive -Path $zip -DestinationPath $dir -Force $user = [Environment]::GetEnvironmentVariable("Path", "User") [Environment]::SetEnvironmentVariable("Path", "$user;$dir", "User") ``` > **Two traps in writing Path back** > > Both snippets read `Path` from the scope they write to, never from > `$env:Path`. `$env:Path` is the process's own copy, which Windows built by > joining the machine list and the user list: write that back into either scope > and you have copied the other one into it, permanently, and it grows again > every time somebody repeats the command. > > The second trap belongs to the machine scope alone. > `[Environment]::GetEnvironmentVariable` expands `%SystemRoot%` and its > relatives while it reads, and `SetEnvironmentVariable` writes the result back > as a plain string, so a machine `Path` that held such entries comes back with > them baked in and its registry value changes kind from `REG_EXPAND_SZ` to > `REG_SZ`, for good. You cannot spot it in `$machine`, because the expansion > has already happened by then. Open System Properties, Environment Variables, > which shows the value unexpanded: if there is a `%VAR%` anywhere in the > machine `Path`, add the directory from that dialog rather than from the > snippet above. A new `Path` reaches only processes started afterwards, so open a new terminal before the check below. The current one keeps the environment it was given. ```powershell ghchronicle -version ``` ```text ghchronicle 1.0.0 (commit 4e5dfc2, built 2026-09-14T23:04:02Z) ``` > **If Windows warns about the file** > > The binary carries no Authenticode signature: the release signs the checksum > file and the SBOMs, and nothing else. SmartScreen and Defender treat an > unsigned executable downloaded from the internet on their own terms, which > vary by version and policy and which this page will not guess at. If Windows > refuses to start it, the attribute to clear is the mark of the web: > > ```powershell > Unblock-File -Path "$dir\ghchronicle.exe" > ``` ## The token, and the rest of the environment Every `${VAR}` in the configuration file is read from the environment when the process starts, so the token never has to be in the file. - **PowerShell** ```powershell # This window only $env:GITHUB_TOKEN = "github_pat_..." # Persisted for this account [Environment]::SetEnvironmentVariable("GITHUB_TOKEN", "github_pat_...", "User") ``` - **cmd** ```bat rem This window only set GITHUB_TOKEN=github_pat_... rem Persisted for this account, and NOT visible in this window setx GITHUB_TOKEN "github_pat_..." ``` > **A persisted variable is a token in the registry** > > `SetEnvironmentVariable` with `User` or `Machine`, and `setx`, write to the > registry in clear text, where anything running as that account can read them. > That is the same bargain as a systemd environment file, without the file mode > to lean on. A scheduled task running as `SYSTEM` reads the machine scope, so > a token put there is readable by every service on the box: prefer the user > scope and a task that runs as that user. ## Run it once The binary needs a configuration file, and [the quickstart](/ghchronicle/start/quickstart/) writes one in six steps. Save it as UTF-8, and mind the two Windows details below it. ```powershell ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` If the binary is in the current directory rather than on the `PATH`, PowerShell needs it named as a path: `.\ghchronicle.exe`. A bare name is a command, and the current directory is not searched for commands. ### Paths in the configuration file YAML treats a backslash as an escape inside a **double-quoted** scalar and as an ordinary character everywhere else. So a Windows path in double quotes is not the path you wrote, and usually not valid YAML either: ```yaml state_file: "C:\ghchronicle\state.json" # ghchronicle: config.yaml: yaml: line N: found unknown escape character state_file: C:\ghchronicle\state.json # correct, plain scalar state_file: 'C:\ghchronicle\state.json' # correct, single quoted state_file: C:/ghchronicle/state.json # correct, and the one to prefer ``` Forward slashes are the simplest answer: Windows accepts them in a path, and they survive being quoted whichever way. ### The file itself has to be UTF-8 Windows PowerShell 5.1, the one that ships in the box, writes neither of the things a YAML parser wants, and it writes a different wrong thing depending on how you ask. `>` and `Out-File` produce UTF-16LE, which the parser reads as binary. `Set-Content` produces the system's active code page, usually ANSI, which parses while the file is pure ASCII and mangles the first accented character in it. PowerShell 7 defaults to UTF-8 without a byte order mark and has neither problem; on 5.1, be explicit: ```powershell Set-Content -Path config.yaml -Value $text -Encoding utf8 ``` `utf8` on 5.1 means UTF-8 **with** a byte order mark, which the value cannot express and the cmdlet does not warn about. The collector's parser reads past one, so the file works; a tool that reads the first bytes for itself may not. ## Keep it running There is no service mode. The collector is a console program: it does not talk to the Service Control Manager, so registering it with `sc.exe create` produces a service that Windows starts and then gives up on, reporting that it "did not respond to the start or control request in a timely fashion". Third-party service wrappers exist and this project neither ships nor tests one. The ordinary way to run something unattended on Windows is a scheduled task, and there are two shapes of it. - **One sweep on a timer** The Windows equivalent of cron, and the one to prefer: nothing has to be stopped, and a missed run costs one sweep. ```powershell $action = New-ScheduledTaskAction ` -Execute "C:\Program Files\ghchronicle\ghchronicle.exe" ` -Argument '-config "C:\ProgramData\ghchronicle\config.yaml" -once' $trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) ` -RepetitionInterval (New-TimeSpan -Hours 1) ` -RepetitionDuration ([TimeSpan]::MaxValue) Register-ScheduledTask -TaskName ghchronicle -Action $action -Trigger $trigger ``` `-RepetitionDuration ([TimeSpan]::MaxValue)` is what spells out "for ever". Task Scheduler's own rule is that a repetition with no duration repeats indefinitely, so leaving it out is not a bug, but the cmdlet has no default of its own and the task is then registered carrying no duration at all. Two conditions come with this shape, and neither is cron's. An hourly task gives every family an hourly cadence at best, so the fifteen-minute rhythm of `actions` is lost, which is the same trade [cron makes on Linux](/ghchronicle/install/systemd/#cron-instead-of-a-service). And `Register-ScheduledTask` here names no `-User` and no `-Principal`, so the task is registered under the calling account with the default logon type and runs **only while that account is logged on**. A cron job does not stop when you log out. For one that behaves the same way, register the task with a principal that has "run whether user is logged on or not" set, which is `New-ScheduledTaskPrincipal`. - **Running all the time** A task triggered at logon or at startup, with the collector left in its long-running mode so each family keeps its own cadence. ```powershell $action = New-ScheduledTaskAction ` -Execute "C:\Program Files\ghchronicle\ghchronicle.exe" ` -Argument '-config "C:\ProgramData\ghchronicle\config.yaml"' $trigger = New-ScheduledTaskTrigger -AtLogOn $settings = New-ScheduledTaskSettingsSet ` -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1) ` -ExecutionTimeLimit ([TimeSpan]::Zero) Register-ScheduledTask -TaskName ghchronicle -Action $action ` -Trigger $trigger -Settings $settings ``` `-ExecutionTimeLimit ([TimeSpan]::Zero)` is the one that matters: the default stops a task after three days, which for a process meant to run for ever is a restart nobody asked for. ```powershell Start-ScheduledTask -TaskName ghchronicle Get-ScheduledTaskInfo -TaskName ghchronicle # last run, last result ``` `Stop-ScheduledTask` ends the process rather than asking it to finish: it is `taskkill /F` by another name, and it closes no sink on its way out. In the timer shape there is nothing to stop, which is most of why it is the better default. > **Where the log goes** > > A task has no console, so the log has to be a file: set `log.file` in the > configuration. Standard output from a scheduled task is discarded, and > `Get-ScheduledTaskInfo` reports only the exit code. ## Build it from source Go 1.27.1 or newer, which is the version `go.mod` declares. `CGO_ENABLED=0` means no MSVC, no MinGW and no Windows SDK. - **go install** ```powershell go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)\bin`, which is `%USERPROFILE%\go\bin` unless you moved it, and that directory has to be on your `Path`. A binary built this way reports its version but not its commit or its build date, because those two are stamped by the release build and a module downloaded through the proxy carries no checkout to read them from. - **From a checkout** ```powershell git clone https://github.com/jmrplens/ghchronicle cd ghchronicle go build -o ghchronicle.exe .\cmd\ghchronicle ``` `go build` rather than `make`: the Makefile is a GNU Makefile whose recipes are POSIX shell, so it wants Git Bash, MSYS2 or WSL. This one line is what `make build` does, minus the version stamping. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. A scheduled task's working directory is not something to rely on, so give every path in the file absolutely. | File | For a machine-wide task | For one account | | ---------------- | -------------------------------- | ----------------------------------- | | Configuration | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | | State and ledger | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | | Log | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | `${LOCALAPPDATA}` is written that way because `${VAR}` is the one form the collector expands, from the environment, as the process starts. `%VAR%` is a shell notation and means nothing to the file: a `%LOCALAPPDATA%` copied into the YAML gives you a directory literally named `%LOCALAPPDATA%`, next to wherever the task happened to be working. It is `%LOCALAPPDATA%` in `cmd` and `$env:LOCALAPPDATA` in PowerShell that create the directory in the first place. A directory under `C:\ProgramData` is writable by its creator and readable by everyone, so create it elevated and then grant write to the account the task runs as. The state file and its write ledger are the two the collector rewrites on every sweep; if one of them is marked read-only the collector clears that attribute itself, because NTFS refuses to replace a read-only file even when asked to replace it, and a sweep should not fail over a file property. --- # systemd A hardened unit for the one process on the host that holds a GitHub token, and what each restriction is for. Source: https://jmrplens.github.io/ghchronicle/install/systemd/ ## The unit ```ini title="/etc/systemd/system/ghchronicle.service" [Unit] Description=ghchronicle, GitHub metrics collector After=network-online.target Wants=network-online.target [Service] Type=simple User=ghchronicle Group=ghchronicle EnvironmentFile=/etc/ghchronicle/ghchronicle.env ExecStart=/usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml Restart=always RestartSec=30s # This is the only process on the host holding a GitHub token, so it gets # nothing it does not need. NoNewPrivileges=true PrivateTmp=true PrivateDevices=true ProtectSystem=strict ProtectHome=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true ProtectClock=true ProtectHostname=true ProtectProc=invisible RestrictNamespaces=true RestrictRealtime=true RestrictSUIDSGID=true LockPersonality=true MemoryDenyWriteExecute=true SystemCallArchitectures=native SystemCallFilter=@system-service CapabilityBoundingSet= AmbientCapabilities= RestrictAddressFamilies=AF_INET AF_INET6 StateDirectory=ghchronicle ReadWritePaths=/var/lib/ghchronicle [Install] WantedBy=multi-user.target ``` ## Why it is hardened The threat model is short and it is the whole justification: **this is very likely the only process on the host holding a GitHub token with read access to every repository of an account.** A token is a bearer credential. Anything that can read this process's memory or its environment file has the account. So the unit gives the process exactly what it needs, which turns out to be almost nothing: an outbound TCP socket and one writable directory. | Directive | What it removes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | `CapabilityBoundingSet=`, `AmbientCapabilities=` | Every Linux capability. It binds no privileged port and owns no device | | `NoNewPrivileges=true` | Any path to gaining privileges through exec, including setuid binaries | | `ProtectSystem=strict` | Write access to the entire filesystem, except `ReadWritePaths` | | `ProtectHome=true` | Every home directory, which is where the interesting credentials on a host usually live | | `PrivateTmp=true`, `PrivateDevices=true` | Shared temporary files, and the physical device nodes | | `ProtectProc=invisible` | The ability to see other users' processes in `/proc`, so it cannot read another process's command line | | `RestrictAddressFamilies=AF_INET AF_INET6` | Unix and netlink sockets. It talks HTTPS and nothing else | | `MemoryDenyWriteExecute=true`, `LockPersonality=true` | The usual shellcode primitives | | `SystemCallFilter=@system-service` | Every syscall outside the ordinary service set, including the module and kernel-tuning ones | | `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictNamespaces`, `RestrictRealtime`, `RestrictSUIDSGID` | Every remaining route to changing the host from inside the service | `StateDirectory=ghchronicle` makes systemd create `/var/lib/ghchronicle` with the right ownership on start, so the state file has somewhere to live without a manual `mkdir` and a `chown` that someone will forget after a reinstall. Two files live there, not one. Beside `state.json` the sweep keeps its write ledger, `state-written.bin` by default, which is what stops an unchanged point being written again; `ReadWritePaths` covers the directory, so both are already allowed. Put either somewhere else and that path needs adding here, and losing the ledger costs one sweep of rewriting: [only what changed is written](/ghchronicle/sinks/#only-what-changed-is-written). ## Installing it 1. Create the user and the directories. ```sh sudo useradd --system --no-create-home --shell /usr/sbin/nologin ghchronicle sudo mkdir -p /etc/ghchronicle ``` 2. Put the configuration in place. - /etc/ghchronicle/ - config.yaml world readable, no secrets in it - ghchronicle.env mode 600, the tokens - /var/lib/ghchronicle/ - state.json created by the service - state-written.bin the write ledger, beside it 3. Write the environment file, and nothing else in it. ```sh title="/etc/ghchronicle/ghchronicle.env" GITHUB_TOKEN=github_pat_... INFLUX_TOKEN=... ``` ```sh sudo chmod 600 /etc/ghchronicle/ghchronicle.env ``` Everything in `config.yaml` reads these through `${VAR}`, which is what lets the config be world readable and version controlled while the secrets are not. 4. Start it. ```sh sudo systemctl daemon-reload sudo systemctl enable --now ghchronicle sudo systemctl status ghchronicle ``` ## What to watch The log says what was written and where. ```text level=INFO msg=written sink=influxdb family=traffic points=629 level=INFO msg="rate budget" bucket=core remaining=4354 limit=5000 ``` Two warnings are worth an alert: - **`rate limit reserve reached`** means a family was skipped to protect the budget. Once is fine; every sweep means the cadences are too fast for the number of repositories. - **`family failed everywhere, not marking it as run`** means every repository failed for one family, so it will be retried rather than treated as done. ```sh journalctl -u ghchronicle -f journalctl -u ghchronicle -p warning --since today ``` > **Check the sandbox after editing the unit** > > `systemd-analyze security ghchronicle` scores the unit and names anything the > sandbox is not covering. It is the fastest way to see that an edit quietly > removed a restriction. ## cron instead of a service `-once` runs a single sweep and exits, which is all a scheduler needs. ```text 0 * * * * /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -once ``` Keep the state file on a persistent path even in this mode. It is what stops the stargazer walk and the year-by-year contribution backfill happening again on every run. Note that an hourly cron gives every family an hourly cadence at best, so the fifteen-minute rhythm of `actions` is lost. --- # Docker The distroless image, what has to be mounted writable, and a compose file next to InfluxDB. Source: https://jmrplens.github.io/ghchronicle/install/docker/ ```sh docker run -d --name ghchronicle \ -v /etc/ghchronicle/config.yaml:/config.yaml:ro \ -e GITHUB_TOKEN -e INFLUX_TOKEN \ -p 9605:9605 \ ghcr.io/jmrplens/ghchronicle -config /config.yaml ``` ## The image Built `FROM gcr.io/distroless/static-debian13:nonroot` over a static, `CGO_ENABLED=0` binary. There is no shell and no package manager in it, so code execution inside the container has nothing to pivot with, and the `nonroot` tag bakes in **uid 65532**, which keeps it off root even when the orchestrator sets no `securityContext` of its own. Two consequences worth knowing before you debug it: - `docker exec ... sh` does not work. There is no `sh`. Read the logs instead. - Anything the container writes must be owned by, or writable by, uid 65532. ## What has to be writable The config file is mounted read-only. Four things are not: | Path | Needed for | | ------------------- | ------------------------------------------------------------------------------------------- | | `state_file` | Always. Without a persistent path the stargazer walk repeats on every restart | | `sinks.dedupe_file` | Always. The write ledger, which defaults to sitting beside the state file | | `sinks.file.path` | Only with the file sink | | `log.file` | Only with a log file configured | Both of the first two live in the same directory by default, so one mounted volume covers them. Mounting only the state file loses the ledger on every restart, and every restart then costs a whole sweep of rewriting, which is the one thing the ledger exists to prevent: [only what changed is written](/ghchronicle/sinks/#only-what-changed-is-written). ```sh docker volume create ghchronicle-state docker run -d --name ghchronicle \ -v /etc/ghchronicle/config.yaml:/config.yaml:ro \ -v ghchronicle-state:/var/lib/ghchronicle \ -e GITHUB_TOKEN \ ghcr.io/jmrplens/ghchronicle -config /config.yaml ``` ## The port `EXPOSE 9605` is the Prometheus exporter, and it is the only listener the process ever opens. Publish it only if you enabled the `prometheus` sink; every other sink is outbound. > **Bind the exporter to 0.0.0.0 inside a container** > > The example configuration listens on `127.0.0.1:9605`, which inside a > container means the container's own loopback and is unreachable from the host. > Set `sinks.prometheus.listen: 0.0.0.0:9605` and let `-p` decide who can reach > it. ## With compose - **Collector only** ```yaml title="compose.yaml" services: ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped environment: GITHUB_TOKEN: ${GITHUB_TOKEN} INFLUX_TOKEN: ${INFLUX_TOKEN} volumes: - ./config.yaml:/config.yaml:ro - state:/var/lib/ghchronicle volumes: state: ``` - **With InfluxDB** ```yaml title="compose.yaml" services: influxdb: image: influxdb:3-core volumes: - influx:/var/lib/influxdb3 ports: - "8181:8181" ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: - influxdb environment: GITHUB_TOKEN: ${GITHUB_TOKEN} INFLUX_TOKEN: ${INFLUX_TOKEN} volumes: - ./config.yaml:/config.yaml:ro - state:/var/lib/ghchronicle volumes: influx: state: ``` The collector reaches the database by service name, so `sinks.influxdb.url` is `http://influxdb:8181`. ## One sweep, then exit The container takes the same flags as the binary, so a scheduler can run it without a long-lived service. ```sh docker run --rm \ -v /etc/ghchronicle/config.yaml:/config.yaml:ro \ -v ghchronicle-state:/var/lib/ghchronicle \ -e GITHUB_TOKEN \ ghcr.io/jmrplens/ghchronicle -config /config.yaml -once ``` Mount the state volume in this mode too. It is what makes the second run cheap. --- # GitHub Actions The composite Action, its three modes, and the two things a hosted runner does not keep. Source: https://jmrplens.github.io/ghchronicle/install/actions/ The repository ships a composite Action, so a workflow needs no Go toolchain: it downloads a release binary and calls it. ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml ``` ## Inputs | Input | Default | What it does | | ----------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------- | | `token` | required | A personal access token. The automatic `GITHUB_TOKEN` is not enough | | `config` | `""` | Path to a configuration file. Omit to run with defaults built from `user` | | `user` | the repository owner | The account to collect when no config file is given | | `mode` | `once` | `once`, `backfill` or `card` | | `backfill-since` | `""` | Bound for `backfill`: a date, `90d`, `2y` or a Go duration. Empty means no bound | | `card` | `""` | Path of the SVG to write. Empty means no card | | `card-layout` | `summary` | One of the thirteen registered layouts | | `card-theme` | `auto` | `dark`, `light`, `auto`, or `both` for a light card and its `_dark` twin | | `card-fields` | `""` | Comma-separated fields. Empty means the layout's default | | `card-motion` | `once` | `once`, `loop` or `off`; `loop` changes only `terminal` and `ticker` | | `card-width` | `""` | Card width in pixels. Empty draws the layout at its own width; each one draws between two ends of its own, stated in its [section](/ghchronicle/card/layouts/). Only `activity-heatmap` spends the room on data, a whole year of the calendar at its far end | | `card-speed` | `""` | How fast an animated layout plays, as a decimal from 0 to 1. Empty means 0.5, the pace every card has always been drawn at; below it the card is slower, above it faster, and every animated layout scales together. 0 is the slowest animation and not a still card, `card-motion: off` is | | `include-private` | `false` | `true` counts private repositories when no config file is given (the default up to v1.0.0). See the warning below | | `version` | `latest` | The release to install | ## The three modes - **once** One sweep against a configuration file you supply, which is how a workflow feeds a database. ```yaml name: Collect on: schedule: - cron: "*/30 * * * *" workflow_dispatch: jobs: collect: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml env: INFLUX_TOKEN: ${{ secrets.INFLUX_TOKEN }} ``` The config file refers to the database credentials as `${VAR}`, so they go in as secrets and never into the repository. - **card** One sweep and one SVG, and nothing written to any database. This is the only mode that needs no store configured at all. ```yaml name: Profile card on: schedule: - cron: "17 6 * * *" workflow_dispatch: permissions: contents: write jobs: card: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/github-stats.svg card-layout: github-stats card-theme: both - name: Commit if it changed run: | git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add generated/ git diff --staged --quiet || git commit -m "Update the profile card" git push ``` The SVG is byte-identical for the same input, so a day with no change produces no commit. - **backfill** Reaches as far back as GitHub allows and waits for the rate limit rather than stopping. Run it once, by hand. ```yaml name: Backfill on: workflow_dispatch: inputs: since: description: "A date, 90d, 2y, or empty for no bound" default: "2y" jobs: backfill: runs-on: ubuntu-latest timeout-minutes: 360 steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: backfill backfill-since: ${{ inputs.since }} config: .github/ghchronicle.yaml ``` Give it a generous `timeout-minutes`: an unbounded backfill parks at every rate limit reset, and a job that is killed half way has spent the quota and kept part of the benefit. ## A card in your profile README GitHub shows the README of the repository named after your account, `/`, at the top of your profile. The card lives in that repository as a file, the README points at it once, and a scheduled workflow replaces the file. The README itself is never rewritten. 1. Create a personal access token (see [the token](/ghchronicle/start/token/)) and store it in `/` as the secret `GHCHRONICLE_TOKEN`. The automatic `GITHUB_TOKEN` will not do, even for public numbers: it is not a user, so the Action cannot list your repositories with it and the card comes out empty. 2. Add `.github/workflows/card.yml`: ```yaml name: Profile card on: schedule: - cron: "17 6 * * *" workflow_dispatch: permissions: contents: write # One run at a time: two that overlap would race each other to push. concurrency: group: profile-card cancel-in-progress: false jobs: card: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: animated-counters card-theme: both card-motion: once - name: Commit if it changed run: | git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add generated/ git diff --staged --quiet || git commit -m "Update the profile card" git push ``` 3. Paste this line into `README.md` wherever the card should appear, once: ```html My GitHub statistics ``` 4. Run the workflow by hand from the Actions tab the first time, so the files exist before the first scheduled run. **More than one card.** Repeat the `uses: jmrplens/ghchronicle@v1` step with another `card` path and layout, and paste one `` per card. Each step is a sweep of its own, and each draws every number again: a run that writes a card collects every family whatever its cadence says, and one in `card` mode writes nothing back to the state file. Several card steps can therefore share one `config`, and so one `state_file`, and each card is the whole account. That is also what each one costs: a card is a cold sweep of every family, so N cards are N sweeps of [the per-family price](/ghchronicle/api/cost/). **A README that is not at the root.** This is for other repositories, since GitHub shows a profile's README only from the root of `/`. The paths in the `` are relative to the README, so a `docs/README.md` points at `../generated/card.svg`. > **Private repositories** > > Without a config file, stars, forks, languages, traffic and repository names > come from public repositories only, while contribution, commit and pull > request counts are account totals, as GitHub reports them on your profile. > Up to v1.0.0, the Action's default configuration counted private > repositories. With `include-private: true` the card adds the stars, forks, > languages and traffic of your private repositories, and the layouts that list repositories > **publish their names**: `repo-list` and `summary` (the Action's default > layout) list them by default, and so does any layout given `top_repos` in > `card-fields`. To count them without naming them, pick the fields yourself and leave > `top_repos` out, for example `card-fields: stars,forks,followers,contributions`. ## Two things a hosted runner does not keep > **The token must be a personal access token** > > Traffic needs push access to *every* repository being collected, and the > automatic `GITHUB_TOKEN` has it only for the repository the workflow is > running in. It also has no `security_events`, no `read:packages`, and is not a > user, so nothing account-wide works with it. See [the > token](/ghchronicle/start/token/). **The state file does not survive between runs.** Each run therefore does the full stargazer walk again. On a small account that is a handful of calls; on an account with many stars, cache it: ```yaml - uses: actions/cache@v4 with: path: ~/.ghchronicle key: ghchronicle-state-${{ github.run_id }} restore-keys: ghchronicle-state- ``` and point `state_file` at `~/.ghchronicle/state.json` in the config. A `card` mode run reads a restored state file, which is what lets it skip the walk, and never writes one back: it delivers its points to the card and to no store, so nothing it learned may tell the next collection that a family is already done. What fills the cache is a `once` or `backfill` step. ## Without the Action The same thing by hand, if you would rather not depend on it: ```yaml - uses: actions/setup-go@v7 with: go-version: stable - run: go install github.com/jmrplens/ghchronicle/cmd/ghchronicle@latest - run: ghchronicle -config .github/ghchronicle.yaml -card profile.svg -card-only env: GITHUB_TOKEN: ${{ secrets.GHCHRONICLE_TOKEN }} ``` --- # The file One YAML file, every value expandable from the environment, and what each top-level block is for. Source: https://jmrplens.github.io/ghchronicle/configuration/ ```sh cp config.example.yaml config.yaml ``` `config.example.yaml` documents every option in comments. It is the file to copy, and it is kept in step with the code: adding a setting means adding it there too. ## The blocks ```yaml github: # the token, the reserve, the timeout targets: # which repositories groups: # which categories of metric are collected at all sinks: # where the points go every: # how often things run: a default, per group, per family heartbeat: # how often the sweep loop wakes, for a test run log: # level, format, and an optional rotating file state_file: # what a restart remembers backfill: # the bound, when run with -backfill ``` Only `github.token` and one of `targets.user`, `targets.orgs` or `targets.repos` are genuinely required, plus at least one sink. Everything else has a default. ## `${VAR}` expansion Every string value is expanded from the environment at start-up. `${GITHUB_TOKEN}` becomes the value of that variable, or an empty string if it is not set. ```yaml github: token: ${GITHUB_TOKEN} sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} ``` This is the whole reason the file can be committed. The configuration is the shape of the deployment and belongs in version control; the tokens are credentials and belong in an environment file with mode 600, or in a secret store. - /etc/ghchronicle/ - config.yaml world readable, version controlled - ghchronicle.env mode 600, never committed ## `github` ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 timeout: 30s # base_url: https://github.example.com/api/v3 # web_url: https://github.example.com ``` | Key | Default | Meaning | | -------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `token` | required | A classic or fine-grained personal access token | | `reserve_rate` | `500` | Calls never spent, so whatever else uses the token keeps working. A non-positive value falls back to the default | | `timeout` | `30s` | Per request. GraphQL over a large account can be slow. A Go duration; anything unparseable or not positive falls back to the default | | `base_url` | api.github.com | A GitHub Enterprise instance uses `https:///api/v3` | | `web_url` | derived from the API | The site the profile page is on, read by `achievements` without the token. Needed only when `base_url` is a proxy in front of the API | The reserve is scaled per bucket; see [rate limits](/ghchronicle/api/). ## `state_file` ```yaml state_file: /var/lib/ghchronicle/state.json ``` Six things, and deleting the file costs a different one for each: - `last_run`, when each family last ran. Without it every family is due at once, so the next sweep is a full one. - `first_saw`, when each repository was first seen. Without it the one-off full walk of the star history is done again. - `last_head`, the commit each repository was on when the dependency diff last ran. Without it the dependency changes in the gap are gone: the next sweep has the photograph and no diff. - `last_full`, when each family that normally reads what changed last read a whole page. Without it a family reads as due, so the next sweep reads them all whole. - `last_notified`, where the inbox window was cut. Without it zero asks for the whole inbox. - `last_event`, the newest event the feed had. Without it empty reads the whole feed. Five of the six cost only quota, because what is collected again is keyed by measurement, tags and timestamp and overwrites what is already stored. `last_head` is the one that loses something: the dependency changes between the head it held and the next one are read from a range that nothing can name once the head is gone. A run with `-card-only` writes none of the six. Its points reach [the card](/ghchronicle/card/) and no store, so a mark it left behind would make the next collection skip a family, or narrow a read, whose data went into a picture and nowhere else. It reads the file as any other run does. ### The other file a sweep remembers itself in ```yaml sinks: dedupe_file: /var/lib/ghchronicle/state-written.bin dedupe_horizon: 720h ``` | Key | Default | Meaning | | ---------------------- | ------------------------------------------- | -------------------------------------------------------------------------- | | `sinks.dedupe_file` | beside `state_file`, as `-written.bin` | The ledger of what has already been written. `off` disables it for every sink | | `sinks.dedupe_horizon` | `720h` | How long the ledger remembers a point nothing offers any more. A Go duration; anything unparseable or not positive falls back to the default | Losing it costs one sweep of rewriting and nothing else, which is exactly what a store that has been wiped and needs filling again wants. Both files want a persistent path: [only what changed is written](/ghchronicle/sinks/#only-what-changed-is-written). ## `backfill` ```yaml backfill: since: 2y ``` Only applies to a run started with `-backfill`, and is overridden by `-backfill-since`. See [backfill](/ghchronicle/how/backfill/). ## When it is wrong, it says so at start-up Configuration is validated before the first call is made, and the messages name the key and what it needs. | Message | Means | | --------------------------------------------------------------------- | --------------------------------------------------------------------- | | `github.token is empty and GITHUB_TOKEN is unset` | Exactly what it says | | `targets: set at least one of user, orgs or repos` | Nothing to collect | | `sinks: enable at least one of ...` | A run that collects and discards is almost never what anyone meant | | `every.families.: unknown collector` | The name is not a family. The message lists the ones that exist | | `sinks.influxdb: url and bucket are required` | Each sink validates its own required keys and says which | | `sinks.sql.dialect: "mysql" is not postgres, the only dialect so far` | The value is not one of the accepted ones, and the message lists them | > **A cadence for a family that does not exist is fatal** > > `every` is checked against the list of known collectors at start-up rather > than being ignored. A typo there would otherwise mean a family silently > running at its default forever, which has bitten twice. ## The rest - [Targets](/ghchronicle/configuration/targets/): which repositories, and the fork and archived defaults. - [Cadences](/ghchronicle/configuration/cadences/): the `every` block, and what `0` means. - [Logging](/ghchronicle/configuration/logging/): level, format and the rotating file. - [Choosing a store](/ghchronicle/sinks/): the `sinks` block, one page per store. --- # Build the configuration A form that writes the config.yaml a normal run takes and the workflow step the Action takes, from the binary's own list of settings. Source: https://jmrplens.github.io/ghchronicle/configuration/builder/ Answer what your deployment is, and this writes the three things that run it: the `config.yaml` a normal run reads, the command that runs it, and the step a workflow gives the composite Action. All three come from one set of answers, and none of them asks for a credential. ## Why it cannot drift The controls are not a copy of the settings. `cmd/gen_config` exports `internal/config`'s own types into a file the page reads, down to the default each key resolves to, which it measures by validating a probe rather than by restating a number. `make check-config-options` fails when that file is no longer what the code produces, in the analysis suite and in CI, so the form cannot offer a setting the binary does not have and cannot miss one it does. The other direction is checked too, and its limit is worth saying out loud. The configurations this form writes for the answer sets in `internal/config/testdata/config-cases.json` go through the real parser in `internal/config`'s own tests, including the shapes that have caught people out: a cadence per family under `every.families`, an `include_private` left off, a sink whose credentials are `${VAR}` references, a card-only run with no destination at all, and answers the parser is expected to refuse. Those cases are generated by the module this page runs, so they cannot be a copy of it, and every branch of that module that can change what it writes has one. What the corpus does not prove is behaviour no answer set produces, which is why the file that owns it lists what it cannot cover and why. ## What the form allows and the parser refuses A form is a form: a destination is a checkbox and the keys it cannot resolve without are not, so a few clicks are enough to write a file that dies at start-up. The page warns about the ones its own generated list of settings can see, above the outputs, and names the rest here. Refusing the rest in the form would mean keeping a second copy of a parser rule on this page, which is the drift the whole arrangement exists to avoid. - A destination switched on with a required key left empty is refused by name, with an example: `sinks.loki: url is required, for example http://loki:3100`. The form warns about this one. - A credential field answered with the credential instead of the name of an environment variable is left out of the file. The form warns about this one. - A file with nothing in it is refused: `the file is empty`. - No account at all is refused: `targets: set at least one of user, orgs or repos`. - `heartbeat: 0` is refused: it must be positive, and leaving the key out is how the loop's tick is derived instead. - A cadence that is not a duration is refused where it is used: `every.families.repo: time: invalid duration "soon"`. - `sinks.elasticsearch.api_key` filled in beside `sinks.elasticsearch.username` is refused: `sinks.elasticsearch: set either api_key or username and password, not both`. - Writing `-` as `sinks.sql.path` while `sinks.stdout` is on is accepted and gives two writers one stream; pick one of them. > **What the form does with a credential** > > Where a setting is a credential, the form asks for the NAME of an environment > variable and writes `${NAME}` into the file. That is the expansion the binary > performs at start-up, and it is what lets the file be committed. An answer > that is not a variable name is not written at all: the form says so and > leaves the key out, rather than folding a pasted token into a reference to a > variable that will never exist. One field is not covered by that, > `sinks.otlp.headers`, which is free text because a header is not always a > credential: what you type there is written as it stands, so put a `${VAR}` > reference in it. The Action's token is a repository secret, which is what the > step references. ## The form This page is a form that writes a configuration. What follows is the inventory it offers, which is generated from the binary's own types, and the three outputs it starts from. - **The account and its API** - `github`: a block of settings - `github.token`: string, required, a credential, like `${GITHUB_TOKEN}` - `github.base_url`: string, like `https://github.example.com/api/v3` - `github.web_url`: string, like `https://github.example.com` - `github.timeout`: string, defaults to `30s`, like `30s` - `github.reserve_rate`: int, defaults to `500`, like `500` - **What to collect** - `targets`: a block of settings - `targets.user`: string, like `your-github-login` - `targets.orgs`: list, like `some-org, another-org` - `targets.repos`: list, like `someone/one-repo` - `targets.exclude`: list, like `someone/experiment-*` - `targets.include_forks`: bool, defaults to `false`, like `false` - `targets.include_archived`: bool, defaults to `false`, like `false` - `targets.include_private`: bool, defaults to `true`, like `true` - **Where the points go** - `sinks`: a block of settings - `sinks.influxdb`: a block of settings - `sinks.influxdb.url`: string, required, like `http://localhost:8181` - `sinks.influxdb.token`: string, a credential, like `${INFLUX_TOKEN}` - `sinks.influxdb.org`: string, defaults to `default`, like `default` - `sinks.influxdb.bucket`: string, required, like `github` - `sinks.influxdb.batch`: int, like `5000` - `sinks.influxdb.exclude`: list, defaults to `gh_job_log`, like `gh_job_log` - `sinks.influxdb.dedupe`: bool, defaults to `true`, like `true` - `sinks.prometheus`: a block of settings - `sinks.prometheus.listen`: string, defaults to `:9605`, like `127.0.0.1:9605` - `sinks.prometheus.path`: string, defaults to `/metrics`, like `/metrics` - `sinks.prometheus.no_prime`: bool, defaults to `false`, like `false` - `sinks.otlp`: a block of settings - `sinks.otlp.endpoint`: string, required, like `http://collector:4318/v1/metrics` - `sinks.otlp.headers`: map, a credential, like `Authorization: Bearer ${OTLP_TOKEN}` - `sinks.otlp.service`: string, like `ghchronicle` - `sinks.otlp.raw`: bool, defaults to `false`, like `false` - `sinks.otlp.batch`: int, like `2000` - `sinks.otlp.repeat`: string, like `1m` - `sinks.loki`: a block of settings - `sinks.loki.url`: string, required, like `http://loki:3100/loki/api/v1/push` - `sinks.loki.tenant_id`: string, like `tenant-one` - `sinks.loki.labels`: map, like `job: ghchronicle` - `sinks.loki.batch`: int, like `1000` - `sinks.loki.max_age`: string, like `1h` - `sinks.file`: a block of settings - `sinks.file.path`: string, required, like `/var/log/ghchronicle/points.lp` - `sinks.file.format`: string, one of `influx`, `json` - `sinks.file.max_bytes`: int, like `67108864` - `sinks.file.keep`: int, like `5` - `sinks.stdout`: bool, defaults to `false`, like `false` - `sinks.stdout_format`: string, one of `influx`, `json` - `sinks.telegraf`: a block of settings - `sinks.telegraf.url`: string, required, like `http://telegraf:8186/telegraf` - `sinks.telegraf.username`: string, like `telegraf` - `sinks.telegraf.password`: string, a credential, like `${TELEGRAF_PASSWORD}` - `sinks.telegraf.batch`: int, like `5000` - `sinks.telegraf.dedupe`: bool, defaults to `true`, like `true` - `sinks.graphite`: a block of settings - `sinks.graphite.addr`: string, required, like `graphite:2003` - `sinks.graphite.prefix`: string, defaults to `github`, like `github` - `sinks.graphite.batch`: int, like `1000` - `sinks.graphite.dedupe`: bool, defaults to `true`, like `true` - `sinks.sql`: a block of settings - `sinks.sql.dialect`: string, defaults to `postgres`, one of `postgres` - `sinks.sql.path`: string, required, like `/var/lib/ghchronicle/points.sql` - `sinks.sql.max_bytes`: int, like `67108864` - `sinks.sql.keep`: int, like `5` - `sinks.sql.dedupe`: bool, defaults to `true`, like `true` - `sinks.elasticsearch`: a block of settings - `sinks.elasticsearch.url`: string, required, like `http://elasticsearch:9200` - `sinks.elasticsearch.prefix`: string, defaults to `ghchronicle`, like `ghchronicle` - `sinks.elasticsearch.username`: string, like `elastic` - `sinks.elasticsearch.password`: string, a credential, like `${ES_PASSWORD}` - `sinks.elasticsearch.api_key`: string, a credential, like `${ES_API_KEY}` - `sinks.elasticsearch.batch`: int, like `1000` - `sinks.elasticsearch.dedupe`: bool, defaults to `true`, like `true` - `sinks.dedupe_file`: string, defaults to `ghchronicle-state-written.bin`, like `/var/lib/ghchronicle/state-written.bin` - `sinks.dedupe_horizon`: string, defaults to `720h`, like `720h` - **How often** - `every`: a block of settings - `every.default`: string, like `15m` - `every.groups`: map, like `ci: 1m` - `every.families`: map, like `deps: 24h` - **The run's own log** - `log`: a block of settings - `log.level`: string, defaults to `info`, one of `debug`, `info`, `warn`, `error` - `log.format`: string, defaults to `text`, one of `text`, `json` - `log.file`: string, like `/var/log/ghchronicle/ghchronicle.log` - `log.max_bytes`: int, like `67108864` - `log.keep`: int, like `5` - **The run** - `heartbeat`: string, like `15s` - `groups`: list, like `audience, account, repos` - `state_file`: string, defaults to `ghchronicle-state.json`, like `/var/lib/ghchronicle/state.json` - `backfill`: a block of settings - `backfill.since`: string, like `2y` - **config.yaml** Save this beside the binary, or at the path you give to -config, and run it with the command below. ```yaml github: token: ${GITHUB_TOKEN} targets: user: octocat sinks: stdout: true ``` - **The command that runs it** ```sh ghchronicle -config config.yaml -once ``` - **Workflow step** These answers need settings no input carries, so the step reads the file above. Commit it at that path. ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} config: .github/ghchronicle.yaml ``` ## What to do with each output There is not one command, which is why the form writes it. A configuration that names a destination is run with `-once`. A configuration that names none is refused by `-once`, with the parser asking for a sink you deliberately did not want, so it is run with `-card-only` and a path for the card instead. The command above changes with the answers, and so does the step. The step goes in the `steps:` of a workflow job. Answers no input can carry make the step read the file, and the file has to be committed at the path the step names; answers the four inputs can carry are written into the step itself, and then there is no file at all. A step with no file that names no destination is a card run, `mode: card` with a `card:` path, because that is the one thing the Action turns into `-card-only`. The two outputs default the opposite way on one setting, and the step is written so it says which one you are getting. `include-private` is off unless you ask for it, because a card that counts private repositories names them in a public README; `targets.include_private` is on unless you say otherwise, because the token already reaches them. So a file-less step always writes the input out rather than leaving it to a default that means the opposite of the file beside it. See [GitHub Actions](/ghchronicle/install/actions/) for the workflow around it, and [the token](/ghchronicle/start/token/) for what the secret needs to be able to do. ## The rest - [The file](/ghchronicle/configuration/): what each block is for, and what happens when a value is wrong. - [Targets](/ghchronicle/configuration/targets/): which repositories. - [Cadences](/ghchronicle/configuration/cadences/): every family, its group and its built-in interval. - [Choosing a store](/ghchronicle/sinks/): one page per destination. --- # Targets Which repositories a sweep touches, and why forks and archived repositories are excluded by default. Source: https://jmrplens.github.io/ghchronicle/configuration/targets/ ```yaml targets: user: your-github-login orgs: [] repos: [] exclude: [] include_forks: false include_archived: false include_private: true ``` ## The keys | Key | Default | What it does | | ------------------ | ------- | --------------------------------------------------------------------- | | `user` | none | The account to collect. Its repositories are discovered automatically | | `orgs` | `[]` | Organisations to include as well | | `repos` | `[]` | Repositories to collect whatever the filters say | | `exclude` | `[]` | Shell globs, matched against `owner/name` | | `include_forks` | `false` | Whether forks are collected | | `include_archived` | `false` | Whether archived repositories are collected | | `include_private` | `true` | Whether private repositories are collected | At least one of `user`, `orgs` or `repos` must be set, or start-up fails with `targets: set at least one of user, orgs or repos`. `user` is also what makes the account-wide families possible. A configuration that names repositories alone has no login to hand to GraphQL, so the contribution calendar, the event feed, notifications, billing, packages and the outbound families are all skipped. ## Naming a repository overrides every exclusion `repos` is not another filter. It is a statement of intent, and it wins: naming `someone/thing` collects it even if it is a fork, even if it is archived, even if a glob in `exclude` would have matched it. ```yaml targets: user: acme repos: - someone-else/a-fork-i-actually-maintain exclude: - "acme/experiment-*" ``` `exclude` takes shell globs, so `someone/experiment-*` drops a whole prefix in one line. ## Why forks and archived are off by default Both defaults exist for the same reason, which is the rate limit. - **A fork's traffic is almost always zero.** GitHub reports views and clones per repository, and for a fork that nobody visits, that is fourteen days of zeroes per sweep. It costs four calls per repository to learn nothing. - **An archived repository cannot change.** Its stars can still move, but nothing else can, and collecting it spends the budget on rows that never move again. Turn either on when the assumption does not hold for you. A fork you actually develop in is a real repository with real traffic, and `repos` is the way to name that one without also collecting the forty bookmarks. Off is not invisible, on two counts. An archived repository still gets the one row it has, `gh_repo_archived`, dated the instant it was archived: the listing a sweep already pays for says which repositories are archived, and the `totals` family asks the date of all of them in one query at its own cadence, so the _Repositories archived_ table exists from the first sweep and costs one point per `totals` sweep after it. And a [backfill](/ghchronicle/how/backfill/) collects archived repositories in full whatever this key says, because their history is the account's history and one walk of it is enough. Forks stay as configured in both cases: an archived fork under `include_forks: false` has no row. > **Check the set before spending quota on it** > > ```sh > ghchronicle -config config.yaml -list > ``` > > That prints the repositories in scope and writes nothing. If something you > expected is missing, this is the command that tells you before a sweep spends > any quota on the wrong set. ## Private repositories `include_private` defaults to `true`, because a token that can see them was given deliberately and the point of the tool is to keep the history of the account, not of its public half. What is stored is the same measurements as for a public repository: counts, durations and names. Two things are worth knowing about what that means: - Repository and branch names appear as tag values, so they will be visible to anyone who can read the dashboard. - Only the _host_ of a webhook URL is stored, never the path, because the path usually carries a secret. Set it to `false` to collect only public repositories. ## The discovery cadence The repository list is rebuilt at most once an hour. Repositories are created rarely and listing them costs a page per hundred, so a new repository can take up to an hour to enter the sweep. Naming it in `repos` does not change that; restarting the process does. --- # Cadences The three layers of the every block, the built-in cadence of every family, the warning when a configuration speeds one up, and the heartbeat. Source: https://jmrplens.github.io/ghchronicle/configuration/cadences/ ```yaml every: default: 15m groups: ci: 1m feeds: 10m families: actions: 30s ``` ## The three layers Most specific wins. A family's own entry beats its group's, a group's beats `default`, and `default` beats the built-in value. Anything omitted falls through to the layer below it, so a file with only `families:` in it behaves exactly as it always did. The value is a Go duration: `30s`, `15m`, `2h`, `36h`. | Layer | Reaches | Written as | | ------------------ | ---------------------------------- | ----------------------- | | `every.families` | one family | `families: {keys: 24h}` | | `every.groups` | every family of one group | `groups: {ci: 1m}` | | `every.default` | every family neither above names | `default: 15m` | | the built-in table | every family none of the above name | nothing | They are three keys of a nested block rather than three flat ones because a flat map cannot hold this vocabulary. `security` and `account` are each both a family name **and** a group name, so `every: {security: 1m}` has two readings and no way to choose between them. Under `families:` the word is the family; under `groups:` it is the group; there is nowhere left for the question to arise. The same nesting is what makes `default` and `groups` safe as words: they are fields of a fixed structure, and an unknown key is refused at start-up, so no family can ever shadow a layer and no layer can ever shadow a family. > **The two broad layers never switch a family on** > > `default` and `groups` cannot reach a family whose built-in cadence is `0`. > `deps`, `history` and `joblogs` ship switched off and are enabled by naming > them under `families:` and by nothing else. `deps` alone is 1.8 MB of SBOM per > repository, and a default written to speed up the fast families must not also > switch on three families you never mentioned. It is the same rule `groups` > already follows. Turning it around gives the shortest way to collect a little: switch everything off with `default`, then name what you want back. ```yaml every: default: 0 families: traffic: 6h actions: 15m ``` ## Every family, its group and its built-in cadence The built-in values are not round numbers picked for tidiness. They came out of a costed audit, and the reason each one is what it is lives next to the number in the code. That is the source, and this table is pinned to it: a test in `internal/config` compares every group, every duration and every reason below against the code in both directions, so a family added, moved or re-costed without this table following it fails the build. `ghchronicle -groups` prints the same membership. | Group | Family | Default | Why that value | | ----------- | ------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `account` | `account` | `12h` | the contribution calendar changes once a day, and the whole family costs one GraphQL point | | `account` | `achievements` | `24h` | the badges on the public profile page, read from the page itself because no API lists them, and the distance to each badge's next tier from the API beside; a badge is earned over weeks and the day costs one page and some thirty GraphQL points | | `account` | `billing` | `6h` | GitHub updates the usage report a few times a day at most | | `account` | `history` | `0` | off until asked for by name: it walks every past year and the year so far, and the rows are idempotent | | `account` | `keys` | `24h` | an SSH or GPG key changes when somebody changes it, and what matters is its expiry date, not the hour it was noticed | | `account` | `outbound` | `12h` | stars given and work in other people's repositories move at the speed of a person | | `account` | `profile` | `12h` | packages, gists and social accounts, all of them edited by hand | | `account` | `totals` | `12h` | twice a day is plenty for a number that only grows | | `audience` | `forks` | `12h` | the whole list fits in one page, and a fork is a rare event | | `audience` | `stars` | `6h` | the full stargazer walk happens once; after that the newest hundred ride in one GraphQL query per ten repositories | | `audience` | `traffic` | `6h` | the fourteen-day window is rewritten whole each time, so a missed sweep repairs itself on the next one | | `ci` | `actions` | `15m` | a workflow run is over in minutes, and its queue time is only worth watching while it is happening | | `ci` | `artifacts` | `1h` | artifacts appear with the run that made them and expire on a scale of days | | `ci` | `deployments` | `1h` | the surface a delivery dashboard reads, and the newest page is cheap: one GraphQL point per five repositories | | `ci` | `joblogs` | `0` | off until asked for by name: it is text rather than a measurement, it costs a request per failure, and it only makes sense with a log store attached | | `collector` | `ratelimit` | `15m` | free, and worth having at the resolution of the busiest family | | `feeds` | `activity` | `30m` | the repository log holds a hundred entries, which covered twenty-six hours on the busiest repository measured | | `feeds` | `events` | `30m` | the feed keeps the last three hundred events whatever their dates, so this is the size of a window, not a speed | | `feeds` | `notifs` | `30m` | read notifications disappear quickly, so this is the size of a window, not a speed | | `repos` | `branches` | `24h` | branches are created and deleted all day, but the question the row answers is which are stale right now, which is a daily one | | `repos` | `deps` | `0` | off until asked for by name: the SBOM is 1.8 MB per repository and has its own budget of a hundred a minute | | `repos` | `inventory` | `24h` | four core requests per repository, for settings that change only when somebody changes them | | `repos` | `policyfiles` | `24h` | SECURITY.md, CODEOWNERS, dependabot.yml and FUNDING.yml move about once a quarter | | `repos` | `repo` | `1h` | stars, forks, languages and topics move slowly, and this is one request per repository | | `repos` | `rulesets` | `24h` | a ruleset is edited a few times a year, every version keeps its own date, and both requests answer 304 until somebody edits one | | `repos` | `settings` | `6h` | webhooks, rulesets, environments and deploy keys change only when somebody changes them | | `security` | `analyses` | `6h` | GitHub prunes code scanning analyses, and a repository produces a handful a day | | `security` | `security` | `1h` | an alert is something to act on today, and the list of open ones is short | | `work` | `commits` | `1h` | one request per commit, so the cost follows how much was pushed rather than how often this asks | | `work` | `discussions` | `2h` | a discussion is answered over hours or days, and few repositories have any | | `work` | `issueevents` | `1h` | the timeline of what moved in two cadences, one GraphQL point a repository, so the hour is how soon a transition is worth seeing | | `work` | `issues` | `1h` | one request per item, so the cost follows how much is open rather than how often this asks | | `work` | `planning` | `6h` | labels and milestones are edited by hand, a few times a week at most | | `work` | `stats` | `12h` | GitHub recomputes these slowly anyway, so asking more often returns the same numbers | ## The warning when a cadence is faster than the value is worth A broad layer is a cheap way to slow everything down and an expensive way to speed everything up. `default: 15m` asks GitHub for the account's SSH keys ninety-six times a day for a value that changes twice a year, and a group cadence does the same thing one level down: `work` holds families the audit measured at `1h` and at `12h`, so one number there is one number over six different answers. So every family a configuration collects **four times more often or more** than its built-in cadence is named at start-up, with the key that set it, both numbers, and the reason that number is what it is: ```text level=WARN msg="every.default sets keys to 15m against a built-in 24h, 96 times more often: an SSH or GPG key changes when somebody changes it, and what matters is its expiry date, not the hour it was noticed" level=WARN msg="every.groups.work sets stats to 1h against a built-in 12h, 12 times more often: GitHub recomputes these slowly anyway, so asking more often returns the same numbers" ``` It is a warning and never a refusal, and it is always per family, never per group. A line saying "the group `work` is too fast" would name nothing you can act on: the reason a cadence is what it is belongs to the family, so the warning has to as well. Four is the threshold because the built-in values are a ladder, `15m` `30m` `1h` `2h` `6h` `12h` `24h`, and the widest gap between two neighbouring rungs is three, from `2h` to `6h`. Four is therefore the smallest factor no single step down that ladder can reach. Moving one rung is a deliberate adjustment made by somebody looking at that family and stays quiet; four or more can only be a broad layer landing somewhere it was never chosen for, or a number typed without reading this table. Nothing warns for going slower. This is about waste, not about taste. ## Groups: collecting less than everything `every` sets how often a family runs. `groups` sets which families exist for this deployment at all. Omit the key and it collects for every group, which is the default and is what "every metric GitHub exposes" has always meant. ```yaml groups: [audience, account, repos, security] ``` Naming any group turns the rest off: their families are never requested and their measurements are never written. `ghchronicle -groups` prints the list, which is: | Group | Families | | ----------- | ------------------------------------------------------------------------ | | `audience` | `forks`, `stars`, `traffic` | | `account` | `account`, `achievements`, `billing`, `history`, `keys`, `outbound`, `profile`, `totals` | | `repos` | `branches`, `deps`, `inventory`, `policyfiles`, `repo`, `rulesets`, `settings` | | `work` | `commits`, `discussions`, `issueevents`, `issues`, `planning`, `stats` | | `ci` | `actions`, `artifacts`, `deployments`, `joblogs` | | `security` | `analyses`, `security` | | `feeds` | `activity`, `events`, `notifs` | | `collector` | `ratelimit` | The top-level `groups` and `every.groups` are different questions about the same eight names: the first decides whether a group is collected at all, the second how often. A cadence under `every.groups` for a group the top-level `groups` leaves out is legal and warns, rather than failing: narrowing a deployment should not also mean pruning an `every` block you tuned last year. The two axes both have to say yes, and only one of them can say no. A group that is not named switches its families off whatever `every` says, and naming a group never resurrects a family whose cadence is zero. That is the whole of how a family ships switched off while the default is everything. Writing `groups: []` is refused rather than read as "collect nothing": omitting the key is how you ask for everything, so an empty list can only be a mistake. Three surfaces cannot be recovered later, whatever you do afterwards, and they sit in three different groups: the event feed and read notifications (`feeds`), the fourteen-day traffic window (`audience`) and job logs, deleted after ninety days (`ci`). A day not collected for those is a day that does not exist. Everything else can be filled in later with `-backfill`. > **Panels for a group you switched off will show an error** > > A measurement that is never written does not exist as a table, and InfluxDB 3 > answers a query naming a table it does not have with an error rather than with > no rows. The dashboards are generated once, for the full set of metrics, so > every panel fed by a group you turned off shows that error. It is expected. > They are a demonstration of what the collector can chart, not a view that > reshapes itself around your configuration. PostgreSQL and Elasticsearch > behave the same way; Prometheus and Graphite have no schema to miss, so the > same panels there read No data instead. The dashboard sections are not aligned to the groups either, so a group switched off empties some panels of several sections rather than one section cleanly. `Stars and forks` reads `gh_repo` from `repos` as well as `gh_star` from `audience`; `Code` reads `gh_workflow_run` from `ci` and `gh_repo_activity` from `feeds` alongside its own commits; `Inventory` reads from `repos` and `account`, and its licence panels read `gh_dependency_license`, which is family `deps` and ships switched off whether or not `repos` is selected. ## What `0` means `0` switches off every family the layer it is written on reaches. It is not "as often as possible" and not "use the default": those families never run, write nothing and cost nothing. ```yaml every: families: billing: 0 # no billing data at all artifacts: 0 # no artifact rows ``` Three families ship off and are enabled by giving them any duration, under `families:` and nowhere else: - **`joblogs`** is the tail of every failed job's log. It is text rather than a measurement, it costs a request per failure, and it only makes sense with a log store attached. The InfluxDB sink excludes it by default. - **`history`** walks every past year's contribution calendar, one GraphQL point per year, back to the day the account was created, and the year in progress from January to now. The past years never change and the rows are idempotent; the current year's row is a snapshot marked `partial`, so a daily cadence keeps it current and a single sweep leaves it frozen on the day it ran. - **`deps`** is the dependency graph. The SBOM is 1.8 MB per repository and has its own budget of a hundred a minute. A configuration that leaves nothing at all enabled says so at start-up rather than running an empty loop in silence. > **An unknown name is fatal at start-up** > > ```text > every.families.trafic: unknown collector (known: account, achievements, actions, activity, ...) > every.families.feeds: "feeds" is a group, not a family; every.groups.feeds is where a whole group's cadence lives > every.groups.actions: "actions" is a family, not a group; it is in group "ci", and every.families.actions is where its cadence lives > ``` > > Every name is checked against the known collectors and the known groups before > the first API call. A typo would otherwise mean a family silently running at its > default forever, and a name written under the wrong layer is told which layer it > belonged under. ## `heartbeat`: the loop's tick, which is not a cadence `heartbeat` forces how often the sweep loop wakes to ask which families are due. It gives no family an interval, it is compared against nothing in the table above, and it earns none of the warnings on this page. ```yaml heartbeat: 15s ``` Omit it and the loop ticks at the shortest cadence configured, held between one minute and one hour, which is what a real deployment wants: a family that runs every quarter of an hour is not delayed by one that runs every twelve hours. The upper end matters to a configuration whose cadences are all slower than an hour: the search for the shortest one starts at an hour, so the loop still wakes hourly and finds nothing due. Set it when you want the loop itself under control, which is almost always a test run. It is the only way to turn the loop faster than that one minute floor, because the floor exists to survive a mistyped cadence and an explicit heartbeat is not one. A heartbeat **longer** than the shortest cadence holds that family back, and start-up says so: ```text level=WARN msg="heartbeat is 1h and the shortest cadence is 15m (actions), so no family can run more often than every 1h" ``` ## Making them slower If the budget is tight, lengthen `artifacts` and then `actions`. They are the only two whose cost scales with how busy the repositories are rather than with how many there are. See [cost of a sweep](/ghchronicle/api/cost/). The symptom of cadences that are too fast is a warning every sweep: ```text level=WARN msg="rate limit reserve reached, family skipped" family=actions ``` ## Cadence is not resolution Lengthening a cadence does not coarsen the history, because the points are dated by the thing that happened rather than by the sweep. Collecting workflow runs every hour instead of every fifteen minutes still records each run at the second it finished. What a longer cadence risks is missing a window entirely: `events`, `notifs` and `activity` are windows rather than speeds, as the table above says, and nothing else here is. --- # Logging Level, format, the rotating file that never replaces standard error, and the two lines worth alerting on. Source: https://jmrplens.github.io/ghchronicle/configuration/logging/ ```yaml log: level: info # debug, info, warn, error format: text # text or json file: /var/log/ghchronicle/ghchronicle.log max_bytes: 67108864 keep: 5 ``` ## The keys | Key | Default | Meaning | | ----------- | ---------- | --------------------------------------------- | | `level` | `info` | `debug`, `info`, `warn` or `error`; anything else behaves as `info` | | `format` | `text` | `text` for a human, `json` for a shipper; anything else is `text` | | `file` | none | A rotating file **as well as** standard error | | `max_bytes` | `67108864` | Rotate at 64 MiB. Anything not positive falls back to the default | | `keep` | `5` | How many rotated files to keep. Anything not positive falls back to the default | ## Both, never instead A configured `file` does not replace standard error, it is written in addition to it. Under systemd the journal is where anyone looks first, and a log file that silently took the journal's place would be a trap: `journalctl -u ghchronicle` would go quiet and the obvious conclusion would be that the service had stopped. Rotation is by size with numbered suffixes, so retention is a count rather than a date and two rotations in the same second cannot collide. The size counter is read from the file at start-up, so a restart does not reset it and let the file grow without bound. ## What the log says on a good day ```text level=INFO msg="repositories discovered" count=18 level=INFO msg=written sink=influxdb family=traffic points=629 level=INFO msg="rate budget" bucket=core remaining=4354 limit=5000 ``` A family that is not due yet simply does not appear. That is normal, and it is the first thing to check when a measurement seems to be missing: with a twelve-hour cadence, half a day of logs can legitimately never mention `account`. ## The two lines worth an alert **`rate limit reserve reached`** means a family was skipped to protect the budget. ```text level=WARN msg="rate limit reserve reached, family skipped" family=artifacts ``` Once is fine. Every sweep means the cadences are too fast for the number of repositories. **`family failed everywhere, not marking it as run`** means every repository failed for one family. ```text level=WARN msg="family failed everywhere, not marking it as run" family=security ``` The family is deliberately not marked as done, so it is retried at the next cadence rather than being treated as complete. That is the line that separates "a feature is off on one repository" from "the token lost a scope". ## Debugging ```yaml log: level: debug ``` Debug adds three lines, and no more: how many points the written-points ledger came back with at start-up, that the account-wide families were skipped because `targets.user` is unset, and how many entries a sink left out for being older than its horizon. That last one is the only way to see that a push was trimmed rather than rejected. There is no per-request log: the client in `internal/ghapi` carries no logger, so which endpoint was called and which answer came back 304 are not visible at any level. > **JSON for a shipper, text for a person** > > `format: json` emits one object per line, which is what Promtail, Vector or > Filebeat want. It is the same information; only the encoding changes. > Combining `format: json` with a `file` is the arrangement for a host that > already ships logs somewhere. ## Not to be confused with the job log collector `log` is the tool's own diary. `every.joblogs` is a _collector_: it fetches the last forty lines of every failed GitHub Actions job and turns them into points. They are unrelated settings, and the second one belongs in a log store rather than in a metrics database. See [Loki](/ghchronicle/sinks/loki/). --- # The sweep What one pass over GitHub does, in what order, and why each family has its own cadence. Source: https://jmrplens.github.io/ghchronicle/how/ A sweep is one pass over every family whose interval has elapsed. It is an increment, not a rebuild: it asks for the little that can have changed since last time, writes what it got to every configured store, and records when each family ran. ## The shape of one pass ```mermaid %% Generated by site/scripts/gen-figures.mjs from internal/config/config.go and internal/run/runner.go flowchart TD T["Timer"] --> D["Discover repositories
(rebuilt hourly)"] D --> A["Account-wide families
account, totals, ratelimit, events, notifs,
billing, profile, outbound, history,
achievements, keys"] D --> R["Per-repository families
traffic, repo, branches, stars, issues,
issueevents, actions, artifacts, security,
stats, discussions, commits, activity,
analyses, forks, planning, joblogs, settings,
rulesets, inventory, deployments, policyfiles,
deps"] A --> B{"Budget above
the reserve?"} R --> B B -- "no" --> S["Skip the family
and warn"] B -- "yes" --> C["Collect"] C --> P["Dated points"] P --> H["Destinations that keep the date
InfluxDB, file, stdout, Telegraf, Graphite,
PostgreSQL, Elasticsearch"] P --> L["Loki
the twenty-two event renderings"] P --> RD["Reducer
current values"] RD --> G["Prometheus
OTLP, when raw: false"] C --> M["Mark the family as run
in the state file"] ``` Two things in that diagram are the whole design. Every collector produces _dated_ points, and the stores that cannot hold a date get them reduced to current values first, by the same reducer, before they ever see the data. That is the subject of [dating a point](/ghchronicle/how/dating/). ## Why families and not one interval The surfaces move at wildly different speeds. Workflow runs finish every few minutes on a busy account. The contribution calendar changes once a day. The list of forks changes a few times a year. One interval for all of them would either waste the rate limit on the slow ones or lose the fast ones, so each family carries its own. | Family | Default | Collects | | --------------- | ------- | -------------------------------------------------------------- | | `actions` | 15m | Workflow runs, jobs, steps, the Actions cache | | `ratelimit` | 15m | What the collector has left to spend, in each budget | | `activity` | 30m | The repository activity log, where a force push is recorded | | `events` | 30m | The account event feed, which keeps only the last 300 | | `notifs` | 30m | The notification inbox | | `artifacts` | 1h | Artifacts and their expiry | | `commits` | 1h | Lines changed and signature state, per commit | | `deployments` | 1h | Deployments and their environments, batched over every repository | | `issueevents` | 1h | The timeline of what moved: labels, assignments, transitions | | `issues` | 1h | Pull requests, issues and reviews, per item | | `repo` | 1h | Stars, forks, languages, topics, releases, rulesets | | `security` | 1h | Dependabot and code scanning alerts | | `discussions` | 2h | The forum half of a repository | | `analyses` | 6h | Code scanning analyses, which GitHub prunes | | `billing` | 6h | Usage per day, product, SKU and repository | | `planning` | 6h | Labels and milestones | | `settings` | 6h | Webhooks and their deliveries, environments, deploy keys | | `stars` | 6h | The stargazer walk once, then the newest hundred | | `traffic` | 6h | The whole 14-day window, rewritten | | `account` | 12h | Profile, contribution calendar, contribution totals | | `forks` | 12h | Who forked, and when | | `outbound` | 12h | Stars given, and work in other people's repositories | | `profile` | 12h | Packages, gists, social accounts | | `stats` | 12h | Commits per week, the punch card, the workflow definitions | | `totals` | 12h | The lifetime numbers, asked of GitHub rather than added up here | | `achievements` | 24h | The profile badges, and the distance to each next tier | | `branches` | 24h | Which branches are live and how stale each tip is | | `inventory` | 24h | What a workflow's own token may do, both secret stores, default code scanning | | `keys` | 24h | The account's SSH and GPG keys, and when each expires | | `policyfiles` | 24h | SECURITY.md, CODEOWNERS, dependabot.yml and FUNDING.yml | | `rulesets` | 24h | Every version of every ruleset's changelog | | `deps` | off | The dependency SBOM of each repository, and what changed | | `history` | off | Each year's contribution calendar, the current one included | | `joblogs` | off | The tail of every failed job's log | Setting any of them to `0` switches it off entirely. See [cadences](/ghchronicle/configuration/cadences/). ## Discovery The repository list is rebuilt at most once an hour. Repositories are created rarely and listing them costs a page per hundred, so anything shorter spends quota to learn nothing. Forks and archived repositories are excluded by default, for the reason set out in [targets](/ghchronicle/configuration/targets/). ## Failure is per repository, not per sweep A family that fails on one repository is logged and skipped; the sweep continues. This matters more than it sounds, because "failure" here is usually a feature being switched off: of fifty repositories, most have Dependabot off, and each of them answers 403. Treating that as an error would lose the other forty-nine. There is one exception, and it is deliberate. A family where _every_ repository failed is not marked as run. Marking it would hide the outage until the next cadence, which for the twelve-hour families is half a day. ```text level=WARN msg="family failed everywhere, not marking it as run" family=security ``` ## The state file `state_file` holds [six things](/ghchronicle/configuration/#state_file), and the two a sweep is judged by are when each family last ran and when each repository was first seen. The second is what makes the one-off full walk of the star history happen once instead of on every sweep. It is written through a temporary file and renamed, so a crash mid-write cannot leave a truncated state that would trigger a full re-collection. > **Three runs collect every family whatever the state says** > > An exporter holds its samples in memory, so a restart empties it and it stays > empty until each family's cadence comes round, which for the twelve-hour ones > is half a day of a dashboard reading zero. Paying for one full sweep is the > cheaper mistake, so the first sweep after start-up runs every enabled family > whatever the state file says. A [backfill](/ghchronicle/how/backfill/) does > the same, because reaching as far back as GitHub allows is the whole point of > asking for one, and so does a run drawing [a card](/ghchronicle/card/), > because every number on the card comes from that one sweep and a family > skipped as not due would be a zero on the picture. ## The brake Before each family the collector checks the three rate buckets it actually spends from. If any of them is at or below its reserve and the window has not reset yet, the family is skipped and a warning is logged rather than the budget being spent to the last call. See [rate limits](/ghchronicle/api/). A [backfill](/ghchronicle/how/backfill/) is the opposite intention: it waits for the window to turn over instead of skipping. --- # Dating a point Every point carries the moment the thing happened, and that single rule decides what the whole project can answer. Source: https://jmrplens.github.io/ghchronicle/how/dating/ **A point carries the date the thing happened, not the date it was collected.** That is the one rule everything else follows from. A workflow run is stamped when it finished. A star is stamped when it was given. A traffic day is stamped at that day's own date. A pull request is stamped when it closed. None of them is stamped at the instant the collector noticed. ## The three kinds of point Not everything GitHub reports has a date of its own, so there are three treatments and each measurement declares which one it gets. | Kind | Stamped at | Example | | --------- | ----------------------------- | --------------------------------------------------------------- | | **Dated** | the moment the thing happened | `gh_star`, `gh_workflow_run`, `gh_traffic`, `gh_commit` | | **Daily** | the start of the UTC day | `gh_traffic_referrer`, `gh_label`, `gh_milestone`, `gh_webhook` | | **Now** | the instant of the sweep | `gh_repo`, `gh_actions_cache`, `gh_dependabot_alert` | Daily is for a snapshot with no date of its own. GitHub returns the top ten referrers of the trailing fourteen days as one list with no day attached, so it is not a series. Stamping it at the instant of the sweep would write four copies a day and any query that summed them would report four times the traffic. Stamping it at the start of the UTC day means a day's sweeps rewrite one row. Now is for something that is genuinely a current state. The size of the Actions cache, the number of open alerts, the inventory of workflows: none of these happened at a moment, so pretending otherwise would be a lie with a timestamp on it. ## Why the rule exists: re-collection has to converge InfluxDB keys a point by measurement, tag set and timestamp. Three fields, one row. Write the same three again and the row is replaced, not added to. Written out, two sweeps six hours apart offer the same day twice, and the second replaces the first because the three keys are identical: ```text gh_traffic,owner=acme,repo=telemetry,kind=views count=220i,uniques=131i 1757203200000000000 gh_traffic,owner=acme,repo=telemetry,kind=views count=238i,uniques=140i 1757203200000000000 ``` ```text gh_traffic,owner=acme,repo=telemetry,kind=views count=238i,uniques=140i 1757203200000000000 ``` One row, carrying the newest number GitHub reported for that day. Stamp those two lines at the moment of collection instead and they are two rows, and every sum over them is wrong by however many sweeps have run. That is what makes the whole design work. GitHub's traffic window is fourteen days, and the collector rewrites _all fourteen_ on every sweep rather than trying to work out which day is new: GitHub keeps 14 days of traffic and the collector re-reads all of them 4 times a day. Dated as built, those 4 sweeps land on the same 14 rows and the newest count replaces the one before it. Stamped at the moment of collection, each sweep adds what it read to what is already there. | After a week, per repository | Dated, as built | Stamped at collection | | --- | --- | --- | | Rows written per sweep | 14 | 14 | | Rows in the store | 14 | 392 | | Copies of each day | 1 | 28 | | A sum of the month's views | the traffic | 28 times the traffic | The same property is what makes a backfill safe to run twice, and what lets you delete the state file without corrupting anything: re-collection rewrites rows it has already written. > **Every history store has this property** > > It is not InfluxDB-specific. The SQL sink's primary key is `(time, tag > columns)`, which is the same series key spelled as a constraint, and its > inserts end in `ON CONFLICT ... DO UPDATE` rather than `DO NOTHING`. > Elasticsearch derives the document id from the measurement, the tags and the > timestamp, and indexes rather than creates. Graphite writes into the whisper > slot the timestamp names. All four converge for the same reason. ## What Prometheus structurally cannot hold Prometheus stamps a sample at scrape time. It does not take a timestamp from the producer, and it rejects anything meaningfully older than now. This was measured rather than assumed. Against **Prometheus 3.14**, with `--web.enable-otlp-receiver` and `out_of_order_time_window: 30m`, a sample dated two days back comes back as **HTTP 400**. Half of what this collects is older than that on purpose: a star from 2020, a pull request merged in July, yesterday's traffic. So there is no configuration of Prometheus in which the dated history survives. Widening the out-of-order window moves the boundary; it does not remove it. > **Do not give the exporter timestamps** > > The obvious "fix" is to have the exporter emit each sample with the point's > own timestamp. Prometheus's exposition format allows it, and Prometheus will > refuse the ones that matter. The exporter deliberately drops the timestamp, > and the reduction below is why that is not a loss. ## The reducer, and what it makes of each measurement Because the store cannot hold the history, the reduction happens _before_ Prometheus or an OTLP backend with `raw: false` ever sees the data. `Summarize` gives each measurement one of four rules. | Rule | What it does | Used for | | ---------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `keepLast` | The most recent point per label set wins | Snapshots: `gh_repo`, `gh_account`, `gh_release` | | `sum` | Every point in the batch is added up | Windows: views over the fourteen days | | `count` | The points become a count plus the mean of each numeric field | Dated items: pull requests become "how many merged" and "how long they took" | | `skip` | Nothing is served | History with no honest current value | A measurement with no rule is skipped rather than guessed at. That is the safe default, and it is deliberate: without it, a new collector could quietly flood an exporter with one series per star. `count` averages each of their numbers except the identifiers. A field that joins one row to another, `run_id`, `workflow_id`, `pull_request`, `number`, `stack` and the rest, is a name and not a quantity: averaged over a count it becomes a number shaped exactly like the identifier it is made of and belonging to nothing, and `gh_deployment` published `run_id_mean` that way. Those fields are left out of the reduction, and so are the markers a point carries only so that it has a field at all, whose mean is 1.0 for ever. The reducer also publishes `total`, a running count of distinct items seen per series. That is what lets a Prometheus dashboard answer "per day" at all, through `increase()` over a monotonic counter, since it has no rows to count. ### The ten that are never served Ten of the measurements carry `skip`, so a Prometheus exporter and an OTLP backend with `raw: false` never see them. Seven are history, two are size, and one is text: | Measurement | Why | | -------------------------- | ------------------------------------------------------------------------------------- | | `gh_artifact` | History. One row per artifact ever, and none of them moves again | | `gh_commit_punchcard` | Size. One series per repository, weekday and hour | | `gh_commits_week` | History. The weekly commit series | | `gh_contribution_day` | History. The green calendar, one row per day | | `gh_contribution_day_repo` | History. The same calendar split per repository, which would mint a series per day | | `gh_job_log` | Text, not a number. It belongs in a log store | | `gh_package_version` | History. The publication date of every tag; the count of them is a field on `gh_package` | | `gh_release_asset` | Size. One series per file ever published | | `gh_traffic_path` | History. The per-day paths | | `gh_workflow_step` | History. The per-step timings | The two skipped for size are the ones worth knowing about, because they are real numbers rather than history: measured, together they were four fifths of the exporter's entire output. Both are drawn properly by the InfluxDB dashboard, and `gh_release` keeps the per-release download counts the assets were being read for. ## Which store keeps what | Store | The dated history | Why | | --------------------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- | | InfluxDB, PostgreSQL, Graphite, Elasticsearch | yes | The timestamp is part of the identity of a row | | Telegraf | as far as its outputs allow | It forwards the timestamps unchanged; an output that stamps at receipt loses them | | File and stdout | yes | The timestamp is in the line | | Loki | recent events only | Loki refuses an entry too far behind the newest in its stream. See [Loki](/ghchronicle/sinks/loki/) | | OpenTelemetry | the backend decides | OTLP data points carry an explicit timestamp; whether it is honoured is not up to this tool | | Prometheus | no | Current values only, by the rules above | ## Three consequences worth knowing **A tag is a series, a field is a value.** Anything unbounded goes in a field. The Actions runner name looks like a good tag until you notice a hosted runner is named uniquely per run (`GitHub Actions 1000163135`), which would create a series for every job ever executed. It is a field. **Weekly rows are anchored to the week, not to today.** `gh_commits_week` is stamped at the Sunday that starts each week. A sweep on Tuesday and one on Friday have to land on the same row, or every re-read writes a second copy of the year. **Prometheus reserves some tag names.** A tag called `job` or `instance` collides with the scrape labels, and the OTLP receiver overwrites it with the service name. Workflow jobs are therefore tagged `job_name`. --- # Backfill One deliberate walk to the end of every surface, how far back it goes, and the three things no backfill can reach. Source: https://jmrplens.github.io/ghchronicle/how/backfill/ ```sh ghchronicle -config config.yaml -backfill ``` A sweep is an increment. A backfill is a walk. They are opposite intentions and the tool treats them as such. ## The difference in one table | | Sweep | Backfill | | --------------------------- | --------------------------------- | -------------------------------------------------- | | Pages per collector | the collector's own small default | until the API runs out, or the bound is reached | | When the reserve is reached | skip the family and warn | wait for the window to reset, then carry on | | Which families run | those whose interval has elapsed | every enabled family, whatever the state file says | | How often | on a schedule, forever | deliberately, usually once | A normal sweep must never block. It protects the reserve so whatever else uses the same token keeps working, and it skips a family rather than sleeping. A backfill is run on purpose and the only thing that matters is that it finishes, so it parks until the budget is whole again. A backfill that gives up half way has spent the expensive part of the budget and keeps only the families it got to the end of: each one is written and marked as soon as it finishes, so the rest is what has to be run again. ## The cooldown When a bucket is at or below its reserve, the backfill waits. The wait is not guessed: every GitHub response says exactly when its window resets, so the collector sleeps until that instant plus a second of slack, and logs what it is doing. ```text level=INFO msg="rate limit reserve reached, waiting for the window to reset" family=commits wait=23m11s ``` Two guards on that wait. It is capped at one hour, so a clock skew or a stale header cannot turn into an unbounded sleep, and it has a floor of one second, so it cannot become a busy loop. A cancelled context ends the wait immediately, which is what makes `Ctrl+C` work during an overnight backfill. ## How far back `-backfill-since` on the command line, or `backfill.since` in the configuration file. Four spellings, because people reach for different ones: | Value | Means | | ------------------------- | ------------------------ | | `2024-01-01` | that date | | `90d` | ninety days ago | | `2y` | two years ago | | `720h` | a Go duration before now | | empty, `all`, `unlimited` | no bound at all | No bound means the walk stops only where the API does, however many hours or days that takes, pausing at every rate limit reset along the way. ```sh ghchronicle -config config.yaml -backfill -backfill-since 2y ``` ## Run it once, first A new install should run a backfill before, or right after, starting the service. A sweep's first pass is a wider increment, not a history: a month of workflow runs, the star history in full, and the newest page of everything else. The points are dated, so every store is fine with that; a dashboard at ninety days or two years is not, because the history it draws begins on the day the collector was installed. Measured after a day of sweeps and no backfill, over repositories holding hundreds of pull requests each: pull requests, about a fifth of what the repositories report, because a sweep reads one page of fifty per repository however many it holds; issues, about half; commits, well under a tenth and none older than thirty days; jobs and steps for a tenth of the workflow runs, so the queue wait, the slowest jobs and the steps that fail were computed over that tenth. At two years, _Pull requests merged_ read a fifth of what _Pull requests merged, ever_ said. Stars, forks, releases, deployments and the alerts were complete, because the first sweep walks those to the end anyway. ## What it reaches that a sweep does not - The whole commit history, rather than the last page. This is the one family where a backfill is qualitatively different rather than merely wider: without it the lines-changed series begins on the day you installed the collector. - Two years of workflow runs, and every run expanded into its jobs and steps. - The archived repositories, in full, whatever `include_archived` says. Their history is the account's history and it never moves again, which is exactly why a sweep leaves them out and why one walk of them is enough. Forks stay as configured. A sweep still writes the one row each archived repository has, the date it was archived: the listing it already pays for says which, and one query per `totals` sweep says when. - Every artifact page, twenty pages of repository activity, and the code scanning analyses. - A hundred pull requests and issues per repository. - The read notifications as well as the unread. - Twenty-four months of billing. - A hundred webhook deliveries per hook. ## What it does not switch on A backfill runs every enabled family whatever the state file says. It enables none of them. The three families that ship with a cadence of `0`, `deps`, `history` and `joblogs`, stay off unless they have been given one by name under `every.families`, and `-backfill` does not change that. `history` is the one that surprises people, because walking every past year of the contribution calendar is exactly what somebody asking for "the whole history" has in mind. Give it a cadence first: ```yaml every: families: history: 24h ``` See [cadences](/ghchronicle/configuration/cadences/) for why `default` and `groups` cannot switch these three on either. ## Two endpoints that needed their own handling Both were found by running it, not by reading documentation. **Dependabot refuses page numbers.** It answers an error to `page=` outright and pages by cursor instead, so the alert walk is written against cursors. **The GraphQL gateway gives up on a hundred pull requests.** Asking for a hundred pull requests with their reviews in one query answers an **HTML 502** after about ten seconds. The pull request walk halves its page size and retries on the same cursor, silently: the collectors carry no logger, so a backfill of a busy repository shows this only as a slower family, never as a line. > **Three things cannot be backfilled at any price** > > No amount of waiting changes these, and they are the reason the project exists > at all. > > - **The event feed keeps three hundred events**, whatever their dates. Past > that ceiling GitHub answers 422 "pagination is limited for this resource", > which the collector reads as the end of the data. > - **Traffic is fourteen days.** Anything older was never stored by GitHub. > - **Job logs are deleted after ninety days** and answer 410 afterwards, while > the run metadata they belong to survives for years. ## Running one safely It is idempotent. Points are keyed by measurement, tags and timestamp, so a backfill run twice rewrites the same rows rather than doubling them, in every store that keeps the history. What it costs is API quota and time. Two things worth doing first: run `-list` to confirm the repository set, and check that the store you are writing to is the one that keeps dates. Backfilling into Prometheus collects a great deal of history and then reduces all of it to a single current value. --- # What is collected The thirty-four families, what each one asks GitHub for, and the reason each exists. Source: https://jmrplens.github.io/ghchronicle/collectors/ Thirty-four families, ninety-one measurements. This page is what each family is _for_; the [measurements reference](/ghchronicle/collectors/measurements/) is every tag and field. The families are also grouped, and `groups:` in the configuration switches a whole area on or off. The binary prints the grouping it actually uses, which is the one to trust: ```sh ghchronicle -groups ``` ```text account the account itself: its lifetime numbers, its profile, its keys, its spending and what it does in other people's repositories account, achievements, billing, history, keys, outbound, profile, totals audience who is looking at the projects, who starred them and who copied them forks, stars, traffic ci continuous integration and deployment: runs, jobs, steps, artifacts, caches and deployments actions, artifacts, deployments, joblogs ... ``` ## Audience **`traffic`** collects the only data GitHub genuinely throws away. Views and clones live for exactly fourteen days and then cease to exist anywhere. The whole window is re-read and rewritten on every sweep, each day stamped with its own date, so a collector that was off for a week loses nothing as long as it comes back inside the window. Referrers and popular paths are different: the API returns a top-ten snapshot with no dates at all, so they are stamped at the start of the UTC day and read as "who was sending traffic when we asked". **`stars`** reconstructs the star curve from its beginning. The stargazers endpoint returns a `starred_at` per user when asked with the star media type, so the entire history is available on the first run: a chart that goes back years, not one that starts the day the collector was installed. After the first sweep only the newest hundred are read, since new stars land at the end, and they are read for every repository at once in one GraphQL query per ten. That is the difference between one point and 280 calls for a repository with 28,000 stars. **`forks`** collects who forked and when. The repository snapshot carries a fork count, which says how many but never when or by whom. The list is walked through REST on the first sweep of a fresh install and in a backfill; after that the newest hundred of every repository ride in the same kind of batch the stars do. A fork row is not static the way a star is: it carries the fork's own stars and how long since it was pushed, so a repository the batch reports holding more than a hundred forks is walked through REST as well, which refreshes those on up to five hundred forks as it always did. ## Repositories **`repo`** collects what a repository is right now, plus the things that accumulate on GitHub's side: languages by bytes, topics, community health, and release downloads **per asset**, which is what tells a Linux build from a macOS one. Bytes per language matter because the dominant-language label cannot show a repository shifting from one language to another over time. **`settings`** collects the configuration that changes, and how well the parts of it that talk to the outside world are working. One thing here is a real time series rather than a snapshot: webhook deliveries carry a status code and a latency. **`rulesets`** collects the changelog of every ruleset, one row per saved version with an actor and a date, which is the only record GitHub keeps of the moment a protection was turned off. The ruleset itself, what it enforces and who may walk past it, is a daily snapshot in `repo`; this is the history that snapshot's `days_since_change` only summarizes. **`branches`** collects the live branch list, one row per branch carrying the age of its tip. Nothing else answers "which branches were abandoned": GitHub's own list sorts by name and forgets, so a branch whose last commit is four months old looks exactly like one pushed this morning. It deliberately does not ask which pull requests point at a branch, which is what makes that query expensive; the join belongs in the panel. **`inventory`** collects the three per-repository policy surfaces that change on a scale of months: what the `GITHUB_TOKEN` of a workflow is allowed to do, how old every stored secret is, and whether code scanning is switched on by GitHub rather than by a workflow of its own. Four core requests per repository a day. All three are settings rather than events, so they are stamped at the start of the UTC day and a change reads as the day the value moved. **`policyfiles`** records which governance files a repository carries, `SECURITY.md`, `CODEOWNERS`, `dependabot.yml` and `FUNDING.yml`, and when each of them last changed. Whether most of them exist today is already in `gh_repo_policy`; when they changed is nowhere else, and `.github/dependabot.yml` is in no other measurement at all, which is what makes "is this repository receiving dependency updates by either route" a question the data can answer. > **Webhooks fail silently** > > Measured, one hook had been answering 403 for seventy-eight of its last > hundred deliveries and nothing anywhere said so. Only the host of a webhook > URL is stored; the path usually carries a secret. ## Development **`issues`** collects pull requests and issues one by one, not as counts. A count of open pull requests says nothing about how the work actually goes; the interesting numbers are durations. How long until someone reviewed it, how long until it merged, how big the diff was, how many review rounds it took. One GraphQL query per repository covers both: a sweep walks what was updated in the last two cadences, ten at a time, and once a day reads a whole page sized to the repository, which is the only read that rewrites an open pull request nobody touched. **`commits`** collects the commit history with its size and its signature. This is what replaces `stats/code_frequency`, which answers 202 with an empty body forever on a personal account. GraphQL gives lines added and removed per commit, attributed to an author and dated to the commit rather than to a week, and the signature comes with it for the same query. **`issueevents`** collects the transitions rather than the state. `issues` says what a pull request ended up as; this says when it was labelled, closed, reopened, renamed or had a review requested. A reopening exists in no other measurement. The repository-level list, `/issues/events`, is the reference for what an event is and how it is named, and it embeds the whole issue in every event: about a megabyte per page of a hundred, of which the collector keeps three per cent. So a sweep asks GraphQL for the timeline of the issues and pull requests updated in its window, ten items a page because the gateway was measured to drop timelines silently at twenty, and a backfill walks the per-issue endpoint, `/issues/{n}/events`, which is the same rows without the issue. Measured over a week of two repositories, the timeline agrees with the list on every event of every type it can name, field for field; a pull request in a stack is read through its own list because `added_to_stack` has no timeline type, and the one thing the list sees that this does not is a commit referencing an issue nobody has touched, three events in 2,217. **`deps`** collects the dependency graph, off by default. Two shapes of the same subject: the SBOM as a photograph, which is a licence histogram, and the difference between two commits, which says what entered and left and what advisory it carried. Only aggregates are stored, because one dependency bump is three hundred and seventy changes and six rows say the same thing. **`discussions`** collects the forum half of a repository. Discussions are invisible to every issue and pull request endpoint, and an answered question is a support cost that never appears in the issue numbers. A repository whose forum is switched off is never asked: the listing that discovered it already said so, and the query costs the same whether or not there is anything to page. **`planning`** collects labels and milestones. A milestone is the only place GitHub records an intention with a due date, and its completion percentage is computed server side. **`activity`** collects a repository's own activity log. It is the only place a force push is recorded: the public event feed does not distinguish one, and nothing else says that a branch was created or deleted or that a merge was a squash rather than a rebase. It is as perishable as traffic, a hundred entries covered twenty-six hours on the busiest repository measured. ## Continuous integration **`actions`** collects workflow runs as dated facts. A run belongs at the instant it completed, not at the instant we noticed, which is what makes "how long did CI take last Tuesday" answerable. Run-level timing hides where the time went: a run that takes twenty minutes because one job waited eighteen for a runner looks exactly like one that spent eighteen executing. Only the job level separates them, and only the job level names the runner and the steps, so the jobs of each run are expanded at one extra request per run. Once: the jobs of a finished attempt never change, so a run whose jobs this process already wrote is not listed again, and a re-run is a new attempt that is. An ordinary sweep reads the run list in pages of thirty and pages on while they are full of runs newer than its window; the first sweep after start and a backfill read pages of a hundred. **`artifacts`** collects what the workflows left behind, with sizes and expiry. **`joblogs`** collects the text a failed job printed. It is the one thing here that is a log rather than a measurement, and it answers the question a chart never can: not "the build failed" but why. Only failures, and only their last forty lines. Off by default. **`deployments`** collects the newest deployments of every repository, which is the surface a delivery dashboard reads. One GraphQL point per five repositories: five rather than ten because the gateway gives up on a query it cannot finish in about ten seconds, and this one asks for connections rather than plain numbers. ## Security **`security`** counts open alerts by severity and state, and records **explicitly which features are switched on**. That last part is why no data and no alerts are distinguishable: without it, a repository with Dependabot off looks exactly like one with nothing to fix. **`analyses`** collects the code scanning analyses themselves, not just the alerts. An alert says what is wrong now. An analysis says the scan ran, when, on which commit, with which version of the tool, and how many results it found, which is what answers "did the scan actually run on that release". GitHub prunes them, so they have to be captured while they are there. ## Account **`account`** is one GraphQL query and the cheapest thing in the project. It returns the full 366-day contribution calendar, every contribution total, the per-repository commit breakdown and the sponsors block for **one point of a five thousand point budget**. The same data over REST would be dozens of calls and would not include the calendar at all. **`profile`** collects packages, gists, social accounts and the follower graph. Packages come from REST deliberately: GraphQL reports zero packages for an account while REST lists them. **`outbound`** is the other half of everything else here. Every other family measures what the account owns; this measures what it reads and what it contributes to: the stars it gave, and the pull requests it opened in other people's repositories. All of it is GraphQL, one point a query: the starred list, five issue searches and the two comment walks. **`totals`** asks GitHub for the numbers that are true since the beginning: pull requests merged ever, commits ever, issues opened ever, and the whole life of each repository. It is the one family that exists because of how a store reads rather than because of what GitHub offers. Every other measurement here is a row per fact, which is the right shape for "how many in July" and the wrong one for "how many ever": answering that from rows means scanning the whole table, and InfluxDB 3 Core refuses a query that would open more than its file limit, forty thousand where this was measured. Search reports a total for any query and GraphQL reports one for any connection, so one query of ten aliased searches, one REST search for the commit count and one batched query give a number that is one row and is right on the first sweep of a fresh install. **`ratelimit`** is the only measurement the collector takes of itself: what is left in each of GitHub's fifteen independent budgets and when each resets. `GET /rate_limit` costs nothing at all, and without it a family skipped for want of budget looks exactly like a family with nothing to report. **`keys`** collects the account's own SSH and GPG keys: which have never been used, and when the key that signs every commit expires. **`stats`** collects commits per week and the hour-of-week punch card, the two `stats` endpoints that actually answer for a personal account. **`history`** walks every past year's contribution calendar, one GraphQL point per year, back to the day the account was created. Off by default because it only needs to happen once. **`achievements`** collects the badges on the public profile page, and it is the one family that does not come from the API at all: GitHub lists achievements in neither REST nor GraphQL, so the page is read once a day as an anonymous visitor, without the token and charged to no budget. Beside each badge it writes how far the account is from that badge's next tier, which does come from the API. The parser is strict on purpose: when GitHub redesigns the page the family writes nothing and says so once, rather than writing wrong numbers that would look exactly like right ones. ## Activity and cost **`events`** collects the account's activity feed, the most perishable surface GitHub has. It keeps roughly the last three hundred events and drops anything older, whatever its date, and nothing else records that a repository was starred, forked, watched or pushed to at a given minute. **`notifs`** collects the inbox. Like the event feed it is a window, not a history: GitHub keeps unread notifications for about a year and read ones for far less, and `per_page` is silently capped at 50 whatever is asked for. **`billing`** collects what the account actually spent, day by day, per product, SKU and repository, which is the only place that says which repository burned the minutes. Gross, discount and net are all kept rather than one being derived from the others, because the net is not always zero: on the account this was developed against it carries the monthly credit. > **A family that fails on one repository does not fail the sweep** > > Of fifty repositories, most have Dependabot switched off, and each of them > answers 403. That is recorded as "not enabled" and the sweep moves on. Only a > family where _every_ repository failed is left unmarked, so it is retried > rather than treated as done. --- # Measurements Every measurement, its tags, its fields, and how each one is dated. Source: https://jmrplens.github.io/ghchronicle/collectors/measurements/ Ninety-one measurements. Each row says how a point is dated, because that is the thing that decides which questions it can answer. ## How to read the tables | Dating | Means | | --------- | ----------------------------------------------------------------------------------------------------------------------------------- | | **dated** | The point carries the moment the thing happened, so the history is real and re-collecting rewrites the same rows | | **daily** | A snapshot with no date of its own, stamped at the start of the UTC day so a day's sweeps converge on one row rather than piling up | | **now** | A current state, which only makes sense as "what is true at this moment" | Every measurement carries `owner`, `repo` and `full_name` as tags unless it is account-wide, in which case it carries `user`. Most of them also carry a `url` field: the page on GitHub for the thing the row is about, so a dashboard row that names an item can also open it. A `url` is absolute or absent, since the dashboards link to the value itself, and every measurement that has one is linked by unit from at least one table, except the eight that only ever draw as a curve or a bar (`gh_pull_request_review`, `gh_workflow_job`, `gh_event`, `gh_issue_event`, `gh_artifact`, `gh_contribution_day`, `gh_contribution_day_repo` and `gh_commit_check`), where a per-item url has no row to sit on. A tag GitHub leaves empty is written as `(none)`, one spelling on every measurement, and the same `(none)` goes into a few string fields that say nothing on most rows, a pull request's `decision` or an issue's `assigned_to`: InfluxDB 3 creates a column the first time a row carries it, and a query naming a column no row has written fails outright, so those are written on every row. The same convention writes `(ghost)` where a login belonged to an account that has since been deleted. The parentheses are the point of the spelling. GitHub answers `unknown` itself in a Dependabot alert's `relationship`, and `none` is a value in more than one of its enums, so a fallback spelled either way could not be told from an answer; `(none)` is never a value GitHub returns. One value does read `none` without them and means it: `gate` on `gh_commit` is the state of the commit's gate, and `none` is the state of a commit no gate ever ran on, beside `SUCCESS` and `FAILURE`. **A value that moves after the row's own date is a field, never a tag.** A tag is part of a row's identity, so a tag that changes after the fact opens a second series at the same instant and the stale row stays beside the new one for ever: measured after eleven hours of sweeps, one artifact in fifty had a row with `expired=false` and another with `expired=true` at the same timestamp, and a commit seen `PENDING` by one sweep and `FAILURE` by the next was counted twice. Every such tag is a field now, under a new name because InfluxDB 3 fixes a column as tag or field at first write: `gh_artifact.expired` is `live`, `gh_commit.checks` is `gate`, `state` on both alert items is `alert_state` and the code scanning `reason` is `resolution`, `gh_issue`'s `state_reason`, `assignee`, `milestone` and `parent` are `resolution`, `assigned_to`, `milestone_title` and `parent_issue`, `gh_pull_request.draft` and `review_decision` are `is_draft` and `decision`, `gh_discussion.answered` is `has_answer`, `gh_pull_request_review.state` is `review_state`, `gh_deployment.state` is `outcome` and `gh_notification.unread` is `is_unread`. `state` on a pull request, an issue and an external contribution stays a tag because its date moves with it: the open row is stamped at the start of the day and the closed row when it closed. The Prometheus exporter reads the demoted values back as labels; Graphite, which keeps no strings, cannot group by them and its panels say so. ### From a row to a query Every row here is a table in the store, its tags are columns you filter and group by, and its fields are the numbers. Read against InfluxDB 3 in SQL mode, the three datings turn into three shapes of query. A **dated** measurement is history, so it is read over a range: ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = 'telemetry' AND time > now() - INTERVAL '90 days' ``` A **daily** snapshot is one row per day, so the newest row is the answer and the difference between two days is the movement: ```sql SELECT time, downloads FROM gh_release_asset WHERE asset = 'ghchronicle_linux_amd64.tar.gz' ORDER BY time DESC LIMIT 30 ``` A **dated item** carries one row per thing that happened, which is what lets a question be asked of the items rather than of a count: ```sql SELECT date_trunc('week', time) AS week, count(*) AS merged, avg(seconds_to_merge) / 3600 AS hours FROM gh_pull_request WHERE state = 'MERGED' GROUP BY week ORDER BY week ``` The same three shapes work in the other history stores; the [dashboards](/ghchronicle/dashboards/) carry one query set per store for every panel, which is the place to copy from. ### Every measurement, alphabetically Ninety-one, each link landing on the table it is in. [`gh_account`](#account) · [`gh_account_total`](#account) · [`gh_achievement`](#account) · [`gh_achievement_progress`](#account) · [`gh_actions_cache`](#continuous-integration) · [`gh_actions_cache_entry`](#continuous-integration) · [`gh_actions_policy`](#security) · [`gh_artifact`](#continuous-integration) · [`gh_artifact_total`](#continuous-integration) · [`gh_billing_usage`](#cost) · [`gh_branch`](#configuration-and-delivery) · [`gh_branch_protection`](#configuration-and-delivery) · [`gh_code_scanning_alert`](#security) · [`gh_code_scanning_alert_item`](#security) · [`gh_code_scanning_analysis`](#security) · [`gh_code_scanning_setup`](#security) · [`gh_commit`](#development) · [`gh_commit_check`](#development) · [`gh_commit_punchcard`](#account) · [`gh_commits_week`](#account) · [`gh_contribution_day`](#account) · [`gh_contribution_day_repo`](#account) · [`gh_contribution_repo`](#account) · [`gh_contribution_year`](#account) · [`gh_contributions_total`](#account) · [`gh_dependabot_alert`](#security) · [`gh_dependabot_alert_item`](#security) · [`gh_dependabot_ecosystem`](#configuration-and-delivery) · [`gh_dependency`](#configuration-and-delivery) · [`gh_dependency_change`](#configuration-and-delivery) · [`gh_dependency_license`](#configuration-and-delivery) · [`gh_deploy_key`](#configuration-and-delivery) · [`gh_deployment`](#configuration-and-delivery) · [`gh_discussion`](#development) · [`gh_discussion_comment`](#account) · [`gh_environment`](#configuration-and-delivery) · [`gh_event`](#activity) · [`gh_external_contribution`](#development) · [`gh_fork`](#stars-and-forks) · [`gh_gist`](#account) · [`gh_issue`](#development) · [`gh_issue_comment`](#account) · [`gh_issue_event`](#development) · [`gh_job_log`](#job-logs) · [`gh_key`](#account) · [`gh_label`](#development) · [`gh_milestone`](#development) · [`gh_notification`](#activity) · [`gh_package`](#account) · [`gh_package_version`](#account) · [`gh_pinned_item`](#account) · [`gh_policy_file`](#configuration-and-delivery) · [`gh_profile_flag`](#account) · [`gh_pull_request`](#development) · [`gh_pull_request_review`](#development) · [`gh_rate_limit`](#configuration-and-delivery) · [`gh_release`](#repositories) · [`gh_release_asset`](#repositories) · [`gh_repo`](#repositories) · [`gh_repo_activity`](#continuous-integration) · [`gh_repo_archived`](#repositories) · [`gh_repo_community`](#repositories) · [`gh_repo_created`](#account) · [`gh_repo_language`](#repositories) · [`gh_repo_policy`](#configuration-and-delivery) · [`gh_repo_topic`](#repositories) · [`gh_repo_total`](#configuration-and-delivery) · [`gh_review_thread`](#development) · [`gh_ruleset`](#configuration-and-delivery) · [`gh_ruleset_rule`](#configuration-and-delivery) · [`gh_ruleset_version`](#configuration-and-delivery) · [`gh_secret`](#security) · [`gh_security_feature`](#security) · [`gh_security_setting`](#security) · [`gh_social_account`](#account) · [`gh_sponsors_listing`](#account) · [`gh_sponsors_tier`](#account) · [`gh_sponsorship`](#account) · [`gh_star`](#stars-and-forks) · [`gh_star_given`](#stars-and-forks) · [`gh_star_list`](#account) · [`gh_traffic`](#audience) · [`gh_traffic_path`](#audience) · [`gh_traffic_referrer`](#audience) · [`gh_webhook`](#configuration-and-delivery) · [`gh_webhook_delivery`](#configuration-and-delivery) · [`gh_workflow`](#continuous-integration) · [`gh_workflow_job`](#continuous-integration) · [`gh_workflow_run`](#continuous-integration) · [`gh_workflow_run_total`](#continuous-integration) · [`gh_workflow_step`](#continuous-integration) ## Audience | Measurement | Dated | Tags | Fields | | --------------------- | ------------------------ | ---------------------- | ---------------------------------- | | `gh_traffic` | dated, one point per day | `kind` (views, clones) | `count`, `uniques`, `url` | | `gh_traffic_referrer` | daily | `referrer` | `count`, `uniques`, `url`, `referrer_url` | | `gh_traffic_path` | daily | `path` | `count`, `uniques`, `title`, `url` | GitHub serves fourteen days and the whole window is rewritten on every sweep, so a collector that was down for a day repairs itself on the next run. The referrers and paths are the top ten of that same window with no dates attached, which is why they are a snapshot rather than a series. ## Stars and forks | Measurement | Dated | Tags | Fields | | --------------- | -------------------------------- | -------------------------- | ------------------------------------------------------ | | `gh_star` | dated, when the star was given | `user` | `starred`, `url`, `user_url` | | `gh_star_given` | dated | `user`, `repo`, `language` | `stars`, `repo_stars`, `url` | | `gh_fork` | dated, when the fork was created | `by` | `forks`, `stars`, `days_since_push`, `advanced`, `url` | `gh_star_given` is the outbound direction: what this account starred in other people's repositories. `advanced` on a fork separates a real derivative from a bookmark, which most forks are. ## Repositories | Measurement | Dated | Tags | Fields | | ------------------- | ----- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_repo` | now | `language`, `visibility`, `license`, `archived`, `fork`, `default_branch` | `stars`, `forks`, `watchers`, `open_issues`, `size_kb`, `age_days`, `days_since_push`, `days_since_config_change`, `network`, `repo_id`, `is_template`, `has_pages`, `web_commit_signoff_required`, `allow_update_branch`, `pull_request_creation_policy`, `url` | | `gh_repo_language` | now | `language` | `bytes` | | `gh_repo_topic` | now | `topic` | `present`, `url` | | `gh_repo_community` | now | | `health_percentage`, `url`, `has_readme`, `has_license`, `has_contributing`, `has_code_of_conduct`, `has_issue_template`, `has_pull_request_template` | | `gh_repo_archived` | dated, when the repository was archived | | `archived`, `age_days_at_archive`, `url` | | `gh_release` | now | `tag`, `draft`, `prerelease` | `downloads`, `assets`, `age_days`, `url` | | `gh_release_asset` | daily | `tag`, `asset` | `downloads`, `size_bytes`, `digest`, `content_type`, `uploader`, `age_days`, `url` | `open_issues` is GitHub's field and GitHub counts pull requests in it. Use `gh_issue` to count issues. The `url` on `gh_release_asset` is the asset's download address, not a page: following it fetches the binary. The assets are inventory, anchored to the start of the UTC day like the cache entries: stamped at the sweep, every asset was a fresh row every hour, which was 15 per cent of the whole database after eleven hours. One row per asset per day still answers "downloads per day", and the newest row is still the value. `gh_repo_archived` is the one row about a repository that carries a date rather than a state: dated at `archivedAt`, a clear-out is visible as the batch it was, and a live repository produces no row at all, so counting the rows is counting the archive. It does not need `include_archived`. The listing a sweep already pays for says which repositories are archived, and the `totals` family asks the date of all of them in one GraphQL query of four scalars per repository, on every `totals` sweep: one point at that cadence, and the rows it rewrites are the same rows, which is what an exporter that keeps only what is rewritten needs; the listing cannot supply the date itself, since REST carries no `archived_at` and its `updated_at` was measured two seconds to eight minutes after the archive. An archived fork under the default fork rule is the one kind with no row. ## Development | Measurement | Dated | Tags | Fields | | -------------------------- | ----------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_pull_request` | dated when closed, daily while open | `number`, `state`, `author` | `is_draft`, `decision`, `title`, `labels`, `label_names`, `author_association`, `additions`, `deletions`, `churn`, `changed_files`, `commits`, `comments`, `total_comments`, `reviews`, `review_requests`, `review_threads`, `base_ref`, `head_ref`, `merged_by`, `merge_commit`, `mergeable`, `merge_state`, `stack`, `stack_size`, `stack_position`, `seconds_to_first_review`, `seconds_to_first_human_review`, `seconds_to_merge`, `seconds_open`, `url` | | `gh_pull_request_review` | dated, when submitted | `number`, `author`, `reviewer`, `bot`, `self` | `review_state`, `reviews`, `seconds_to_review`, `url` | | `gh_issue` | dated when closed, daily while open | `number`, `state`, `author` | `resolution`, `assigned_to`, `milestone_title`, `parent_issue`, `comments`, `reactions`, `labels`, `label_names`, `sub_issues_total`, `sub_issues_completed`, `pull_request`, `seconds_to_close`, `seconds_open`, `url` | | `gh_commit` | dated, when committed | `sha`, `author`, `branch`, `signature` | `gate`, `additions`, `deletions`, `churn`, `changed_files`, `commits`, `signed`, `oid`, `headline`, `url`, `pull_request`, `checks_total`, `checks_failed` | | `gh_commit_check` | dated, when the check finished | `sha`, `app`, `check`, `conclusion` | `checks`, `failed`, `url` | | `gh_issue_event` | dated, when it happened | `event`, `actor`, `kind`, `bot`, `label`, `milestone`, `requested_reviewer`, `review_requester`, `mentioned` | `events`, `number`, `title`, `url`, `commit_id`, `rename_from`, `rename_to` | | `gh_review_thread` | dated, when the thread's first comment was written | `thread`, `number`, `author`, `bot` | `path`, `comments`, `resolved`, `outdated`, `subject_type`, `resolved_by` | | `gh_discussion` | dated, when created | `category`, `answerable`, `author`, `number` | `has_answer`, `comments`, `replies`, `reactions`, `upvotes`, `closed`, `state_reason`, `seconds_to_answer`, `seconds_to_close`, `title`, `url` | | `gh_label` | daily | `label` | `issues`, `pull_requests`, `used`, `url` | | `gh_milestone` | daily | `milestone`, `state` | `progress`, `issues`, `pull_requests`, `days_to_due`, `seconds_to_close`, `url` | | `gh_external_contribution` | dated | `user`, `repo`, `number`, `kind`, `state` | `contributions`, `merged`, `title`, `comments`, `seconds_to_merge`, `seconds_open`, `url` | `gh_commit` is what replaces `stats/code_frequency`, which returns 202 with an empty body forever on a personal account. `signature` is `unsigned` when there is no signature at all, which is a different fact from one that failed to verify. `gate` is the state of the whole gate on that commit, which is not the same claim as a workflow run having failed: a run says one job failed, the rollup says the commit came out red. It is a field because the verdict lands after the commit's own date. `gh_commit_check` holds only the checks that are not GitHub Actions, since everything Actions runs is already collected in far more detail. `seconds_to_first_review` counts any review, and on an account with review bots that is the bot: measured over 140 pull requests, its median was five seconds, because 132 were first reviewed by sourcery-ai or coderabbitai within a minute of opening. `seconds_to_first_human_review` is the wait for somebody else, over the first twenty reviews the query fetches: not a bot, and not the author. The author's reply in a review thread arrives as a review of state `COMMENTED` under their own name, and on this account it was the earliest non-bot review on every one of the 91 pull requests that had one, so a wait that counted it measured how fast the owner answers sourcery-ai. A pull request whose fetched reviews are all bots and the author's own replies carries no such field rather than a wrong one. A bot is a GitHub App (`__typename` Bot) or a login ending in `[bot]`; a deleted account is not one. `gh_pull_request_review.bot` draws the same line per review and `self` marks the author's own, so a reviewers table can leave both out or show them apart; `bot` is what `gh_review_thread.bot` already does for threads. `title`, `label_names` and `author_association` are fields because a title is unbounded and nine labels on one pull request are one row, not nine series. `labels` is the count and `label_names` the names joined by commas, absent when there are none, on pull requests and issues alike. `author_association` is `OWNER`, `MEMBER`, `COLLABORATOR`, `CONTRIBUTOR`, `FIRST_TIME_CONTRIBUTOR` or `NONE`: what separates an outside contribution from the owner's own work. `mergeable` and `merge_state` are written only while a pull request is open. A merged one keeps answering `CONFLICTING` long after it was merged, which is stale rather than false but reads as a repository full of conflicts. A sweep reads only what was updated in the last two cadences, so an open pull request nobody touches has its `seconds_open`, `mergeable` and `merge_state` rewritten once a day by the whole-page read rather than every hour; anything that moves `updatedAt`, a review, a comment, a push, a close, is rewritten by the sweep that follows it. `stack`, `stack_size` and `stack_position` describe a stack of dependent pull requests and are absent on a pull request that is in none. `stack` is the stack's own number, not a member's, so the honest way to count deliveries is distinct `stack` values plus the rows carrying no stack fields at all. `review_requests` and `review_threads` are the two counts a reviewing flow is measured with, and `total_comments` counts every comment on the pull request rather than the ones in `comments`, which are the ones on the conversation. `sub_issues_total` and `sub_issues_completed` are how far an epic has got, from the checklist GitHub keeps on the parent. `parent_issue` is the other end of the same relation, on the child, and is `0` on an issue with no parent. `pull_request` is the pull request that closed the issue, `0` when none did. `gh_issue_event` is the transition rather than the state. `gh_issue` and `gh_pull_request` say what something ended up as; this says when it was labelled, closed, reopened, renamed or had a review requested. A reopening exists nowhere else. `mentioned` is the person a `mentioned` or `subscribed` event happened to, which GitHub files as the actor without saying who wrote the comment; here that person has a tag of their own and `actor` reads `(none)` on those two types, so the account named in "@coderabbitai" no longer shares a column with the app that reviews. An app is spelled the way REST spells it, with the `[bot]` suffix, on every measurement. `gh_discussion` counts comments and replies apart: comments answer the discussion, replies answer those, and GitHub's own number on the page is the two added together. ## Continuous integration | Measurement | Dated | Tags | Fields | | ------------------------ | ----------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_workflow_run` | dated, when it finished | `workflow` (the file path), `event`, `conclusion`, `actor` | `duration_seconds`, `queued_seconds`, `attempt`, `success`, `run_id`, `run_number`, `pull_request`, `pull_requests`, `headline`, `head_repo`, `head_sha`, `head_branch`, `name`, `title`, `workflow_id`, `initial_actor`, `url` | | `gh_workflow_run_total` | now | | `runs` | | `gh_workflow_job` | dated, when it finished | `workflow`, `job_name`, `attempt`, `conclusion`, `runner_group`, `labels` | `duration_seconds`, `queued_seconds`, `steps`, `success`, `runner`, `run_id`, `head_sha`, `head_branch`, `url` | | `gh_workflow_step` | dated, when it finished | `workflow`, `job_name`, `attempt`, `step`, `conclusion` | `duration_seconds`, `step_number` | | `gh_workflow` | now | `workflow`, `path`, `state` | `active`, `age_days`, `days_since_change`, `url` | | `gh_artifact` | dated, when created | `artifact` | `live`, `size_bytes`, `retention_days`, `digest`, `run_id`, `head_sha`, `head_branch`, `url` | | `gh_artifact_total` | now | | `live_bytes`, `count`, `walked` | | `gh_actions_cache` | now | | `size_bytes`, `count` | | `gh_actions_cache_entry` | daily | `cache`, `ref` | `size_bytes`, `caches`, `key`, `days_since_use`, `age_days` | | `gh_repo_activity` | dated | `activity`, `actor` | `events`, `id`, `ref_name` | > **The workflow tag changed meaning** > > `workflow` used to hold the run's name and now holds the workflow's file path. > GitHub rewrites the name of anything dynamic, so a Dependabot run is named > after the bump it made and a code scanning run after the pull request that > triggered it: measured over three hundred runs of one repository, the name > took thirty-five values against seven paths, which is thirty-five series for > seven workflows. The human name is not lost, it is the `name` field, and > `gh_workflow` keys that same path to that same name, so a panel joins > `gh_workflow.path` to `gh_workflow_run.workflow` to print "CodeQL" again. > Nothing merges across the change: every series has a new identity, so rows > written before it sit beside the new ones rather than continuing them. `branch` was a tag on `gh_workflow_run` and on `gh_artifact`, and is now the field `head_branch` on both, and on `gh_workflow_job` as well. A branch is an identity, but not a reusable one: every pull request and every Dependabot bump mints a name that never comes back, so the tag grows without bound, and the bounded question a reader actually asks, whether this was a push or a pull request, is already the `event` tag. InfluxDB 3 also fixes a column as a tag or a field the first time it sees it and refuses every later write that disagrees, so keeping the name would have meant dropping both tables to publish a value the API itself calls `head_branch`. `attempt` is a tag on the job and on the step because the job listing is now asked for every attempt rather than the last one. Without it the two tries of a re-run are one series, told apart only by the second they finished in, and a flaky test cannot be distinguished from a broken one. `queued_seconds` on a run is written on first attempts only. GitHub keeps the run's `created_at` across re-runs, so on a second attempt the gap to `run_started_at` is the time a person took to press the button, not a runner queue: 0.4 s on average on first attempts against 1,747 s on second ones, measured. The queue of a retry exists only per job. `run_number` is the "#1483" GitHub shows and people quote; `run_id` is what the API keys by. `pull_request` is the number of the first pull request GitHub linked to the run and `pull_requests` how many it linked, both absent when it linked none. `headline` is the first line of the commit that ran, and `head_repo` is written only when the run came from another repository, which is what a fork's pull request looks like. `gh_workflow_run_total` is the run listing's own `total_count`, which is the whole history rather than the few hundred runs the walk sees. It is current state, so it is stamped now, and it is the only place "how many runs ever" can be answered without scanning the table. An ordinary sweep asks for the run list in pages of thirty rather than a hundred. The page is thirteen kilobytes a run, of which the collector keeps six hundred bytes, and at a hundred runs it was a megabyte and a half per active repository every quarter of an hour, forty six percent of everything a day downloads. The stores lose nothing: the walk still pages on while a page is full of runs newer than the window, up to seven pages, which is the two hundred and ten runs two pages of a hundred reached, and the first sweep after start and a backfill still ask for a hundred. What changes is the Prometheus exporter, which holds only what the last sweep collected and shows the newest thirty runs between builds rather than the newest hundred. The jobs of a run are listed once. The jobs of a completed attempt never change, and listing them again every sweep was a request per run in the window, nearly all of them 304s that cost no quota but a third of a second of waiting each, ninety six times a day. The collector remembers each attempt whose jobs it wrote, in memory like the ETag cache, so after a restart the first sweep lists the newest twenty per repository once and then asks only for new runs and new attempts; a backfill lists every run regardless. A re-run keeps the run's id and is a new attempt, so it is listed again. The cap of twenty bounds what a sweep pays, not which runs get jobs: a window with more runs than that fills in twenty a sweep. A run is remembered only once the sweep that listed it succeeded, because the runner keeps nothing of a collector that failed partway. `retention_days` is the retention an artifact actually got, which is rarely the configured default: eighty-eight of a hundred artifacts measured lived one day against a setting of ninety. `gh_repo_activity` is one row per activity type, actor and second. The branch is the field `ref_name`, one name per pull request and per Dependabot bump, and without it in the key the branches one push moved in the same second would be one row in every store, the last one written standing for all of them: eight force pushes in one second, measured. The entries that share a key are folded into one point, `events` counting them, `ref_name` naming every branch joined by commas, `id` the newest entry's, so a sum of `events` is the number of activities everywhere. Queue time only exists at the job level. The run-level figure folds the wait into the duration, and the job-level one includes waiting for a dependency, so a job that waits nineteen minutes for another job to finish is not evidence of a runner shortage. `runner` is a field, not a tag: a hosted runner is named uniquely per run, so as a tag it would create a series for every job ever executed. The workflow job tag is `job_name` rather than `job`, because `job` collides with the labels Prometheus adds at scrape time. When `walked` is lower than `count`, the live size is a floor and the repository has more artifacts than the page cap reached. `gh_actions_cache` says a repository holds twelve gigabytes; `gh_actions_cache_entry` says which key holds them and which has not been touched for a week, which is what decides what GitHub evicts at the ten gigabyte ceiling. The tag is the key without its content hash, because the whole key is a series per build. ## Security | Measurement | Dated | Tags | Fields | | ----------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `gh_dependabot_alert` | now | `severity`, `ecosystem` | `open`, `url` | | `gh_dependabot_alert_item` | dated, when raised | `number`, `severity`, `ecosystem`, `package`, `ghsa`, `scope`, `relationship`, `manifest` | `alert_state`, `alerts`, `cvss`, `cvss_v4`, `epss`, `epss_percentile`, `cve`, `cwe`, `summary`, `vulnerable_range`, `first_patched`, `dismissed_reason`, `dismissed_by`, `dismissed_comment`, `seconds_to_detect`, `seconds_to_resolve`, `seconds_open`, `url` | | `gh_code_scanning_alert` | now | `severity`, `tool` | `open`, `url` | | `gh_code_scanning_alert_item` | dated, when raised | `number`, `severity`, `tool`, `rule`, `path`, `category`, `ref` | `alert_state`, `resolution`, `alerts`, `commit`, `line`, `cwe`, `seconds_to_resolve`, `seconds_open`, `url` | | `gh_code_scanning_analysis` | dated, when the scan ran | `tool`, `version`, `ref`, `category` | `analyses`, `results`, `rules`, `commit` | | `gh_security_feature` | now | `feature` | `enabled`, `open_alerts`, `alerts`, `url` | | `gh_security_setting` | now | `setting`, `status` | `enabled` | | `gh_code_scanning_setup` | daily | `state`, `query_suite`, `schedule` | `setups`, `languages`, `days_since_change` | | `gh_secret` | daily | `kind` (actions, dependabot), `secret` | `secrets`, `age_days`, `days_since_rotation` | | `gh_actions_policy` | daily | `permissions` | `policies`, `can_approve_pr` | The tags on the two item measurements cost nothing: both listings were already carrying them, and the series was always keyed by `number`, so they group rows that exist one per alert rather than multiplying them. All of them are always written, falling back to `(none)` when GitHub omits one. A tag written only sometimes gives the measurement two Graphite path depths, and the panels index their nodes from one fixed table. `alert_state` on both items and `resolution` on the code scanning one are fields, since an alert is dated when it was raised and both move when it closes; `resolution` reads `open` until then, so the column exists before any alert has closed. The fields are the opposite: `cve`, `cwe`, `first_patched`, `epss`, `epss_percentile` and `seconds_to_detect` are written only when the advisory carries them, because a missing EPSS score is not a score of zero. `cvss` and `cvss_v4` need the same guard for a different reason: GitHub always sends both keys and fills the one it lacks with `0.0`, and an advisory published with a v4 vector only was 78 of the 225 alerts of one repository, enough for "worst CVSS" over a severity group of them to read zero. Neither score is written unless it is above zero; a panel that wants one number per alert reads `COALESCE(cvss_v4, cvss)`. `summary` is the advisory's title, `vulnerable_range` the range it covers, which next to `first_patched` is the action to take, and `dismissed_reason`, `dismissed_by` and `dismissed_comment` say why a person closed an alert without fixing it; they exist only on an alert in state `dismissed`. `seconds_to_detect` is the gap between the advisory being published and the alert being raised here. It is negative when the alert came first, which happens when an advisory is written up after the fact. The two `cwe` fields cannot be joined. Dependabot writes `CWE-400` and code scanning writes `cwe-079`, both GitHub's own spelling, and neither is normalised here. `line` is the start line of the alert's most recent instance, and its zero is GitHub's own value for an alert about a whole file, not a missing reading. A Dependabot alert closes three ways, not two: `auto_dismissed_at` is how GitHub closes a development-dependency alert on its own, leaving the other two null. An alert closed that way used to grow `seconds_open` forever. `gh_security_feature` exists so that no data and no alerts are distinguishable. Without it, a repository with Dependabot switched off looks exactly like one with nothing to fix. `enabled` is read from the first full page of the listing, which answers 403 when the feature is off and, for code scanning, 404 when nothing has been analysed yet. `alerts` counts what the sweep read, which is one page, so at most a hundred, rather than the repository's total. ## Account | Measurement | Dated | Tags | Fields | | ------------------------ | ------------------------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_account` | now | | `followers`, `following`, `following_users`, `public_repos`, `gists`, `packages`, `projects`, `starred`, `watching`, `sponsors`, `sponsoring`, `account_age_days`, `pronouns`, `url` | | `gh_contributions_total` | now | | `calendar_total`, `commits`, `pull_requests`, `reviews`, `issues`, `repositories`, `restricted`, `repos_with_commits`, `repos_with_issues`, `repos_with_pulls`, `repos_with_reviews`, `url` | | `gh_contribution_day` | dated, one point per calendar day | | `contributions`, `level`, `url` | | `gh_contribution_year` | dated, end of the year; the year in progress daily | `year` | `contributions`, `commits`, `issues`, `pull_requests`, `reviews`, `repositories`, `restricted`, `repos_with_commits`, `repos_with_issues`, `repos_with_pulls`, `repos_with_reviews`, `partial` | | `gh_contribution_repo` | now | `repo`, `kind` (commits, issues, pulls, reviews) | `contributions`, `commits`, `days`, `commits_dated`, `url` | | `gh_contribution_day_repo` | dated, the day the commits belong to | `repo`, `private`, `own` | `commits`, `url` | | `gh_commits_week` | dated, the Sunday of its week | | `commits`, `owner_commits` | | `gh_commit_punchcard` | now | `weekday`, `hour` | `commits` | | `gh_package` | now | `package`, `type`, `visibility`, `repo` | `versions`, `tagged_versions`, `age_days`, `days_since_update`, `url` | | `gh_package_version` | dated, when published | `package`, `type`, `visibility`, `repo`, `tag` | `digest`, `published`, `url` | | `gh_gist` | now | `gist`, `public` | `files`, `comments`, `size_bytes`, `description`, `url`, `age_days`, `days_since_update` | | `gh_achievement` | daily | `achievement` | `name`, `tier_number`, `tier_name`, `present`, `image`, `url` | | `gh_achievement_progress` | daily | `achievement` | `name`, `count`, `tier_number`, `next_threshold`, `percent`, `page_tier`, `agrees`, `image`, `url` | | `gh_social_account` | now; the `orcid` row daily | `provider` | `url`, `present` | | `gh_pinned_item` | now | `repo` | `pinned`, `position`, `kind`, `stars`, `days_since_push`, `url` | | `gh_profile_flag` | now | `flag` | `enabled`, `message`, `age_days`, `url` | | `gh_sponsorship` | dated, when the sponsorship was made | `direction` (sponsor, maintainer), `sponsorable` | `sponsorship`, `active`, `one_time`, `privacy`, `tier`, `amount_cents`, `url` | | `gh_sponsors_listing` | now | | `has_listing`, `listing_name`, `listing_public`, `listing_age_days`, `tiers`, `monthly_income_cents`, `next_payout_cents`, `next_payout_date`, `sponsor_spend_cents`, `lifetime_received_cents`, `sponsorships_received`, `goal_kind`, `goal_title`, `goal_target`, `goal_percent`, `url` | | `gh_sponsors_tier` | daily | `tier` | `tiers`, `price_cents`, `one_time`, `retired`, `age_days`, `url` | | `gh_star_list` | daily | `list` | `lists`, `items`, `private`, `name`, `age_days`, `days_since_add`, `url` | | `gh_account_total` | now | | `pulls_opened`, `pulls_merged`, `pulls_open_now`, `pulls_merged_elsewhere`, `pulls_reviewed`, `issues_opened`, `issues_closed`, `issues_elsewhere`, `commented_elsewhere`, `commits`, `repositories`, `url` | | `gh_repo_created` | dated, when created | `repo`, `fork` | `created`, `private`, `url` | | `gh_key` | daily | `kind` (ssh, gpg), `key` | `keys`, `age_days`, `days_since_use`, `never_used`, `days_to_expiry`, `verified`, `revoked`, `can_sign`, `emails`, `url` | | `gh_discussion_comment` | dated | `repo`, `own`, `is_answer`, `is_reply`, `author`, `comment`, `number` | `comments`, `answers`, `upvotes`, `title`, `reply_to`, `discussion_answered`, `discussion_answerable`, `discussion_closed`, `answered_by`, `answer_chosen_by`, `state_reason`, `category`, `seconds_to_answer`, `seconds_to_close`, `url` | | `gh_issue_comment` | dated | `repo`, `own`, `number` | `comments`, `url` | `following` is the profile's own number, and it counts organisations as well as people. GraphQL's `following` connection counts only users, which on this account read four where the profile read nine, so more than half of it was invisible. Both are kept: `following` is what the profile page shows, `following_users` is the connection's people-only count. When the profile request fails the connection's count fills both, which is the tell that the profile was not reached. `versions` on a package is the count the package element declares, with the walked count as the fallback. `tagged_versions` counts named releases only. It used to count every container tag, and about half of those are the OCI referrers fallback tag GitHub publishes for each attestation and signature manifest: `sha256-` followed by the digest the row already carries. Nobody pulls one, there is a fresh one on every build, and excluding them halved the count on two packages, from a hundred and twenty six to fifty seven on one. `gh_package_version` no longer writes a row for one either. `gh_pinned_item` and `gh_profile_flag` are the profile page itself as data. A pin has no date of its own, so both are stamped now. `position` is a field and not a tag: a repository that moves from slot two to slot three is the same pin, and as a tag every rearrangement would fork the series. `flag` is a closed list of eight: `hireable`, `developer_program`, `campus_expert`, `github_star`, `bounty_hunter`, `employee`, `sponsors_listing`, which is whether the account has a Sponsors profile at all, and `limited_availability`, which is the availability status carried as an eighth flag rather than a measurement of its own, with the `message` it displays and the `age_days` since that status was set. `age_days` is written on that row alone and is the age of the status message, not of the flag; GitHub says when none of the other flags was granted, so no row carries a date for them. `gh_sponsorship` is the only dated record of the money. `gh_account.sponsors` and `gh_account.sponsoring` are counts as of now that say neither when nor to whom, and `gh_sponsors_listing.lifetime_received_cents` is a total with no dates in it. Both connections are read with `activeOnly` off, which is what recovers a lapsed one. `sponsorable` is the other party, and it is the literal word `private` when the sponsorship hides it, in which case no URL is written rather than one being guessed at. `gh_sponsors_tier` is standing inventory, the way an SSH key is. Dating a tier at its creation would put all eight of them in 2021, outside every dashboard range, where they would read as "no tiers"; anchored to the start of the UTC day they converge on one row per tier per day, and `age_days` keeps the creation date recoverable. `gh_star_list` is the same shape for the same reason: the lists the account files its stars into, one row per list with how many it holds, anchored to the start of the UTC day. A list carries two dates, when it was made and when a star last went into it, and both survive as `age_days` and `days_since_add` rather than dating the row, which would put a list made in 2024 outside every dashboard range. The tag is the slug, which the list's page is addressed by; the display name is a field. Whether the slug outlives a rename is not verified, since checking it means renaming a list. It rides in the account query that was already being paid for: measured on 2026-09-11, eleven lists with their item counts added nothing to a cost of one. `gh_contribution_day` is the only place the green squares exist as data. With `every.history` set, it reaches back to the year the account was created, at one GraphQL point per year. Its `level` is the square's shade, GitHub's own quartile of the year as the 0 to 4 the profile draws, which is not a function of the count: on one account 83 contributions on one day and 52 on another were both the second quartile. The quartile is of the window asked for, the trailing twelve months for the sweep and the calendar year for `history`, so a day both write can change shade between the two, as it does on the profile when a year is picked. Which is why the dashboard's grid shades a day by its own count instead: measured on 2026-09-14, the profile page shades by the fifths of the busiest day of the window, a rule that reproduced all 366 of its squares from GitHub's own counts, where `level` disagreed with the page on 33 of those days. `gh_achievement` is the one measurement that does not come from the API. GitHub lists achievements nowhere in REST or GraphQL, so the family reads the public profile page, `https://github.com/?tab=achievements`, once a day as an anonymous visitor: no token travels to it and it is charged to no budget. One row per badge, stamped at the start of the UTC day: `name` is the badge, `tier_number` is the number on its label (1 with no label, 2 to 4 for x2 to x4) and `tier_name` the colour that goes with it (default, bronze, silver, gold); the number is not called `tier` because Elasticsearch maps a field name once across every measurement's index and `tier` is already a string on sponsorships. The parser is strict about the markup it accepts and holds each part of a card to the others, so when GitHub changes the page the family logs one warning and writes nothing until the parser is updated; the rows it wrote before stay, and a panel reading the newest row per badge goes stale rather than wrong. `image` is the badge image the page shows at that tier, for a panel to draw. The site the page is read from is derived from `github.base_url`. A `base_url` that is a proxy in front of the API has to name the site with [`github.web_url`](/ghchronicle/configuration/#github), because the API host answers the page's url with a JSON 404, and the family refuses that rather than reading it as no badges. An account with no badge at all has no achievements tab: its url answers 404 while the profile answers 200, which is no rows and no warning, the same reading every family gives a 404. A 200 with no card in it is refused as a changed page rather than read as none. `gh_achievement_progress` is written by the same family beside the badges: one row per badge that has tiers (Pull Shark, Galaxy Brain, Starstruck, Pair Extraordinaire), whether or not the page shows it yet, saying how far the account is from the next tier. GitHub publishes neither the rule a badge is earned by nor the count it has reached, so the count is recomputed from the API and the thresholds are the community's, the Tiers table of [Schweinepriester/github-profile-achievements](https://github.com/Schweinepriester/github-profile-achievements) as read on 2026-09-12: Pull Shark counts merged pull requests anywhere and its tiers begin at 2, 16, 128 and 1024; Galaxy Brain counts the discussions whose accepted answer the account wrote, at 2, 8, 16 and 32; Starstruck takes the stars on the most starred repository of the account's own, forks left out, at 16, 128, 512 and 4096; Pair Extraordinaire counts merged pull requests in public repositories with a co-authored commit, one per pull request, at 1, 10, 24 and 48, cross-checked the same day against a hand count (the two counts differed by two, the difference falling in a range where the hand count ran past the thousand results a search pages, and the page showed the same tier either way; the co-authored pull requests of a private repository moved nothing). The single-tier badges and the two GitHub is still testing have no row: there is no next tier to measure against. `tier_number` is the tier the count implies (0 below the first threshold), `page_tier` the tier the profile page shows (0 when the badge is not on it) and `agrees` whether the two are the same; when they are, `next_threshold` is where the next tier begins (0 at the top) and `percent` the count against it (100 at the top). A row that disagrees is a rule the page contradicts, or a page GitHub has not recomputed yet, said once per process in the log, and it carries neither field, so no bar is drawn from a rule the page contradicts. Three of the counts are one GraphQL query; the fourth is a walk over the account's merged pull requests in public repositories with their commit messages, split by merge date where a range holds more than the thousand results a search will page, one point a page: a few dozen points and about a minute for the day over a whole account's life. A count the API would not give is a day without progress rows, never a day without badges. `gh_social_account` carries one more row than the social accounts listing: the homepage, under the provider `website`, from the `blog` of the profile. The ORCID iD the profile page shows is in no endpoint, so the `achievements` family, which already reads that page once a day, writes it from the page's vcard under the provider `orcid`, stamped at the start of the UTC day like the badges. The links the API does list (Mastodon, LinkedIn, Bluesky) are written by the `profile` family from the API and skipped on the page, so no account is written twice. The read is as strict as the badges': a page without the vcard is a changed page and one warning, never "no accounts". `pronouns` is the profile's pronouns line, `he/him`, as a field on the headline row, absent when the profile shows none. A field and not a tag: it is free text the owner can edit, and as a tag every edit would fork the account's one series. `gh_issue_comment` and `gh_discussion_comment` are read from the newest end of their connections, which list oldest first: a sweep's one page is the hundred newest comments, and a comment that became the accepted answer after it was first written is seen again on the next sweep. `gh_contribution_year` has one row per past year, dated the thirty-first of December, and one for the year in progress, asked for on every run from the first of January to now. That row is a snapshot, stamped at the start of the UTC day and marked `partial`, so a panel comparing years can tell a bar that is still growing from one that is finished; read it as the newest row per `year`. The first run of the next year replaces it with the final row dated December. `gh_account.packages` is counted from the REST listings the profile family walks, not from GraphQL's `packages` connection, which does not see the container registry and answered 0 for an account whose four packages are all containers. If a listing fails the GraphQL count stands. `gh_account_total` is the answer to "how many ever". Every other measurement here is a row per fact, which is the right shape for "how many in July" and the wrong one for a lifetime count. GitHub counts them itself, in one search request each, so the number is one row and is right on the first sweep of a fresh install. `gh_dependency_change` writes a row on every sweep of the `deps` family, not only when a dependency moved: a range with no change, a head that did not move and the first sweep, which has no base yet, each write one row with `change` and `ecosystem` at `(none)` and both counters at zero. InfluxDB 3 creates a table at its first point and answers a query naming a table it has not seen with an error, so a measurement written only on a change did not exist until the first bump, and the panel over it was an error until then. The zero row costs no request and the panels leave the `(none)` series out. The three measurements about other people's repositories, `gh_discussion_comment`, `gh_issue_comment` and `gh_repo_created`, exist because a sweep over one's own repositories cannot see any of it. Each carries `own` so the two can be told apart. > **Why the weekly row is anchored to Sunday** > > `gh_commits_week` is stamped at the Sunday that starts each week, not at the > sweep. A sweep on Tuesday and one on Friday have to land on the same row, or > every re-read writes a second copy of the year. ## Activity | Measurement | Dated | Tags | Fields | | ----------------- | ------------------ | ------------------------------------------------------- | ------------------------------- | | `gh_event` | dated | `type`, `repo`, `action`, `ref_type` | `events`, `public`, `commits`, `url` | | `gh_notification` | dated, last update | `reason`, `repo`, `private`, `subject_type` | `is_unread`, `notifications`, `title`, `url` | Both are windows, not histories. GitHub keeps the last three hundred events whatever their dates and discards read notifications quickly. What is captured is what was there when the sweep ran. `gh_event` carries no actor, since the feed is the account's own and the actor was the login on every row; `action` and `ref_type` are written on every row, `(none)` on a push. `is_unread` is a field: reading a thread does not move its `updated_at`, so the daily read with `all=true`, the one that lists a thread read without a reply, rewrites the same row rather than opening a second one beside it. A notification's `url` is derived from the API address of its subject, and a subject shape the mapping does not recognise is left without one rather than guessed at, so a good part of the rows carry no link. ## Configuration and delivery | Measurement | Dated | Tags | Fields | | ----------------------- | --------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_webhook` | daily | `hook` (the id), `host`, `active` | `events`, `hooks` | | `gh_webhook_delivery` | dated, when delivered | `hook`, `host`, `event`, `status`, `code`, `ok` | `deliveries`, `duration_seconds`, `redelivery` | | `gh_ruleset` | daily | `ruleset`, `target`, `enforcement` | `rulesets`, `active`, `days_since_change`, `url` | | `gh_ruleset_rule` | daily | `ruleset`, `rule` | `rules`, `bypass_actors`, `bypass_always`, `bypass_sampled`, `ref_include`, `ref_exclude` | | `gh_ruleset_version` | dated, when saved | `ruleset`, `target`, `actor_type` | `versions`, `version_id`, `ruleset_id`, `actor_id`, `url` | | `gh_branch_protection` | daily | `pattern` | `rules`, `admin_enforced`, `allows_deletions`, `allows_force_pushes`, `blocks_creations`, `dismisses_stale_reviews`, `requires_approving_reviews`, `required_reviews`, `requires_code_owner_reviews`, `requires_commit_signatures`, `requires_conversation_resolution`, `requires_linear_history`, `requires_status_checks`, `requires_strict_status_checks`, `required_checks`, `requires_deployments`, `restricts_pushes`, `restricts_review_dismissals`, `url` | | `gh_branch` | daily | `branch`, `is_default` | `branches`, `oid`, `days_since_commit` | | `gh_deployment` | dated, when the deployment was created | `deployment`, `environment`, `task` | `outcome`, `deployments`, `deployment_state`, `success`, `superseded`, `creator`, `commit`, `ref`, `log_url`, `environment_url`, `run_id`, `seconds_to_status`, `seconds_live`, `url` | | `gh_policy_file` | dated, when the path last changed | `file` (dependabot, codeowners, security, funding) | `present`, `bytes`, `changes`, `path`, `blocks`, `ecosystems`, `url` | | `gh_dependabot_ecosystem` | dated, when dependabot.yml last changed | `ecosystem`, `interval` | `blocks` | | `gh_environment` | daily | `environment` | `environments`, `days_since_change`, `age_days`, `protection_rules`, `has_branch_policy`, `protected_branches`, `custom_branch_policies`, `can_admins_bypass`, `url` | | `gh_deploy_key` | daily | `key`, `read_only` | `keys`, `days_since_use` | | `gh_repo_policy` | now | | `security_policy`, `forking_allowed`, `discussions`, `issues`, `wiki`, `sponsorships`, `blank_issues`, `auto_merge`, `delete_branch_on_merge`, `merge_commit`, `rebase_merge`, `squash_merge`, `funding_links`, `issue_templates`, `pull_request_templates`, `branch_protection_rules`, `codeowners`, `codeowners_errors`, `vulnerability_alerts`, `url` | | `gh_repo_total` | now | `visibility`, `archived`, `fork` | `commits`, `stars`, `forks`, `watchers`, `issues`, `issues_open`, `issues_closed`, `pulls`, `pulls_open`, `pulls_merged`, `pulls_closed`, `releases`, `discussions`, `labels`, `milestones`, `branches`, `tags`, `size_kb`, `repo_id`, `age_days`, `days_since_push`, `url` | | `gh_dependency` | daily | `ecosystem` | `packages` | | `gh_dependency_license` | daily | `license` | `packages` | | `gh_dependency_change` | now | `change`, `ecosystem` | `packages`, `vulnerable`, `base`, `head` | | `gh_rate_limit` | now | `resource` | `limit`, `used`, `remaining`, `used_ratio`, `seconds_to_reset`, `own_cost`, `own_queries` | Webhooks fail silently. Measured, one hook had been answering 403 for seventy-eight of its last hundred deliveries and nothing anywhere said so. A 404 from branch protection does not mean unprotected: a repository can be governed entirely by rulesets, which that endpoint knows nothing about. `gh_ruleset_version` is the changelog behind `gh_ruleset`: one row per saved version of a ruleset, dated the moment GitHub saved it, with the actor that saved it. `days_since_change` only summarizes that history: a ruleset switched off on a Tuesday and back on the Friday after reads as "changed three days ago", and nothing else collected says a protection was ever absent. GitHub names the actor by id and type and not by login, so the row carries `actor_type` as a tag and `actor_id` as a field. The family is `rulesets`, daily: one list request per repository and one history request per ruleset, both with an ETag, so a day on which nobody edited a protection costs nothing from the budget. Measured on 2026-09-11 against the ruleset guarding the busiest repository measured: twenty versions across five months, 3 KB, one core request. Only the host of a webhook URL is stored. The path usually carries a secret. `gh_repo_policy` and `gh_repo_total` arrive in one batched GraphQL query that costs a single point for ten repositories, which is why settings that a REST sweep would price at a hundred and ninety eight calls are collected at all. `codeowners_errors` is the one that fails silently: a broken CODEOWNERS file stops requesting reviews and says nothing. `vulnerability_alerts` rides in that same query at no extra cost and is a second, independent reading of the switch `gh_security_feature{feature="dependabot"}.enabled` reports: one is the repository's own setting, the other is whether the listing actually answered. Two sources that disagree is the case worth seeing. The three dependency measurements are off by default. The SBOM is one call and a megabyte or two per repository, and only the aggregate is kept: a single dependency bump is three hundred and seventy changes, and what is stored is six rows. The commit a diff ends at, and the next one starts from, is read as the bare SHA of `HEAD` under the `application/vnd.github.sha` media type: forty bytes, where the one-commit listing it used to read was five and a half kilobytes. The answer carries an ETag and is asked for conditionally, so on a repository nobody pushed to the day's read is a free 304, as the listing's was. The SBOM is read only when that head moved: GitHub regenerates it on every request, so its ETag never matches and each read is charged from its own bucket, and a repository without a commit has the packages it had. `gh_rate_limit` is the only measurement the collector takes of itself. GitHub runs fifteen independent budgets, and without this a family skipped for want of budget looks exactly like a family with nothing to report. `GET /rate_limit` reports the budgets and charges for none of them, which is what makes almost all of this free. Not every one it reports is true: measured with the token this runs under, the endpoint answered `graphql` as `used=0, remaining=5000` in the same minute GraphQL itself answered `used=162` and moved by one on every query, and the two do not even share a clock. So the `graphql` row is built from the `rateLimit` block GraphQL answers with, and the endpoint's version of it is dropped rather than published beside it. That reading is a request every fifteen minutes and costs nothing in points, measured. `graphql` is not the only bucket the endpoint invents. Measured on 2026-09-12, an SBOM request's headers said `dependency_sbom` used 1, remaining 99, reset in 59 s, and `GET /rate_limit` two seconds later said used 0, remaining 100, its reset sliding forward a second per call. Every REST answer names the bucket it charged in its headers, so a row is built from the newest headers the client saw whenever they are still inside their own window and say more was spent than the endpoint admits. The windows of `dependency_sbom` and `search` are one minute, so a `dependency_sbom` row that reads zero between runs of the `deps` family is a refilled bucket, not the defect. The `graphql` row can therefore be absent, where the others are written whenever the endpoint answers at all. It is not written when nothing has been read from GraphQL yet and no earlier reading is still inside its window: a missing row says "not measured" where a zero says "nothing spent", and the zero was the defect. `own_cost` and `own_queries` are on that row alone. `limit`, `used` and `remaining` describe the whole token's window, shared with whatever else holds it; these two are the part this process is answerable for. They count from process start, so a restart returns them to zero and a panel has to read them as a counter rather than a value. ## Job logs | Measurement | Dated | Tags | Fields | | ------------ | -------------------------------- | ----------------------------- | --------------------- | | `gh_job_log` | dated, when the line was printed | `workflow`, `job_name`, `run` | `line`, `head_branch` | Off by default: set `every.joblogs`. It is text rather than a measurement, so it is excluded from the InfluxDB sink by default and skipped by the Prometheus exporter; Loki is where it belongs. The exported dashboards carry a text panel, "Where failure output went", in the place the lines would take, since an importer may have no Loki; `cmd/publish_dashboard -loki ` publishes the dashboard with the lines drawn from Loki in that panel's place (see [the dashboards](/ghchronicle/dashboards/panels/#delivery-and-access)). Only failed jobs, and only the last forty lines of each. A successful job's output is thousands of lines nobody will read, each log costs a request, and the tail is where a failure explains itself. GitHub keeps logs for exactly ninety days and answers 410 after that, so there is no backfilling them. `workflow` here is the same file path, through the same helper, so a log line joins to the run that printed it; and `branch` became the field `head_branch` for the same two reasons as on the run. `run` is deliberately a series per run, which is what a log line genuinely belongs to, and it is affordable only because this family is off by default. Colour codes are stripped and the byte order mark GitHub writes before the first timestamp is removed, so a search for a word does not fail because the word happened to be coloured. The failure list is asked only for the runs created in the thirty one days before the window opened, rounded down to the day. Thirty one days because a re-run keeps the created_at of its first attempt and GitHub allows one for thirty days: measured on 2026-09-11, the newest failure of the busiest repository measured was the third attempt of a run created two hours before it finished, and a margin the length of a job would have missed every re-run of a failure older than a morning. Unfiltered the list was the newest hundred failures the repository ever had, six hundred kilobytes per repository per sweep for a window of an hour that is nearly always empty; filtered, a month of failures, sixty eight rows and a megabyte decompressed on that busiest repository, a few rows or none on most. Rounding keeps the URL, and with it the ETag, the same across the sweeps of a day, which is what makes the repeat a free 304 rather than a charged 200 on a fresh URL; within the day the page changes only when a failure is created or re-run. The cut at the window itself is still made here, by when the run finished. ## Cost | Measurement | Dated | Tags | Fields | | ------------------ | -------------- | --------------------------------------- | --------------------------------------------------------------- | | `gh_billing_usage` | dated, per day | `product`, `sku`, `unit`, `repo`, `org` | `quantity`, `price_per_unit`, `gross`, `discount`, `net`, `url` | `unit` is GitHub's own `unitType`, capitalised as GitHub sends it: `Minutes`, `GigabyteHours`, `AICredits`, `Requests`. It is passed through rather than normalised, and the panels that read minutes filter on the capital. `net` is not always zero. On the account this was developed against it carries the monthly credit, which is why gross, discount and net are all stored rather than one being derived from the others. The row is stamped at the start of its day: GitHub's `date` arrives as the first billed minute of the day on half the rows, which would double the row had GitHub reported a different minute on the next read. ## Columns that exist only once written InfluxDB 3 creates a column the first time a row carries it, and a query that names a column no row has written fails at planning rather than answering null: the whole panel goes red. So a field written only when GitHub has a value for it does not exist on a database where that has never happened. The ones a fresh database is most likely to lack: `gh_discussion.state_reason`, `seconds_to_answer` and `seconds_to_close`; `gh_milestone.days_to_due` and `seconds_to_close`; `gh_ruleset_rule.ref_exclude`; and `gh_workflow_run.initial_actor`. The same rule covers every `seconds_to_*` that needs a closing, every `url` on an item GitHub sends no address for, the optional advisory fields on an alert, `checks_total` and `checks_failed` on a commit a gate ran on, `label_names` on an item with a label, `resolved_by` on a resolved thread, `queued_seconds` on a run's first attempt, and `pull_request`, `pull_requests`, `headline` and `head_repo` on a run GitHub linked, described or took from a fork. Two more are worth naming, because the tables above list them beside fields that are always there: `gh_dependabot_alert_item.dismissed_comment`, written only when whoever dismissed an alert typed a reason, and `gh_event.commits`, written only on a push event. Neither column exists on the production database this documentation was checked against. `gh_label` writes only the labels somebody has used; `gh_repo_total.labels` is the declared count. --- # Choosing a store Ten stores, what each one can and cannot answer, and the one property that decides between them. Source: https://jmrplens.github.io/ghchronicle/sinks/ Ten sinks, and running more than one is the normal arrangement. Every one of them pushes: the tool is meant to run wherever it is convenient and reach its stores from there, not to be scraped. The Prometheus exporter is the one exception, and it exists because Prometheus insists. ## The comparison | Store | Keeps | Good for | Config | | -------------------------------------------------- | -------------------------------------------- | ---------------------------------------------- | ------------------------------- | | [InfluxDB](/ghchronicle/sinks/influxdb/) | the dated history | "how fast were we merging in July" | `url`, `token`, `org`, `bucket` | | [PostgreSQL](/ghchronicle/sinks/postgres/) | the dated history, as SQL you pipe into psql | a Grafana user with a Postgres and no InfluxDB | `dialect`, `path` | | [Graphite](/ghchronicle/sinks/graphite/) | the dated history | a Graphite that is already there | `addr`, `prefix` | | [Elasticsearch](/ghchronicle/sinks/elasticsearch/) | the dated history, as documents | search across everything collected | `url`, `prefix`, `api_key` | | [Prometheus](/ghchronicle/sinks/prometheus/) | the current value | alerting, and a number on a wall | `listen`, `path` | | [OpenTelemetry](/ghchronicle/sinks/otlp/) | either, depending on the backend | an existing collector pipeline | `endpoint`, `raw` | | [Loki](/ghchronicle/sinks/loki/) | the events, as log lines | "what happened, in order" | `url`, `labels`, `max_age` | | [Telegraf](/ghchronicle/sinks/telegraf/) | whatever its outputs keep | reaching anything Telegraf can reach | `url` | | [File and stdout](/ghchronicle/sinks/file/) | line protocol or JSON | a shipper you already run, and a buffer | `path`, `format` | There is no sink for a specific hosted vendor, and that is deliberate. A managed backend is reached through one of the two sinks that exist to route onward: Telegraf, whose own outputs cover Datadog, New Relic, Wavefront, Azure Monitor and a hundred more, or OpenTelemetry, which most of them now accept directly. A sink per vendor is a key to rotate, an API to track and a test that needs a paid account, for a hop those two already make. ## The thing that decides everything A point carries the date the thing happened. A star is dated when it was given, a workflow run when it finished, a traffic day at that day's own date. InfluxDB keys a point by measurement, tag set and timestamp, so writing the same fourteen-day traffic window every six hours converges on the right answer rather than accumulating copies. That is what makes the whole backfill design work, and it is why InfluxDB is the sink that keeps history. Prometheus cannot do that. It stamps a sample at scrape time and rejects anything meaningfully older: measured against Prometheus 3.14 with the OTLP receiver enabled and a thirty-minute out-of-order window, a sample dated two days back comes back as HTTP 400. So the exporter reduces the per-item rows to current values before serving them. The full argument, and what the reduction does to each measurement, is on [dating a point](/ghchronicle/how/dating/). ## Only what changed is written A sweep offers the same history every time: the fourteen day traffic window, every open pull request, the contribution calendar. Writing it all again is harmless to what the store holds, since a row is keyed by series and timestamp and simply overwrites, and it is how a collector that was down for a day repairs itself. It is not harmless to the store's files. InfluxDB 3 Core writes one Parquet file per partition per write request and never compacts them, and it refuses any query that would open more than its file limit. Measured before the fix below and with the limit then at ten thousand: `gh_notification` held barely more than one row per Parquet file, and a query over fourteen days came back with "Query would scan 10000 Parquet files, exceeding the file limit". Asking for a coarser interval does not help, because the limit counts the files the planner opens, before any aggregation. So the tool keeps a small ledger of what it has already written and sends only the points whose values have moved: ```yaml sinks: dedupe_file: /var/lib/ghchronicle/state-written.bin # default: beside state_file dedupe_horizon: 720h # forget a point nothing offers any more influxdb: dedupe: true # the default, here and for telegraf, graphite, sql and elasticsearch ``` The ledger stores two 64 bit hashes and a day per point, so a large account costs a few megabytes. It is keyed by sink, so a store that was unreachable still receives everything on its next write. Losing it, or setting `dedupe_file: off`, costs one sweep of rewriting and nothing else, which is exactly what a store that has been wiped and needs filling again wants. It is the second file worth putting on a persistent path, next to the [state file](/ghchronicle/configuration/). A run that ends when its sweep does never opens the ledger at all. `-once`, `-backfill` and a card render write once and exit, so there is nothing to save and nothing to prune, and each of them offers the whole history again. That is what a backfill is for; it is also why a scheduled `-once` job, which is the shape [the Action](/ghchronicle/install/actions/) runs in, writes every point every time. Where that matters, run the loop instead. The sweep log says what this saved: ```text level=INFO msg=written sink=influxdb family=events points=0 unchanged=300 ``` If the files have already accumulated, the writer fix stops the growth but does not remove them: raise `--query-file-limit` on the server, rewrite the affected tables, or move to InfluxDB 3 Enterprise, which compacts on its own and is free for home use. ## Start here - [You want the history](/ghchronicle/sinks/influxdb/): InfluxDB is the reference implementation and the dashboard is built against it. PostgreSQL, Graphite and Elasticsearch keep the same facts in their own shapes. - [You want alerts](/ghchronicle/sinks/prometheus/): Prometheus serves the current value of everything that has one. Run it alongside a history store rather than instead of one. - [You want to read what happened](/ghchronicle/sinks/loki/): Loki turns twenty-two of the measurements into log lines that read as sentences and carry every tag after them in logfmt. - [You already have a pipeline](/ghchronicle/sinks/telegraf/): Telegraf and OpenTelemetry hand the points to something that already knows where they should go. ## Running several Normal, and cheap: the collection happens once and the points are handed to every configured sink. The usual arrangement is one history store plus one of the current-value ones. ```yaml sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} org: default bucket: github prometheus: listen: 127.0.0.1:9605 path: /metrics ``` > **One combination to avoid** > > `sinks.sql.path: "-"` and `sinks.stdout: true` both write to standard output, > and interleaving SQL statements with line protocol produces a stream that > neither psql nor Telegraf can read. Choose one. ## What every sink does with a failure A sink that fails is logged and the sweep continues; a database being down does not stop collection, and with the file sink configured the data is still on disk when it comes back. Two failures are reported specially rather than as errors, because they are partial successes: - **rejected lines**, where everything parseable was written and the refused lines are logged individually with the store's own reason. - **dropped entries**, which is Loki's age horizon leaving out what its out-of-order window would have refused, rather than losing the whole push. --- # InfluxDB The reference store for the dated history, why re-collection converges, and the one write error worth recognising. Source: https://jmrplens.github.io/ghchronicle/sinks/influxdb/ ```yaml sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} org: default bucket: github batch: 5000 # exclude: [gh_job_log] ``` ## The wire contract Line protocol posted to the **v2 write endpoint**, which InfluxDB 2 and 3 both serve, so one sink covers both. The token goes in an `Authorization` header, `org` and `bucket` in the query string. Precision is **nanoseconds**, because the traffic points are days and the workflow ones are seconds and one precision has to cover both. Batches default to 5000 lines per request. `batch` lowers that for a server with a smaller body limit. ## What it can answer that the others cannot Everything dated, which is most of the project: - The traffic of a particular Tuesday, months later. - The star curve since the first star, drawn from one row per star. - The merge time of a pull request closed in July. - Lines added and removed per commit, per author, over years. InfluxDB keys a point by measurement, tag set and timestamp, so **rewriting a point that already exists is not a duplicate**. Replaying the same fourteen-day traffic window every six hours converges instead of accumulating, which is what makes the whole [backfill](/ghchronicle/how/backfill/) design work. The [dashboards](/ghchronicle/dashboards/) are generated against this store first; the other four query sets are translations of it. ## Rewriting is free in rows and not in files InfluxDB 3 Core writes one Parquet file per partition per write request and never compacts them, and it refuses any query that would open more than its file limit. So the row that overwrites harmlessly still costs a file, and a sweep that offers the same history every six hours buys "Query would scan 10000 Parquet files, exceeding the file limit" a few weeks later. That is why the write ledger exists, why `dedupe` is on by default here, and why turning it off is a decision rather than a tidy-up: [only what changed is written](/ghchronicle/sinks/#only-what-changed-is-written). ## `exclude` Names measurements this sink should not receive. It defaults to `gh_job_log`, which is text meant for a log store: writing thousands of lines of build output into a metrics database is a lot of storage for something nobody will query as a number. ## Column types are fixed on first sight > **InfluxDB 3 will not let a name change side** > > InfluxDB 3 fixes a column as a tag or a field the first time it sees it, and > rejects later writes that disagree. If a version of this tool ever moves a name > between the two, the table has to be dropped rather than repaired: > > ```http > DELETE /api/v3/configure/table > ``` > > This is the usual cause of `influx write: 400`. ## Rejected lines A write that comes back 400 is bisected: the sink halves the batch, retries, and narrows down to the individual lines the server refuses to parse. Those are logged one by one with the server's own reason, everything else is written, and the sweep reports a warning rather than a failure. ```text level=WARN msg="sink rejected some lines" sink=influxdb family=actions rejected=2 ``` That behaviour matters because a batch is five thousand lines. Failing the whole batch on one malformed value would lose four thousand nine hundred and ninety-nine good points. ## Grafana Use the **InfluxDB 3 datasource in SQL mode** for the shipped dashboard, and point it at the database the sink writes to. The dashboard file declares `DS_INFLUXDB` as an input, so importing asks you to choose your own datasource rather than carrying somebody else's uid. ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = 'ghchronicle' ``` ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares InfluxDB with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Prometheus An exporter, not a pusher, and what the reduction to current values does to each measurement. Source: https://jmrplens.github.io/ghchronicle/sinks/prometheus/ ```yaml sinks: prometheus: listen: 127.0.0.1:9605 path: /metrics ``` An exporter, not a pusher: point a scrape at it. It is the one sink in the project that is not outbound, and it exists because Prometheus insists on pulling. ## What it serves Metric names are `github__`, with the tags as labels. ```text github_repo_stars{repo="ghchronicle",language="Go",visibility="public"} 283 github_workflow_runs_count{repo="ghchronicle",conclusion="success"} 412 ``` ## What it cannot serve The dated history, and not by choice. Prometheus stamps a sample at scrape time and rejects anything meaningfully older: measured against Prometheus 3.14 with `--web.enable-otlp-receiver` and a thirty-minute out-of-order window, a sample dated two days back comes back as **HTTP 400**. Through this sink, GitHub's fourteen-day traffic window collapses to its most recent day, and the star history to the current total. That is worth stating plainly rather than hiding. Run it alongside a history store rather than instead of one; both can run at once and the collection happens only once. ## The reduction Before serving, `Summarize` reduces each measurement according to a rule. | Rule | What survives | | ---------- | ----------------------------------------------------------------- | | `keepLast` | The most recent value per label set. Snapshots | | `sum` | The batch added up. Windows, such as views over the fourteen days | | `count` | A count plus the mean of each numeric field. Dated items | | `skip` | Nothing | A measurement with no rule is skipped, so a new collector cannot quietly flood the exporter with one series per star. The reducer also publishes `total`, a running distinct-item count per series. That is what lets a Prometheus dashboard say "per day" through `increase()`, since it has no rows to count. ## Two measurements are skipped for size The commit punch card is one series per repository, weekday and hour, and the release assets one per file ever published. Measured, together they were **four fifths of the exporter's entire output**. Both are drawn properly by the InfluxDB dashboard. Eight more are skipped for a reason other than size, seven of them as history and one as text, so ten measurements in all never reach the exporter. The list, with the reason for each, is on [dating a point](/ghchronicle/how/dating/#the-ten-that-are-never-served). The exporter holds only what the last sweep collected, and for workflow runs that is the newest thirty per repository between builds: an ordinary sweep reads the run list in pages of thirty, and pages on only while a page is full of runs newer than its two hour window. The stores keep every run the walk ever saw; this is the one place the smaller page is visible. Workflow jobs are the other: a sweep carries the jobs of the runs it listed for the first time, so `gh_workflow_jobs` counts the jobs new to this process rather than those of the newest twenty runs, and a sweep in which no run finished carries none, leaving the last count standing until it goes stale a day later. The `total` beside it is unaffected, since it counts every distinct job the process has ever seen; the stores are unaffected too, since a job is written once and dated when it finished. > **A tag called job would collide** > > Prometheus adds `job` and `instance` labels at scrape time, and the OTLP > receiver overwrites a `job` attribute with the service name. Workflow jobs are > therefore tagged `job_name`. ## Scraping it ```yaml scrape_configs: - job_name: ghchronicle static_configs: - targets: ["127.0.0.1:9605"] ``` The exporter holds its samples in memory, so a restart empties it. That is why the first sweep after start-up runs every enabled family whatever the state file says: without it, a twelve-hour family would leave its panels reading zero for half a day. ```yaml sinks: prometheus: listen: 0.0.0.0:9605 path: /metrics no_prime: true # leave the first sweep on its ordinary schedule ``` `no_prime: true` switches that priming sweep off, for an account whose quota is tight enough that a full sweep on every restart is not affordable. The price is exactly the behaviour the priming exists to avoid: until each family's cadence comes round, the panels that read it have nothing, and for a twelve-hour family that is half a day of zeros. The stores are unaffected either way, since they keep what was collected before the restart. A series that has not been rewritten for **24 hours** is dropped, so a repository that leaves the sweep stops being reported as if it were still there. The horizon is not configurable. ## Errors at start-up, not in a goroutine ```text prometheus exporter: listen tcp :9605: bind: address already in use ``` Reported when the process starts rather than swallowed in a background goroutine, so a port clash cannot leave you with a running collector and a silently missing exporter. ## In a container The example listens on `127.0.0.1`, which inside a container is the container's own loopback and unreachable from the host. Use `0.0.0.0:9605` there and let the port publication decide who can reach it. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares Prometheus with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # OpenTelemetry OTLP over HTTP with the JSON encoding, and why raw is false by default. Source: https://jmrplens.github.io/ghchronicle/sinks/otlp/ ```yaml sinks: otlp: endpoint: http://collector:4318/v1/metrics service: ghchronicle headers: Authorization: Bearer ${OTLP_TOKEN} raw: false batch: 2000 ``` ## The wire contract OTLP over HTTP with the **JSON encoding**, which every receiver worth using accepts on the same endpoint. That is a deliberate trade: JSON is larger on the wire, and it keeps a code generator, a protobuf runtime and their transitive dependencies out of a tool whose entire dependency list is one YAML parser. `endpoint` is the full URL of the metrics path, not a base. `headers` go with every request, which is where an API key or a tenant id belongs. `service` becomes the resource attribute the backend groups by. `batch` is how many data points go in one request, 2000 unless it is lowered for a receiver with a smaller body limit. Metric names use dots, following the OpenTelemetry convention: `github.workflow.run.duration.seconds`. A receiver exporting onward to Prometheus converts them itself; going the other way it cannot. ## `raw` - **raw: false (default)** Sends the same reduced current values as the Prometheus exporter. Safe with any backend, including Prometheus's own OTLP receiver, because nothing in the payload is older than the sweep. - **raw: true** Sends the dated points. Whether they survive is entirely the backend's decision: OTLP data points carry an explicit timestamp, so a store that accepts old ones keeps the history, and one that does not rejects them. > **Prometheus's OTLP receiver is one that does not** > > Measured: against Prometheus 3.14 with `--web.enable-otlp-receiver` and > `out_of_order_time_window: 30m`, a sample dated two days back comes back as > HTTP 400. With `raw: true` pointed at Prometheus, roughly half of every sweep > is refused. Use `raw: false` there, or send to a store that keeps the dates. ## `repeat` A gauge asserted once and then never again fades from the dashboards, and a family that runs every twelve hours would be a flat line with a dot in it. With `repeat` set, the sink keeps asserting the newest value of every series it has seen until the next sweep replaces it. ```yaml sinks: otlp: endpoint: http://collector:4318/v1/metrics repeat: 1m ``` This matters for a backend that answers an instant query from the last sample within a lookback window. ## A collector in front The usual arrangement is a collector that receives from here and exports onward, which is what makes this sink worth having at all: the tool speaks one protocol and the pipeline decides where the data ends up. ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 exporters: prometheusremotewrite: endpoint: http://prometheus:9090/api/v1/write service: pipelines: metrics: receivers: [otlp] exporters: [prometheusremotewrite] ``` With that pipeline, keep `raw: false`: the exporter at the far end is Prometheus, and the constraint follows the data rather than the protocol. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares OpenTelemetry with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # PostgreSQL INSERT statements you pipe into psql, the schema they declare, and why the conflict clause updates rather than does nothing. Source: https://jmrplens.github.io/ghchronicle/sinks/postgres/ ```yaml sinks: sql: dialect: postgres path: /var/lib/ghchronicle/points.sql max_bytes: 67108864 keep: 5 ``` INSERT statements in PostgreSQL's dialect, written to a rotating file or, with `path: "-"`, to standard output. ```sh ghchronicle -config config.yaml -once | psql "$DATABASE_URL" ``` This is the sink for the Grafana user who self-hosts PostgreSQL or TimescaleDB and runs no InfluxDB. The tool cannot speak the PostgreSQL wire protocol without a driver, and a driver is a dependency this repository does not take, so it emits the SQL and leaves the connection to psql. Grafana's PostgreSQL datasource then has a real schema to query. ## The schema is the contract - One table per measurement, named after it: `gh_repo`, `gh_traffic`, `gh_workflow_run`. - `time TIMESTAMPTZ NOT NULL`, the date the thing happened. - One `TEXT NOT NULL DEFAULT ''` column per tag, an empty string where the point had no value. - One column per field, typed from the value: `BIGINT` for an integer, `DOUBLE PRECISION` for a float, `BOOLEAN`, `TEXT`, and `TIMESTAMPTZ` for a field that is itself a time. - `PRIMARY KEY (time, )`. - Every identifier is double-quoted, because `user`, `type` and `state` are tag names here and reserved words there. That primary key is InfluxDB's series key spelled as a constraint, and it is what makes a rewrite of the fourteen-day traffic window converge instead of accumulate. ```sql CREATE TABLE IF NOT EXISTS "gh_traffic" ("time" TIMESTAMPTZ NOT NULL, "full_name" TEXT NOT NULL DEFAULT '', "kind" TEXT NOT NULL DEFAULT '', "owner" TEXT NOT NULL DEFAULT '', "repo" TEXT NOT NULL DEFAULT '', "count" BIGINT, "uniques" BIGINT, "url" TEXT, PRIMARY KEY ("time", "full_name", "kind", "owner", "repo")); INSERT INTO "gh_traffic" ("time", "full_name", "kind", "owner", "repo", "count", "uniques", "url") VALUES ('2026-09-07T00:00:00Z'::timestamptz, 'acme/telemetry', 'views', 'acme', 'telemetry', 41, 12, 'https://github.com/acme/telemetry/graphs/traffic') ON CONFLICT ("time", "full_name", "kind", "owner", "repo") DO UPDATE SET "count" = EXCLUDED."count", "uniques" = EXCLUDED."uniques", "url" = EXCLUDED."url"; ``` > **DO UPDATE, not DO NOTHING** > > Today's traffic row is rewritten with a higher count on every sweep. A row > frozen at its first value would be the one bug the whole dated-point design > exists to avoid. ## How the declarations arrive The `CREATE TABLE IF NOT EXISTS` is emitted the first time a measurement is seen in a file, with the union of the columns that batch carries. A column that turns up in a later batch arrives as `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. A rotated file starts its declarations again, so any one file can be replayed on its own. A tag first seen after the table was declared cannot join the primary key without rewriting it, so it becomes a plain column. That only happens when a collector changes its tag set between sweeps. ## TimescaleDB Turn each table into a hypertable once it exists. The primary key already includes `time`, which is the one condition TimescaleDB puts on it. ```sql SELECT create_hypertable('gh_traffic', 'time', if_not_exists => TRUE); SELECT create_hypertable('gh_workflow_run', 'time', if_not_exists => TRUE); ``` Nothing in the dashboard changes. ## Setting it up 1. Point the sink at a file, or at standard output for a direct pipe. 2. Load it. ```sh psql "$DATABASE_URL" -f /var/lib/ghchronicle/points.sql ``` Or, for the streaming arrangement, run the collector with `-once` from a scheduler and pipe it straight in. 3. Point Grafana's PostgreSQL datasource at the database and import `ghchronicle-postgres.json`. ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = $repo ``` `dashboards/ghchronicle-postgres.json` has the same 152 panels as the InfluxDB one, with every query translated to PostgreSQL against this schema. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares PostgreSQL with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Graphite The plaintext protocol over TCP, and the exact metric path, which is the dashboard's contract. Source: https://jmrplens.github.io/ghchronicle/sinks/graphite/ ```yaml sinks: graphite: addr: graphite:2003 prefix: github batch: 1000 ``` The plaintext protocol over TCP: `path value timestamp`, one line per numeric field. String fields are skipped, because Graphite has no way to hold one. Graphite keeps a point at the time it was given, so the raw dated points go out as they are and the traffic window lands on its own days. Writing the same point twice fills the same slot of the same whisper file, which is exactly what a rewrite of the window wants. The connection is reopened on a failed write, once, before the write is reported as failed. ## The path is the dashboard's contract ```text ... ``` - Every tag is one node, holding its value. The nodes are ordered by the tag's **key**, alphabetically, never by the order a collector set them. - An empty tag value is written as `none`, so a measurement's depth never changes from one point to the next and `github.repo.*.*.*.*.*.*.*.*.*.stars` keeps matching. - A node keeps ASCII letters, digits, `_`, `-` and `:`. Everything else becomes `_`: the dot, the space, the comma, and the slash in `owner/repo`. A dot would split the node and a slash would nest a directory. So a `gh_repo` point tagged `archived=false default_branch=main fork=false full_name=acme/edge-cache language=Go license=MIT owner=acme repo=edge-cache visibility=public` with a `stars` field becomes: ```text github.repo.false.main.false.acme_edge-cache.Go.MIT.acme.edge-cache.public.stars 37 1757280000 ``` A field that shares its name with a tag is skipped, the same rule the line protocol applies: the tag wins, because it is the one that can be grouped by. > **A new tag changes the path depth** > > Adding a tag to a collector inserts a node into every path of that > measurement, so every existing dashboard target for it stops matching. This is > the cost of a store with no schema, and it is why `TAGS` in the dashboard > specification mirrors the tag set of each measurement. ## What it cannot answer Graphite keeps the dated points but has no rows. A series is a path and a number, so: - A table that needs several fields of one row cannot be built. The shipped dashboard keeps the column it is sorted by and says in the panel description which columns it dropped. - A boolean is not a metric there at all, and neither is a title or any other string. Those panels say so. Everything time-shaped works normally, which is most of the dashboard. ## The dashboard `dashboards/ghchronicle-graphite.json` has the same 152 panels as the InfluxDB one, written against these paths with the default prefix. It needs **Graphite 1.1 or later** for the functions it uses, and any panel a series cannot carry says so in its description. Change `prefix` and the dashboard targets have to change with it, since the prefix is the first node of every path. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares Graphite with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Elasticsearch The bulk API, one index per measurement, and a document id that makes a rewrite replace rather than duplicate. Source: https://jmrplens.github.io/ghchronicle/sinks/elasticsearch/ ```yaml sinks: elasticsearch: url: http://elasticsearch:9200 prefix: ghchronicle api_key: ${ES_API_KEY} # or username and password batch: 1000 ``` The `_bulk` API, which Elasticsearch and OpenSearch share, so the same sink serves both, and Kibana or the OpenSearch dashboards on top of either. ## Credentials - **API key** ```yaml api_key: ${ES_API_KEY} ``` Sent as `Authorization: ApiKey`. - **Basic** ```yaml username: ghchronicle password: ${ES_PASSWORD} ``` Sent as basic auth. Not both. Configuring an API key alongside a username fails validation at start-up with `sinks.elasticsearch: set either api_key or username and password, not both`. ## The documents One index per measurement, named `-`: `ghchronicle-gh_repo`, `ghchronicle-gh_traffic`. One document per point, with the time as `@timestamp` in RFC 3339, `measurement`, and every tag and field as a top-level key, so nothing has to be unnested before it can be filtered on. ```json { "@timestamp": "2026-09-07T00:00:00Z", "measurement": "gh_traffic", "owner": "acme", "repo": "telemetry", "full_name": "acme/telemetry", "kind": "views", "count": 220, "uniques": 131 } ``` ## Why re-collection converges here too The document id is the **SHA-256 of the measurement, the tags that are set and the timestamp**, and the action is `index` rather than `create`. So writing the same fourteen-day traffic window every six hours replaces fourteen documents instead of adding fourteen more, which is the same convergence InfluxDB gives for free. > **A bulk request answers 200 even when items failed** > > Elasticsearch reports per-item verdicts inside a successful response. The sink > reads them, logs each refused document with the cluster's own reason, and > reports the count as a warning rather than a failure, because everything else > in the batch was written. ## No mapping is written The sink creates no index template. Dynamic mapping gives every string field a `.keyword` sub-field, which is what the dashboard aggregates on. If you want explicit mappings, create the index templates before the first write. Nothing in the sink depends on them; only the panels' choice of `.keyword` does. ## The dashboard `dashboards/ghchronicle-elasticsearch.json` has the same 152 panels as the InfluxDB one, as Lucene filters and aggregations over **one** datasource pointing at `-*`, because each target names its own index in its query. Set the datasource's time field to `@timestamp`. A per-item table there is the newest documents themselves; everything else is a bucket aggregation. OpenSearch works through the same plugin. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares Elasticsearch with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Loki The twenty-two measurements that are events rather than numbers, and the age horizon that keeps a push from being refused. Source: https://jmrplens.github.io/ghchronicle/sinks/loki/ ```yaml sinks: loki: url: http://loki:3100/loki/api/v1/push tenant_id: "" labels: job: ghchronicle max_age: 1h batch: 1000 ``` ## Measurements and events Some of what GitHub reports is a measurement and some of it is an event. "The repository has 148 stars" is a measurement. "Someone starred it at 03:03, this release was published, that workflow failed on main, this alert was raised" are events: each happened once, at a known moment, and what you want later is to read them in order and search them, not to average them. **Twenty-two measurements have an event rendering**: stars in both directions, forks, releases, published package versions, pull requests, reviews, review threads, issues, commits, workflow runs, job logs, repository activity, Dependabot alerts, code scanning analyses, the event feed, notifications, discussions, webhook deliveries, deployments, ruleset versions and external contributions. Everything else is a gauge in disguise and is not sent. A measurement with no rendering is dropped in silence, which is right for a gauge and wrong for an event nobody has got round to: deployments and review threads were dated events with no log line for months, and nothing said so. So every dated measurement now has to appear in one of two tables in `internal/sink/loki.go`, the renderings or the refusals, and each refusal carries the reason it is not a log line. A test fails on a dated measurement that appears in neither. ## The line format Each line reads as a sentence first and carries every tag and field after it in logfmt, so the same line is greppable in a terminal and queryable in Grafana without keeping two copies of the data. ```text someone starred acme/telemetry full_name="acme/telemetry" user="someone" starred=1 ``` `batch` is how many entries go in one push, 1000 unless it is lowered. The stream label is `kind`, which is what you filter on first. ```text {job="ghchronicle", kind="workflow_run"} |= "failure" {job="ghchronicle", kind="job_log"} ``` > **Keep the other labels few** > > Loki indexes labels, and a high cardinality label costs far more than a wide > line. `labels` in the configuration is for the fixed ones that identify this > collector, not for anything that varies per point. ## `max_age`, and why the reason is not the obvious one Loki refuses an entire push when one entry predates `reject_old_samples_max_age`, a week by default, and half of what this collector produces is older than that on purpose: a star from 2020, a pull request from 2024. But the limit that actually bites is the other one. Loki also refuses an entry more than its out-of-order window behind the newest entry already in that stream, about two hours by default. **Measured against a real Loki 3**: once the stream held an entry from 19:14, one from 00:35 the same day came back as "entry too far behind". So the horizon is applied three ways: 1. against the wall clock, 2. against the newest entry of each stream inside the batch, 3. against the newest entry that stream has been sent before. What falls outside is left out and counted, at debug level, rather than costing the whole push. `max_age` defaults to one hour, which is inside Loki's default window. Raise it only if you have raised `out_of_order_time_window` to match. ## What Loki is not for The dated history. That is what a metrics store is for, and it is why the two run together rather than one replacing the other. A log answers "what happened recently, in order"; a time series answers "how much, over which period". ## Job logs belong here `every.joblogs` collects the last forty lines of every failed GitHub Actions job. It is text rather than a measurement, so the InfluxDB sink excludes it by default and the Prometheus exporter skips it. Loki is where it belongs, and the query is: ```text {job="ghchronicle", kind="job_log"} ``` The exported dashboards do not show it, because a dashboard bound to one datasource cannot query two and an importer may have no Loki: they carry a text panel, "Where failure output went", with that query. On a Grafana that has a Loki datasource, `cmd/publish_dashboard -loki ` publishes the dashboard with the lines drawn from Loki in that panel's place, newest first, filtered by the dashboard's repository variable where the store's variable can be read as a regular expression. The steps are in `dashboards/PUBLISHING.md`. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares Loki with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Telegraf Line protocol posted to http_listener_v2, and the reason this is one sink instead of a hundred. Source: https://jmrplens.github.io/ghchronicle/sinks/telegraf/ ```yaml sinks: telegraf: url: http://telegraf:8186/telegraf username: "" password: "" batch: 5000 ``` Line protocol posted to Telegraf's `http_listener_v2` input, as `text/plain`, with basic auth when a username is set. > **A URL with no path gets /telegraf** > > The listener answers 404 to `/` without saying why, so a URL with an empty > path is given the input's default path rather than being sent somewhere that > will silently fail. ## The matching input ```toml [[inputs.http_listener_v2]] service_address = ":8186" paths = ["/telegraf"] data_format = "influx" ``` ## Why this exists as one sink Telegraf has an output for Kafka, Datadog, New Relic, Graphite, Loki, Elasticsearch, Wavefront, Azure Monitor, Google Cloud Monitoring and a hundred more. Rather than grow a sink here for each of them, the points go to Telegraf as the line protocol it already speaks, and its own `[[outputs.*]]` blocks route them onward. That is the whole argument. It is one sink that reaches everything Telegraf reaches, and it costs this project no new dependency and no new wire format. ## What happens to the dates Telegraf keeps the timestamps as given, so the dated history survives as far as each output lets it. An output that stamps at receipt, or one that rejects old samples, loses it, exactly as those backends would if this tool wrote to them directly. So the question to ask is not "does Telegraf keep the history" but "does the output I configured keep it". Kafka and InfluxDB do. A Prometheus remote write does not. ## An example that fans out ```toml [[inputs.http_listener_v2]] service_address = ":8186" paths = ["/telegraf"] data_format = "influx" [[outputs.influxdb_v2]] urls = ["http://influxdb:8181"] token = "$INFLUX_TOKEN" organization = "default" bucket = "github" [[outputs.kafka]] brokers = ["kafka:9092"] topic = "github-metrics" ``` One sweep, both destinations, and the collector knows about neither. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares Telegraf with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # File and stdout A rotating file for a shipper you already run, the simplest durable buffer there is, and line protocol on standard output. Source: https://jmrplens.github.io/ghchronicle/sinks/file/ ## File ```yaml sinks: file: path: /var/log/ghchronicle/points.lp format: influx # or json max_bytes: 67108864 keep: 5 ``` For the setups that already run a shipper. Telegraf tails line protocol, Promtail and Vector tail JSON, and neither needs this process to know anything about their backend. It is also the simplest durable buffer there is: when the database is down, the file still has the data. ### Rotation By size, with numbered suffixes, so retention is a count and two rotations in the same second cannot collide. The size counter is read from the file on start-up, so a restart does not reset it and let the file grow without bound. - **influx** ```text gh_traffic,full_name=acme/telemetry,kind=views,owner=acme,repo=telemetry count=220i,uniques=131i 1788739200000000000 ``` - **json** ```json {"time":"2026-09-07T00:00:00Z","measurement":"gh_traffic","tags":{"full_name":"acme/telemetry","kind":"views","owner":"acme","repo":"telemetry"},"fields":{"count":220,"uniques":131}} ``` One object per line in JSON, which is the shape a line-oriented shipper wants. ## Standard output ```yaml sinks: stdout: true stdout_format: influx # or json ``` Line protocol on standard output, for piping into Telegraf or for seeing what would be written before pointing this at a database. It is the fastest way to answer "what does this actually collect". ```sh ghchronicle -config config.yaml -once | head -20 ghchronicle -config config.yaml -once | grep gh_workflow_run ``` `stdout_format: json` prints one object per line instead, the same shape the file sink writes. > **Do not combine it with the SQL sink on standard output** > > `sinks.sql.path: "-"` also writes to standard output. Interleaving SQL > statements with line protocol produces a stream neither psql nor Telegraf can > read. ## As a buffer The file sink is the answer to "what happens when the database is down". Run it alongside the real store: the write to the database fails and is logged, the sweep continues, and the points are on disk. Replaying them afterwards is a `curl` for InfluxDB, or `psql -f` for the SQL sink, and it converges rather than duplicating because the points carry their own timestamps. ```sh curl -s -XPOST "http://localhost:8181/api/v2/write?org=default&bucket=github&precision=ns" \ -H "Authorization: Token $INFLUX_TOKEN" \ --data-binary @/var/log/ghchronicle/points.lp ``` ## Permissions The dump is created `0600` inside a `0750` directory. A sweep of a private account puts private repository names, Dependabot severities and whole job log lines in it, so it is not created readable by the whole machine just because a shipper is coming to read it. ### Letting the shipper read it Telegraf, Promtail, Vector and Fluent Bit all read as their own user. None of them documents a required mode, and every one of them documents the same remedy, which is a group: their own answers say `usermod -aG adm`. So the grant is yours to make, and it is two commands: ```sh usermod -aG ghchronicle telegraf # the shipper joins the service's group chmod 0640 /var/log/ghchronicle/points.lp # or create it that way beforehand ``` The grant sticks. The sink never changes the mode of a file that is already there, and a rotation creates the replacement with the mode of the file it is renaming away, so the `chmod` is not undone the first time the dump fills up. > **The failure this prevents is a silent one** > > A shipper that cannot open the file it is tailing does not announce it, and > this process keeps writing to a handle it already holds. Neither end logs > anything, so the first symptom is a dashboard that stopped days ago. If you would rather the dump belonged to the shipper's own group, own the directory with it and set the setgid bit, which is what makes every file created in it inherit the group, rotations included: ```sh install -d -o ghchronicle -g telegraf -m 2750 /var/log/ghchronicle ``` That settles which group owns the dump, not what the group may do with it, so the `chmod 0640` above is still the other half. Making the directory yourself is worth doing whichever route you take: the one this process creates when the directory is missing is `0750` less whatever the process umask takes off it, and a shipper that cannot traverse the directory fails as quietly as one that cannot read the file. An ACL is not carried, because it belongs to the file it was set on and a rotation creates a new one, so grant through the group rather than through `setfacl` on the dump. What a rotation does carry is the mode, not the owner: this process cannot `chown` a file to a user it is not. The write ledger behind `sinks.dedupe_file` is the opposite case and gets the opposite answer: nothing but this process is meant to read it, so it is written `0600` and every save puts it back to `0600`. ### Where it can write at all Under the hardened systemd unit, the directory has to be in `ReadWritePaths`. In a container it has to be writable by uid 65532. Both are the same mistake in two clothes: the process is deliberately allowed to write almost nowhere. ## Where to go next - [Choosing a store](/ghchronicle/sinks/) compares the file sink with the other nine, and holds the write ledger every one of them shares. - [The dashboards](/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Importing Five generated Grafana dashboards, one per store, and how to import each of them. Source: https://jmrplens.github.io/ghchronicle/dashboards/ Five dashboards, all in English, all in Grafana's shareable export format: the datasource is a `${DS_...}` placeholder and the `__inputs` block asks the importer to choose their own. | File | Panels | Store | | -------------------------------- | ------ | -------------------------------------------------------- | | `ghchronicle-influxdb.json` | 152 | InfluxDB 3, queried with SQL | | `ghchronicle-prometheus.json` | 152 | Prometheus | | `ghchronicle-postgres.json` | 152 | PostgreSQL or TimescaleDB, from the SQL sink | | `ghchronicle-graphite.json` | 152 | Graphite, from the Graphite sink | | `ghchronicle-elasticsearch.json` | 152 | Elasticsearch or OpenSearch, from the Elasticsearch sink | The five hold the same panels in the same order. What differs is how many of them the store behind each one can answer. Each cell is the panels that store answers with a query, out of the panels in that section. A panel a store cannot answer ships as a text panel with the same title, so every dashboard has the same 152 panels; the 2 that are prose in all five are left out here. | Section | InfluxDB | PostgreSQL | Elasticsearch | Graphite | Prometheus | Panels | | --- | --- | --- | --- | --- | --- | --- | | Overview | 4 | 4 | 4 | 4 | 4 | 4 | | Lifetime | 5 | 5 | 5 | 5 | 5 | 5 | | Audience | 6 | 6 | 6 | 6 | 5 | 6 | | Stars and forks | 5 | 5 | 5 | 5 | 5 | 5 | | Contributions | 11 | 11 | 10 | 10 | 6 | 11 | | Pull requests and issues | 14 | 14 | 14 | 14 | 10 | 14 | | Continuous integration | 14 | 14 | 13 | 13 | 10 | 14 | | Code | 9 | 9 | 8 | 8 | 8 | 9 | | Planning and community | 10 | 10 | 10 | 10 | 8 | 10 | | Delivery and access | 13 | 13 | 13 | 13 | 13 | 13 | | Releases | 4 | 4 | 3 | 4 | 2 | 4 | | Security | 14 | 14 | 14 | 13 | 13 | 14 | | Cost | 6 | 6 | 6 | 6 | 6 | 6 | | Activity | 9 | 9 | 9 | 8 | 7 | 9 | | Inventory | 16 | 16 | 15 | 15 | 14 | 16 | | Profile and sponsorship | 8 | 8 | 8 | 8 | 8 | 8 | | The collector itself | 2 | 2 | 2 | 2 | 2 | 2 | | **Total** | **150** | **150** | **145** | **144** | **126** | **150** | ![The InfluxDB dashboard over ninety days of the demonstration database: the repository picker and the range across the top, the Overview with the ghchronicle badge and four tile groups reading 5 repositories with 350 stars and 51 forks, 37.5 thousand views with 21.1 thousand unique visitors and 19.6 thousand clones, 117 followers and 58 following with 4 sponsors and 2 sponsored, and 3.22 thousand contributions over 7.78 years, then the collapsed Lifetime header and the Audience section with views, unique visitors and clones per day, the top referrers, the top paths and clone amplification](../../../assets/dashboard-influxdb-demo.png) The account in that capture is the invented one every capture in this documentation uses, `acme` and five repositories, described beside [what the panels show](/ghchronicle/dashboards/panels/). Every section but the Overview ships collapsed, which is why Lifetime is a header there. ## Importing from the UI 1. In Grafana, go to **Dashboards**, then **New**, then **Import**. 2. Upload the `ghchronicle-.json` for the store you are using. 3. Choose the datasource Grafana asks for. - **InfluxDB** The InfluxDB 3 datasource for the database the sink writes to, **in SQL mode**. - **Prometheus** The Prometheus that scrapes ghchronicle's exporter. - **PostgreSQL** The PostgreSQL datasource for the database the SQL sink's statements were piped into. TimescaleDB is the same datasource with the TimescaleDB switch on; the queries do not change. - **Graphite** The Graphite the sink writes to. The paths assume the default prefix, `github`, and the functions need Graphite 1.1 or later. - **Elasticsearch** An Elasticsearch datasource whose index pattern is `ghchronicle-*` and whose time field is `@timestamp`. One datasource serves every panel, because each target names its own index in the query. OpenSearch works through the same plugin. ## Importing with the API Name the input the file declares: ```sh curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d "{\"dashboard\": $(cat ghchronicle-influxdb.json), \"inputs\": [ {\"name\":\"DS_INFLUXDB\",\"type\":\"datasource\",\"pluginId\":\"influxdb\",\"value\":\"\"}], \"overwrite\": true}" \ "$GRAFANA/api/dashboards/import" ``` The inputs are `DS_INFLUXDB` (`influxdb`), `DS_PROMETHEUS` (`prometheus`), `DS_POSTGRES` (`grafana-postgresql-datasource`), `DS_GRAPHITE` (`graphite`) and `DS_ELASTICSEARCH` (`elasticsearch`). ## The uid Each file carries a fixed `uid` (`ghchronicle-`). That is deliberate for a repository import, where a stable uid means a stable URL and a re-import updates in place rather than duplicating. Importing two of these into one Grafana is fine, because the uids differ per store and cannot collide. ## They are generated, never hand-edited `cmd/internal/dashboards` holds one ordered list of sections and panels, and every panel carries one query set per store. The generator picks one set and emits the JSON, so every file has the same panels in the same places with the same titles, and it refuses to write files whose layouts have drifted apart. ```sh go run ./cmd/gen_dashboards # writes all five files go run ./cmd/gen_dashboards -check # writes nothing, fails if they are stale ``` Edit `cmd/internal/dashboards/sections_*.go`, not the JSON. `panels.go` holds the panel constructors it uses, `query.go` the query helpers for each store, and `stores.go` only chooses a query set and a datasource. > **No builder is trusted without running the queries** > > The raw database API accepts things the Grafana plugin then fails to render, so > the checkers go through Grafana's own query path where a datasource exists. > > ```sh > GRAFANA_TOKEN=... go run ./cmd/check_dashboards influxdb > GRAFANA_TOKEN=... go run ./cmd/check_prometheus > go run ./cmd/check_postgres > ``` > > `check_dashboards` reports every panel as ok, empty or failing. > `check_prometheus` additionally checks each metric name against a live dump of > the exporter's own `/metrics`, because a typo in a metric name is not a syntax > error: PromQL parses it happily and returns nothing forever. > > Both of them, and `cmd/publish_dashboard`, read two variables: `GRAFANA_TOKEN` > for the credential, and `GRAFANA_URL` for the server. The compiled-in default > is `http://localhost:3000`, which is the address Grafana itself ships with, so > anything else has to be named: the first symptom of not naming it is a > connection refused. ## Publishing to the Grafana directory The files are already in the shape the directory requires: `__inputs` declares the datasource the importer must choose, `__requires` names the Grafana version and the plugin, and there is no `id` key, which the directory assigns on publication. Keep the listing names distinct, because five dashboards with the same title are indistinguishable in search results: "ghchronicle for InfluxDB", "ghchronicle for Prometheus", "ghchronicle for PostgreSQL and TimescaleDB", "ghchronicle for Graphite", "ghchronicle for Elasticsearch and OpenSearch". Publishing again against the same listing adds a revision rather than replacing it, so a regeneration that changes panels is a new revision of the same five listings, not five new listings. That is what a reader needs to know. The step by step, with the listing names, what each screenshot should show and what to do when a revision is rejected, is a maintainer's task and lives in [`dashboards/PUBLISHING.md`](https://github.com/jmrplens/ghchronicle/blob/main/dashboards/PUBLISHING.md) in the repository. --- # What they show The seventeen sections and their one hundred and fifty two panels, one capture each, and what changes when the store cannot answer. Source: https://jmrplens.github.io/ghchronicle/dashboards/panels/ One dashboard, rendered once per store. Seventeen sections, one hundred and fifty two panels, in the same places with the same titles whichever database you chose. One capture per section below, in the order the dashboard puts them. Every section but the Overview opens collapsed. Open, the first seven were twenty-three phone screens before the reader found out there were ten more; collapsed, the second screen of a phone is the index of the sixteen, each a tap away, and Grafana keeps what was opened in the URL. On a desktop it costs a click per section, and the first render asks for the Overview's panels rather than all of them. > **The section captures are of a demonstration database** > > Every section capture below is the InfluxDB dashboard over ninety days of a > demonstration database, generated so that every panel has something to draw. A > real account leaves several of them legitimately empty: no star this week, no > failed workflow in the window, no open milestone, and a capture of an empty > panel teaches nothing. The database was filled by a generator that is not part > of this repository, and it took every measurement, tag, field and column type > from the [measurements reference](/ghchronicle/collectors/measurements/) and > from a live database, so the shape is real. Only the account is invented: > `acme` and five repositories that are obviously examples. The one capture that > is not of it is the last on this page, of the Prometheus dashboard, which > comes from the containerised end-to-end suite instead, and the text beside it > says so. No capture on this page is of a real account. ## Overview A masthead rather than a panel: the mark, large and centered, the name under it, and under the name a button to this documentation and one to the source, on the page itself with no box around them. Then four groups of numbers: Repositories (repositories, stars, forks), Traffic in range (views, unique visitors, clones), Community (followers, following, sponsors, sponsoring) and Account (contributions in the last year, account age, watching, stars given, gists, packages). A group is one stat panel of several values rather than a tile per number: on a desktop it reads as the row of tiles it replaces, and on a phone, where every panel is a column, sixteen tiles were four screens of single numbers and four groups are one. The repository variable beside them filters every section at once. Stars and forks add up the newest row per repository rather than every row in the range, because both are current state and a sum over the range would count every sweep. ![The Overview row: the repository picker and the 90 day range across the top, the ghchronicle badge with its Docs and Source buttons, then four tile groups reading 5 repositories with 350 stars and 51 forks, 37.5 thousand views with 21.1 thousand unique visitors and 19.6 thousand clones, 117 followers and 58 following with 4 sponsors and 2 sponsored, and an account with 3.22 thousand contributions over 7.78 years](../../../assets/dashboards/overview.png) Reads `gh_account`, `gh_repo`, `gh_traffic` and `gh_contributions_total`. ## Lifetime Six numbers that are true since the account began, in one panel, and one row per repository with its whole life in it: pull requests merged and reviewed ever, commits ever, issues opened, what was merged in other people's repositories and what was commented there. ![The Lifetime section: a tile group reading 1.18 thousand pull requests merged, 386 reviewed, 6.68 thousand commits, 148 issues opened, 27 merged elsewhere and 96 comments elsewhere, the table of every repository ever with commits, merges, issues, releases, stars, branches and tags, and below it the repositories created, the single archived repository and workflow runs ever as a bar per repository](../../../assets/dashboards/lifetime.png) Every other section counts rows inside the dashboard range. These do not: GitHub counts them itself, in one search request or one GraphQL field each, and the collector stores the answer as a single row. That is why they are instant and why they are already right on the first sweep of a new install, which a running total kept by this tool would not be. It is also the shape a store can answer without reading everything it holds: InfluxDB 3 Core refuses a query that would open more than its file limit, forty thousand where this was measured, and "how many ever" from a row per fact is exactly that query. Reads `gh_account_total` and `gh_repo_total`. ## Audience Views, unique visitors and clones over time, the top referrers and the top paths. GitHub serves fourteen days and rewrites the whole window on every sweep, so the series extend as far back as the collector has been running rather than fourteen days. Referrers and paths carry no dates of their own, so they are the newest snapshot of that window rather than a series. ![The Audience section: views, unique visitors and clones per day as stacked bars per repository, the top referrers table led by Google with 250 views, the top paths table with each path's title, and the clone amplification table giving clones per cloner, around 1.3 in every repository](../../../assets/dashboards/audience.png) > **Clones do not count people** > > Continuous integration clones a repository thousands of times for every human > visit. Measured on one repository, seventy three clones for every unique > cloner. The last panel of the section divides one by the other, which is the > only figure that separates adoption from machinery. Reads `gh_traffic`, `gh_traffic_referrer` and `gh_traffic_path`. ## Stars and forks Stars gained over time, the cumulative curve, stars by repository, the fifty most recent stars with the user and the moment, and forks over time. The eight repositories that gained the most in the range are named and the rest are `other`: of the seventeen that gained a star in two years here, nine gained between one and three, and seventeen legend entries hid a third of the plot. The cumulative curve reaches back to the first star because the stargazer walk collected each one with its own `starred_at`, not because the collector has been running that long. Forks over time is drawn the same way from `gh_fork`, each fork dated when it was made, so it too reaches back past the day the collector started rather than beginning at the first snapshot. Both curves are drawn to both edges of the range: a running count over dated rows has a point only where a row is, so a fork curve over a quiet month was a line from the first fork to the last and blank on either side, which reads as collection having stopped. Each end carries a bucket of zero, so the line holds its value to the end of the range. ![The Stars and forks section: stars gained per day as bars per repository, the cumulative star curve rising to 350, stars by repository as a bar chart, a table of recent stars with timestamps and users, and forks over time rising to 51](../../../assets/dashboards/stars-and-forks.png) Reads `gh_star` and `gh_repo`. ## Contributions The contribution calendar as a series and then as the grid GitHub draws, a column per week and a row per weekday with a cell's shade a function of that day's own count; commits per week, the totals of the last year and the mix those totals make, the four kinds of contribution as shares of their sum, which is the radar the profile draws, beside the grid as the profile puts it; commits by hour of day and by weekday, commits by repository, and one row per past year. The grid is a status history panel over one field per weekday, because Grafana has no calendar panel and that is the one core panel that draws a grid of cells colored by value. It is GitHub's grid measured off the profile page on 2026-09-14 and copied: ten units wide by five draws the profile's own cell, ten pixels square in a 1920 pixel window, in a gutter of three to four pixels across and five to six down where the profile's is three both ways; the four greens are GitHub's dark theme read off that page the same day, over the gray of an empty day; three rows are named, Monday, Wednesday and Friday, as the profile names them; and the shade is the fifths of the busiest day of the year, the rule that reproduced all 366 of the profile's squares when applied to GitHub's own counts. The panel keeps that year whatever the dashboard range is set to, the way the profile's grid is always a year: at five years the same 53 columns would be 262 of six pixels, and at a week two bars. Three things a core Grafana panel cannot copy. The month name over the first week of each month: a status history builds its own x ticks, one every few columns, so a tick always lands on a week and the stride is a function of the panel's pixel width, which at three weeks writes the same month twice; the ticks name the Sunday each column starts on instead, which no two columns share. The square at every size: a cell is a fraction of the band its row gets, so the square survives at 1920 and again at 768, where the panel takes the whole width of a phone, and between the two the ten pixel height holds while the width narrows, eight at 1600, seven at 1280, five at 430; and where the panel sits on the dashboard it stretches into a tall bar when maximized, 27 by 67 pixels in a 1920 by 900 window and 27 by 84 in a taller one. And a tooltip naming the day and the count, since the cell is colored by the value it carries, so that value has to be the shade and a column is a week. The two punch cards are current state rather than history: every sweep writes the whole grid again, so a sum over a range counts every sweep and what the panels are read for is the shape. ![The Contributions section: contributions per day, the contribution calendar drawn as GitHub's own grid for the last year with its Mon, Wed and Fri labels, the contribution mix at 70.1 percent commits, commits per week with own commits overlaid, the totals table, the hour of day histogram peaking in the afternoon, the weekday bars, commits by repository, the five year table, commits per day by repository, and the bar chart splitting the commits the profile hides into yours and other people's, public and private](../../../assets/dashboards/contributions.png) Reads `gh_contribution_day`, `gh_commits_week`, `gh_contributions_total`, `gh_commit_punchcard`, `gh_contribution_repo` and `gh_contribution_year`. ## Pull requests and issues Fourteen panels, and the section where the per-item collection pays for itself. Merged count, time to merge, time to first review, issues closed, time to close and lines changed as one group of six values; then the same split by state over time, the largest merged pull requests, and the breakdowns by author, by repository and by reviewer. An item still open is written once per day for as long as it stays open, and those rows stay when it closes, so the dated panels count the open ones as distinct numbers under "Open that day" and draw them as a line beside the stack of what closed, and the "open the longest" tables read each item from its newest row. ![The Pull requests and issues section: a tile group reading 467 pull requests merged, 10.3 hours to merge, 9.81 hours to a first review, 137 issues closed, 1.75 days to close one and 75 lines per pull request, pull requests and issues per day split by state, time to merge and pull request size over time, the largest merged pull requests with their titles, associations and labels, pull requests by author, the per repository table, the reviewer table led by review-bot at 209 reviews, reviews per day, the two open the longest tables, review threads per day split into bot and human, and the review debt table](../../../assets/dashboards/pull-requests-and-issues.png) > **Read the time to first review carefully** > > The tile counts the first review by somebody other than the author and other > than a bot, so on an account reviewed by bots alone it reads No data rather > than the bots' few seconds. Measured, nine pull requests in ten had a bot > review inside a minute; the Reviewers table shows that, with each bot marked > as one and the author's own replies as one row. Reads `gh_pull_request`, `gh_pull_request_review` and `gh_issue`. ## Continuous integration Fourteen panels: run count, success rate over the runs that succeeded or failed, the canceled and skipped runs beside it, run duration, queue wait, artifact storage and cache size, all seven as one group; then the same figures over time, the workflows, the slowest jobs and the slowest steps, and then the six that say what to do about it: minutes spent on runs that failed, the workflows that keep failing, the steps that fail rather than the jobs, the workflows declared and never run, the artifacts created over time, and how much of the artifact storage was actually counted. ![The Continuous integration section: a tile group reading 1.51 thousand runs, a 90.2 percent success rate, 84 undecided runs, 15.2 minutes per run, 34 seconds of queue wait, 141 mebibytes of artifacts and 3.08 gibibytes of cache, runs per day by outcome, run duration and queue wait over time, artifact storage over time with a line per repository, and the workflows, slowest jobs, slowest steps, minutes spent on failed runs, repeatedly failing workflows, failing steps and artifact tables](../../../assets/dashboards/continuous-integration.png) Queue wait is a job-level number. The run-level figure folds the wait into the duration, so a run that took twenty minutes because one job waited eighteen for a runner looks identical to one that spent eighteen executing. Two of these are the ones worth looking at first. "Workflows that keep failing" is not about flakiness: measured, two workflows had failed on every single run they ever had, dozens of runs apiece, and nobody had switched them off. "Workflows that never ran" is the other side of the same question, and it needs both `gh_workflow` and `gh_workflow_run`, which is why it is the one panel here that only the two SQL stores can answer. "Artifact storage counted" exists because the total is a floor. GitHub reports how many artifacts a repository has, the collector records how many it actually walked, and when the second is smaller the live size is short: on one repository here, by a factor of fifty six. Reads `gh_workflow_run`, `gh_workflow_job`, `gh_workflow_step`, `gh_workflow`, `gh_artifact`, `gh_artifact_total` and `gh_actions_cache`. ## Code Commits, lines added and removed, the share of signed commits, lines changed over time, repository activity by kind, commits by author and by signature, and the force pushes with who made them and on which branch. ![The Code section: a tile group reading 752 commits, 49.0 thousand lines added, 20.0 thousand removed and 55.6 percent signed over the last 90 days, lines added and removed per day, repository activity by kind, the commits by author table, commits by signature, the force push table, commits per day by gate state, the table of checks that are not Actions, and the table of commits sitting behind a red branch](../../../assets/dashboards/code.png) `signature` separates `unsigned`, meaning there was no signature at all, from a signature that failed to verify. They are different facts and the bar chart keeps them apart. The last three panels are about the gate rather than the code. "Commits by gate state" is not the same claim as a workflow run failing: a run says one job failed, the rollup says the commit itself came out red, and on the measured account twenty seven of fifty commits on the default branch did. "Checks that are not Actions" holds what the two sections above cannot see, the code quality service and the dependency bot. And "Commits behind a red branch" joins the two by commit hash, which is the one panel that justifies storing `oid` and `head_sha` at all. > **This section keeps its own range** > > The six panels over `gh_commit` are pinned to ninety days whatever the range > picker is set to, and Grafana says so beside each title. One row per commit > is one Parquet file per commit in InfluxDB 3 Core, which refuses a query that > would open more than forty thousand: measured, a range of 270 days opened > just under that limit and answered, 300 days was refused, and the refusal > reaches the reader as an empty panel rather than as an error. Ninety days > opens a third of the limit, which leaves room for the history to keep > growing. Reads `gh_commit`, `gh_commit_check`, `gh_workflow_run` and `gh_repo_activity`. ## Planning and community Labels with how often each is used, milestones with their progress, forks gained over time, the fork list, discussions by category and whether they were answered, and beside that count the discussions themselves: the newest fifty one by one, with their category, comment count, whether each was answered and a link to each, whatever the range, since an account has a handful and a range of a month hid all but one of them under a row that read "Ideas". ![The Planning and community section: the labels table led by dependencies, the milestones table with progress bars, forks per day, the fork list with who forked and whether they pushed, the discussions table by category, the latest discussions with their answered state, issue transitions per day, comments left per repository, discussion answers, and the answers elsewhere table](../../../assets/dashboards/planning-and-community.png) The fork list carries `advanced`, which is what separates a real derivative from a bookmark. Most forks are bookmarks. Three more panels are about the conversation rather than the plan. Transitions over time is when something was labelled, closed, reopened or renamed, which the state of an issue does not record: a reopening exists in no other measurement. The two comment tables count what was written anywhere, including in repositories the account does not own, which is where most of it happens: the comments left in other people's repositories outnumbered the discussions inside the account by more than an order of magnitude when this was measured. Beside the per-repository count of discussion comments, the comments themselves: every one left in a discussion of somebody else's repository, newest first, with whether the maintainer accepted it as the answer and a link to the comment in its thread. Reads `gh_label`, `gh_milestone`, `gh_fork`, `gh_discussion`, `gh_issue_event`, `gh_issue_comment` and `gh_discussion_comment`. ## Delivery and access The webhook failure rate, deliveries by status code, the endpoints ranked by failures, the rulesets and the deploy keys with how long each has gone unused. ![The Delivery and access section: a 20.1 percent webhook failure rate gauge, deliveries per hour by status code, the endpoint table showing legacy.example.net failing most of its deliveries, the rulesets table, the deploy keys table, the text panel about failed job output, the configured webhooks, the environments and stale branches tables, branch protection rules per repository, ruleset rules with their bypasses, ruleset changes, deployments per day by environment, and the deployments by environment table](../../../assets/dashboards/delivery-and-access.png) Webhooks fail silently. Measured, one hook had been answering 403 for seventy-eight of its last hundred deliveries and nothing anywhere said so. Only the host of a webhook URL is stored, because the path usually carries a secret. One panel is text in the exported files, "Where failure output went", because the output of a failed job is text and belongs in a log store, and an importer may have none: a dashboard bound to one datasource cannot query two. Published to a Grafana that has a Loki datasource, with `cmd/publish_dashboard -loki `, the same panel draws the last lines of every failed job from Loki instead, newest first, with the workflow, job and run of each line in its logfmt tail. The repository variable is applied in the InfluxDB, PostgreSQL and Prometheus dashboards; the Graphite and Elasticsearch variables name the glob star as their All value, which is not a regular expression, so there the panel shows every repository and says so. The last two are inventories rather than traffic. "Webhooks configured" exists because the endpoint table is built from deliveries, so a hook that has never delivered anything appears in it nowhere, and an active hook with no traffic is exactly the interesting row. "Environments" asks the deploy key question about deployment targets: one environment here had not been touched in one thousand one hundred and seventy seven days. Reads `gh_webhook_delivery`, `gh_webhook`, `gh_ruleset`, `gh_deploy_key` and `gh_environment`. ## Releases Total downloads, downloads by release, and every asset with its size and its own download count. ![The Releases section: 11.6 thousand downloads over 14 releases, a bar chart of downloads per release tag, the release asset table with per asset downloads and sizes, and the table of downloads gained in the range](../../../assets/dashboards/releases.png) The first three are current state. GitHub reports a running total per asset and never a history, so the series that a downloads-per-day panel would need does not exist to be collected. The fourth panel is what can be recovered from it: the difference between the first and last value inside the range, which is what each asset actually gained. A day of that on the measured account showed ninety seven per cent of the downloads going to one Linux binary and its checksum file, which is an installer rather than a person. Reads `gh_release` and `gh_release_asset`. ## Security Open Dependabot and code scanning alerts, the breakdowns by severity and by ecosystem, open alerts over time, the feature table, and the time taken to resolve an alert by severity. ![The Security section: the open alerts tile reading 25 from Dependabot and 29 from code scanning, alerts by severity and by ecosystem, open alerts per severity over time, the security feature table with on and off per repository, code scanning runs per day by tool, the time to resolve table with each advisory and its CVSS, the resolved scanning alerts, scan results by tool, the oldest open alerts, the security settings and default code scanning setup tables, workflow token permissions, and the secret rotation table](../../../assets/dashboards/security.png) The feature table is what tells "no alerts" apart from "the feature is switched off". Without `gh_security_feature`, a repository with Dependabot disabled looks exactly like one with nothing to fix. The alert counts are current state, rewritten on every sweep, so every panel here takes the newest row of each series and adds those up rather than summing the range. Summing the range counts each alert once per sweep: before this was fixed the tile read 3.61 thousand where the account had a few dozen. The last two panels are new information rather than a different view. Time to resolve a code scanning alert comes from dates that were being downloaded and thrown away, so until now only the open count existed; of thirty four alerts on one repository, thirty were fixed and three dismissed, and all thirty three were invisible. Scan results by tool says what a scan found rather than that it ran, which is what explains a jump in the alert count. Reads `gh_dependabot_alert`, `gh_dependabot_alert_item`, `gh_code_scanning_alert`, `gh_code_scanning_alert_item`, `gh_code_scanning_analysis` and `gh_security_feature`. ## Cost Gross, the part covered by the plan, what was actually billed, Actions minutes, cost over time by product, minutes over time by SKU, and usage by repository. ![The Cost section: a tile group reading 360 dollars gross, 249 covered by the plan, 111 actually billed and 21.9 thousand Actions minutes, cost per day by product, minutes per day by SKU, the usage by repository table with SKU, quantity, unit and price, the cache entries by key, and the cache against the ceiling](../../../assets/dashboards/cost.png) Gross, discount and net are all stored rather than one being derived from the others, because net is not always zero and the discount is where a monthly credit shows up. The price per unit is in the table for the same reason: it is what explains thirty thousand macOS minutes costing more than two hundred and forty thousand Linux ones. A repository can appear in that table and in no other panel, because the list that bills and the list that is swept are not the same list. The two cache panels are about the ceiling. GitHub caps a repository at ten gigabytes and evicts the least recently used entry past it, so the bar is each repository against that cap and the panel is read for the distance left; the entry table says which key is being thrown away and which has not been touched for a week. Reads `gh_billing_usage`, `gh_actions_cache` and `gh_actions_cache_entry`. ## Activity Events over time, events by type and by repository, notifications by reason and kind, notifications over time, and the work done in other people's repositories. Both dated panels bucket by the range rather than by a fixed hour or day. ![The Activity section: events per day by type, the donut of events by type led by PushEvent at 39 percent, events by repository, the notifications table by reason, notifications per day, the latest notifications, the work elsewhere table of pull requests and issues in other people's repositories, the languages starred chart, and the recently starred table](../../../assets/dashboards/activity.png) Events and notifications are windows, not histories. GitHub keeps the last three hundred events whatever their dates and discards read notifications quickly, so what is stored is what was there when the sweep ran. The donut carries the share of each type in its legend and writes nothing on the slices, because Grafana leaves a label off a slice it does not fit and the thin slices never got theirs. The legend is a list under the chart rather than a column beside it, measured rather than chosen: under 992 pixels Grafana puts every legend under the chart and caps it at 35 per cent of the panel, whatever the panel asks for, and a legend placed beside the chart is drawn there as a column, one entry per line, that ended after seven entries on a phone; a list placed under the chart wraps, and at the height the pie has, the eleven types of a month all fit at 360 pixels. The last two panels are the mirror of the stars section: what this account starred rather than what was starred, by language and by project, dated when each star was given. Whether the projects are small or famous is a different question from how many there are, so the second table carries their own star counts. Reads `gh_event`, `gh_notification`, `gh_external_contribution` and `gh_star_given`. ## Inventory Code by language, the community profile score per repository, the repository table, the topics, the packages, the gists, and the container tags with the moment each was published. ![The Inventory section: code by language, the community profile table, the repository table with stars, forks, size, age and licence, the topics, packages and gists tables, the container tags published, the repository settings and account keys tables, dependencies by licence, the social accounts, configuration changes, policy files and Dependabot ecosystems tables, and dependencies by ecosystem and dependency changes](../../../assets/dashboards/inventory.png) `open_issues` in the repository table is GitHub's own field, and GitHub counts pull requests in it. `gh_issue` is what to count issues with. The community profile shows the boxes behind the score as well as the score, because the percentage hides which one is missing: on the measured account eight of thirty-five repositories have no licence. The issue template box is not GitHub's: the API's `issue_template` flag reports only the legacy single `ISSUE_TEMPLATE.md` file, not a templates directory, while the community page and `health_percentage` do count the directory, so the flag said "no" for every repository of an account whose repositories score 100 with four issue forms each (the gap is filed as community/community#207706). The column is the count of templates the repository actually has, forms and Markdown, from `gh_repo_policy.issue_templates`, joined to the profile row per repository in the InfluxDB, PostgreSQL and Prometheus dashboards; Graphite and Elasticsearch cannot join two measurements in one panel and keep the API's flag, saying so. The settings table is the same idea for what a repository allows, and `codeowners_errors` is the entry in it that fails silently, since a broken CODEOWNERS file stops requesting reviews and says nothing. The account keys table is where an SSH key that has never been used shows up, and where the expiry of the key that signs every commit is written down. The last panel is empty almost always, and that is the point: a row in "Configuration changes" means a repository was renamed, archived, made private, relicensed or had its default branch moved. Identity counts distinct `repo_id` values, which is the only way to tell a rename from a new repository, because GitHub publishes no rename history at all. Reads `gh_repo`, `gh_repo_language`, `gh_repo_community`, `gh_repo_topic`, `gh_repo_policy`, `gh_package`, `gh_package_version`, `gh_gist`, `gh_key`, `gh_social_account` and `gh_dependency_license`. ## Profile and sponsorship What the profile advertises and what Sponsors moves: the money in and out, the sponsorships in both directions, the tiers on offer, the pinned items, the profile flags, the star lists, and the achievement badges with the distance from each to its next tier. ![The Profile and sponsorship section: the four sponsorship figures, $1.28 K received over the lifetime, $65 a month, $32.5 at the next payout and $24 spent sponsoring, then the sponsorships table beside the tiers, the pinned items beside the profile flags, the star lists, the achievements led by Pull Shark at gold, and the achievement progress table with a bar per badge](../../../assets/dashboards/profile-and-sponsorship.png) The four figures are the newest reading of a snapshot rewritten every sweep and not a sum over the range, which would report the lifetime total once per sweep. The lifetime total is the one that keeps the history: the monthly estimate goes to zero the moment the last sponsorship lapses, and the money that did arrive stops being visible anywhere else. What is spent sponsoring is not the spend in the Cost section, which is what GitHub charges for Actions, packages and storage; this is money that leaves for somebody else's work. The tables under it are inventories rather than histories, and they say so by naming their own window: a sponsorship made in 2021 is listed at the default thirty days, where a panel on the dashboard's range would draw an empty axis over a handful of rows spread across years. A sponsorship is dated the day it began and not the day of a payment, both connections are read with `activeOnly` off so a lapsed one is recovered, and the sponsorable is the literal word private when the other party is hidden, with no link guessed at. The tiers are anchored to the start of the UTC day for the same reason: dated at their creation they would all fall outside every range and read as no tiers at all. Pinned items carry the position as a field and not a tag, because a repository that moves from slot two to slot three is the same pin, and as a tag every rearrangement would fork the series. The repository filter at the top of the dashboard does not reach that panel: a pin is named owner/name, or is a gist, and the variable holds neither. The achievements are read once a day from the public profile page, because no API lists them. Next tier at is the community-observed threshold (Schweinepriester/github-profile-achievements) rather than a number GitHub publishes, so the last column says whether the profile page agrees with the tier the count implies; a row that disagrees is a rule the page contradicts, and it carries no target and no bar. Both achievement panels are empty until that family has run once. Reads `gh_sponsors_listing`, `gh_sponsorship`, `gh_sponsors_tier`, `gh_pinned_item`, `gh_profile_flag`, `gh_star_list`, `gh_achievement` and `gh_achievement_progress`. ## The collector itself What the collector has left to spend: the budget of each of GitHub's fifteen independent rate limits over time, and a table of all of them ordered by how much of each has been used. ![The collector itself section: the rate budget used per bucket over the range, and the table of every bucket with its limit, most used and lowest remaining, led by core at 3134 used of 5000 and 1866 left](../../../assets/dashboards/the-collector-itself.png) The chart shows only the buckets with more than thirty requests in them, because search has thirty a minute and would flatten the axis. Reading all of this costs nothing: `GET /rate_limit` is the one endpoint GitHub does not charge for. Without it, a family skipped because a bucket was spent looks exactly like a family with nothing to report. Reads `gh_rate_limit`. ## The Link column Every table whose rows are items on GitHub selects a column called Link, the row's `url`, and hides it: the link hangs on the table's first column, the name of the thing, and opens the row's own page in a new tab. On a phone the column at the far right was reached in two tables of twenty-eight, and the first column is always on the screen; for the same reason the column a table is sorted by is its second, and a date is shown to the minute rather than the second. A few tables link from another url the row carries: Recent stars opens the person who gave the star, Top referrers the host that sent the visitors, Deployments by environment has a second column, Live, for the environment's own address. Release assets keeps its file under Download, which fetches the binary, and adds the release page as its Link; a click on a bar of Downloads by release opens the release too. Where the page is one GitHub shows to the owner alone, the settings of a repository, its traffic graph, an alert, the link's hover title says so. The column reaches a store only where that store's query returns it: the two SQL stores select it by name and Elasticsearch buckets on `url.keyword`, so a document without one lands in an empty cell rather than out of the table. Prometheus carries no url and Graphite keeps no string, and each of their tables says in its description that the Link column of the InfluxDB dashboard is absent there. `cmd/check_dashboards` holds every link column of a rendered dashboard to that: the column has to come back, and hold nothing but absolute urls and empty cells, a null from the SQL stores or the empty string of that bucket. ## When a store cannot answer The stores cannot answer identical questions, and the panels say so rather than pretend. - **InfluxDB and PostgreSQL** hold a row per fact, dated when the fact happened, so they draw the traffic of a particular Tuesday, the star curve since 2018 and the merge time of a pull request closed in July. The PostgreSQL set is the InfluxDB SQL translated, because the SQL sink writes the same facts as tables. - **Prometheus** stamps every sample at scrape time, so the exporter reduces the per-item rows to current values plus, for the counted measurements, a monotonic `_total` of distinct items seen since the exporter started. `increase()` over that is how the "per day" and "over the range" panels are answered. - **Graphite** keeps the dated points but has no rows: a table there is one number per series reduced over the range, so a table that needs several fields of one row keeps the column it is sorted by and says which it dropped, and a boolean is not a metric there at all. - **Elasticsearch** keeps the dated documents, so a per-item table is the newest documents themselves and everything else is a bucket aggregation, on the `.keyword` sub-field of each tag. Each panel whose twin in another store is richer says so in one sentence of its description. ## Twenty-four panels have no Prometheus answer at all Top paths, the contribution calendar, the two commit punch cards and the two per-repository splits of the calendar, the largest pull requests, pull requests by author, the issues open the longest, the review debt, the slowest steps, the steps that fail, the workflows that never ran, the artifacts created over time, the commits that left the branch red, the latest discussions and the answers left elsewhere, release assets, what each asset gained, the oldest open alerts, events by repository, the latest notifications, container tags and the configuration that changed. The exporter either skips the measurement, drops the identity the panel is about, or the panel is a join between two of them. Where failure output went is a text panel in every store, this one included: it is a note about where to look, not a query. Three of those are joins or set differences and have no answer in Graphite or Elasticsearch either, for the same reason: an aggregation runs inside one index or one tree, and these need two. The calendar has none in either for a different one: the grid needs the week along one axis and the weekday along the other, and a date histogram or a summarize buckets by one interval. Each of the two loses a little more on its own account: Graphite the oldest open alerts and the latest notifications, because it stores numbers and not the strings those tables are made of, and Elasticsearch what each asset gained, which is the difference between the first and the last value of the range. They are still emitted, as text panels with the same title saying what they would show and why the store cannot, so the layouts stay identical. ![The Prometheus dashboard over ten minutes: the Overview with the octocat fixture's current values (42 repositories, 80 stars, 9 forks; 361 views, 54 unique visitors, 19 clones; 1.20 thousand followers; 1.11 thousand contributions over 15.6 years), then the Audience section in which Views, Unique visitors and Clones over time all read No data while Top referrers and Clone amplification carry rows and Top paths is the text panel saying the exporter skips gh_traffic_path, the fourteen collapsed section headers, and at the foot the collector's own Rate budget used drawn as a flat line at 0.06 per cent](../../../assets/dashboard-prometheus-e2e.png) That capture is of no account at all. It is the Prometheus dashboard of the containerised end-to-end suite (`test/e2e/docker`), where the collector sweeps the fake GitHub of `test/e2e/fakegh`, whose account is `octocat` with one repository, and Prometheus scrapes the exporter every five seconds. The range is the ten minutes it had been collecting when the capture was taken, which is what makes it worth showing: every number on the Overview, and both tables of the Audience section, is a current value, because a current value is all the exporter can serve; `Rate budget used` is a flat line, because a gauge repeated at every scrape is a flat line; `Top paths` is the text panel described above; and `Views`, `Unique visitors` and `Clones over time` read "No data" because they floor their bucket at a day and this Prometheus is minutes old. On a server that has been scraping for months those three draw, starting the day the exporter did. --- # Overview A self-contained SVG for a profile README, drawn from the same points the databases receive. Source: https://jmrplens.github.io/ghchronicle/card/ ```sh ghchronicle -config config.yaml -card card.svg -card-only ``` > **This is a side feature** > > The point of the project is the ingestion. The card exists because the numbers > were already there, and it is drawn from exactly the same points the databases > receive, so the card and the dashboard cannot disagree. ## The flags | Flag | Does | | --------------------------------------- | ------------------------------------------------------------ | | `-card ` | Runs one sweep and writes the SVG there | | `-card-only` | Writes the SVG and nothing else, so no database is needed | | `-card-layout ` | One of the thirteen [layouts](/ghchronicle/card/layouts/) | | `-card-theme ` | Both writes the light card and a `_dark` twin from one sweep | | `-card-motion ` | How an animated layout moves; the others ignore it | | `-card-fields ` | Comma-separated, in drawing order | | `-card-width ` | The layout's own width when left out; on `activity-heatmap` it buys weeks | | `-card-speed <0 to 1>` | How fast an animated layout plays; `0.5`, the default, is the pace it always had | | `-card-layouts` | Prints the layouts with their fields and widths, then exits | Without `-card-only` the card is written **as well as** everything the sinks would normally get, which is the arrangement for a host that is already collecting and wants a picture too. ## The card run and the state file A run given `-card` collects **every family**, whatever the cadences say, because every number on the card comes from the points of that one sweep and a family skipped as not due would be a zero on the picture. With `-card-only` it also leaves the [state file](/ghchronicle/configuration/#state_file) exactly as it found it. Nothing that run collected reached a store, so nothing it learned may tell the next collection that a family is already done. It still reads the file, which is what lets it skip the one-off walk of the star history: what the state remembers makes the sweep cheaper, never the card smaller. So a card, a second card and a collection can all share one `state_file`, and each of them is the whole account. ## What it draws Stars, forks, followers and repository count; contributions over the last year; views and unique visitors over GitHub's fourteen-day traffic window; a sparkline of the contribution calendar; and the top repositories by stars with their language. Stars and forks are **summed from the repositories the sweep collected**, because GitHub's account endpoint reports neither total. That means excluding forks or archived repositories in `targets` is visible on the card too, which is the honest reading rather than a hidden discrepancy. ## The fields `-card-fields` takes a comma-separated list, in drawing order, from: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `visitors`, `clones`, `commits`, `pull_requests`, `reviews`, `issues`, `languages`, `top_repos`, `sparkline`. A layout skips a field it cannot draw. An unknown name is an error that lists the valid ones, rather than a silent skip: a typo would otherwise remove a number and nobody would notice until the card was already committed. ```sh ghchronicle -config config.yaml -card card.svg -card-only \ -card-layout github-stats -card-theme dark \ -card-fields repos,stars,forks,followers,commits,pull_requests,languages ``` Empty means the layout's default set. ## Themes - **dark and light** One palette each, and the choice for a README. `-card-theme both` (or `card-theme: both` in the Action) writes the light card at the path given and the dark one beside it with `_dark` before the extension, from the same sweep, so the two can never disagree. Put both in a ``, the way GitHub documents showing a different picture per theme: ```html My GitHub card ``` This project's own README does exactly that with its cards. - **auto** Both palettes in one file, behind a `prefers-color-scheme` query inside the picture. The query follows the reader's operating system rather than the theme they chose on the page, and a browser does not reliably evaluate it again inside an image: on GitHub, auto cards were seen switching palettes on a dark page after the tab was left and come back to. It suits a page with no theme of its own. For a README, use the two files. ## Three constraints it respects **Self-contained.** No external stylesheet, no webfont, no `` pointing at a URL, no script. GitHub's camo proxy serves README images from its own domain and blocks all of that, so anything external would simply not render. **Deterministic.** The same input produces a byte-identical file, so a scheduled job that commits the card does not produce a diff on every run. **Written atomically.** Rendered to a temporary file and renamed, so a reader watching the path never sees half a document. ## Motion Animation, where a layout has it, is CSS inside the SVG, and there is no script, ever. Every animation ends on the complete static card, so a renderer that ignores animation shows the finished state, and `prefers-reduced-motion` switches it off in every mode. | `-card-motion` | What the card does | | -------------- | --------------------------------------------------------------------------- | | `once` | Plays when it loads and settles. The default | | `loop` | The same, and whatever the layout has that never ends goes on for ever | | `off` | No animation at all, and a smaller file | ### A loop never replays An animation here reveals content: a number counts up to what it is, a bar grows to its share, a line draws itself. Playing that again would take back something the reader has already been shown, and a card that keeps hiding its own figures is worse than a still one. So `loop` does not mean "and again". The reveal plays once and settles in both modes, and the only thing that may go on for ever is motion that puts nothing on the card and takes nothing off it. Two layouts have such a thing: - `terminal`, whose cursor blinks at its prompt from the moment the window is drawn. The numbers still type themselves in once. Played `once` instead, the cursor waits for the last of them, blinks a couple of times and settles lit. - `ticker`, whose band of pills keeps scrolling. Nothing disappears; the same pills come round again. On every other layout `loop` draws exactly the card `once` draws, to the byte. It is accepted rather than refused, so a workflow can set it once and change `card-layout` freely. `once` is still the considerate default for a profile, and more so than before. A card that loops moves for everyone who opens the README, and neither of the two that can is paced: the cursor blinks and the band scrolls with no pause between passes, where the looping cards this replaced held still for seven seconds between plays. A README gives a reader no way to stop it. Their only escape is `prefers-reduced-motion`, which switches the animation off for them but is a setting for their whole machine, not a control over your card. That is the same reason the layouts page never shows a card looping until the reader presses the toggle (WCAG 2.2.2, Pause, Stop, Hide). ![The terminal layout played once: a terminal window whose lines of output each have a number typing itself in, with a lit block cursor at the prompt below them that blinks once the last number lands, and the page can also play it in a loop, where the typing still happens once and only the cursor goes on, blinking from the start](../../../assets/card-terminal.svg) - **Binary** ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both card-motion: loop ``` ### How fast it plays `-card-speed` is a decimal from 0 to 1, and 0.5 is the default. It is one number for the whole card: every animated layout scales by it, the two continuous motions with the reveals, so a card set slower has a band that takes longer to come round and a cursor that blinks more slowly at the same time. | `-card-speed` | What the card does | | ------------- | ---------------------------------------------------------------------- | | `0` | The slowest animation, twice as long as the default | | `0.5` | Exactly the card this renderer always drew, to the byte. The default | | `1` | The fastest, half as long as the default | There is one knob and not one per layout for the reason there is one width and not one per layout: the cycles here were paced against each other, and scaling them together is what keeps the pacing the motion was designed with. A speed outside the range is refused before the sweep runs, naming both ends. > **0 is the slowest animation, not none** > > A range that starts at zero reads like a switch, and this one is not. A card > drawn at `0` still animates, as slowly as this renderer will draw it. The one > that draws no animation at all is `-card-motion off`, and it is also the one > that makes the file smaller. `prefers-reduced-motion` is untouched by any of this: a reader who has asked their machine for less motion gets no animation to slow down or speed up. ## In a README ```html My GitHub statistics ``` The workflow that keeps it current, for a profile README or any other, is in [A card in your profile README](/ghchronicle/install/actions/#a-card-in-your-profile-readme). ## There is no Go library The renderer lives in `internal/render`, and Go refuses that import from outside the module, so there is no way to draw a card from your own program by calling into this one. A program that wants an SVG runs the binary with `-card` and reads the file, the same way it would run any other command: see [calling it from a program](/ghchronicle/reference/subprocess/). --- # Layouts Thirteen layouts in two visual families, with what each one draws by default, which of them animate and which of them can keep going. Source: https://jmrplens.github.io/ghchronicle/card/layouts/ ```sh ghchronicle -card-layouts ``` Prints them with the fields each shows by default. Every layout below has a section of its own, stating its family, its motion, the width it is drawn at and those fields, over a picture of the card. Each card is shown in the palette this page is in, light or dark; every layout draws both, and `auto` puts both in one file. Where a layout animates, the animation plays once and settles on the complete static card, so every picture below is that settled frame and not a moment in the middle of one. How that is set, and what switches it off, is [Motion](/ghchronicle/card/#motion). ## The two families The **chronicle** family is this tool's own look. The **github** family uses GitHub's Primer palette and monospace numbers so the card sits in a profile README as if GitHub had drawn it. > **The command under each card** > > Under each picture, **Command for** opens the command that draws that card > from your own account, and the same card as a step of the > [Action](/ghchronicle/install/actions/#a-card-in-your-profile-readme). > `config.yaml` is your configuration, and `-card-theme both` writes two files, > `card.svg` and `card_dark.svg`, which is how a page or a README shows the card > in its own palette. On the two cards that loop, pressing the loop toggle adds > `-card-motion loop`. The numbers a card draws are chosen with `-card-fields`, > and every flag is on [the card](/ghchronicle/card/#the-flags). ## summary The original card: title, two rows of numbers, a sparkline and the most starred repositories. - **Family**: chronicle - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `visitors`, `sparkline`, `top_repos` ![The summary layout: a title, two rows of large numbers, a contribution sparkline and a list of the most starred repositories](../../../assets/card-summary.svg) - **Binary** ```sh ghchronicle -card-layout summary -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: summary card-theme: both ``` ## github-stats GitHub's own box: a header band, rows of four monospace numbers and a language share bar with its legend. The widest of them, and the one that looks most native in a profile README. The numbers count up, and the bar grows in from its left edge once they have landed. - **Family**: github - **Motion**: plays once - **Width**: 800 px, drawn from 600 to 1200 - **Default fields**: `repos`, `stars`, `forks`, `followers`, `commits`, `pull_requests`, `views`, `clones`, `languages` ![The github-stats layout played once: a header band over rows of four monospace numbers counting up, and a horizontal language share bar growing in from the left beside its legend](../../../assets/card-github-stats.svg) - **Binary** ```sh ghchronicle -card-layout github-stats -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: github-stats card-theme: both ``` ## github-compact One row of monospace numbers under a thin header band. - **Family**: github - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `commits` ![The github-compact layout: a thin header band above a single row of monospace numbers](../../../assets/card-github-compact.svg) - **Binary** ```sh ghchronicle -card-layout github-compact -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: github-compact card-theme: both ``` ## badge-row A row of 20 pixel pill badges, one per number, for a README line. - **Family**: chronicle - **Motion**: still - **Width**: follows its content - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions` ![The badge-row layout: a horizontal row of small pill badges, each with a label and a number](../../../assets/card-badge-row.svg) - **Binary** ```sh ghchronicle -card-layout badge-row -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: badge-row card-theme: both ``` ## wide-banner A full-width 60 pixel banner: login on the left, numbers spread across, the sparkline drawing itself behind them. - **Family**: chronicle - **Motion**: plays once - **Width**: 800 px, drawn from 500 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `contributions`, `sparkline` ![The wide-banner layout played once: a wide, short banner with the login on the left, numbers spread across and a sparkline drawing itself behind them](../../../assets/card-wide-banner.svg) - **Binary** ```sh ghchronicle -card-layout wide-banner -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: wide-banner card-theme: both ``` ## sparkline-hero The contribution sparkline is the whole card, with up to three numbers overlaid. The line draws itself on load. - **Family**: chronicle - **Motion**: plays once - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `contributions`, `stars`, `followers`, `sparkline` ![The sparkline-hero layout played once: a large contribution sparkline drawing itself across the card with three numbers overlaid](../../../assets/card-sparkline-hero.svg) - **Binary** ```sh ghchronicle -card-layout sparkline-hero -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: sparkline-hero card-theme: both ``` ## language-ring A donut of language shares with the legend beside it and a row of headline numbers. Each slice draws itself around the ring after the one before it, and the legend appears when the donut is whole. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 400 to 1200 - **Default fields**: `languages`, `stars`, `repos` ![The language-ring layout played once: a donut chart whose language slices draw themselves one after another, with a legend appearing beside it and a row of headline numbers](../../../assets/card-language-ring.svg) - **Binary** ```sh ghchronicle -card-layout language-ring -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: language-ring card-theme: both ``` ## repo-list The most starred repositories as the main content: language dot, stars and a bar per row, totals underneath. - **Family**: github - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `top_repos`, `stars`, `forks`, `repos` ![The repo-list layout: one row per repository with a language dot, the star count and a proportional bar, with totals underneath](../../../assets/card-repo-list.svg) - **Binary** ```sh ghchronicle -card-layout repo-list -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: repo-list card-theme: both ``` ## activity-heatmap As much of the contribution calendar as the width holds, up to a year of it, as GitHub's green squares, with up to three numbers beside it. The week count is not a fixed number: the grid takes whatever room the numbers beside it leave, so it ends where the card does rather than stopping a third of the way short. At the width this layout declares that is twenty-three weeks, at its minimum sixteen, and `-card-width` at the far end its facts state draws the whole year the collector keeps. An account whose numbers reach seven digits takes a wider column for them and leaves the grid a week or two fewer, which is the same rule seen from the other side: at the width this layout declares, a million contributions is twenty-two weeks rather than twenty-three, and six digits still fits inside the labels. The far end is where the year lands for the three numbers this layout draws by default, so a card asked for fewer, or for numbers with shorter labels, reaches the year before it and has room to spare at the end: `-card-fields sparkline` draws its whole year well short of the far end. The weeks fade in from the left, the wave crossing the grid in 0.22 s however many weeks it holds, so the calendar fills in as a wave of the same length at every width. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 400 to 891 - **Default fields**: `sparkline`, `contributions`, `commits`, `pull_requests` ![The activity-heatmap layout played once: twenty-three weeks of contribution squares in GitHub's green scale fading in from the left, with three numbers beside them](../../../assets/card-activity-heatmap.svg) - **Binary** ```sh ghchronicle -card-layout activity-heatmap -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: activity-heatmap card-theme: both ``` ## animated-counters Numbers that count up on load over a sparkline that draws itself, settling to the static card. - **Family**: chronicle - **Motion**: plays once - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `sparkline` ![The animated-counters layout played once: a grid of large numbers counting up over a contribution sparkline that draws itself](../../../assets/card-animated-counters.svg) - **Binary** ```sh ghchronicle -card-layout animated-counters -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: animated-counters card-theme: both ``` ## terminal A terminal window with the project's mark in its title bar, one line of output per number and one per repository. The numbers type themselves in, a line at a time, under a cover painted in the card's own background colour. The cursor at the prompt is the card's one piece of endless motion, and it does a different thing in each motion. Played `once` it sits lit while the numbers arrive, blinks a couple of times when the last one lands and settles lit, which is the state the finished card rests in. Under `loop` it blinks from the moment the window is drawn and never stops, because a cursor blinks for the reason a terminal is open and not for the reason a card is finished. The typing itself happens once either way. - **Family**: chronicle - **Motion**: plays once, or in a loop - **Width**: 495 px, drawn from 360 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `top_repos` ![The terminal layout played once: a terminal window with the project's mark in its title bar, whose lines of output each have a number typing itself in under a lit block cursor that begins blinking when the last number lands, and the page can also play it in a loop, where the typing still happens once and the cursor blinks from the start and never stops](../../../assets/card-terminal.svg) - **Binary** ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both card-motion: loop ``` ## ticker A band of pills, one per number and one per repository, scrolling from right to left. The content is repeated end to end and the band moves by exactly one copy, so the picture at the end of a pass is the picture at its start and the loop has no seam. Played once, it makes a single pass and comes back to the beginning. The band scrolls at a fixed speed, so a card with more in it takes longer to come round than any other layout takes to settle. - **Family**: chronicle - **Motion**: plays once, or in a loop - **Width**: 800 px, drawn from 400 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `commits`, `views`, `top_repos` ![The ticker layout played once: a wide band of rounded pills, one per number and one per repository, scrolling from right to left under the account name, and the page can also play it in a loop](../../../assets/card-ticker.svg) - **Binary** ```sh ghchronicle -card-layout ticker -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout ticker -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: ticker card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: ticker card-theme: both card-motion: loop ``` ## language-bars One full-width bar per language, each growing from its own left edge after the one above it, with the name and the share arriving once their own bar has stopped. The layout that shows the share of a language as its own line, where `language-ring` shows all of them in one donut and `github-stats` in one bar. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 360 to 1200 - **Default fields**: `languages` ![The language-bars layout played once: one full-width bar per language growing from its left edge, one after another, with the language name and its percentage arriving behind each bar](../../../assets/card-language-bars.svg) - **Binary** ```sh ghchronicle -card-layout language-bars -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v1 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: language-bars card-theme: both ``` ## Motion Each section above states its layout's motion, and one fact decides what that line can say: a reveal is never replayed. A number that has counted up, a bar that has grown and a line that has drawn itself are not taken back, so `-card-motion loop` draws exactly the card `once` draws on every layout but the two whose motion ends nothing, `terminal`'s blinking cursor and `ticker`'s scrolling band. Those two are the ones with a loop toggle under their picture, and the reasoning is in [A loop never replays](/ghchronicle/card/#a-loop-never-replays). ## Width Each layout declares the width it is drawn at and the two ends it refuses to go outside, and its own section above states all three. `-card-width` on the binary, and `card-width` on the Action, ask for another: anything between that layout's own two ends. A width outside them is refused before the sweep runs, naming both, and `-card-layouts` prints them. Left out, a card comes out at its layout's own width, which is what every card came out at before the flag existed. `badge-row` declares none of the three, because a pill row stretched to a fixed width would have gaps in it; its width follows its content, and the flag neither changes it nor is refused by it. The near end is where a column stops fitting. The far end is usually only a guard against a typo, because a layout given more room spreads the same content over it, and one asked for twenty thousand used to be drawn twenty thousand units wide. `activity-heatmap` is the one with a real one, and it is the reason the ends are each layout's own rather than one pair for all of them: it reads the width rather than only being sized by it, working out how many weeks of the contribution calendar fit in the room the width leaves, so the same card is sixteen weeks at its near end, twenty-three at the width it declares and the whole year the collector keeps at its far end. Past that there is no more calendar to draw, so the far end is exactly the width where the year lands and the card is never asked to fill space it has nothing for. ## Fields a layout cannot draw Each layout declares which fields it supports. Asking for one it does not, such as `top_repos` on `github-compact`, drops it silently. Asking for a name that is not in the vocabulary at all is an error listing the valid ones. Twelve of the fifteen fields are numbers, and every layout takes all twelve. Only the three that need room of their own are restricted: | Field | Drawn by | | ------------ | ----------------------------------------------------------------- | | `languages` | `summary`, `github-stats`, `language-ring`, `language-bars` | | `top_repos` | `summary`, `github-stats`, `repo-list`, `terminal`, `ticker` | | `sparkline` | `summary`, `github-stats`, `wide-banner`, `sparkline-hero`, `activity-heatmap`, `animated-counters` | `ghchronicle -card-layouts` prints the layouts with the fields each one draws by default. ## Where to go next - [The card](/ghchronicle/card/) is what draws these, and how to put one in a README. - [Calling it from a program](/ghchronicle/reference/subprocess/) is the supported way to get one out of another language. --- # Rate limits GitHub runs fifteen independent budgets; three of them matter here, and the brake is scaled to each. Source: https://jmrplens.github.io/ghchronicle/api/ GitHub does not have a rate limit. It has fifteen of them, and every response says which one it just charged in the `x-ratelimit-resource` header. ## The three that matter here | Bucket | Limit | Spent by | | --------- | -------------------- | ---------------------------------------------------------------- | | `core` | 5000 per hour | Every REST call | | `graphql` | 5000 points per hour | The account, commits, discussions, labels and milestones queries, and since 2026-09-11 the newest stars and forks, the starred list, the outbound searches and ten of the eleven totals counts | | `search` | 30 per minute | The commit count of `totals`, and nothing else: GraphQL search has no COMMIT type | Two more are charged by one family each: `webhook_deliveries` (500 a minute) by the delivery list of every hook in `settings`, and `dependency_sbom` (100 a minute) by the SBOM in `deps`. Neither is one of the fifteen `/rate_limit` reports; they exist only in the headers of the endpoints that charge them, and both are named in the [cost table](/ghchronicle/api/cost/) where they apply. The brake looks at the three above, not at whichever bucket was charged last, which matters for the reason below. ## The brake `github.reserve_rate` is how many calls are never spent. It defaults to 500. The collector stops a family rather than crossing that line, so whatever else uses the same token keeps working. ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 ``` The reserve is scaled to each bucket: a fifth of its limit, or the configured value, whichever is smaller. > **The scaling is a bug fix, not a refinement** > > Search allows thirty requests a minute. One call to it leaves "29 remaining", > and comparing that against a flat reserve of 500 read as exhausted, so every > remaining family in the sweep was skipped. Judging a thirty-request bucket by > a five-thousand-request reserve stops the collector dead. When a bucket is below its reserve, the collector waits for the reset the response already told it about, rather than sleeping a guessed interval and retrying. In a normal sweep the family is skipped with a warning: ```text level=WARN msg="rate limit reserve reached, family skipped" family=artifacts ``` Once is fine. Every sweep means the cadences are too fast for the number of repositories; see [cost of a sweep](/ghchronicle/api/cost/) for what to lengthen first. ## ETags, and why a 304 is free Every response is cached by its ETag. A repeat request sends `If-None-Match`, and GitHub answers 304 Not Modified when nothing has changed. **A 304 costs no quota at all.** That is not an optimisation detail, it is the thing that makes short cadences affordable. Most of what this collects barely moves: a repository's language breakdown, its topics, its community profile, the list of workflows. Asking for them every hour would be unaffordable if each question cost a call. Asking whether they changed costs nothing. The consequence is that the measured cost of a sweep in the next page is an upper bound reached on the first sweep and after a change, not the steady-state figure. What the cache keeps beside each ETag is the value the collector decoded, encoded again, not the body GitHub sent. A page of a hundred workflow runs is 1.4 MB of which the collector keeps a few hundred bytes per run, so a sweep's cache holds about a ninth of what the raw bodies would, and a 304 decodes a ninth of the bytes. A test runs every REST collector twice against a fake GitHub that answers the repeat 304 and fails on the first point that differs, which is what keeps the replay the same answer as the original. An entry is keyed by the URL and the type that decoded it, so the one URL two collectors read differently, `GET /repos/{owner}/{repo}` (four flags for discovery of a repository named in `targets.repos`, forty fields for the repo family), has an entry per reader and neither is answered from the other's. The cache is bounded, and the bound is 256 MB. It is an LRU, and an entry is charged the bytes it holds plus two hundred for the map slot, the list element and the struct around them, so a cache full of small answers is accounted at what it really costs rather than at half of it. There is no key for it in the configuration file: a deployment whose live set does not fit raises it from Go with `SetCacheLimit`, and `CacheStats().Evicted` is the number that says whether it had to, because it stays at zero for as long as one sweep's live set fits. Two things follow. A limit below one sweep's live set is not a smaller cache but no cache, since every sweep would evict what the next one is about to ask for. And the bound is resident memory once the cache is full, which is the number to know before giving the process a container with a memory limit: it can hold that much of decoded bodies on top of its own footprint. ## A refusal is remembered too A 403 or a 404 is how GitHub says a feature is switched off: Dependabot on a repository that does not use it, code scanning where it was never enabled, a forum on a repository with discussions off. That is not an error, and the collector turns it into "there is nothing here". What it is not is free. A refusal carries no ETag, so where a page that did not change costs nothing, a feature that is off is charged in full on every sweep, for ever. So a refusal is remembered for a day, keyed by family, repository and endpoint, and the sweeps inside that day ask nothing. Two consequences worth knowing: - Switch a feature on and it is noticed a day later at the latest, not on the next sweep. - Restarting the process asks again straight away. The memory lives in the process, like the ETag cache, not in the state file. A spent budget is a 403 as well and is never remembered as one: the client types it apart precisely so that a rate limit is not read as a feature that is off. ## The budget in the log ```text level=INFO msg="rate budget" bucket=core remaining=4354 limit=5000 ``` Worth an alert on the warning above rather than on this line. A budget that dips is normal; a family skipped every sweep is a configuration problem. ## A backfill inverts the rule A [backfill](/ghchronicle/how/backfill/) is the opposite intention and says so. When a bucket runs out it waits for the window to reset rather than giving up, because a backfill that stops half way has spent the expensive part of the budget and keeps only the families it finished: each one is written and marked as it completes, and the rest has to be run again. --- # Cost of a sweep The measured price of one sweep, per family, and which families to lengthen when the budget is tight. Source: https://jmrplens.github.io/ghchronicle/api/cost/ Measured on 2026-09-11 with the real binary against the live API, every family switched on and every request logged with the `x-ratelimit-resource` and `x-ratelimit-used` headers of its own response. **Cold** is the first sweep of a fresh install or of a restarted service: an empty ETag cache, a month of workflow runs. **Steady** is the third sweep of the same process a few minutes later, when about four fifths of its requests were answered 304 and cost nothing; the second sweep is the one that pays the fill-in the actions row describes. A 304 is free, and it is only ever available to REST: GraphQL carries no ETag, so its column is the same in both sweeps. The figures are per repository where the family asks per repository, and per sweep where it asks about the account. `core` is the REST bucket, `pt` a GraphQL point. Families with a request that is not `core` say which bucket it went to: `search` (30 a minute), `webhook_deliveries` (500 a minute) and `dependency_sbom` (100 a minute) each have their own. A row the round of 2026-09-11 changed describes the requests the code makes now and says what the two sweeps measured where that differs. There is no total row on purpose: a total belongs to one account and ages with it, so multiply the per-repository rows by the number of repositories `-list` prints and you have your own. ## Per family | Family | Cadence | Cold | Steady | Notes | | ----------- | ------- | --------------------------------------------------- | --------------------------------- | ----- | | account | 12h | 1 pt | 1 pt | one query: the calendar, the totals, the pins, the sponsors block and the star lists; 66 KB, never compressed, never conditional | | achievements | 24h | 1 page off budget, 1 pt, plus the co-authored walk | the same | the badges come from the public profile page, read without the token and charged to no bucket. The progress rows beside them do come from the API: one counts query, and a walk over the merged pull requests whose co-authored commits no count exposes. Search stops at a thousand results, so the walk asks the whole life of the account and splits any range that overflows into two, one point per page and one per split, which is tens of points on a long-lived account and one on a new one. All of it once a day | | totals | 12h | 1 core, 1 search, 3 pt | 0 core, 1 search, 3 pt | the profile is a 304 from the second sweep on; ten of the eleven counts are one GraphQL query of ten aliases, 406 bytes at cost 1, measured identical to the REST answers of the same minute; only the commit count is still a search, charged every time; a refused counts query is asked again one count at a time, ten points on that sweep only, so one failed search loses one number as it did in REST | | ratelimit | 15m | 1 pt | 1 pt | `GET /rate_limit` is free; the point is the GraphQL half of the same question | | events | 30m | 3 core | 1 core | before the round: three pages of a feed that changes every time, so `If-None-Match` never matches. Now: the walk stops at the page that carries the newest event of the previous sweep, remembered as `last_event` in the state file, which is usually the first page: 1 core a sweep, 96 a day fewer; a first sweep and a backfill still read the three | | notifs | 30m | 20 core | 1 core | before the round: twenty pages of fifty; one new notification shifts every page, so the ETag never matches, for six or seven rows that were new. Now: `since=` the newest `updated_at` seen minus two cadences (`last_notified` in the state file), which is one page for a two hour window: 1 core a sweep, and the twenty once a day (`last_full`), on a backfill, and on the first sweep after an upgrade, read threads included (`all=true`): that bounds to a day whatever GitHub's filter left out of the windowed reads, and it is the read that closes a thread, since a sweep lists unread threads only and a thread read without a reply leaves that listing rather than coming back with its `unread` tag changed. 912 core a day fewer | | billing | 6h | 2 core | 1 core | one per month walked; the month in progress changes, the previous one is a 304 | | profile | 12h | 4 core plus 1 per package | 0 core | the profile, the social accounts, the gists, the packages and one page of versions per package; all 304 after the first sweep | | outbound | 12h | 8 pt | 8 pt | all GraphQL: the starred list is one query of a hundred (13.5 KB against 617 KB decompressed from REST), the five searches one point each (7 KB against 100 KB), the two comment walks one each; nothing here has an ETag on either path, so the two columns are the same | | history | off | 1 pt per past year | the same | the account's whole calendar, once, at a point per year it has existed | | keys | 24h | 2 core | 0 core | the SSH and the GPG keys | | traffic | 6h | 4 core per repo | 0 core | views, clones, referrers, paths; the fourteen-day window is a 304 until it moves | | repo | 1h | 3 core per repo, 2 pt per 10 repos | 0 core for a repo that did not change, 2 pt per 10 repos | the repository, its community profile and one page of releases each; languages, topics, rulesets and branch protection ride in one GraphQL query per ten repositories | | branches | 24h | 1 pt per 14 repos | 1 pt per 14 repos | one query per fourteen repositories | | stars | 6h | 1 core per repo | 1 pt per 10 repos | the full walk through REST the first time a repository is seen, then the newest hundred of every repository in one GraphQL query per ten, about a kilobyte per repository; a restart with a state file starts at that query. Before the round the last page was asked of every repository on every sweep, all but one of them 304 | | issues | 1h | up to 8 pt per repo | 1 to 2 pt per repo | before the round: a page of fifty with ten review threads each, 8 points, which the gateway refused with a 502 once a sweep on the busiest repository. Now: 2 points of GraphQL per repo for what changed in two cadences, ten at a time (1 point where the repository holds five or fewer), and once a UTC day a whole page sized to the repository from the last totals (5, 10, 20 or 50 cost 1, 2, 3 or 8). The sweep that carries the daily page is the expensive one, and the busiest repositories still draw the gateway's 502 or 504 at fifty once a day and are retried at twenty-five | | issueevents | 1h | several pt per repo for the month, plus 1 core per stacked pull request in it | 1 pt per repo, 0 to 1 core | before the round: one page of a hundred events each, up to a megabyte per repository because every event embeds its whole issue; a 304 on all but the repository that moved. Now: one GraphQL query per repository, the timeline of the ten most recently updated issues and ten pull requests for the events of the last two cadences (1 pt, 2 to 6 KB, a second page only when more than ten items moved), plus one core per pull request in a stack, whose `added_to_stack` event the timeline cannot name; thirty days on a first sweep, once, which is minutes rather than seconds where the pull requests are nearly all stacked, because almost every one updated in the month takes the per-issue road for its `added_to_stack`; a steady sweep is a point per repository and one such read at most. And from a cadence before the last run after a gap, so a stopped process does not leave its hours out of the series. Measured against the list over a week of two repositories: every event of every type agrees, field for field, except a commit referencing an issue nobody has touched, 3 of 2,217, which does not move the issue and so is not asked for. A backfill walks `/issues/{n}/events` per item instead of the list, 1.4 KB compressed per twelve events against 45 KB per event | | actions | 15m | a month of runs per repo, plus 1 core per run | 1 core per repo that had a run, plus 1 per run completed since | before the round: pages of a hundred runs reaching a month back, a job list per run, and the caches and workflows of every repository, which was three fifths of the cold sweep's bytes, and those job lists again on every sweep, nearly all of them 304. Now: pages of 30, one on a quiet repository and up to 7 while they come full of runs newer than the window, plus one per run not yet expanded, jobs listed once per attempt, at most 20 new runs a sweep; pages of 100 on the first sweep and in a backfill. Measured over three sweeps of one process, the second is the one that pays the fill-in, because the page of thirty is a new URL for every repository and holds runs the first sweep's twenty did not cover; from there a sweep costs one page per repository that had a run and one job list per run completed since, so what it costs is how many runs the account completes | | artifacts | 1h | up to 5 core per repo | 0 to 5 core per repo | a busy repository fills its pages, and all five of them are charged again whenever an artifact was added or expired: 5 on one sweep, 0 on the next | | security | 1h | 2 core per repo | 0 core | Dependabot and code scanning alerts; most are the 403 of a repository with Dependabot switched off or the 404 of one without code scanning, and a refusal carries no ETag, so each was charged again on every sweep before the round remembered them. Now: a refusal is answered from memory for a day, per family, repository and endpoint, so the steady figure is 0 core on the second sweep and one request per refusal once a day. The rest are conditional requests, all 304 | | stats | 12h | 3 core per repo | 0 core | participation and the punch card; GitHub recomputes them slowly and answers 304 | | discussions | 2h | 2 pt per repo with a forum | the same | before the round: fifty threads with twenty comments and twenty replies each, 11 points, asked of every repository, including every one with no forum. Now: 2 points per repo with a forum, the ten most recently updated threads with the same twenty comments and twenty replies each (comments come oldest first, so a shorter page there would stop recording the eleventh comment of a thread); a repository whose forum is off is never asked | | commits | 1h | 1 pt per repo | 1 pt per repo | the last two cadences of the default branch; a month back on the first sweep, which is a few hundred kilobytes for a busy repository | | activity | 30m | 2 core per repo | 0 to 2 core per repo | the repository log, a hundred entries per page; charged only for a repository whose log moved, 2 on one sweep and 0 on the next | | analyses | 6h | 1 core per repo | 0 core | most are the 403 and 404 of repositories without code scanning, charged again each sweep before the round; now remembered for a day, so the steady sweep is conditional requests and 0 core | | forks | 12h | 1 core per repo | 1 pt per 10 repos | one page each through REST on the first sweep of a fresh install and in a backfill, then the newest hundred of every repository in one GraphQL query per ten, under a kilobyte per repository; a repository the batch reports holding more than a hundred forks is walked through REST as well, since a fork row's stars and days_since_push move and the batch cannot refresh the rows past its page. Before the round: one request per repository a sweep, all 304 | | planning | 6h | 1 pt per repo | 1 pt per repo | labels and milestones | | joblogs | off | 1 core per repo, 1 blob per failed job | 0 core | the failed runs of the last hour per repository, asked of a list filtered to the month a re-run can reach back to, and one blob per failed job from object storage, outside the API's quota; the filter's URL changes once a day, so a day costs one charged page per repository and the rest are 304 | | settings | 6h | 2 core per repo, 1 webhook_deliveries per hook | 0 core, 1 webhook_deliveries per hook that moved | webhooks, environments and deploy keys; the deliveries of every hook are charged to their own bucket | | rulesets | 24h | 1 core per repo plus 1 per ruleset | 0 core | one list per repository and one history per ruleset, both with an ETag; none of it charged on a day nobody edited a ruleset, when the steady sweep asks the same questions and every one of them is conditional | | inventory | 24h | 4 core per repo | 0 core | the workflow token policy, the secrets, the code scanning setup; the refusals were the 403 of repositories without it, now remembered for a day, which at this cadence is the next sweep anyway; the rest are conditional requests, all 304 | | deployments | 1h | 1 pt per 5 repos | 1 pt per 5 repos | one query per five repositories, the newest hundred deployments each | | policyfiles | 24h | 1 pt per 5 repos | 1 pt per 5 repos | one query per five repositories, the history of four paths each | | deps | off | 1 core and 1 dependency_sbom per repo | 0 core, 1 dependency_sbom per repo that received a commit | one commit read and one SBOM per repository; the SBOM has its own bucket and most were 404, now remembered for a day. GitHub regenerates a SBOM on every read and its ETag never matches, so the photograph is taken only when the head moved since the last sweep (the head itself is a free 304 when it did not): 0 dependency_sbom on a repository without a commit, one per repository that received one, and one of the SBOM reads timed out on GitHub's side on each of the three sweeps measured | What weighed before the round was not REST but GraphQL: `issues` and `discussions` were 88 % of the points, because both queries were sized for a backfill and asked on every sweep. The only search is the commit count of `totals`, one a sweep against a budget of thirty a minute. The round of 2026-09-11 lowered several of these rows, and the first measurement is the baseline it is measured against: the job lists a run has already had expanded are not asked for again, the runs page is thirty on an ordinary sweep, `discussions` is asked only of repositories with a forum and for ten threads, `pulls` is sized to the repository and windowed to two cadences, notifications and the event feed stop at what the previous sweep saw, a 403 or 404 is remembered for a day instead of being charged again every hour, and the newest stars and forks, the starred list, the outbound searches and ten of the eleven totals counts moved from REST to GraphQL, the same rows for a point apiece instead of a request apiece. Measured again after the round, three sweeps of one process on the evening of the same day, the steady sweep charges about a third of the `core` and a third of the GraphQL points it charged before, moves half the bytes on the wire, and answers about four fifths of its requests with a 304. Projected to a day at the built-in cadences that is roughly an eighth of the REST calls and a third of the points, plus one job list per workflow run completed, which is the one term that grows with how busy the repositories are rather than with how many there are. Two things did not fall. The cold sweep charges more `core` than before, because the first sweep of issue events now reads the whole month through the timeline and, where pull requests are stacked, one per-issue list for nearly every pull request updated in it: minutes, once per process. And a sweep that runs every family at once still takes minutes rather than seconds, because its conditional requests cost a third of a second each and its GraphQL queries nine tenths, one after the other; the sweeps production runs are the 15 minute one and the hourly one, each a fraction of the whole. ## GraphQL is the cheap one, by a wide margin One query returns the full 366-day contribution calendar, every contribution total, the per-repository commit breakdown and the social counts, for **one point of a five thousand point budget**. The same data over REST would be dozens of calls and would not include the calendar at all, because the calendar exists nowhere else. That is why the account family runs on a twelve-hour cadence and still costs almost nothing, and why the expensive families are the REST ones that scale with how busy the repositories are. ## Two families scale with activity, not with size Everything else costs a roughly fixed number of calls per repository. Two do not: - **`actions`** costs a page of thirty runs, up to seven while they come full of runs newer than the window, plus one request per run whose jobs this process has not written yet, at most twenty a sweep. A repository with continuous integration on every push generates runs continuously; a quiet one costs the one page, answered 304, and nothing else. - **`artifacts`** walks up to five pages per repository, and a busy repository fills them. > **What to lengthen first** > > If the budget is tight, lengthen `artifacts` and then `actions`. They are the > only two whose cost grows with how busy the repositories are rather than with > how many there are, so they are also the only two where lengthening the > cadence buys back a variable amount rather than a fixed one. ## Switching a family off Set its interval to `0`. ```yaml every: families: artifacts: 0 joblogs: 0 ``` A family that is off writes nothing and costs nothing. Its dashboard panels go empty, which is the honest reading. See [cadences](/ghchronicle/configuration/cadences/). ## Reading the arithmetic for your own account A first sweep is the expensive one: the full stargazer walk, a month of workflow runs, and (with `every.history` set) every past year's contribution calendar. After that, multiply the per-repository rows above by the number of repositories `-list` prints, and divide the hourly budget by the cadence. A [card](/ghchronicle/card/) is priced as a cold sweep whatever the cadences say: the run collects every family, because every number it draws comes from that one sweep, and its process starts with an empty ETag cache. So N cards are N sweeps, and a workflow drawing three of them pays three. The signal that the sum came out wrong is a warning, not a guess: ```text level=WARN msg="rate limit reserve reached, family skipped" family=actions ``` Once in a while is fine. Every sweep means the cadences are too fast for the number of repositories. --- # What GitHub will not give The endpoints that are verified not to work on a personal account, written down so nobody rediscovers them. Source: https://jmrplens.github.io/ghchronicle/api/limits/ Every entry here was checked against the live API. It is written down so nobody spends an afternoon finding it out again, and so that a missing panel can be told apart from a broken collector. ## Statistics that never arrive `stats/code_frequency` and `stats/contributors` answer **202 with an empty body, indefinitely**, on a personal account. A 202 normally means "still being computed, ask again", and for these two the next answer is another 202. They are deliberately not called. The lines added and removed that `code_frequency` would have given come from the commits collector instead, per commit rather than per week, attributed to an author and dated to the commit. `stats/participation` and `stats/punch_card` do work and are used. Ten seconds is all it takes to see it on your own account: ```sh curl -s -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer $GITHUB_TOKEN" \ https://api.github.com/repos/OWNER/REPO/stats/code_frequency # 202, for ever curl -s -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer $GITHUB_TOKEN" \ https://api.github.com/repos/OWNER/REPO/stats/participation # 200 ``` ## Billing | Endpoint | Answer | | --------------------------------------- | -------- | | `/settings/billing/actions` | 410 Gone | | `/settings/billing/packages` | 410 Gone | | `/settings/billing/shared-storage` | 410 Gone | | `/user/settings/billing/usage` | 404 | | `/users/{login}/settings/billing/usage` | works | Only the last form works for a personal account, and it returns full RFC 3339 timestamps in a field its documentation describes as a date. ## Organisation and enterprise only Custom repository properties, classic projects, cost centres and the audit log. A personal account cannot see any of them, however the token is scoped. ## Endpoints that answer, but say nothing - **`workflows/{id}/timing`** returns 200 with an always-empty `billable` object. It looks like the source for per-workflow minutes and is not. - **`stargazers/history`** returns only the last thirty weeks. It does not replace the `starred_at` walk; this was checked on three repositories. - **`/user/installations`** returns 403 without a GitHub App. ## GraphQL is wrong about packages GraphQL reports zero packages for an account while REST lists them. The pretty query is simply wrong here, so packages come from REST, at one call per package for its versions. ## Traffic goes stale rather than empty A repository with no traffic does not return an empty window. GitHub keeps returning the last fourteen days that _had_ data, so the window can end weeks ago. The collector records what it is told. > **This is why gh_security_feature exists** > > Several of the entries above have the same shape: an endpoint that answers > nothing is indistinguishable from a feature that is switched off, which is > indistinguishable from a repository with nothing to report. > `gh_security_feature` records explicitly which features are enabled, so "no > alerts" and "no data" stop looking alike on a dashboard. ## What this means for a sweep None of the above is treated as a failure. `ghapi.UnavailableError` (403 or 404: the feature is switched off) and `ghapi.NotReadyError` (202: GitHub is still computing) both mean "there is nothing here", and the sweep continues to the next repository. The activity feeds have their own version of this: past their ceiling GitHub answers **422 "pagination is limited for this resource"**, which is read as the end of the data rather than as an error. ## Where to go next - [Rate limits](/ghchronicle/api/) is the other half of this: what the calls that do work cost, and why a 304 costs nothing. - [Cost of a sweep](/ghchronicle/api/cost/) prices every family. --- # The command line The flags, what each one does, and which of them print something and exit. Source: https://jmrplens.github.io/ghchronicle/reference/cli/ The binary takes these flags and no subcommands. Everything else is in the configuration file, because a schedule is not something to retype. ```sh ghchronicle -config /etc/ghchronicle/config.yaml ``` ## Every flag | Flag | Default | What it does | | ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `-config` | `config.yaml` | Path to the configuration file | | `-once` | off | Run one sweep and exit instead of scheduling; a family that is not due by its cadence is still skipped | | `-list` | off | Print the repositories that would be collected, and which are set aside, then exit | | `-version` | off | Print the version, the commit and the build date, then exit | | `-groups` | off | Print the groups with the families in each, then exit | | `-backfill` | off | Reach as far back as each surface allows, waiting for the rate limit to reset rather than stopping | | `-backfill-since` | none | Bound the backfill: a date (`2024-01-01`), a duration (`720h`), days (`90d`) or years (`2y`) | | `-card` | none | Run one sweep and write a summary SVG to this path; that sweep runs every family, whatever the cadences say | | `-card-only` | off | With `-card`, write the SVG and nothing else: no sink is needed, none is written to, and the state file is left as it was | | `-card-theme` | `auto` | `dark`, `light`, `auto`, or `both`: the light card at `-card` and the dark one beside it with `_dark` before the extension, from one sweep | | `-card-motion` | `once` | `once`, `loop` or `off`; `loop` changes only `terminal` and `ticker` | | `-card-layout` | `summary` | Which of the thirteen [layouts](/ghchronicle/card/layouts/) to draw | | `-card-fields` | none | Comma-separated fields the card shows, from [the fields](/ghchronicle/card/#the-fields); empty means the layout's own default | | `-card-width` | the layout's | Card width in pixels. Each layout draws between two ends of its own, which its [section](/ghchronicle/card/layouts/) and `-card-layouts` both state; `badge-row` ignores it, its width following its pills | | `-card-speed` | `0.5` | How fast an animated layout plays, as a decimal from 0 to 1. `0` is the slowest animation and `1` the fastest; `0.5` is the pace every card has always been drawn at. `0` is not a still card, `-card-motion off` is | | `-card-layouts` | off | Print the layouts with the fields and the widths each draws, then exit | ## The four that print and exit `-version`, `-groups`, `-card-layouts` and `-list` answer and stop. The first three need no token and no configuration; `-list` reads the configuration and asks GitHub which repositories the targets resolve to, which is the cheap way to check a change before spending quota on a sweep. ```sh ghchronicle -version ghchronicle -groups ghchronicle -card-layouts ghchronicle -config config.yaml -list ``` ## The three ways to run a sweep ```sh ghchronicle -config config.yaml # the loop: each family on its cadence ghchronicle -config config.yaml -once # one sweep, in the foreground, then exit ghchronicle -config config.yaml -backfill # the history walk, once ``` The loop is what a service runs. `-once` is what a scheduled job runs, and it is also the quickest way to see what a configuration change does. A [backfill](/ghchronicle/how/backfill/) is a different intention and says so: it walks every surface to the end and waits for a spent budget to refill rather than giving up. ```sh ghchronicle -config config.yaml -backfill -backfill-since 2y ``` ## The card, in one line ```sh ghchronicle -config config.yaml -card profile.svg -card-only \ -card-layout github-stats -card-theme dark ``` `-card-only` is the one combination worth remembering: it makes a run that writes no points, so it needs no sink configured and a configuration that would otherwise be refused at start-up is accepted. [The card](/ghchronicle/card/) has the layouts and the fields. `-card-width` is the one flag that changes what a card says rather than only how it looks, and on one layout only. [`activity-heatmap`](/ghchronicle/card/layouts/#activity-heatmap) spends the room on data: sixteen weeks of the contribution calendar at its near end, twenty-three at the width it declares, and the whole year the collector keeps at its far end, which sits exactly where the year lands so the card is never asked to fill space it has nothing for. Every other layout spreads the same content over whatever width it is given, so widening one of those buys proportions and not information, and its far end is only a guard against a typo. A width outside a layout's two ends is refused before the sweep runs, naming both, and `-card-layouts` prints them for every layout. ```sh ghchronicle -config config.yaml -card calendar.svg -card-only \ -card-layout activity-heatmap -card-width 700 ``` ## The speed, and what its slow end is not `-card-speed` is one number for the whole card. Every animated layout scales together, the continuous motions with the rest: the ticker's band takes longer to come round and the terminal's cursor blinks more slowly at the same setting. It is one knob and not one per layout because the motion the card has was paced against itself, one layout's cycle chosen beside another's, and a reader who finds the band slow finds the typing slow with it. `0.5` is the middle of the range and is exactly the card this renderer has always drawn, to the byte, so leaving the flag out and asking for `0.5` are the same command, and each end reaches the same distance from it: `0` draws the animation twice as long as the default, `1` half as long as the default. > **0 is the slowest animation, not none** > > A range that starts at zero reads like a switch, and this one is not. At `0` > the card still animates, as slowly as this renderer will draw it. What draws > a card with no animation at all is `-card-motion off`. ```sh ghchronicle -config config.yaml -card slow.svg -card-only \ -card-layout ticker -card-motion loop -card-speed 0.25 ``` ## Where the rest lives Everything that is not in that table is a configuration key, not a flag: [the file](/ghchronicle/configuration/) is the map of them. --- # Calling it from a program There is no Go library. What there is instead is one sweep, NDJSON on standard output, and good reasons not to call it in a loop. Source: https://jmrplens.github.io/ghchronicle/reference/subprocess/ > **There is no Go library** > > Every package this tool has lives under `internal/`, and Go refuses that > import from outside the module. The subprocess route on this page is not a > workaround for something you have missed; it is the interface. ```text main.go:6:2: use of internal package github.com/jmrplens/ghchronicle/internal/collect not allowed ``` What exists instead is a binary that runs one sweep, writes one JSON object per line to standard output and exits. Anything that can spawn a process and read a pipe can use it, in any language, and it gets the same points a database would rather than a second pass over the API. ## A configuration for one question ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 targets: repos: [acme/telemetry] sinks: stdout: true stdout_format: json state_file: /tmp/ghchronicle-adhoc.json log: level: warn ``` Two things about `targets` are worth getting right the first time. **Name repositories, not the account.** `repos` is an inclusion, not a filter: adding `user: acme` beside it does not narrow anything, it discovers the whole account and collects that as well. `-list` is the cheap way to check before spending quota on it. ```sh ghchronicle -config adhoc.yaml -list ``` ```text acme/telemetry ``` **Leaving `user` out switches off eleven families for free.** The account-wide collectors (`account`, `totals`, `ratelimit`, `events`, `notifs`, `billing`, `profile`, `outbound`, `history`, `achievements` and `keys`) have no login to ask about and are skipped, which is most of what an application asking about one repository does not want to pay for. The log goes to standard error, always and whatever the level, so standard output carries data and nothing else. That is what makes the pipe safe to parse without filtering it first. ## The command ```sh ghchronicle -config adhoc.yaml -once > points.ndjson 2> sweep.log ``` `-once` runs a single sweep and exits. It starts no exporter and holds no port, so it does not collide with a long-running instance on the same host. ## What comes out Three lines of it, from the demonstration account every example here uses: ```json {"time":"2026-09-08T16:08:17.651862791Z","measurement":"gh_repo","tags":{"archived":"false","default_branch":"main","fork":"false","full_name":"acme/telemetry","language":"Go","license":"apache-2.0","owner":"acme","repo":"telemetry","visibility":"public"},"fields":{"age_days":1290,"days_since_push":0,"forks":21,"network":21,"open_issues":9,"repo_id":1043778215,"size_kb":18422,"stars":148,"url":"https://github.com/acme/telemetry","watchers":11}} {"time":"2026-09-07T00:00:00Z","measurement":"gh_traffic","tags":{"full_name":"acme/telemetry","kind":"clones","owner":"acme","repo":"telemetry"},"fields":{"count":94,"uniques":71,"url":"https://github.com/acme/telemetry/graphs/traffic"}} {"time":"2026-09-08T16:08:17.651862791Z","measurement":"gh_release","tags":{"draft":"false","full_name":"acme/telemetry","owner":"acme","prerelease":"false","repo":"telemetry","tag":"v2.4.0"},"fields":{"age_days":22,"assets":4,"downloads":1840,"url":"https://github.com/acme/telemetry/releases/tag/v2.4.0"}} ``` Four keys, and they are the same four for every point: | Key | What it holds | | ------------- | ---------------------------------------------------------------------- | | `time` | The date the thing happened, RFC 3339. Not the time of the sweep | | `measurement` | What kind of thing this is, always prefixed `gh_` | | `tags` | Strings, and only strings. Together they identify the series | | `fields` | The values: numbers, booleans, and the occasional string such as `url` | The `time` distinction is the whole design, and it shows in those three lines. The traffic point is dated `2026-09-07T00:00:00Z` because that is the day those 94 clones happened, and every sweep for the next fortnight will offer it again with the same date. The repository point is stamped at the moment of the sweep, because "148 stars" is true now and has no other date to carry. The release carries its age in a field for the same reason. Every measurement and every field is listed in [Measurements](/ghchronicle/collectors/measurements/). `stdout_format: influx` prints InfluxDB line protocol instead, which is the better choice when the receiving end already speaks it. ## Narrowing it to what you want Even against one repository a sweep runs twenty-one families, the twenty-three per-repository ones less `deps` and `joblogs`, which ship switched off. `every` switches them off, and `default` is the layer that does it in one line: put everything to `0`, then name back what you want. ```yaml every: default: 0 families: traffic: 6h ``` That run costs five API calls, one to resolve the repository named in `targets` and four for the traffic family, and prints 48 objects: 28 `gh_traffic`, 10 `gh_traffic_path` and 10 `gh_traffic_referrer`. > **A default of 0 never switches the off ones on** > > `deps`, `history` and `joblogs` ship with a built-in cadence of `0`, and > neither `default` nor `every.groups` can reach a family that ships switched > off. Naming it under `families:` is the only way to turn one on, so the > configuration above collects traffic and nothing else, the dependency graph > included. A family is not a measurement, which is the other half of this: `repo` alone emits `gh_repo`, `gh_repo_language`, `gh_repo_topic`, `gh_release` and several more. Switching the family off is what saves the API calls; filtering the stream is what saves you reading them. ```sh ghchronicle -config adhoc.yaml -once | grep '"gh_traffic"' ghchronicle -config adhoc.yaml -once | jq -c 'select(.measurement == "gh_repo") | {repo: .tags.full_name, stars: .fields.stars}' ``` A family name spelled wrong is a start-up error, and the message lists every name there is, so there is no need to keep a copy of the list anywhere. ## Reading it from another program Any language, since the contract is a pipe and a line of JSON. Python here, because a Go program cannot import this and would be running the same subprocess. ```python import json import subprocess proc = subprocess.Popen( ["ghchronicle", "-config", "adhoc.yaml", "-once"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, ) stars = {} for line in proc.stdout: point = json.loads(line) if point["measurement"] == "gh_repo": stars[point["tags"]["full_name"]] = point["fields"]["stars"] if proc.wait() != 0: raise SystemExit("the sweep failed; run it again without stderr suppressed") print(stars) ``` ```text {'acme/telemetry': 148} ``` Read the pipe as it fills rather than after the process exits. The sink flushes once per family, so a consumer sees the traffic points while the commit collector is still working, and a sweep over a whole account takes minutes. Waiting for the end also means holding all of it in memory: over a handful of busy repositories the `actions` family alone can produce more than ten thousand objects in a single sweep. Sending the log to `DEVNULL` is right for a program that only needs to know whether the sweep succeeded. It is the wrong default while you are still writing that program: keep it, and read it. ## Two smaller questions - **-list** Prints the repositories in scope, one full name per line, and costs the discovery calls and nothing else. ```sh ghchronicle -config adhoc.yaml -list ``` - **-card** Still runs a full sweep, but writes a self-contained SVG with no database configured at all. The card is about an account rather than a repository, so it needs a config with `targets.user` in it: the one above has none, and gets `render: card has no login` and a 1. See [the card](/ghchronicle/card/). ```sh ghchronicle -config account.yaml -card summary.svg -card-only ``` ## Exit codes | Code | Means | | ---- | -------------------------------------------------------------------------------------------------------------- | | 0 | The sweep ran | | 1 | It could not start, or could not list the repositories: no config file, an invalid one, a token GitHub refused | | 2 | An unknown flag. This is Go's flag package, and it has already printed the usage | Zero answers "did the sweep run", not "did everything work". A collector that fails is logged and the sweep continues, because a repository with a feature switched off must not stop the sweep for the other forty. ```text level=ERROR msg="collector failed" family=issueevents repo=acme/parser err="/repos/acme/parser/issues/events?per_page=100&page=1: 504 Gateway Timeout" ``` A caller that needs to know about that has to read standard error. There is no exit code for it, deliberately: on a large account some family fails somewhere most sweeps, and a status that reported it would be permanently red. Killing the process usually gets a 1 as well. `SIGTERM` and `SIGINT` cancel the sweep, what is already on the pipe stays there, and the run ends with `ghchronicle: context canceled`. Usually, because the code depends on where the signal lands: during discovery it is a 1 carrying the cancelled request's own error instead, and in the last repository of a family it is logged as a collector failure and the sweep still finishes with a 0. A caller with its own deadline should treat a kill it ordered as "incomplete" rather than read the exit code for it. ## Calling it in a loop is the wrong shape A sweep costs API calls: four per repository for traffic, three per repository plus two GraphQL points per ten repositories for the repository family, one GraphQL point per repository for commits and one to two for issues. [Cost of a sweep](/ghchronicle/api/cost/) has the measured table. The budget is 5000 REST calls an hour for the whole token, shared with everything else that uses it, and the collector brakes before spending the last `reserve_rate` of them: it stops a family rather than crossing that line, and says so in the log. So a program that calls this on every request, or on a tight timer, is answered with an empty sweep and a warning rather than with fresher numbers. Two things follow, and the second one surprises people. **Cache the answer.** These numbers move on the order of hours. GitHub's own traffic window updates once a day. **Expect the second run to print nothing.** A family is collected only when its cadence says it is due, and `-once` marks it as run in the state file. Two sweeps a minute apart therefore give a full stream and then an empty one, exit code 0 both times, which reads like a bug and is the brake working. At `info` the log says so by omission: ```text level=INFO msg="repositories discovered" count=1 level=INFO msg="rate budget" bucket=core remaining=3949 limit=5000 level=INFO msg="sweep finished" ``` No `written` line, because nothing was due. Deleting the state file collects everything again at full price, including the one-off walk of every star, so point `state_file` somewhere the application controls and leave it there. > **When it is not ad hoc** > > If the program wants a continuous feed rather than an answer now, stop > spawning it: run it as a service and give it a [sink](/ghchronicle/sinks/). > The file sink writes this same JSON to a rotating file for anything that tails > it, and the push sinks reach a store the program can query without touching > GitHub at all. --- # Troubleshooting The messages that look like errors and are not, the ones that are, and the data that looks wrong and is not. Source: https://jmrplens.github.io/ghchronicle/reference/troubleshooting/ ## Things that look like errors and are not **`not available (403)` or `(404)`.** The feature is switched off for that repository, or the token cannot see it. Dependabot, code scanning, discussions and the dependency graph all answer this way when disabled. The collector records the fact and moves on: a repository with a feature off must not stop the sweep for the other forty. If it is _every_ repository rather than one, it is the token. Traffic needs push access; alerts need `security_events`. See [the token](/ghchronicle/start/token/). **A feature switched on, and nothing collected from it.** Code scanning enabled this morning, Dependabot turned on, a forum opened: the collector asked before you did it, was refused, and remembers the refusal for a day rather than paying for it on every sweep. It is noticed a day later at the latest, and restarting the process asks again straight away. See [a refusal is remembered too](/ghchronicle/api/#a-refusal-is-remembered-too). **`still being computed by GitHub (202)`.** GitHub computes the `stats/*` endpoints asynchronously and answers 202 with an empty body while it works. The next sweep usually gets the numbers. Two of them never do. On a personal account `stats/code_frequency` and `stats/contributors` return 202 with an empty body indefinitely, which is why this project does not call them: the lines added and removed come from the commits collector instead. **`pagination is limited for this resource` (422).** The end of an activity feed, not a failure. GitHub serves three pages of the event feed and refuses the fourth. **A repository with no traffic showing a window that ended weeks ago.** GitHub keeps returning the last fourteen days that _had_ data, not the last fourteen days. The collector records what it is told. **A family that never appears in the log.** It is not due yet. With a twelve-hour cadence, half a day of logs can legitimately never mention `account`. ## Things that are errors **`github.token is empty and GITHUB_TOKEN is unset`.** Exactly what it says. **`every.families.: unknown collector`.** The name is not a family. The message lists the thirty-four that exist, in one parenthesis after the colon. **`groups: is empty`.** `groups: []` would collect nothing at all. Omit the key to collect everything, which is what it means when it is absent. **`groups[N]: "" is not a group`.** The name is not a group. The message lists the ones that exist, and `ghchronicle -groups` prints each with its families. **`groups[N]: "" is a family, not a group`.** Families and groups are both lowercase nouns from the same table, so this is an easy one to hit. The message names the group the family is in, which is probably what you wanted, and points at `every.families.`, which is where a single family's cadence lives. **`sinks: enable at least one of ...`.** A run that collects and discards is almost never what anyone meant. `-card-only` is the exception and needs no sink at all. **`prometheus exporter: listen tcp :9605: bind: address already in use`.** Reported at start-up rather than swallowed in a goroutine, so a port clash cannot leave you with a running collector and a silently missing exporter. **`influx write: 400`.** Almost always a column type collision. InfluxDB fixes a column as a tag or a field the first time it sees it and rejects later writes that disagree. If a collector changed which one a name is, the table has to be dropped: `DELETE /api/v3/configure/table`. **`family failed everywhere, not marking it as run`.** Every repository failed for one family, so it will be retried rather than treated as done. One repository failing is normal; all of them is the token, the network or an outage. **`rate limit reserve reached, family skipped`.** Once is fine. Every sweep means the cadences are too fast for the number of repositories. Lengthen `artifacts` and then `actions`; see [cost of a sweep](/ghchronicle/api/cost/). ## The data looks wrong **A number is a multiple of the sweep count.** Something that is a snapshot is being summed over time. Referrers, paths, labels and milestones are snapshots of a window with no date of their own; they are stamped at the start of the UTC day so a day's sweeps rewrite one row, and the dashboard takes the newest rather than the sum. **Median time to first review reads No data.** The panel reads `seconds_to_first_human_review`, which leaves out review bots and the author's own replies; on an account where nobody else reviews, no pull request carries it and the tile is honestly empty. The bots' speed is in the Reviewers table, where each one is marked as a bot: measured, nine pull requests in ten had a bot review inside a minute. **Clones are enormous compared with views.** Continuous integration clones a repository thousands of times for every human visit. One repository measured here took more than a hundred clones for every view. `clones` does not count people. **`open_issues` disagrees with the issue count.** That field is GitHub's, and GitHub counts pull requests as issues in it. The `gh_issue` measurement is the one that counts issues. **Artifact storage looks too small.** Check the `walked` field against `count` in `gh_artifact_total`. When they disagree, the live size is a floor: the repository has more artifacts than the page cap walked. **The traffic chart only goes back fourteen days.** That is a first sweep. The window is rewritten day by day on every sweep, so the series extends as the collector keeps running. It cannot be backfilled: GitHub never stored anything older. **A panel says "Query would scan 10000 Parquet files".** InfluxDB 3 Core writes one file per partition per write request and never compacts them, so a store fed by a version of this tool older than the write ledger holds its rows in far more files than it needs. Widening the panel's interval does not help: the limit counts the files the planner opens, before any aggregation. What helps is the ledger, which is on by default and stops the growth, and then one of three things for what has already accumulated: `--query-file-limit` raised on the server, the affected tables rewritten, or InfluxDB 3 Enterprise, which compacts on its own and is free for home use. See [only what changed is written](/ghchronicle/sinks/#only-what-changed-is-written). **A Prometheus panel shows one flat line.** That is the store, not the data. The exporter serves current values, so the fourteen-day traffic window collapses to its most recent day and the star history to the current total. See [dating a point](/ghchronicle/how/dating/). ## Nothing is being written Run one sweep in the foreground and read what it says. Then check, in order: 1. that `-list` prints the repositories you expect, 2. that the sweep log says `written`, 3. that the sink is reachable. ```sh ghchronicle -config config.yaml -list # the repositories, and which are set aside ghchronicle -config config.yaml -once # one sweep in the foreground, then exit journalctl -u ghchronicle -f # under systemd ``` `debug` adds three lines to that and nothing else: the size of the written-points ledger at start-up, the account-wide families skipped for want of a `targets.user`, and the entries a sink left out for being too old. There is no per-request log at any level. See [logging](/ghchronicle/configuration/logging/). ```yaml log: level: debug ``` A family that is not due yet simply does not appear. > **Deleting the state file costs quota, and one thing more** > > It remembers six things, and five of them cost only quota when they go: what > is collected again is keyed by measurement, tags and timestamp and overwrites. > The sixth, `last_head`, is the commit each dependency diff started from, and > without it the next sweep has the photograph and no diff. See > [the state file](/ghchronicle/configuration/#state_file). ## Loki drops entries Look for the debug line counting them. Loki refuses an entry more than its out-of-order window behind the newest entry already in that stream, about two hours by default, so the sink leaves the older ones out rather than losing the whole push. Raise `max_age` only alongside Loki's own `out_of_order_time_window`. See [Loki](/ghchronicle/sinks/loki/). --- # The test layers Three layers, only one of them free: the contract test of the bytes, the containerised stores that accept them, and the dashboards' own queries. Source: https://jmrplens.github.io/ghchronicle/reference/testing/ The tests come in three layers, and they are worth telling apart because only the first one is free. Two of them start nine containers, and anyone deciding whether to wait for that deserves to know what it buys. | Layer | Command | Docker | Time | What it proves | | ------------------ | ---------------------- | ------ | -------------- | -------------------------------------------------------------------- | | L1, the contract | `make test` | no | about 5 s | the exact bytes each sink puts on the wire | | L2, the stores | `make test-e2e-docker` | yes | 53 s warm | a real store accepts those bytes, and keeps the date of the event | | L3, the dashboards | the same target | yes | included above | the five dashboards' own queries answer against what the sinks wrote | L1 runs on every push. L2 and L3 are one suite behind the `dockere2e` build tag, so `go test ./...` never starts a container; in CI they run weekly, on demand, and as a release gate. ## L1: the bytes on the wire `test/e2e` builds the real binary, runs it against a fake GitHub whose fixtures live in `test/e2e/testdata` and whose route table is `test/e2e/fakegh`, and points every sink at an `httptest` capture server. Then it asserts the bytes: the line protocol, the `_bulk` envelope, the SQL statements, the Graphite path, the OTLP payload. The fake also prices its answers the way api.github.com does: one ETag per REST fixture and none on a GraphQL answer, a 304 charged nothing for a request that presents it, one for every other answer on the API, nothing for the object storage a job log redirects to, and the budget block on every GraphQL answer. That is what lets `TestTheSecondSweepIsPricedByTheCache` run two sweeps in a single process and hold that every URL answered 200 the first time came back 304 the second, that the second sweep charged fewer than half the core requests of the first, and that `own_cost` on the `gh_rate_limit` row is the number of queries the process made rather than zero. That is the right test for a format, and it is fast enough to run while a sink is being changed. What it cannot catch is anything the receiver has an opinion about. A capture server answers 204 to everything. It has no column types, no mapping, no query planner and no schema. ## L2: the stores themselves `test/e2e/docker` starts the real stores in containers, runs one sweep from the same fake GitHub into all of them, then reads each store back and asserts the value, the tags and above all the timestamp. The dating rule is the product of this tool: a star is stamped when it was given, a workflow run when it finished, a traffic day at that day's own date. No capture test can prove a store kept the date of the event rather than the date of the sweep, because storing it is the store's job. These are the defects this layer exists for, each of them real: **InfluxDB fixes a column's type on first sight.** InfluxDB 3 decides that a column is a tag or a field the first time it sees it and refuses every later write that disagrees: `400 invalid column type for column 'owner', expected iox::column_type::tag`. A capture server answers 204 and notices nothing. This has already cost a database wipe, and reproducing it was the first thing the containerised stack was used for. **Elasticsearch's dynamic mapping decides whether the dashboards can aggregate.** One panel pulls a url through a `top_metrics`, which normally needs a keyword field rather than a text one. Two separate audits recorded that as unverifiable for want of a real Elasticsearch. This layer answers it, by indexing through `_bulk` and reading back the mapping the cluster built for itself. **PostgreSQL has to accept the DDL.** The SQL sink emits statements rather than speaking the wire protocol, so until now nothing ever had them parsed. The suite pipes them through `psql`, inserts, and plans the panels' queries against the schema the sink created rather than against one transcribed from InfluxDB. **Graphite paths have to have the depth the dashboards index.** The dashboards address path nodes by position, and the agreement between those positions and what the sink writes was kept by a hand-maintained table that nothing checked. Here the sink writes to carbon and the render API is asked for the path back. ## L3: the dashboards' own queries With the stores loaded, the five generated dashboards are run through Grafana's `/api/ds/query`, which is the path `cmd/check_dashboards` takes against a live Grafana. Every datasource is provisioned at boot with a fixed uid and the harness mints a service account token, so a panel query goes through Grafana exactly as it would for a person looking at the dashboard. That is what turns three manual checkers into something CI runs, and it is what settles the Elasticsearch question above: a panel that cannot aggregate returns no frame. ## What none of them catch Every layer runs against the fake GitHub, so nothing here notices GitHub changing a payload, retiring an endpoint or throttling differently. That is what `ghchronicle -once` against a real token is for. They also prove nothing about a store the suite does not start. The answer covers InfluxDB 3 Core, PostgreSQL 18, Elasticsearch 9, Graphite 1.1, Prometheus 3, Loki 3, the OpenTelemetry collector and Telegraf, at the pinned versions. OpenSearch, TimescaleDB and anything behind the Telegraf or OTLP hop are still an inference from the format. There are five more things, and none of them is a layer. Each is switched on by an environment variable and skipped when it is absent, so an ordinary `go test ./...` stays offline. **The store you actually run.** `test/live` pushes a handful of points at a Loki or an OpenTelemetry collector named in `GHC_LIVE_LOKI` or `GHC_LIVE_OTLP`. It answers the one question containers cannot: whether your instance accepts them. ```sh GHC_LIVE_LOKI=http://localhost:3100 go test ./test/live/ ``` **The real API, end to end.** `GHC_E2E_LIVE=1` runs `TestLiveAPI` against GitHub itself rather than the fake, with a real `GITHUB_TOKEN`, sweeping the account named in `GHC_E2E_USER`. ```sh GHC_E2E_LIVE=1 GHC_E2E_USER=octocat GITHUB_TOKEN=ghp_... go test ./test/e2e/ -run TestLiveAPI ``` **What a sweep costs in cache.** `GHC_LIVE_CONFIG` points `TestLiveSweepCacheFootprint` at a configuration file and sweeps the account it names, reporting the entries and the bytes the conditional-request cache holds after each sweep. Those are the figures the 256 MB bound and the [cost of a sweep](/ghchronicle/api/cost/) rest on, and this is how to reproduce them for your own account. `GHC_LIVE_DUMP=1` adds the per-URL list to standard output. ```sh GHC_LIVE_CONFIG=config.yaml go test ./internal/ghapi/ -run TestLiveSweepCacheFootprint -v ``` **One repository, one family.** `cmd/probe` runs the collectors against a single repository and prints the line they would write, writing nothing anywhere. `GHC_DUMP=` prints every point of that family in full, which is the fastest way to see what a collector actually produces. ```sh go run ./cmd/probe owner/name GHC_DUMP=actions go run ./cmd/probe owner/name ``` **The pictures of the card.** `GHC_CARD_GALLERY` names an existing directory and `TestCardGallery` renders one card per layout into it, from the fake GitHub rather than from anybody's account. That is where the pictures on [the layouts page](/ghchronicle/card/layouts/) come from, and a layout that changes shape is one command away from a set that agrees with it. The account is the base fixtures with `test/e2e/testdata/gallery/` laid over them: a year of contributions, GitHub's whole fourteen days of traffic, five repositories to rank and one of them in six languages, which the smaller account every other suite asserts on cannot give a picture. A fixture named `~` there answers for that one repository, and any other repository borrows hello-world's. Each layout comes out of one sweep under `-card-theme both` as two files, `card-.svg` in the light palette and `card-_dark.svg` in the dark one, which is what the site's `ThemeImage` and the README's `` read. The two layouts that loop come out a second time under `-card-motion loop`, as `card--loop.svg` and its `_dark` twin. Only those two: on every other layout `loop` draws the same card as `once`, so a looping picture of one would be a second copy of the first under a name that promises something else. Which layouts they are is the registry's `Loops`, and the gallery reads it rather than keeping its own list. ```sh mkdir -p /tmp/cards GHC_CARD_GALLERY=/tmp/cards go test ./test/e2e/ -run TestCardGallery ``` `make check-gallery` renders the gallery into a scratch directory and fails, naming every difference, if the committed set no longer matches it byte for byte; `make gallery` regenerates it in place. CI's "Generated artifacts" job runs the check on every pull request. ## Running the stack Docker with the compose plugin, and room for the images. Then: ```sh make test-e2e-docker ``` Up, run, down on every path including a failing assertion, and then a check that `docker ps` shows nothing of the project left. A suite that leaves nine containers behind on a failure is a suite nobody runs twice. > **Nothing of this listens outside loopback** > > Every port is published on `127.0.0.1`, on a free port Docker picks out of > 49200 to 49299, never a store's default: 9200, 8086, 5432, 2003, 9090, 3100, > 3000 and 4318 belong to whatever else is on the machine. The harness reads the > chosen ports back with `docker compose port`, and the project is named > `ghchronicle-e2e` in the compose file as well as on every command, so nothing > here can reach a container it did not start. Boot, measured cold with the images already pulled: Elasticsearch 29 s, Loki 21 s, Grafana 13 s, the Graphite render API 10 s, InfluxDB 8 s, PostgreSQL 6 s, the rest 6 s. The stack is ready in 30 s; the target end to end, teardown included, is 53 s. ## Debugging with the stack up The reason to fail an assertion is to go and look at the store, and a suite that tore the store down first cannot be looked at. So the two halves are separate targets, and the harness reuses a stack it finds already running and leaves it running. 1. Start the stores and leave them up. The command prints the port each service ended up on. ```sh make e2e-docker-up ``` 2. Run the suite, or one test of it, as many times as it takes. ```sh go test -count=1 -tags dockere2e -timeout 30m -v ./test/e2e/docker/ ``` `GHCHRONICLE_E2E_KEEP=1` also stops the test binary tearing down a stack it started itself, which is what you want when a single `-run` is failing. 3. Ask the store what it thinks, then tear it down. ```sh make e2e-docker-logs SERVICE=influxdb make e2e-docker-down ``` With the ports from step 1: ```sh # What InfluxDB thinks each column is. This is the answer to a 400 on write. curl -s "http://127.0.0.1:/api/v3/query_sql?db=ghchronicle" \ --data-urlencode "q=SELECT * FROM information_schema.columns WHERE table_name = 'gh_repo'" # The mapping Elasticsearch built for itself. curl -s "http://127.0.0.1:/ghchronicle-*/_mapping?pretty" # What the SQL sink actually created. psql "postgres://ghchronicle:ghchronicle@127.0.0.1:/ghchronicle" -c '\d+ gh_repo' # The Graphite path, node by node. curl -s "http://127.0.0.1:/metrics/find?query=github.repo.*" ``` Grafana is at the port it published, with `admin` and `admin`, and every datasource is already provisioned, so a panel query can be pasted into Explore and run by hand. ## Three things the stack had to be told Each of these silently produced a wrong answer before it was found, and each is in the compose file or its configuration with the measurement beside it: - **Carbon drops a point older than its longest archive without saying so.** A star dated 2020 vanished under a six year retention and the write was reported as accepted. The retention is `1d:12y` for that reason. - **Carbon's default `MAX_CREATES_PER_MINUTE` is 50**, fewer paths than one sweep creates, so most of a first sweep would be dropped. - **Loki answers a push with 204 and will not serve it until the chunk is flushed.** The test polls rather than asking once, and `chunk_idle_period` is 5 s. > **A firewalled machine needs one rule to scrape the exporter** > > The Prometheus exporter is the one sink that is scraped rather than pushed to, > so the scrape has to cross from the container back out to the host. Where the > default INPUT policy is deny, no container on any bridge reaches any host > port. One narrow rule is enough: allow TCP 49300-49399, the range > `scrapePortLow` and `scrapePortHigh` bound in `sweep_push_stores_test.go`, > from the container address space and from nothing else. Under ufw that is > > ```sh > ufw allow proto tcp from 172.16.0.0/12 to any port 49300:49399 > ``` > > Without it the harness reports `ErrExporterUnreachable` and the Prometheus > assertions skip with a reason instead of failing, which means they have never > run on that machine. A CI runner needs no rule and neither does an ordinary > workstation. A machine that adds the rule runs those assertions for the first time, which is when `promNeedsHistory` in `dashboards_test.go` starts to matter: the exporter is alive only for the length of the test, so every timeseries panel and every panel built on `increase()` is held to nothing and only the instant panels are asserted. ## In CI `.github/workflows/e2e.yml` runs `make test-e2e-docker` and has three ways in: manual dispatch with an optional ref, a weekly schedule on main, and `workflow_call`, so a release pipeline gates a tag on it with one line rather than a copy of the job that drifts from the original. It is not a required check on a pull request: nine containers and around 10 GB of image is too much for every push, and L1 is what covers every push. The weekly run is the point of the schedule. Nothing else in the repository ever starts a container, so without it the suite would run only when somebody remembered it, which is how a suite ends up broken for weeks with nobody knowing. The same suite also runs under the race detector, in `.github/workflows/race.yml`: weekly, at every release beside the E2E gate, and by hand. The harness builds the collector with `-race` and starts it with `GORACE=halt_on_error=1`, so a race inside the collector fails the test that started it, with the report. Locally it is `make test-e2e-docker-race`.