Troubleshooting

Things that look like errors and are not

not available (403) or (404). The feature is switched off for that repository, or the token cannot see it. Dependabot, code scanning, discussions and the dependency graph all answer this way when disabled. The collector records the fact and moves on: a repository with a feature off must not stop the sweep for the other forty.

If it is every repository rather than one, it is the token. Traffic needs push access and, on a fine-grained token, the repository permission Administration (read); alerts need security_events. See the token.

A feature switched on, and nothing collected from it. Code scanning enabled this morning, Dependabot turned on, a forum opened: the collector asked before you did it, was refused, and remembers the refusal for a day rather than paying for it on every sweep. It is noticed a day later at the latest, a restart included; deleting the cache file beside the state file asks again straight away. See a refusal is remembered too.

still being computed by GitHub (202). GitHub computes the stats/* endpoints asynchronously and answers 202 with an empty body while it works. The next sweep usually gets the numbers.

Two of them never do. On a personal account stats/code_frequency and stats/contributors return 202 with an empty body indefinitely, which is why this project does not call them: the lines added and removed come from the commits collector instead.

pagination is limited for this resource (422). The end of an activity feed, not a failure. GitHub serves three pages of the event feed and refuses the fourth.

A repository with no traffic showing a window that ended weeks ago. GitHub keeps returning the last fourteen days that had data, not the last fourteen days. The collector records what it is told.

cache file not read, the first pass of each family pays in full. The cache file beside the state file was cut short, damaged or written in another format, so it is not read at all, and the next save writes over it. It costs one pass of each family at full price and loses nothing; see the cache beside it.

which repositories moved could not be asked, reading those it did not answer for. The query that tells commits, issues and issueevents which repositories to read failed, or answered for some of them. Every repository it said nothing about is read as if it had not been asked, so nothing is missed and the pass costs what it did before the query existed; see asking first what moved.

outbound search read fewer items than it counts, GitHub serves a thousand at most. GitHub serves a thousand results of a search and no more, and the account has more than that in one of the states outbound searches, merged pull requests for instance. The thousand that moved most recently are kept, and the line is said once per count; see what a backfill reaches that a sweep does not.

NOTICE: column ... already exists, skipping from psql. Replaying the SQL file sink’s file into a database that already has its tables. The file declares every table and every field column it writes, since it cannot ask which exist, and PostgreSQL says so for each one that does, with a relation ... already exists for the table. Nothing failed.

A family that never appears in the log. It is not due yet. With a twelve-hour cadence, half a day of logs can legitimately never mention stats. A family of six hours or more can also be due and waiting its turn behind another one, which the log names under waiting in slow families due together take turns; see the slow families take turns.

Things that are errors

github.token is empty and GITHUB_TOKEN is unset. Exactly what it says.

every.families.<name>: unknown collector. The name is not a family. The message lists the thirty-four that exist, in one parenthesis after the colon.

groups: is empty. groups: [] would collect nothing at all. Omit the key to collect everything, which is what it means when it is absent.

groups[N]: "<name>" is not a group. The name is not a group. The message lists the ones that exist, and ghchronicle -groups prints each with its families.

groups[N]: "<name>" is a family, not a group. Families and groups are both lowercase nouns from the same table, so this is an easy one to hit. The message names the group the family is in, which is probably what you wanted, and points at every.families.<name>, which is where a single family’s cadence lives.

sinks: enable at least one of .... A run that collects and discards is almost never what anyone meant. -card-only is the exception and needs no sink at all.

prometheus exporter: listen tcp :9605: bind: address already in use. Reported at start-up rather than swallowed in a goroutine, so a port clash cannot leave you with a running collector and a silently missing exporter.

influx write: 400. Almost always a column type collision. InfluxDB fixes a column as a tag or a field the first time it sees it and rejects later writes that disagree. If a collector changed which one a name is, the table has to be dropped: DELETE /api/v3/configure/table.

family failed everywhere, not marking it as run. Every repository failed for one family, so it will be retried rather than treated as done. One repository failing is normal; all of them is the token, the network or an outage.

collector failed naming a 500, a 502 Bad Gateway, a 503 or a 504 Gateway Timeout. GitHub did not finish the answer to a request, at its gateway or in the application, or the storage a job log is read from did not. The client asks once more two seconds later and says nothing about it, so a line like this is a request that failed twice, and the next sweep asks again. A GraphQL query is asked again the same way, but for one answer: a 502 or a 504 that took GitHub’s ten seconds to come back is a query too large, which the collectors that have a smaller page ask again with it. A line that says the query is too large for one request is one that timed out even at the smallest page, or in a walk with no smaller page to ask. What was read before it is written, a backfill does not record that repository as walked, and the next sweep reads it again.

state not saved or cache file not saved. The user the collector runs as could not put the file in place. Each of the two is written under a temporary name in the directory the state file names and then renamed into place, so either that directory cannot be written, or it does not exist and cannot be created, or the file itself cannot be renamed over. Nothing a sweep learns then survives a restart: the next start collects every family and walks the stargazer list, the whole star history and the co-authored pull requests again. In a container this is a host directory mounted without handing it to uid 65532, a new volume mounted anywhere but /var/lib/ghchronicle, or any new volume under an image before 2.6.1, none of which that uid owns, or a state file mounted on its own, which cannot be renamed over; see what has to be writable.

the state file ... cannot be read or does not parse, and the run stops. The state file is there and this run could not read it, or it is not whole. A run that writes to the stores does not take it for a new one, which would forget the refills a migration still owes and put a new file in its place on its first save. A file root left behind, after -migrate -yes or a backfill run with sudo, is the usual cause: chown it back to the user the collector runs as. Moved aside instead, the next run starts from a new one, and what it recorded, a refill still owed among it, is gone; see the state file.

rate limit reserve reached, family skipped. Once is fine. Every sweep means the cadences are too fast for the number of repositories. If the bucket that runs short is core, lengthen artifacts and then actions; if it is graphql, issues, issueevents and commits. See what scales with activity, not with size.

refill owed: a store a migration cleared does not hold that history yet. A migration cleared the measurement named in that store and reading its history back from GitHub did not finish: the run was stopped, a store refused a write, or GitHub did not answer. The store now looks like one that never held the old shape, so only the state file knows. Under migrate: warn a start says this and sweeps; -migrate -yes, named in the line, reads it back from where it stopped, and so does any start under migrate: auto. See reading the history back.

The data looks wrong

reconciled at WARN, with not_served. After reading a cleared measurement back, ghchronicle compared the copy of the old rows with the table and found items GitHub no longer serves: a repository deleted or no longer covered, a comment deleted, an alert whose feature was switched off. first names a few, and only_in is the copy that still holds them, until it is purged: a day after the migration for PostgreSQL’s and Elasticsearch’s, and when the server’s schedule says for InfluxDB 3’s, 72 hours by default, or never on a server before 3.2. Carrying them over is by hand, from that copy, while it is there.

A number is a multiple of the sweep count. Something that is a snapshot is being summed over time. Referrers, paths, labels and milestones are snapshots of a window with no date of their own; they are stamped at the start of the UTC day so a day’s sweeps rewrite one row, and the dashboard takes the newest rather than the sum.

Median time to review by someone else reads No data. The panel reads seconds_to_first_human_review, which leaves out review bots and the author’s own replies; on an account where nobody else reviews, no pull request carries it and the tile is honestly empty. It does not mean nothing was reviewed. The bots’ speed is in the Reviewers table, where each one is marked as a bot: measured, nine pull requests in ten had a bot review inside a minute.

Clones are enormous compared with views. Continuous integration clones a repository thousands of times for every human visit. One repository measured here took more than a hundred clones for every view. clones does not count people.

open_issues disagrees with the issue count. That field is GitHub’s, and GitHub counts pull requests as issues in it. The gh_issue measurement is the one that counts issues.

Artifact storage looks too small. Compare walked against count in gh_artifact_total. When they disagree the live size is a floor: the repository has more artifacts than the page cap walked, or a page of the listing failed on that sweep and the row was written from the pages before it. It can also be smaller than expected for a second reason: count is GitHub’s total and includes the artifacts it has already expired, while the size is over the live ones alone, which live_count counts.

A panel answers with a schema error rather than No data. InfluxDB 3 creates a column the first time a row carries it, so a query naming one that no row has written fails at planning. A release that adds a field does that to an existing database until the family that writes it has swept once: gh_fork.seconds_to_push and gh_artifact_total.live_count were two, which the Forks and “Artifact storage counted” panels name, and the forks family runs every twelve hours. A release that adds a measurement a panel joins does the same to the whole panel, in PostgreSQL as well, where the table rather than the column is missing: the “Work elsewhere” table joins gh_upstream_repo for each repository’s stars, and until the first outbound pass of the new version writes it, which is within an hour, both stores refuse the table rather than leave its Stars column empty. Either wait for that pass or publish the dashboards after it. A -once does not hurry it: it runs only the families that are due, and outbound is not until an hour after its last pass. The other columns a database can lack are listed under Columns that exist only once written.

The traffic chart only goes back fourteen days. That is a first sweep. The window is rewritten day by day on every sweep, so the series extends as the collector keeps running. It cannot be backfilled: GitHub never stored anything older.

A panel says “Query would scan 10000 Parquet files”. InfluxDB 3 Core writes one file per partition per write request and never compacts them, so a store fed by a version of this tool older than the write ledger holds its rows in far more files than it needs. Widening the panel’s interval does not help: the limit counts the files the planner opens, before any aggregation. What helps is the ledger, which is on by default and stops the growth, and then one of three things for what has already accumulated: --query-file-limit raised on the server, the affected tables rewritten, or InfluxDB 3 Enterprise, which compacts on its own and is free for home use. See only what changed is written.

A Prometheus panel shows one flat line. That is the store, not the data. The exporter serves current values, so the fourteen-day traffic window collapses to its most recent day and the star history to the current total. See dating a point.

Why is nothing being written?

Run one sweep in the foreground and read what it says. Then check, in order:

1. that -list prints the repositories you expect,
2. that the sweep log says written,
3. that the sink is reachable.

ghchronicle -config config.yaml -list        # the repositories, and which are set aside
ghchronicle -config config.yaml -once        # one sweep in the foreground, then exit
journalctl -u ghchronicle -f                 # under systemd

debug adds these to that and nothing else: the size of the written-points ledger at start-up, a first start with no cache file, a start listing jobs again because a store keeps no ledger, the account-wide families skipped for want of a targets.user, the entries a sink left out for being too old, a card-only sweep leaving the state file alone, each save of the cache file, the migrations a start finds not needed or applied before, the ones noted or frozen that the first start said at INFO, and each copy a migration set aside that the state file forgets because the store purges it or keeps it for good. There is no per-request log at any level. See debugging.

log:
  level: debug

A family that is not due yet simply does not appear.

Deleting the state file costs quota, and two things more

It remembers nine things, and seven of them cost only quota when they go: what is collected again is keyed by measurement, tags and timestamp and overwrites. last_head is the commit each dependency diff started from, and without it the next sweep has the photograph and no diff. stores is what the file records of each store, a refill a migration still owes among it, and without it nothing reads that history back until a -backfill -families does. See the state file. The cache file beside it costs quota and nothing more: see the cache beside it.

Why does Loki drop entries?

Look for the debug line counting them. Loki refuses an entry more than its out-of-order window behind the newest entry already in that stream, half of the ingester’s max_chunk_age and so one hour by default, so the sink leaves the older ones out rather than losing the whole push. Raise max_age only alongside Loki’s own max_chunk_age. See Loki.

Publishing the dashboard

Permissions needed: datasources:create. The token publishes dashboards and cannot make datasources. An Editor can do the first and not the second, so either give the service account that permission, make it an Admin, or let it adopt a datasource you made yourself by naming it in grafana.datasource.uid, which only ever reads. Correcting a datasource that is already there asks for datasources:write instead, with the same answers. That stops the run for the store’s datasource, which every panel reads. For the Loki one it is only a warning, below.

datasource ... does not answer: connection refused. Grafana reached the address and nothing was listening, which almost always means the address is right for the collector and wrong for Grafana. A collector on the host writes to a published port; Grafana in a container reaches the same store by its name on the container network, and 127.0.0.1 there is the container. Put the address Grafana would use in grafana.datasource.url. The run refuses rather than publishing, because a dashboard against a datasource that cannot be reached draws nothing and says nothing about why.

asking datasource ... whether it answers: ... datasources:query. The token may not query the datasource, which is not the same thing as a datasource that cannot reach its store, and the address is not what to change. Measured on Grafana 13, a service account with no role gets this and an Editor does not, so give it the Editor role.

Invalid API key. The token reached Grafana as the characters ${GRAFANA_TOKEN} rather than as a token, which means the variable is not set where the collector runs. In a service unit that is an EnvironmentFile the unit does not read, or a variable named in the file and not exported.

The datasource is there and has no password. Only a password the DSN itself writes is sent. One that pgx found in PGPASSWORD or a .pgpass on the machine that published is named in a line and left there, because a Grafana is shared and a credential the configuration never mentioned is not one to copy into it.

the dsn asks for sslmode=prefer and Grafana's datasource has no such mode. libpq’s default is to try TLS and carry on without it, and the datasource either insists or refuses. It was given disable; set grafana.datasource.sslmode to require, verify-ca or verify-full if that is wrong for your server.

Two dashboards, and the one you are looking at stops changing. The generated dashboard has its own uid and every publish overwrites it, so this only happens when yours lives somewhere else: imported as new, or renamed by hand. Name the one you read in grafana.dashboard_uid.

... is still in Grafana and no ... sink is configured here. Something this made is under a uid nothing writes to any more, which is what changing store leaves behind. It is a note and nothing was touched; delete it in Grafana when you are done with it.

no sink this builds a dashboard for is configured. Dashboards exist for five stores, and the sink you have is not one of them. A forwarding sink has no dashboard of its own because the dashboard belongs to wherever it forwards to, and file, stdout and loki are not metric stores.

warning: the Loki datasource ghchronicle-loki could not be set up. The token may not make the Loki datasource the failed job output panel reads, and the line names the permission it lacks: datasources:create for an Editor, since the datasource is not there yet. Nothing else waits on it: that one panel is left as the note the exported files carry, and the dashboards are published as long as the store’s own datasource is. Name a Loki datasource Grafana already has in grafana.datasource.loki_uid, or grant the permission. The line lists the Loki datasources Grafana has, with their addresses, because the one to name is often there already under an address the collector does not push to, such as Loki’s name on Grafana’s container network. 2.4.0 and 2.5.0 stopped the whole publish here instead, with the Loki datasource: writing datasource ghchronicle-loki, and loki_uid is the fix for those too.

warning: the Loki datasource ghchronicle-loki is used as Grafana has it. An earlier run with a token that could write datasources made it, and this token may not rewrite it: the line names datasources:write. A sink with a tenant_id asks for that rewrite on every start, because the tenant is a secret Grafana never hands back to compare, and so does a datasource whose address was changed by hand. The panel reads it as it is, so nothing is lost unless the sink has moved since. Grant the token datasources:write to keep its address and tenant in step with the sink, or set grafana.datasource.loki_uid to ghchronicle-loki to read it without trying.

datasource ... (loki) adopted. Grafana already had a Loki datasource at the address the sink’s push endpoint comes from, so the panel reads that one and no second one was made. Adopting only reads, so a token that may not create datasources gets this far. A sink with a tenant_id is never matched this way: the tenant travels in a secret Grafana never hands back, so nothing can tell whether the datasource at that address reads the same tenant, and one is made instead.

The failed job output panel is still a note. That panel becomes the log lines when there is a Loki sink whose address ends in /loki/api/v1/push, which is what says where the query endpoint is. A sink writing anywhere else says so and leaves the panel alone; name a Loki datasource in grafana.datasource.loki_uid. A token that may not make the datasource leaves it too, with the warning above.

Nothing was published and nothing failed. publish_on_start is off unless you turn it on, and it is skipped for -once, -backfill and -card, because turning it on in a service unit did not mean rewriting the dashboard on every run of a scheduled card render.

The collector started and the dashboard did not appear. A publish that fails at start-up warns and the sweep goes on. The metrics of an hour spent not running cannot be recovered and a dashboard published on the next restart can, so the log line is the place to look.

Taking things away

-uninstall printed a list and removed nothing. That is the default. Add -yes.

The datasource survived the uninstall. One named in grafana.datasource.uid is adopted, not made, so it was somebody else’s before this ran and stays theirs.

unknown -uninstall target. The whole list is refused rather than the part that parsed, so a typo removes nothing. The targets are dashboard, data, state and all.

The rows are still in the database after -uninstall data. Three stores cannot be emptied from here and say so: Graphite has no delete, the Prometheus sink was scraped rather than written to, and the SQL sink’s file goes but rows already loaded from it into a real database were loaded by you.

Tables or indices named gh_...-20261001T091004. A migration set the measurement’s old rows aside under that name, the measurement, a dash and the instant in UTC: PostgreSQL’s renamed table, Elasticsearch’s clone, lower case there, and the name InfluxDB 3 gives a table it deleted. None of the shipped panels reads one. ghchronicle purges PostgreSQL’s and Elasticsearch’s once they have been kept 24 hours, and InfluxDB 3 purges its own on its own schedule, 72 hours after the delete by default, except one a release before 3.2 deleted, which is never purged, not even after an upgrade (see before 3.2 the copy stays); -migrate lists them under their store as kept aside. -uninstall data removes PostgreSQL’s and Elasticsearch’s with everything else, and leaves out InfluxDB 3’s with a note saying which of them stay, since the server refuses a delete of a table it has already deleted, or, before 3.2, renames it once more.

After an upgrade

migration pending, at every start. A store holds rows of a measurement in a shape this release no longer writes, and this start did not bring it along. not_applied says why: migrate: warn, or a reason the change needs somebody’s word, such as an InfluxDB 2, whose only way is a final delete, a SQL file, a Graphite or a Telegraf, where ghchronicle keeps nothing aside, rows of accounts this configuration does not collect, rows reading it again would not bring back, or rows whose accounts or repositories could not be read, which an InfluxDB 3 Core past its query file limit refuses to say; or a one-shot run on a new state file, every run of the Action without one restored, which applies nothing on its own. plan and apply are the two commands, and first, on the service, says to stop it before the second. It is said at every start until the store is brought along: see Migrations.

applying a migration before the first sweep. Not an error. Under migrate: auto the start found a change it can apply without losing anything, and is applying it: found is what the store holds, action where the old rows go, and refill what is read back from GitHub. It comes once per store and change, followed by refill starting, refill complete and reconciled, and then the first sweep.

migration noted: nothing is changed or migration frozen. Not errors either. A note is a change nothing can put right, such as the gh_actions_cache_entry rows dated before 2.6.0, and a frozen measurement is one nothing this configuration runs writes any more. Each is said once at INFO and then at DEBUG.

migration check failed: the store did not answer. The store was asked at start-up whether it holds an old shape, and refused or did not answer within 30 seconds. Nothing was recorded and nothing changed; the sweep goes on, and the next start asks again.

migration failed. The store refused what applying asked of it, and the change stays pending; the other stores went ahead. err says why, and the usual reason is a credential that may write and may not delete: InfluxDB 3’s token has to be allowed to delete a table, and the error carries the same delete as a curl to run by hand; PostgreSQL’s user has to own the table, as it does when its own sink made it; an Elasticsearch key needs manage and delete_index on the indices of its prefix. PostgreSQL also gives up after three waits of 5 seconds each behind a query that holds the table, which a long Grafana query can be. Run the same command again once the cause is fixed: what was applied is recorded and is not done twice.

refill did not finish, and is still owed. Reading back what was cleared stopped: the run was stopped, a store refused a write or GitHub did not answer. resume says how to go on. The walk keeps its checkpoint beside the state file, <name>-refill.json, and refill owed, under things that are errors, is what a later start says of it.

the store may have been cleared, so the refill stays owed. Clearing the store failed in a way that does not say whether it was carried out: a proxy’s 502, a timeout, a connection that broke after the store did what it was asked. The refill was recorded as owed before the store was touched and stays owed, so the history is read back whether or not the clear happened; the change is asked about again at the next start. At worst that reads the history once for nothing.

the state file records a refill owed to ... where it no longer points. A migration cleared a store, the refill did not finish, and the sink now points at another store. The record is kept, and the refill is read the next time the sink points there; until then nothing reads it. -migrate shows it under the store as owed there.

not reconciled: the copy and the table could not be compared. The history was read back, and the copy of the old rows could not be read, or was already due to be purged. What came back is in the table; only the list of the items GitHub no longer serves is missing, and while the copy is there it can be read by hand.

set-aside copy not purged or set-aside copies not listed. A day after a migration set it aside, ghchronicle could not drop PostgreSQL’s copy or delete Elasticsearch’s clone, or could not ask the store which copies it holds. It tries again after every sweep and at every start; aside names the copy, which can also be removed by hand. InfluxDB 3 purges its own on its own schedule, or, before 3.2, never.

the cache file still claims what the cleared store held. A migration could not rewrite the cache file to forget the refusals, and the workflow runs already written, of the families it reads back, so those families may go on skipping what the cleared store no longer holds. With the service stopped, deleting <name>-cache.bin costs one sweep at full price: the cache beside it.

-migrate -yes answers process N (ghchronicle ..., service, since ...) holds ...-lock. The service, or another -migrate -yes, holds the state file, and nothing was changed. Stop it, or let it finish, and run the command again. In Docker, N is the process as the service’s own container numbers it.

waiting for the process holding the state file to finish. The service was started while -migrate -yes ran on the same state file. It waits for it to end and then starts on what it left.

another ghchronicle service keeps the same state file. A second service was started on a state file a running one holds, and refused, since two services saving one state file undo each other’s marks. Stop one, or give each a state_file of its own.

the state file cannot be locked, so -migrate -yes cannot tell this service is running. The service could not create or open <name>-lock beside the state file: the directory is not writable, or a run as another user, root most often, made the file. The service runs without the lock, and -migrate -yes then refuses to run at all, since nothing would keep the service from starting under it half way. Run it as the service’s user: after an upgrade.

The PostgreSQL that connects

could not reach its database. The sink connects on the first write rather than at start-up, so this is the first sweep and not the configuration being read. The DSN takes either shape libpq does, and the message is the driver’s.

A column is missing rather than the row. A field with no value does not get a cell: an empty string is left out, and a float that is not a number is stored as NULL. The column appears the first time a point carries something for it.

A field changed type and the insert fails. A column is created with the type of the first value seen and is never altered afterwards, so a field that was a number and is now a string has nowhere to go. Rename the field, or drop the table, and the collector’s next write declares it again. No restart is needed: the sink declares a table once per process, and the first insert after the drop is refused because the table is not there, so the sink forgets the tables of that batch, declares them again and sends the batch once more.

The SQL file’s gh_discussion_comment statements are refused. With there is no unique or exclusion constraint matching the ON CONFLICT specification, replayed into a table a release before 2.6.1 made. That table keeps is_answer in its key, which 2.6.1 no longer writes, and the file cannot read a table’s key the way the connecting sink does, so its upserts name a key the table does not have. The rest of the file loads. -migrate -yes writes the drop of that table into the file, ahead of the rows it reads again, so the file replayed in order loads whole: see what applying does in each store. By hand, drop the table, as the measurements page describes, and fill it again with a backfill.

The file sink and this one disagree. Apart from the key above, they cannot: both render the same schema from the same code, and cmd/check_postgres plans every panel query against it. If the tables differ, one of them was loaded from an older version.