Repository navigation
Description
When github-metadata-backup runs with a personal access token, GitHub includes some timeline events that it shows only to that user, not to the public. I found two kinds in my backup:
- the token owner's own pending (unsubmitted) reviews, and
- references from commits in private repositories that the owner's account can read.
Privacy: the generator renders these like any other event, so a public mirror can publish a reviewer's unsent draft review, or reveal that a private repository has commits mentioning an issue, and when.
Crash: both kinds have null fields the HTML renderer doesn't expect (submitted_at, and commit_id/commit_url), so the generator exits with a traceback and writes no pages. My mirror stopped updating for two weeks this way.
Fix: drop both kinds of event when the backup JSON is loaded, so neither the index nor the renderers see them. The diff at the end does that, and also makes the "referenced" rendering null-safe (event.get("commit_url", "") returns None when the key is present with a null value; markdown_page.py already uses or "" for this). Feel free to apply it as is or fix this a different way. I can turn it into a PR as well if that would help.
Details
Pending reviews
GitHub shows a pending review only to its author, so a backup made with a reviewer's token contains their pending reviews as reviewed events with "state": "PENDING" and "submitted_at": null. My backup has four (bitcoin/bitcoin#17493, #30361, #30437, bitcoin-core/libmultiprocess#361). timeline_event passes event.get("submitted_at", event.get("created_at", "")) to comment_card, which is None, so any pending review with a body crashes:
File "mirror/html/components.py", line 110, in comment_card
time_str = format_time_short(created_at)
File "mirror/util.py", line 34, in _parse_iso
return datetime.strptime(iso, DATE_FORMAT).replace(tzinfo=timezone.utc)
TypeError: strptime() argument 1 must be str, not None
Private references
My backup has three referenced events whose commit_id and commit_url are null, all by the same user:
| Issue | Event id | Created |
|---|---|---|
| bitcoin/bitcoin#35361 | 30879047178 | 2026-09-10T05:48:42Z |
| bitcoin-core/libmultiprocess#319 | 30879046848 | 2026-09-10T05:48:42Z |
| bitcoin/bitcoin#36049 | 31722068529 | 2026-09-24T01:41:40Z |
They come from commits in a private repository my account can read, while the backup token is limited to public repositories, so GitHub lists the event but hides the commit:
- With the backup token, each issue's
/timelineincludes the event. Through GraphQL (by node id) it is aReferencedEventwithisCrossRepository: trueandisDirectReference: true(a commit in another repository naming the issue in its message),commit: null, and asking forcommitRepositoryfails with a GitHubINTERNALerror. - With a token from an account that can't read that repository, the events are missing from
/timelineand/events, and the GraphQL node isNOT_FOUND. They can still be fetched by id from/repos/{owner}/{repo}/issues/events/{id}, with the same null fields. This is why your mirror doesn't have them: your #35361 page shows the other ten "referenced" entries on that issue but not this one.
Rendering one crashes:
File "mirror/html/components.py", line 396, in timeline_event
commit_url = commit_url.replace("api.", "").replace("/repos", "").replace("commits", "commit")
AttributeError: 'NoneType' object has no attribute 'replace'
Patch
This is what my mirror runs now (applied to da474e2).
diff -ru a/mirror/data.py b/mirror/data.py
--- a/mirror/data.py
+++ b/mirror/data.py
@@ -20,6 +20,28 @@
])
+def drop_private_events(data: dict[str, Any]) -> dict[str, Any]:
+ """Remove timeline events that GitHub shows only to some users, modifying data in place.
+
+ A backup made with a user's token gets events that are not public:
+ - that user's own unsubmitted reviews ("reviewed", state PENDING, submitted_at null);
+ - "referenced" events from commits in private repos the user can access. If the
+ token itself cannot read that repo, commit_id and commit_url are null.
+ Neither is shown to the public on GitHub, and neither has the data the renderers
+ expect, so both are dropped before anything reads the events.
+ """
+ events = data.get("events")
+ if events:
+ data["events"] = [e for e in events if not _is_private_event(e)]
+ return data
+
+
+def _is_private_event(e: dict[str, Any]) -> bool:
+ if e.get("event") == "reviewed" and e.get("state") == "PENDING":
+ return True
+ return e.get("event") == "referenced" and e.get("commit_id") is None
+
+
def remove_nested_keys(obj: Any) -> Any:
"""Remove unneeded API keys to reduce size, modifying obj in place."""
if isinstance(obj, dict):
diff -ru a/mirror/html/components.py b/mirror/html/components.py
--- a/mirror/html/components.py
+++ b/mirror/html/components.py
@@ -391,13 +391,16 @@
return _simple_event(b, "arrow-clockwise", f"<b>{html_escape(name)}</b> restored the branch {date_str}")
if ev == "referenced":
- commit_url = event.get("commit_url", "")
+ # GitHub can send commit_url and commit_id as null rather than leaving them out
+ # (bitcoin/bitcoin events 30879047178 and 31722068529), so .get(key, "") is not
+ # enough. Render like markdown_page.py does: no link when there is no URL.
+ commit_url = event.get("commit_url") or ""
# Rewrite API URL to web URL
commit_url = commit_url.replace("api.", "").replace("/repos", "").replace("commits", "commit")
- commit_id = truncate(event.get("commit_id", ""), 10)
+ commit_id = truncate(event.get("commit_id") or "", 10)
+ commit = f'<a href="{commit_url}">{commit_id}</a>' if commit_url else commit_id
return _simple_event(b, "link-45deg",
- f'<b>{html_escape(name)}</b> referenced this in commit '
- f'<a href="{commit_url}">{commit_id}</a> {date_str}')
+ f'<b>{html_escape(name)}</b> referenced this in commit {commit} {date_str}')
if ev == "cross-referenced":
source = event.get("source", {})
diff -ru a/mirror/html/renderer.py b/mirror/html/renderer.py
--- a/mirror/html/renderer.py
+++ b/mirror/html/renderer.py
@@ -8,7 +8,7 @@
from typing import Any
from mirror.config import Config
-from mirror.data import build_pull_timeline, remove_nested_keys
+from mirror.data import build_pull_timeline, drop_private_events, remove_nested_keys
from mirror.markdown import MarkdownRenderer
from mirror.models import SiteIndex
from mirror.util import format_date_long, urlize
@@ -186,4 +186,4 @@
def _read_json(path: Path) -> dict[str, Any]:
with open(path, "r") as f:
- return json.load(f)
+ return drop_private_events(json.load(f))
diff -ru a/mirror/index.py b/mirror/index.py
--- a/mirror/index.py
+++ b/mirror/index.py
@@ -8,7 +8,7 @@
from typing import Any
from mirror.config import Config
-from mirror.data import extract_issue_meta, extract_pull_meta
+from mirror.data import drop_private_events, extract_issue_meta, extract_pull_meta
from mirror.models import EntryMeta, SiteIndex
SUBSET_SIZE = 100
@@ -16,7 +16,7 @@
def _read_json(path: Path) -> dict[str, Any]:
with open(path, "r") as f:
- return json.load(f)
+ return drop_private_events(json.load(f))
# Bots whose cross-references are noise and should be excluded from the graph.
Activity
0xB10C commentedon Sep 29, 2026
makes sense to incorporate this here.
My general recommendation would be to use another GH account and/or a GH token with very limited scope.
ryanofsky commentedon Sep 29, 2026
Yeah it is a fine-grained token with no permissions, repository access set to "Public repositories - Read-only access to public repositories." I don't think it is possible to create a token with more limited permissions than this. The public repository setting probably explains why the references to events in private repositories are null even though the events are still in the timeline.
fix: drop non-public timeline events when loading backup JSON
0xB10C commentedon Sep 29, 2026
Can you test #21 in your setup?
ryanofsky commentedon Sep 29, 2026
Confirmed 98c589e works in my setup.
But I realized one thing this doesn't cover is the backup JSON. The events are still in the JSON files, and the github-metadata-backup README suggests committing each backup to git (the NixOS module can push it to remotes), so anyone publishing their backup publishes them too. My mirror also serves tarballs of the raw backup, and they contain all 7 events. What leaks is limited:
PENDING,submitted_atnull, and the summary text if one was written). Draft inline comments aren't included, since/pulls/N/commentsdoesn't return them.reposcope), the event arrives fully populated and looks like a public one.Restricting the token doesn't prevent this: my token is fine-grained, with no permissions and public-repository read-only access, and still gets both kinds of event, because GitHub decides based on what the account can see, not the token.
Planning to look into a rust patch.
ryanofsky commentedon Sep 29, 2026
Now using the following which drops nonpublic events from the github-metadata-backup stage:
LLM also suggested a documentation update there:
README.md
Non-public data in backups
The GitHub API returns some timeline events to a token's user that it does not
show to the public. This happens even with a fine-grained token that has no
permissions and read-only access to public repositories, because GitHub decides
what to return based on what the account can see, not the token. Two kinds are
known:
PENDING).These include the review summary text if one was written, but not the draft
inline comments.
("referenced" events). If the token cannot read the private repository, the
event includes who made the reference and when, but
commit_idandcommit_urlare null, so neither the commit nor the repository name isincluded. If the token can read it, the event is fully populated and looks
like a public one.
The tool drops pending reviews and references with a null commit before writing
the backup. It cannot recognize fully populated references from private
repositories, so if you publish your backup, use a token whose account cannot
see any private repositories that might reference the backed-up repository.
Files written by versions before this filtering was added keep these events
until the issue or pull request is fetched again. To remove them, delete
state.jsonto make a full backup. [github-metadata-mirror] also drops theseevents when generating HTML, so they only appear in the JSON files.