Skip to content

Commit 59f7aa8

Browse files
hotlongclaude
andcommitted
ci(board-snapshot): upload only the records that changed to R2, not all ~41k objects every run
`aws s3 sync` uploads when the size differs or the local mtime is newer than the object's LastModified, and a fresh checkout gives every file a new mtime, so every run re-PUT the whole archive: 1.2M PutObject in 11 days on the bucket (~$10/month of Class A operations against a 1M/month free tier). The archive branch is now fetched with its whole history but no blobs (`--filter=blob:none`, ~7 MB of commits and trees today), and before the sync each file's mtime is set to the committer time of the last commit that changed it. Every upload happens after the commit whose content it carries, so a current object is newer than its file and is skipped, while a file changed after its object's last upload (this run's edits, or an earlier run's failed upload) is newer and is sent. `--delete`, the target prefix, the daily tarball, the schedule and the fail-red/notice behaviour are unchanged. A truncated history only over-uploads; an unreadable one falls back to the old full re-upload with a warning. Verified by running the checkout, commit/push and R2 steps verbatim against a throwaway git server and a fake aws applying aws-cli's size+mtime rule: incremental uploads, deletions, a same-size edit, catch-up after a failed upload, a no-change run and a depth=1 history all leave the mirror byte-identical to the archive. With the mtime restore ablated, the same harness sees every run re-upload all objects. Co-authored-by: Claude <noreply@anthropic.com>
1 parent 018b077 commit 59f7aa8

1 file changed

Lines changed: 108 additions & 6 deletions

File tree

‎.github/workflows/board-snapshot.yml‎

Lines changed: 108 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -217,11 +217,21 @@ jobs:
217217
# The archive branch is an ORPHAN: it shares no history with `main`, so a
218218
# checkout of it carries the archive and nothing else. First run creates it
219219
# locally; the push below is what publishes it.
220+
#
221+
# It is fetched with its WHOLE history but without file contents
222+
# (`--filter=blob:none`), because the R2 step reads each record's
223+
# last-change time out of that history; see "Only changed records are
224+
# uploaded" there. The worktree checkout then fetches the tip's blobs in
225+
# one batch, which is the same content a `--depth=1` fetch downloaded, so
226+
# the history adds only commits and trees: ~7 MB for 127 commits over
227+
# 41,047 files, measured 2026-10-12. ⛔ Do not put `--depth` back: a
228+
# truncated history makes the R2 step re-upload every record older than
229+
# the cut on every run (safe, but the cost that step exists to avoid).
220230
- name: Check out or create the archive branch
221231
if: github.event_name != 'pull_request'
222232
run: |
223233
set -euo pipefail
224-
git fetch --depth=1 origin '+refs/heads/board-archive:refs/remotes/origin/board-archive' || true
234+
git fetch --filter=blob:none origin '+refs/heads/board-archive:refs/remotes/origin/board-archive' || true
225235
if git rev-parse --verify --quiet refs/remotes/origin/board-archive > /dev/null; then
226236
git worktree add archive -b board-archive refs/remotes/origin/board-archive
227237
echo "archive branch exists; checked out at $(git -C archive rev-parse --short HEAD)"
@@ -319,6 +329,43 @@ jobs:
319329
# rather than by the clock, so a missed 02:07 run, a re-run and a
320330
# `workflow_dispatch` all converge on one tarball per day.
321331
#
332+
# ## Only changed records are uploaded
333+
#
334+
# `aws s3 sync` uploads a file when its size differs from the object's,
335+
# or when the file's mtime is newer than the object's LastModified
336+
# (aws-cli `SizeAndLastModifiedSync`). A fresh checkout stamps every file
337+
# with the checkout time, so each run re-PUT all ~41,000 objects:
338+
# measured on the bucket, 1.2M PutObject in 11 days (~110k a day, ~$10 a
339+
# month of R2 Class A operations against a 1M-a-month free tier).
340+
#
341+
# So before syncing, every file's mtime is set to the committer time of
342+
# the last commit on the archive branch that changed it. That makes the
343+
# comparison mean what sync assumes it means:
344+
#
345+
# - an object's LastModified is the time it was last uploaded, and every
346+
# upload happens AFTER the commit whose content it carries (the commit
347+
# step runs before this one), so an object that is current has a
348+
# LastModified newer than its file's mtime and is skipped;
349+
# - a file changed by a commit made after its object was last uploaded
350+
# has the newer mtime and is uploaded. That covers this run's changes
351+
# AND anything an earlier run failed to upload, with no state kept
352+
# between runs;
353+
# - a file the archive no longer has is still removed by `--delete`.
354+
#
355+
# ⛔ Not `--size-only`: a record edit can keep its byte count (every
356+
# `updated_at` move does, and so does a count gaining no digit), and a
357+
# size-only comparison would leave the stale copy in R2 for good.
358+
#
359+
# This relies on two things being true. The archive branch is written
360+
# only by this workflow, so its committer times follow real time. And
361+
# nothing else writes under the mirror prefix: an out-of-band write
362+
# with the same size and a newer LastModified would not be overwritten,
363+
# where the old full re-upload replaced it within six hours. If the
364+
# history cannot be read, the step does not guess. It leaves the
365+
# checkout times in place, which re-uploads everything, and says so in
366+
# a warning and in the run's R2 status line: more Class A operations,
367+
# never a weaker copy.
368+
#
322369
# ## Unset secrets are a notice, not a failure
323370
#
324371
# The bucket and its scoped API token are the maintainer's to create.
@@ -378,6 +425,59 @@ jobs:
378425
# No `.git` filter is needed here: `archive/` is a linked worktree,
379426
# so its `.git` is a FILE at `archive/.git`, which is outside this
380427
# source (filters match paths relative to the source directory).
428+
#
429+
# mtimes first: see "Only changed records are uploaded" above. The
430+
# history is walked newest-first, so the first time a path appears
431+
# is the last commit that touched it; a path whose newest commit
432+
# deleted it has no file and is skipped. `-z` keeps every byte of a
433+
# path intact. A truncated history errs in the safe direction only:
434+
# its oldest commit looks like the one that created every file older
435+
# than it, which stamps those files NEWER than their real last
436+
# change, so they are uploaded again, and no stale object is ever
437+
# skipped.
438+
#
439+
# The script is written to a file rather than fed through a heredoc
440+
# inside the command substitution, which keeps bash's parser out of
441+
# the backticks and quotes it contains.
442+
cat > "$RUNNER_TEMP/archive-mtimes.cjs" <<'JS'
443+
const { execFileSync } = require('node:child_process');
444+
const { utimesSync } = require('node:fs');
445+
const { join } = require('node:path');
446+
const git = (...args) =>
447+
execFileSync('git', ['-C', 'archive', ...args], { encoding: 'utf8', maxBuffer: 1 << 30 });
448+
const log = git('log', '--no-renames', '--name-only', '-z', '--format=%x01%ct', 'HEAD', '--', 'board');
449+
const seen = new Set();
450+
let time = NaN;
451+
let stamped = 0;
452+
for (const token of log.split('\0')) {
453+
if (token.startsWith('\x01')) { time = Number(token.slice(1)); continue; }
454+
const path = token.replace(/^\n/, '');
455+
if (!path || seen.has(path)) continue;
456+
seen.add(path);
457+
if (!Number.isInteger(time)) throw new Error(`no commit time read before ${path}`);
458+
try {
459+
utimesSync(join('archive', path), time, time);
460+
stamped += 1;
461+
} catch (e) {
462+
if (e.code !== 'ENOENT') throw e;
463+
}
464+
}
465+
const tracked = git('ls-files', '-z', '--', 'board').split('\0').filter(Boolean).length;
466+
if (stamped !== tracked) throw new Error(`stamped ${stamped} file(s) but the archive tracks ${tracked}`);
467+
const commits = git('rev-list', '--count', 'HEAD').trim();
468+
console.log(`${stamped} file(s) stamped from ${commits} commit(s)`);
469+
JS
470+
if mtimes=$(node "$RUNNER_TEMP/archive-mtimes.cjs" 2>&1); then
471+
# The last line only: the status line below goes into
472+
# $GITHUB_OUTPUT, where a second line would break its format.
473+
mtime_note="mtimes from history: ${mtimes##*$'\n'}"
474+
echo "${mtime_note}"
475+
else
476+
printf '%s\n' "$mtimes" >&2
477+
echo "::warning::Could not set the archive's mtimes from its git history (output above), so this sync re-uploads every object. The copy stays complete, but every file costs a Class A operation."
478+
mtime_note="FULL re-upload, mtimes not restored"
479+
fi
480+
381481
# No pipe between the command and `$?` — see the snapshot step.
382482
set +e
383483
aws s3 sync archive/board/ "$mirror" \
@@ -392,9 +492,11 @@ jobs:
392492
echo "::error::aws s3 sync to R2 exited $code. This run IS archived on board-archive; the out-of-GitHub copy is now behind and every later run will stay behind until this is fixed."
393493
exit 1
394494
fi
395-
# One line per object moved. The first configured run moves the whole
396-
# board and every later run moves a handful, so the log is truncated
397-
# here and the counts below are taken over all of it.
495+
# One line per object moved. With the mtimes restored above, a run
496+
# moves the records that changed since the last upload; a run that
497+
# fell back (or the first one configured) moves the whole board, so
498+
# the log is truncated here and the counts below are taken over all
499+
# of it.
398500
transfers=$(wc -l < "$RUNNER_TEMP/r2-sync.log")
399501
head -n 50 "$RUNNER_TEMP/r2-sync.log"
400502
if [ "$transfers" -gt 50 ]; then
@@ -414,8 +516,8 @@ jobs:
414516
tarred="written for ${day}"
415517
fi
416518
417-
echo "status=synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}" >> "$GITHUB_OUTPUT"
418-
echo "R2: synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}"
519+
echo "status=synced ${uploaded} object(s), deleted ${deleted} (${mtime_note}); daily tarball ${tarred}" >> "$GITHUB_OUTPUT"
520+
echo "R2: synced ${uploaded} object(s), deleted ${deleted} (${mtime_note}); daily tarball ${tarred}"
419521
420522
- name: Publish the run to the summary
421523
if: always()

0 commit comments

Comments
 (0)