You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 59f7aa8
Browse filesBrowse the repository at this point in the historyBrowse files
ci(board-snapshot): upload only the records that changed to R2, not all ~41k objects every run
`aws s3 sync` uploads when the size differs or the local mtime is newer
than the object's LastModified, and a fresh checkout gives every file a
new mtime, so every run re-PUT the whole archive: 1.2M PutObject in 11
days on the bucket (~$10/month of Class A operations against a 1M/month
free tier).
The archive branch is now fetched with its whole history but no blobs
(`--filter=blob:none`, ~7 MB of commits and trees today), and before the
sync each file's mtime is set to the committer time of the last commit
that changed it. Every upload happens after the commit whose content it
carries, so a current object is newer than its file and is skipped,
while a file changed after its object's last upload (this run's edits,
or an earlier run's failed upload) is newer and is sent. `--delete`, the
target prefix, the daily tarball, the schedule and the fail-red/notice
behaviour are unchanged. A truncated history only over-uploads; an
unreadable one falls back to the old full re-upload with a warning.
Verified by running the checkout, commit/push and R2 steps verbatim
against a throwaway git server and a fake aws applying aws-cli's
size+mtime rule: incremental uploads, deletions, a same-size edit,
catch-up after a failed upload, a no-change run and a depth=1 history
all leave the mirror byte-identical to the archive. With the mtime
restore ablated, the same harness sees every run re-upload all objects.
Co-authored-by: Claude <noreply@anthropic.com>
console.log(`${stamped} file(s) stamped from ${commits} commit(s)`);
469
+
JS
470
+
if mtimes=$(node "$RUNNER_TEMP/archive-mtimes.cjs" 2>&1); then
471
+
# The last line only: the status line below goes into
472
+
# $GITHUB_OUTPUT, where a second line would break its format.
473
+
mtime_note="mtimes from history: ${mtimes##*$'\n'}"
474
+
echo "${mtime_note}"
475
+
else
476
+
printf '%s\n' "$mtimes" >&2
477
+
echo "::warning::Could not set the archive's mtimes from its git history (output above), so this sync re-uploads every object. The copy stays complete, but every file costs a Class A operation."
478
+
mtime_note="FULL re-upload, mtimes not restored"
479
+
fi
480
+
381
481
# No pipe between the command and `$?` — see the snapshot step.
382
482
set +e
383
483
aws s3 sync archive/board/ "$mirror" \
@@ -392,9 +492,11 @@ jobs:
392
492
echo "::error::aws s3 sync to R2 exited $code. This run IS archived on board-archive; the out-of-GitHub copy is now behind and every later run will stay behind until this is fixed."
393
493
exit 1
394
494
fi
395
-
# One line per object moved. The first configured run moves the whole
396
-
# board and every later run moves a handful, so the log is truncated
397
-
# here and the counts below are taken over all of it.
495
+
# One line per object moved. With the mtimes restored above, a run
496
+
# moves the records that changed since the last upload; a run that
497
+
# fell back (or the first one configured) moves the whole board, so
498
+
# the log is truncated here and the counts below are taken over all
0 commit comments