Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -473,6 +473,8 @@ jobs:
options: --user root
env:
BUILD_TARGET: linux64_multiprocess
# One full lint pass is about 237 MiB; retain it plus room for changed inputs.
CTCACHE_MAXSIZE_MB: 512
steps:
- name: Checkout code
uses: actions/checkout@v6
Expand Down Expand Up @@ -526,16 +528,19 @@ jobs:
test -s "build-ci/dashcore-${BUILD_TARGET}/compile_commands.json"
shell: bash

# ctcache hashes preprocessed headers, comments, flags and clang-tidy config.
# Only toolchain/plugin changes need a new namespace, not all depends recipes.
- name: Restore ctcache cache
uses: actions/cache/restore@v5
with:
path: /cache/ctcache
key: ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/devtools/bitcoin-tidy/*', 'depends/packages/*') }}-linux64_multiprocess-${{ github.sha }}
key: ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/containers/ci/ci-slim.Dockerfile', 'contrib/devtools/bitcoin-tidy/**') }}-linux64_multiprocess-${{ github.sha }}
restore-keys: |
ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/devtools/bitcoin-tidy/*', 'depends/packages/*') }}-linux64_multiprocess-
ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/containers/ci/ci-slim.Dockerfile', 'contrib/devtools/bitcoin-tidy/**') }}-linux64_multiprocess-

- name: Run linters
run: |
python3 -m unittest discover -s ci/dash -p test_cleanup_ctcache.py
CACHE_DIR="/cache"
source ./ci/dash/matrix.sh
./ci/dash/lint-tidy.sh
Expand All @@ -548,7 +553,7 @@ jobs:
uses: actions/cache/save@v5
with:
path: /cache/ctcache
key: ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/devtools/bitcoin-tidy/*', 'depends/packages/*') }}-linux64_multiprocess-${{ github.sha }}
key: ctcache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'contrib/containers/ci/ci-slim.Dockerfile', 'contrib/devtools/bitcoin-tidy/**') }}-linux64_multiprocess-${{ github.sha }}

test-linux64:
name: linux64-test
Expand Down
34 changes: 34 additions & 0 deletions ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,40 @@ In order to avoid rebuilding all dependencies for each build, the binaries are
cached and reused when possible. Changes in the dependency-generator will
trigger cache-invalidation and rebuilds as necessary.

### clang-tidy result cache

`linux64_multiprocess-lint` restores only `/cache/ctcache`, separately from
compiler ccache, cppcheck and depends. Its `CTCACHE_MAXSIZE_MB` is 512 MiB:
a full lint pass currently occupies about 237 MiB, so this retains a full
pass with room for changed inputs. Other jobs' cache limits and the local
fallback in `00_setup_env.sh` are unchanged. `cleanup-ctcache.py` removes
oldest files until allocated disk usage is within the limit, rather than
removing a percentage of files with very different sizes. ctcache refreshes
an output's modification time on a hit.

The archive key includes both CI Dockerfiles (including the inherited LLVM
version) and the bitcoin-tidy plugin sources. Within that namespace, restore
falls back to the latest commit's archive. Do not add all depends recipes to
the namespace: ctcache already hashes preprocessed source and headers,
comments (for NOLINT), compiler/tidy arguments and the effective tidy config.
A recipe change should invalidate affected translation units, not every
cached result. Toolchain/plugin changes still require a cold cache.

Only successful default-branch push jobs save archives; pull requests only
restore them. The existing `prune-ccache` job keeps the newest default-branch
ctcache generation per target. Overlapping runs can restore before a previous
run saves, so cold starts are still possible. Separate keys do not reserve
separate GitHub cache quotas: all jobs share the repository's storage limit
(10 GB by default), with least-recently-accessed eviction and removal of
entries unused for seven days. Account for overlapping generations and other
jobs before increasing this budget further.

Run the cleanup regression tests without a compiler or build artifacts:

```sh
python3 -m unittest discover -s ci/dash -p test_cleanup_ctcache.py -v
```

### amd64 runner tiers

`.github/workflows/select_dynamic_runner.py` places the amd64 jobs in two
Expand Down
58 changes: 58 additions & 0 deletions ci/dash/cleanup-ctcache.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
#!/usr/bin/env python3
# Copyright (c) 2026 The Dash Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.

"""Trim the local clang-tidy cache to an allocated-size limit, oldest files first."""

import argparse
from pathlib import Path
import subprocess


def cache_size(cache_dir: Path) -> int:
return int(subprocess.check_output(["du", "-sk", str(cache_dir)], text=True, encoding="utf8").split()[0]) * 1024


def cleanup(cache_dir: Path, max_size_mb: int) -> None:
limit = max_size_mb * 1024 * 1024
size = cache_size(cache_dir)
if size <= limit:
return

print(f"Cache size {size / 1024 / 1024:.1f} MiB exceeds limit {max_size_mb} MiB, cleaning old entries...")
files = sorted((path.stat().st_mtime_ns, path) for path in cache_dir.rglob("*") if path.is_file())
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
removed = 0
for _, path in files:
path.unlink()
removed += 1
parent = path.parent
while parent != cache_dir:
try:
parent.rmdir()
except OSError:
break
parent = parent.parent
# Outputs vary greatly in size, and stats files count too. Check bytes,
# not a fixed percentage of files, including the remaining directories.
size = cache_size(cache_dir)
if size <= limit:
break
Comment on lines +36 to +40

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💬 Nitpick: Avoid a full cache traversal after every deleted file

Each iteration invokes cache_size(), which starts a new du process and traverses the remaining cache tree. Deleting k files from an n-file cache therefore performs O(k*n) metadata work, becoming quadratic when most entries must be removed. The early return avoids this cost below budget, and remeasurement correctly includes directory allocation, so this is not a correctness problem. Track allocated blocks for files and removed directories, then verify with du near the limit, rather than rescanning after every unlink; retain the final measured-budget check.

source: glm-5.3-flash (phase1-reviewer: general)


print(f"Removed {removed} oldest cache files; cache size after cleanup: {size / 1024 / 1024:.1f} MiB")
if size > limit:
raise RuntimeError("Could not trim ctcache to its size limit")


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("cache_dir", type=Path)
parser.add_argument("max_size_mb", type=int)
args = parser.parse_args()
if args.max_size_mb <= 0:
parser.error("max_size_mb must be positive")
cleanup(args.cache_dir.resolve(), args.max_size_mb)


if __name__ == "__main__":
main()
25 changes: 1 addition & 24 deletions ci/dash/lint-tidy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -11,29 +11,6 @@ set -eo pipefail
# only on nor do they set the requisite build parameters. Make sure you do
# that *before* running this script.

# Cleanup old ctcache entries to keep cache size under limit
cleanup_ctcache() {
local cache_dir=$1
local max_size_mb=$2
local cache_size_mb

cache_size_mb=$(du -sm "${cache_dir}" 2>/dev/null | cut -f1)
if [ "${cache_size_mb}" -gt "${max_size_mb}" ]; then
echo "Cache size ${cache_size_mb}MB exceeds limit ${max_size_mb}MB, cleaning old entries..."
# Remove files older than 7 days, or if still too large, oldest 20% of files
find "${cache_dir}" -type f -mtime +7 -delete 2>/dev/null || true
cache_size_mb=$(du -sm "${cache_dir}" 2>/dev/null | cut -f1)
if [ "${cache_size_mb}" -gt "${max_size_mb}" ]; then
local file_count remove_count
file_count=$(find "${cache_dir}" -type f | wc -l)
remove_count=$((file_count / 5))
find "${cache_dir}" -type f -printf '%T+ %p\0' | sort -z | head -z -n "${remove_count}" | cut -z -d' ' -f2- | xargs -0 rm -f 2>/dev/null || true
echo "Attempted to remove ${remove_count} oldest cache entries"
fi
echo "Cache size after cleanup: $(du -sh "${cache_dir}" 2>/dev/null | cut -f1)"
fi
}

# Verify clang-tidy is available
CLANG_TIDY_BIN=$(command -v clang-tidy)
if [ -z "${CLANG_TIDY_BIN}" ]; then
Expand Down Expand Up @@ -107,7 +84,7 @@ du -sh "${CTCACHE_DIR}" 2>/dev/null || echo "Cache directory not found"
python3 "${CLANG_TIDY_CACHE_PY}" --show-stats 2>&1 || true

# Limit cache size (ctcache has no built-in size management)
cleanup_ctcache "${CTCACHE_DIR}" "${CTCACHE_MAXSIZE_MB}"
python3 "${BASE_ROOT_DIR}/ci/dash/cleanup-ctcache.py" "${CTCACHE_DIR}" "${CTCACHE_MAXSIZE_MB}"
echo "=========================="

cd "${BASE_ROOT_DIR}/build-ci/dashcore-${BUILD_TARGET}"
Expand Down
77 changes: 77 additions & 0 deletions ci/dash/test_cleanup_ctcache.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
#!/usr/bin/env python3
# Copyright (c) 2026 The Dash Core developers
# Distributed under the MIT software license, see the accompanying
# file COPYING or http://www.opensource.org/licenses/mit-license.php.

"""Regression tests for the lint cache's disk budget (no clang-tidy required)."""

import os
from pathlib import Path
import subprocess
import sys
import tempfile
import time
import unittest


SCRIPT = Path(__file__).with_name("cleanup-ctcache.py")
MIB = 1024 * 1024


class CleanupCTCacheTest(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.cache = Path(self.temp.name)

def write_entry(self, name, size, timestamp):
path = self.cache / name
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(b"x" * size)
os.utime(path, (timestamp, timestamp))
return path

def size_kib(self):
return int(subprocess.check_output(["du", "-sk", str(self.cache)], text=True, encoding="utf8").split()[0])

def trim(self, limit):
return subprocess.run(
[sys.executable, str(SCRIPT), str(self.cache), str(limit)],
check=True, capture_output=True, text=True, encoding="utf8",
).stdout

def test_under_budget_keeps_even_old_entries(self):
old = self.write_entry("ab/old", MIB // 2, 1)
before = self.size_kib()
self.trim(1)
self.assertTrue(old.exists())
self.assertEqual(before, self.size_kib())

def test_unequal_outputs_and_stats_trim_to_budget_oldest_first(self):
# Removing 20% of these files only removes tiny stats, not enough bytes.
now = int(time.time())
stats = [self.write_entry(f"stats/{i:02x}", 4, now - 200 + i) for i in range(10)]
large = self.write_entry("ab/old large output", 2 * MIB, now - 100)
hot = self.write_entry("cd/recent output", MIB // 2, now)
self.assertGreater(self.size_kib(), 1024)
output = self.trim(1)
self.assertLessEqual(self.size_kib(), 1024)
self.assertTrue(hot.exists())
self.assertFalse(large.exists())
self.assertTrue(all(not path.exists() for path in stats))
self.assertFalse((self.cache / "ab").exists())
self.assertIn("Removed 11 oldest cache files", output)
self.assertEqual(self.trim(1), "")

def test_empty_cache_and_invalid_limit(self):
self.trim(1)
result = subprocess.run(
[sys.executable, str(SCRIPT), str(self.cache), "0"],
capture_output=True, text=True, encoding="utf8",
)
self.assertNotEqual(result.returncode, 0)
self.assertIn("must be positive", result.stderr)


if __name__ == "__main__":
unittest.main()
Loading