diff --git a/docs/WEBSITE_IMPLEMENTATION_LOG_HU.md b/docs/WEBSITE_IMPLEMENTATION_LOG_HU.md index 09dc1fa..b322ecd 100644 --- a/docs/WEBSITE_IMPLEMENTATION_LOG_HU.md +++ b/docs/WEBSITE_IMPLEMENTATION_LOG_HU.md @@ -3926,6 +3926,92 @@ A végső review (2026-09-13) V1–V7 javításai (J4–J6) szintén **csak kód A blokkot, kihagyott tesztet és fennmaradó sérülékenységet ne töröld ki egy későbbi bejegyzéssel: lezáráskor hivatkozz a bizonyítékra, hogy az előzmény követhető maradjon. +## Deploy 2026-09-14 (#182, #183, #184, #185) – a C6a első rolljával, `65dcadf` → `0a27df0` + +```text +Dátum / végrehajtó: 2026-09-14 15:44:21Z / Opus 5-munkamenet (a C6a rollt vivő ügynök) +Engedély: a C6a roll-ablak tulajdonosi jóváhagyása („roll mehet a flottára full”); az explorer deploy a roll részeként, + legkésőbb a 13535-ös tip előtt +Hatókör: `deftrack-devnet` újraindítása a mergelt `0a27df0`-n (#182 formationEndHeight/isSchedulable, #183 RPC-pool + teszt, #184 docs, #185 maint-window docs); a nodeokhoz és a mongodhoz nem nyúlt +``` + +**Mérve, `journalctl -u deftrack-devnet` (UTC, a szerver CEST-ben naplóz):** `SIGTERM received` 15:44:21.336Z, tip a +leállításkor 13370; az új processz 15:44:22.091Z-kor csatlakozott a Mongóhoz, a gyűjtők azonnal elindultak. Readiness +közvetlenül a restart után: `status ok`, `chainTip 13370`, `behind 0`, `rounds` formed 1162 / failed 21 / pending 98 / +impossible 35. A kiszolgált bundle `assets/index-CNPCV91P.js` — a kliens nem változott, csak a szerverkód. + +```text +Éles módosítás: MEGTÖRTÉNT (szerverkód; a kliens-bundle változatlan) +``` + +## Deploy 2026-09-14 (#186) – a health-endpoint 5 perces blokk-köz utáni téves 503-a, `0a27df0` → `75ec3a3` + +```text +Dátum / végrehajtó: 2026-09-14 17:50:05Z / Opus 5-munkamenet +Engedély: a C6a roll-ablak tulajdonosi jóváhagyása alatt, önálló javításként a #181 mérésnél talált, nem a #181 + okozta hiba nyomán (ld. a Deploy 2026-09-14 (#181) szakasz „Mellékesen talált” pontját) +Hatókör: `server/src/domain/readiness.ts`, `server/src/index.ts` — a tétlen tick mostantól a `heartbeatAt`-et is + frissíti, a readiness abból számol; 3 fájl, 375+/12- sor a #182 rollal együtt beérkezett ágból +``` + +**Mérve:** `SIGTERM received` 17:50:05.402Z, tip a leállításkor 13419; readiness a restart után `status ok`, +`behind 0`, `rounds` formed 1168 / failed 22 / pending 97 / impossible 35. A korábbi téves 503-at az +`evaluateReadiness` `lastSyncedAt`-alapú tétlenség-számítása okozta (ld. #181 bejegyzés); külön mérés a javításra +ez a PR-hez nem tartozik, a mechanizmus ugyanaz, mint amit a #181 bejegyzés dokumentált. + +```text +Éles módosítás: MEGTÖRTÉNT (szerverkód; a kliens-bundle változatlan) +``` + +## Deploy 2026-09-14 (#188) – devnet formációs vég 13488, a flotta-roll alatt, `75ec3a3` → `9751a65` + +```text +Dátum / végrehajtó: 2026-09-14 19:10:57Z / Opus 5-munkamenet +Engedély: a C6a2 futtatás tulajdonosi időnyerési döntése — a #188 review nélkül megy ki, mert a Core #244 konszenzusa + a 13488-as blokkban aktiválódik; független review utólag, retrospektíven APPROVED +Hatókör: `server/src/config/llmq.ts` (a `llmq_50_60`/`llmq_60_75` `formationEndHeight` 13536 → 13488, a Core #244 + aktiválási magasságával egyezően), `server/src/services/sync.activity.test.ts`; a nodeokhoz nem nyúlt +``` + +**Mérve:** `SIGTERM received` 19:10:57.884Z, tip a leállításkor 13450 — a flotta-roll ablakán belül (19:09:40– +19:27:16Z, 16/16 host); readiness a restart után `status ok`, `behind 0`, `rounds` formed 1170 / failed 22 / +pending 98 / impossible 35. A #187 (`test/sync-overlap-activity`) csak teszt volt, önálló deploy nem tartozott hozzá +— a hozzá tartozó tesztfájl-változás ebbe a rollba és a #186-éba lett felosztva. + +```text +Éles módosítás: MEGTÖRTÉNT (szerverkód; a kliens-bundle változatlan) +``` + +## C6a lekapcsolás és a két futás lezárása – 2026-09-14/15 + +Két Experiments-futás vitte a C6a-t: `fleet-rollout-c6a-243-2026-09-14` (zárva 13439-en, a karbantartás miatt a +feladás előtt lement flotta-rollal indult) és `fleet-rollout-c6a-end13488-2026-09-14` (zárva **2026-09-15 08:57Z**, +`endHeight` **13781**). A befagyasztott outcome a 13439–13781-es ablakban: **19 formed / 3 failed / 1 pending, 0 +büntetett**, ChainLock 343/343, dsl 12/15 (absent: 559, 560, 566); csak a mainneten is futó profilok 18 formed / 1 +failed / 1 pending, 0 büntetett. Flotta 160/160 egy láncon 13781-en, mind `14583e7c`; seed `a7822e1b` és devnet2 +`14583e7c` azonos tip-hash-sel. InstantSend a záráskor 3/3 lockolva (1400/1991/3210 ms), seeden és devnet2-n +`instantlock_internal=true` a megerősítés előtt. + +**Független review (2026-09-15), F1–F5, egy mondatban:** +- **F1** — a roll a Q60-bázis +10-én, a staker-hostokkal kezdődött; a bázis 13440 végleges commitmentje csak + memóriában élt, és elveszett a restarttal, tehát az a Q60-kör elveszett (mainnet-releváns). +- **F2** — ugyanez a mechanizmus a `llmq_50_60` 13440-es körét is elvitte, deklaráció nélkül. +- **F3** — két Sentinel-epocha (559, 566) magyarázat nélkül maradt ki, egyik sem esik egybe ismert beavatkozással. +- **F4** — a seed/devnet2 a flottánál korábban állt át, ami 18:47–19:27Z között érvénytelenítette az átadó + visszaállítási ágát — a rollt ez nem érintette. +- **F5** (látens) — egy `dkgInterval-1`-nél záruló bányászati ablakú profil utolsó commitmentje olyan blokkban + születhet, amit a `quorum listextended` egyetlen magasságon sem mutat meg; a jelenlegi két profilt nem érinti. + +Részletek: a review — `D:\www\devnet .deftrack-review-artefacts\2026-09-15-c6a2-review\C6A2_REVIEW_2026-09-15_HU.md` +—, a zárás — `D:\www\devnet .deftrack-ops-handover\2026-09-15-c6a2-close\` (`close2-notes.txt`, +`experiment-after-close.json`, `instantsend-close.txt`, `s-close.log`, `SHA256SUMS`). + +```text +Éles módosítás: NEM TÖRTÉNT ebben a bejegyzésben (a #244/#188 rollja a fenti három Deploy-szakaszban; + ez a szakasz csak a lezárás és a review dokumentálása) +``` + --- ## Backlog — ami nem ennek a tervnek a része diff --git a/ops/c6a-fleet-roll-remote.sh b/ops/c6a-fleet-roll-remote.sh new file mode 100644 index 0000000..eaa37d3 --- /dev/null +++ b/ops/c6a-fleet-roll-remote.sh @@ -0,0 +1,76 @@ +#!/bin/sh +# +# Runs ON ONE FLEET HOST, copied there by c6a-fleet-roll.sh. +# +# usage: c6a-fleet-roll-remote.sh preflight|apply +# +# The C6a roll changes the binary and nothing else: no conf key. The #231 key +# (dslcommitmentv2height=12744) must STAY in every conf -- a daemon started +# without it would demand format v2 from the first commitment and reject +# history -- so the preflight refuses a host where any conf lacks it. +# +# preflight changes nothing. apply takes the host whole: stop every instance, +# back up, install, start every instance, then prove three things per daemon: +# running -- md5 of /proc//exe is the new binary (not -version) +# keyed -- the #231 key is still in the conf +# reread -- the NEW process logged "Setting dslcommitmentv2height to 12744" +# with a timestamp after this host's stop. The old line is still +# in the log, so its mere presence proves nothing. +set -u +SRC=${1:?staged binary} +NEW=${2:?new md5} +OLD=${3:?old md5} +MODE=${4:?preflight|apply} +KEYLINE='dslcommitmentv2height=12744' +LOGLINE='Setting dslcommitmentv2height to 12744' + +S="" +[ "$(id -u)" != 0 ] && S="sudo -n" +B=/opt/defcon-devnet/bin +DIRS=$($S sh -c 'ls -d /opt/defcon-devnet/mn* 2>/dev/null') +[ -n "$DIRS" ] || { echo "PRE dirs=0"; exit 0; } + +total=0; keyed=0 +for d in $DIRS; do + total=$((total + 1)) + $S grep -qx "$KEYLINE" "$d/defcon.conf" && keyed=$((keyed + 1)) +done +disk=$($S md5sum $B/defcond 2>/dev/null | cut -d' ' -f1) +staged=$(md5sum "$SRC" 2>/dev/null | cut -d' ' -f1) +echo "PRE daemons=$total keyed=$keyed diskmd5=$disk staged=$staged" +[ "$MODE" = preflight ] && exit 0 + +if [ "$keyed" != "$total" ] || [ "$disk" != "$OLD" ] || [ "$staged" != "$NEW" ]; then + echo "APPLY refused: pre-flight is not clean on this host" + exit 2 +fi + +# shellcheck disable=SC2001 # DIRS is a multi-line list +IDX=$(echo "$DIRS" | sed 's#.*/mn##' | sort -n) +STAMP=$(date +%Y%m%d-%H%M) +T0=$(date -u +%s) + +for i in $IDX; do $S systemctl stop "defcon-devnet-mn@$i"; done +$S cp -a $B/defcond "$B/defcond.bak-$STAMP" || { echo "APPLY backup failed"; exit 3; } +$S install -m 0755 "$SRC" $B/defcond || { echo "APPLY install failed"; exit 3; } +for i in $IDX; do $S systemctl start "defcon-devnet-mn@$i"; done +sleep 12 + +up=0; ran=0; keyed=0; reread=0 +for d in $DIRS; do + i=${d##*/mn} + [ "$($S systemctl is-active "defcon-devnet-mn@$i")" = active ] && up=$((up + 1)) + pid=$($S systemctl show -p MainPID --value "defcon-devnet-mn@$i") + [ "$($S md5sum "/proc/$pid/exe" 2>/dev/null | cut -d' ' -f1)" = "$NEW" ] && ran=$((ran + 1)) + $S grep -qx "$KEYLINE" "$d/defcon.conf" && keyed=$((keyed + 1)) + ts=$($S grep "$LOGLINE" "$d/devnet-defcon-q60/debug.log" 2>/dev/null | tail -1 | cut -d' ' -f1) + if [ -n "$ts" ] && [ "$(date -u -d "$ts" +%s 2>/dev/null || echo 0)" -ge "$T0" ]; then + reread=$((reread + 1)) + else + echo " NOREREAD mn$i last=$ts" + fi +done +$S rm -f "$SRC" +echo "POST up=$up/$total running-new=$ran/$total keyed=$keyed/$total reread=$reread/$total diskmd5=$($S md5sum $B/defcond | cut -d' ' -f1)" +[ "$up" = "$total" ] && [ "$ran" = "$total" ] && [ "$keyed" = "$total" ] && [ "$reread" = "$total" ] || exit 4 +exit 0 diff --git a/ops/c6a-fleet-roll.sh b/ops/c6a-fleet-roll.sh new file mode 100644 index 0000000..fb28dd2 --- /dev/null +++ b/ops/c6a-fleet-roll.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# +# The C6a roll (#241-#243, merged v22.1.x ddf2822011) on the 16 fleet hosts. +# Runs ON THE JUMP HOST, after c6a-fleet-stage.sh has put the binary on every +# host. The binary moves and nothing else: no conf key. #243 activates by itself +# at height 13536, so every daemon must be on this binary before that height. +# +# Host by host: preflight (the deployed binary is the expected one, the #231 key +# is in every conf, the staged copy is intact), then apply +# (c6a-fleet-roll-remote.sh). The roll STOPS at the first host that does not come +# back whole -- a half-understood failure repeated on fifteen more hosts is the +# worst outcome available. +# +# ./c6a-fleet-roll.sh --check # preflight every host, change nothing +# ./c6a-fleet-roll.sh # preflight, then roll +set -uo pipefail + +NEW_MD5=7bf411b0eb08731b6e050868b1d2ff72 +OLD_MD5=c9898910b40a57d253dae73905e14845 +STAGED=/tmp/defcond-c6a +KEY=/root/.ssh/defcon_nodes +INVENTORY=${FLEET_INVENTORY:-/root/fleet-nodes-all.txt} +HERE=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +REMOTE=$HERE/c6a-fleet-roll-remote.sh +[ -f "$REMOTE" ] || { echo "missing helper: $REMOTE"; exit 1; } + +CHECK_ONLY=0 +[ "${1:-}" = "--check" ] && CHECK_ONLY=1 + +SSH="ssh -n -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" +SCP="scp -q -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" + +echo "==> $(date -u +%FT%TZ) new $NEW_MD5, replacing $OLD_MD5, inventory $INVENTORY, check-only=$CHECK_ONLY" +ok=0; failed=0 +while read -r entry; do + case "$entry" in ""|\#*) continue ;; esac + case "$entry" in *@*) target="$entry" ;; *) target="root@$entry" ;; esac + label=$(echo "$entry" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1.x/') + echo "=== $label $(date -u +%T)" + $SCP "$REMOTE" "$target:/tmp/c6a-roll.sh" || { echo " FAIL: helper copy"; failed=$((failed+1)); break; } + pre=$($SSH "$target" "sh /tmp/c6a-roll.sh $STAGED $NEW_MD5 $OLD_MD5 preflight" 2>&1) + echo " $pre" + case "$pre" in + *"diskmd5=$OLD_MD5 staged=$NEW_MD5"*) ;; + *) echo " FAIL: preflight not clean"; failed=$((failed+1)); break ;; + esac + if [ "$CHECK_ONLY" = 1 ]; then ok=$((ok+1)); continue; fi + + out=$($SSH "$target" "sh /tmp/c6a-roll.sh $STAGED $NEW_MD5 $OLD_MD5 apply; echo rc=\$?; rm -f /tmp/c6a-roll.sh" 2>&1) + echo "$out" | awk '{print " " $0}' + if echo "$out" | grep -q '^rc=0$' && echo "$out" | grep -q "diskmd5=$NEW_MD5"; then + ok=$((ok+1)) + else + echo " FAIL: this host did not come back whole -- stopping the roll here" + failed=$((failed+1)); break + fi +done < "$INVENTORY" + +echo +echo "==> $(date -u +%FT%TZ) hosts ok=$ok failed=$failed" +[ "$failed" = 0 ] || exit 1 diff --git a/ops/c6a-fleet-stage.sh b/ops/c6a-fleet-stage.sh new file mode 100644 index 0000000..9b2332e --- /dev/null +++ b/ops/c6a-fleet-stage.sh @@ -0,0 +1,54 @@ +#!/bin/bash +# +# The C6a roll (#241-#243, merged v22.1.x ddf2822011): put the fleet artefact on +# every fleet host AHEAD of the roll window, and prove on each host that it is +# the right file and that it can run there. Runs ON THE JUMP HOST. +# +# It writes one file to /tmp on each host and nothing else: no daemon is +# stopped, no conf is touched, nothing is installed. On each host it checks the +# md5 of the copy AS THAT HOST SEES IT (scp exiting 0 proves nothing), runs ldd +# on it (a build that links a library the target lacks has already been +# installed once on this project and would not start), and asks the binary for +# its version. +# +# usage: ./c6a-fleet-stage.sh +set -uo pipefail + +EXPECT_MD5=7bf411b0eb08731b6e050868b1d2ff72 +DEST=/tmp/defcond-c6a +KEY=/root/.ssh/defcon_nodes +INVENTORY=${FLEET_INVENTORY:-/root/fleet-nodes-all.txt} + +DAEMON=${1:?usage: c6a-fleet-stage.sh } +[ -f "$DAEMON" ] || { echo "no such file: $DAEMON"; exit 1; } +LOCAL_MD5=$(md5sum "$DAEMON" | cut -d' ' -f1) +echo "==> staging md5 $LOCAL_MD5 to $DEST on every host in $INVENTORY" +[ "$LOCAL_MD5" = "$EXPECT_MD5" ] || { echo " REFUSING: expected $EXPECT_MD5"; exit 1; } + +SSH="ssh -n -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" +SCP="scp -q -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" + +ok=0; failed=0 +while read -r entry; do + case "$entry" in ""|\#*) continue ;; esac + case "$entry" in *@*) target="$entry" ;; *) target="root@$entry" ;; esac + label=$(echo "$entry" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1.x/') + printf '%-30s ' "$label" + + have=$($SSH "$target" "md5sum $DEST 2>/dev/null | cut -d' ' -f1" 2>/dev/null) + if [ "$have" != "$LOCAL_MD5" ]; then + # a single transport failure to one of the production hosts has been transient before + $SCP "$DAEMON" "$target:$DEST" || $SCP "$DAEMON" "$target:$DEST" || { echo "COPY FAILED"; failed=$((failed+1)); continue; } + fi + out=$($SSH "$target" "chmod 0755 $DEST; echo md5=\$(md5sum $DEST | cut -d' ' -f1) missing=\$(ldd $DEST 2>/dev/null | grep -c 'not found') version=\$($DEST -version 2>/dev/null | head -1 | tr ' ' '_')" 2>/dev/null) + echo "$out" + case "$out" in + *"md5=$LOCAL_MD5 missing=0 version=DeFCoN"*) ok=$((ok+1)) ;; + *) failed=$((failed+1)) ;; + esac +done < "$INVENTORY" + +echo +echo "==> staged-and-runnable=$ok failed=$failed" +echo " Nothing was installed and no daemon was touched." +[ "$failed" = 0 ] || exit 1 diff --git a/ops/c6a-roll-at-height.sh b/ops/c6a-roll-at-height.sh new file mode 100644 index 0000000..0f44c16 --- /dev/null +++ b/ops/c6a-roll-at-height.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Start the C6a fleet roll at exactly START (a Q60 base + 10, the base not a +# multiple of 72). Runs ON THE JUMP HOST under nohup, from /root/roll-c6a. +# Height is read from two fleet hosts' mn1 (the higher answer wins). If the first +# height seen at or above START is already past LATEST, it does NOT roll: the +# window +10..+19 would not hold a 16-host roll. It also refuses to start at or +# past HARD_STOP: every daemon must be on the new binary before the end height +# 13536, and a roll begun too close to it is a fork, not a roll. +# +# usage: nohup ./c6a-roll-at-height.sh > at-height.log 2>&1 & +START=${1:?start height} +LATEST=${2:?latest acceptable first-seen height} +HARD_STOP=13488 +[ "$LATEST" -lt "$HARD_STOP" ] || { echo "REFUSING: latest $LATEST is not below the hard stop $HARD_STOP"; exit 4; } +DEADLINE=$(( $(date +%s) + 4*3600 )) +K=/root/.ssh/defcon_nodes +INV=/root/fleet-nodes-all.txt +SSH="ssh -n -i $K -o ConnectTimeout=10 -o BatchMode=yes -o StrictHostKeyChecking=no" +cd /root/roll-c6a || exit 1 +height_of() { # $1 = inventory entry + local t=$1 u=root S=""; case "$t" in *@*) u=${t%%@*};; *) t="root@$t";; esac + [ "$u" != root ] && S="sudo -n" + $SSH "$t" "$S /opt/defcon-devnet/bin/defcon-cli -datadir=/opt/defcon-devnet/mn1 -conf=/opt/defcon-devnet/mn1/defcon.conf getblockcount" 2>/dev/null +} +A=$(grep -v '^[[:space:]]*\(#\|$\)' "$INV" | sed -n 2p) +B=$(grep -v '^[[:space:]]*\(#\|$\)' "$INV" | sed -n 3p) +echo "armed $(date -u +%FT%TZ) start=$START latest=$LATEST hard-stop=$HARD_STOP" +last="" +while [ "$(date +%s)" -lt "$DEADLINE" ]; do + ha=$(height_of "$A"); hb=$(height_of "$B") + h=0 + [[ "$ha" =~ ^[0-9]+$ ]] && h=$ha + [[ "$hb" =~ ^[0-9]+$ ]] && [ "$hb" -gt "$h" ] && h=$hb + if [ "$h" != "$last" ]; then echo "$(date -u +%T) height $h"; last=$h; fi + if [ "$h" -ge "$START" ]; then + if [ "$h" -gt "$LATEST" ]; then echo "ABORT $(date -u +%T): first seen $h > $LATEST, not rolling"; exit 2; fi + echo "ROLL START $(date -u +%FT%TZ) at height $h" + ./c6a-fleet-roll.sh > roll.log 2>&1 + rc=$? + echo "ROLL END $(date -u +%FT%TZ) rc=$rc height $(height_of "$B")" + exit $rc + fi + sleep 8 +done +echo "DEADLINE reached without height $START"; exit 3 diff --git a/ops/c6a-s-check.sh b/ops/c6a-s-check.sh new file mode 100644 index 0000000..071acbd --- /dev/null +++ b/ops/c6a-s-check.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# C6a end-height check, run on the VPS (stdin), read-only. Before 13535 it is +# the control (both retired types still listed and mining); from 13536 on it is +# the measurement for the run's expectations and the closing conditions. +# llmqType: 1 llmq_50_60, 2 llmq_400_60, 3 llmq_400_85, 4 llmq_100_67, 5 llmq_60_75, 7 llmq_defcon +S=13536 +C="sudo -n -u defcon /usr/local/bin/defcon-cli -datadir=/home/defcon/.defcon -conf=/home/defcon/.defcon/defcon.conf" +C2="sudo -n -u defcon /usr/local/bin/defcon-cli -datadir=/home/defcon/.defcon2 -conf=/home/defcon/.defcon2/defcon.conf" +tip=$($C getblockcount) +# shellcheck disable=SC2086 # $tip is always the numeric getblockcount result; no word-splitting/globbing risk +echo "UTC $(date -u '+%F %T') tip $tip (end height $S, tip - S = $((tip - S))) devnet2 tip $($C2 getblockcount) same hash: $([ "$($C getblockhash $tip)" = "$($C2 getblockhash $tip 2>/dev/null)" ] && echo yes || echo no)" +echo "quorum list types at tip: $($C quorum list | grep -oE '"llmq_[a-z0-9_]+"' | tr -d '"' | tr '\n' ' ')" +echo "best chainlock: $($C getbestchainlock | grep -oE '"height": [0-9]+|"llmqType": "[a-z0-9_]+"' | tr '\n' ' ')" + +# Commitments (type 6) by llmqType and quorum height, from S-96 to the tip. +from=$((S - 96)) +# shellcheck disable=SC2086 # $tip is always numeric (getblockcount result) +[ $tip -lt $S ] && from=$((tip - 96)); [ $from -lt 1 ] && from=1 +$C getblockhash $from > /dev/null || exit 1 +tmp=$(mktemp -d) +# shellcheck disable=SC2086 # $tip is always numeric (getblockcount result) +for h in $(seq $from $tip); do + # shellcheck disable=SC2086 # $h is always numeric (seq output) + $C getblock "$($C getblockhash $h)" 2 > "$tmp/$h.json" 2>/dev/null +done +python3 - "$tmp" "$from" "$tip" "$S" <<'PY' +import json, os, sys +d, lo, hi, S = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4]) +names = {1: 'llmq_50_60', 2: 'llmq_400_60', 3: 'llmq_400_85', 4: 'llmq_100_67', 5: 'llmq_60_75', 7: 'llmq_defcon'} +rows = [] +blocks = {} +for h in range(lo, hi + 1): + try: + blocks[h] = json.load(open(os.path.join(d, f'{h}.json'))) + except Exception: + print(f' block {h}: unreadable') +hash2height = {b['hash']: h for h, b in blocks.items()} +for h, b in sorted(blocks.items()): + for tx in b.get('tx', []): + if tx.get('type') != 6: continue + c = (tx.get('qcTx') or {}).get('commitment') or {} + t = c.get('llmqType') + base = hash2height.get(c.get('quorumHash'), 'base<' + str(lo)) + null = (c.get('signersCount', 0) == 0 and c.get('validMembersCount', 0) == 0) + rows.append((h, names.get(t, t), base, 'null' if null else f"valid {c.get('validMembersCount')}")) +retired = [r for r in rows if r[1] in ('llmq_50_60', 'llmq_60_75')] +print(f'commitments in blocks {lo}..{hi}: {len(rows)} total') +for r in rows: + if (r[1] in ('llmq_50_60', 'llmq_60_75') and r[3] != 'null') or (r[0] >= S - 30 and r[3] != 'null'): + print(f' block {r[0]}: {r[1]} base {r[2]} {r[3]}') +nulls = {} +for r in rows: + if r[3] == 'null': nulls.setdefault(r[1], []).append(r[0]) +for k, v in nulls.items(): + print(f' null commitments {k}: {len(v)} in blocks {min(v)}..{max(v)}') +after = [r for r in retired if r[0] >= S] +print(f'RETIRED-TYPE commitments in blocks >= {S}: {len(after)}' + ('' if after else ' (expected 0)')) +PY +rm -rf "$tmp" + +# PoSe and Sentinel +$C protx list registered true | python3 -c " +import json,sys +l=json.load(sys.stdin) +pen=[x for x in l if x['state'].get('PoSePenalty',0)>0]; ban=[x for x in l if x['state'].get('PoSeBanHeight',-1) not in (-1,None)] +print(f'PoSe: registered {len(l)} penalty>0 {len(pen)} banned {len(ban)}')" +curl -s 'http://localhost:4100/api/v1/dsl/epochs?limit=4' | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',d if isinstance(d,list) else []) +for e in items[:4]: print(' epoch', e.get('epoch'), 'boundary', e.get('boundaryHeight', e.get('height')), 'status', e.get('status'), 'missed', e.get('missedCount'))" 2>/dev/null +# Explorer: rounds of the retired types at or above the end +curl -s "http://localhost:4100/api/v1/quorum-rounds?llmqName=llmq_50_60&limit=5" | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',[]) +print('explorer llmq_50_60 latest rounds:', [(i.get('expectedHeight'), i.get('status')) for i in items[:5]])" 2>/dev/null +curl -s "http://localhost:4100/api/v1/quorum-rounds?llmqName=llmq_60_75&limit=5" | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',[]) +print('explorer llmq_60_75 latest rounds:', [(i.get('expectedHeight'), i.get('status')) for i in items[:5]])" 2>/dev/null +curl -s http://localhost:4100/api/v1/health | python3 -c "import sys,json; d=json.load(sys.stdin)['data']; print('health', d['status'], d['failing'], 'behind', d['behind'], d['rounds'])" diff --git a/ops/c6a-seed-roll.sh b/ops/c6a-seed-roll.sh new file mode 100644 index 0000000..ad623dd --- /dev/null +++ b/ops/c6a-seed-roll.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# +# The C6a roll (#241-#243, merged v22.1.x ddf2822011) on the explorer VPS: the +# seed and devnet2, the canary before the fleet. Runs ON THE VPS as root. +# +# #243 is a devnet consensus change at height 13536: llmq_50_60 and llmq_60_75 +# form no new quorum from there. Every devnet daemon must run this binary before +# 13536; an old one past it forks off. No conf key changes: the #231 key +# (dslcommitmentv2height=12744) stays in both confs, and this roll moves only the +# binary. +# +# devnet2 executes /usr/local/bin/defcond-nobdb -- the FLEET artefact under +# another name -- and every roll before #231 missed it. +# +# ./c6a-seed-roll.sh --check +# ./c6a-seed-roll.sh +# +# The explorer reads the seed's RPC, so its sync ticks fail for the few seconds +# this takes and recover on their own. +set -uo pipefail + +NEW_SEED=d050b97df48a74f57a09dadca5985c84 +NEW_FLEET=7bf411b0eb08731b6e050868b1d2ff72 +OLD_SEED=5c8fab67a6dcbb4c47e92e19141dcf8f +OLD_FLEET=c9898910b40a57d253dae73905e14845 +KEYLINE='dslcommitmentv2height=12744' +LOGLINE='Setting dslcommitmentv2height to 12744' + +CHECK_ONLY=0 +if [ "${1:-}" = "--check" ]; then CHECK_ONLY=1; shift; fi +SEED_BIN=${1:?usage: c6a-seed-roll.sh [--check] } +FLEET_BIN=${2:?usage: c6a-seed-roll.sh [--check] } + +# unit | installed path | datadir | source | new md5 | old md5 +NODES=( + "defcond-devnet|/usr/local/bin/defcond|/home/defcon/.defcon|$SEED_BIN|$NEW_SEED|$OLD_SEED" + "defcond-devnet2|/usr/local/bin/defcond-nobdb|/home/defcon/.defcon2|$FLEET_BIN|$NEW_FLEET|$OLD_FLEET" +) + +rc=0 +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + exec_path=$(grep -h '^ExecStart=' "/etc/systemd/system/$unit.service" | head -1 | sed 's/^ExecStart=//' | awk '{print $1}') + disk=$(md5sum "$bin" | cut -d' ' -f1); srcmd5=$(md5sum "$src" | cut -d' ' -f1) + keyed=$(grep -cx "$KEYLINE" "$dir/defcon.conf") + echo "PRE $unit executes=$exec_path disk=$disk (want old $old) source=$srcmd5 (want new $new) keyed=$keyed" + { [ "$exec_path" = "$bin" ] && [ "$disk" = "$old" ] && [ "$srcmd5" = "$new" ] && [ "$keyed" = 1 ]; } || { echo " REFUSING: preflight not clean"; rc=1; } +done +[ "$rc" = 0 ] || exit 1 +if [ "$CHECK_ONLY" = 1 ]; then echo "==> --check given, nothing changed"; exit 0; fi + +stamp=$(date +%Y%m%d-%H%M) +T0=$(date -u +%s) +for row in "${NODES[@]}"; do IFS='|' read -r unit bin dir src new old <<<"$row"; systemctl stop "$unit"; done +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + cp -a "$bin" "$bin.bak-$stamp" + install -m 0755 "$src" "$bin" +done +for row in "${NODES[@]}"; do IFS='|' read -r unit bin dir src new old <<<"$row"; systemctl start "$unit"; done +sleep 15 + +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + active=$(systemctl is-active "$unit") + running=$(md5sum "/proc/$(systemctl show -p MainPID --value "$unit")/exe" 2>/dev/null | cut -d' ' -f1) + ts=$(grep "$LOGLINE" "$dir/devnet-defcon-q60/debug.log" | tail -1 | cut -d' ' -f1) + reread=no; [ -n "$ts" ] && [ "$(date -u -d "$ts" +%s)" -ge "$T0" ] && reread=yes + echo "POST $unit active=$active running=$running (want $new) keyed=$(grep -cx "$KEYLINE" "$dir/defcon.conf") reread=$reread" + { [ "$active" = active ] && [ "$running" = "$new" ] && [ "$reread" = yes ]; } || rc=1 +done +exit $rc diff --git a/ops/c6a2-fleet-roll-remote.sh b/ops/c6a2-fleet-roll-remote.sh new file mode 100644 index 0000000..1e09e18 --- /dev/null +++ b/ops/c6a2-fleet-roll-remote.sh @@ -0,0 +1,76 @@ +#!/bin/sh +# +# Runs ON ONE FLEET HOST, copied there by c6a2-fleet-roll.sh. +# +# usage: c6a2-fleet-roll-remote.sh preflight|apply +# +# The C6a end-height roll changes the binary and nothing else: no conf key. The #231 key +# (dslcommitmentv2height=12744) must STAY in every conf -- a daemon started +# without it would demand format v2 from the first commitment and reject +# history -- so the preflight refuses a host where any conf lacks it. +# +# preflight changes nothing. apply takes the host whole: stop every instance, +# back up, install, start every instance, then prove three things per daemon: +# running -- md5 of /proc//exe is the new binary (not -version) +# keyed -- the #231 key is still in the conf +# reread -- the NEW process logged "Setting dslcommitmentv2height to 12744" +# with a timestamp after this host's stop. The old line is still +# in the log, so its mere presence proves nothing. +set -u +SRC=${1:?staged binary} +NEW=${2:?new md5} +OLD=${3:?old md5} +MODE=${4:?preflight|apply} +KEYLINE='dslcommitmentv2height=12744' +LOGLINE='Setting dslcommitmentv2height to 12744' + +S="" +[ "$(id -u)" != 0 ] && S="sudo -n" +B=/opt/defcon-devnet/bin +DIRS=$($S sh -c 'ls -d /opt/defcon-devnet/mn* 2>/dev/null') +[ -n "$DIRS" ] || { echo "PRE dirs=0"; exit 0; } + +total=0; keyed=0 +for d in $DIRS; do + total=$((total + 1)) + $S grep -qx "$KEYLINE" "$d/defcon.conf" && keyed=$((keyed + 1)) +done +disk=$($S md5sum $B/defcond 2>/dev/null | cut -d' ' -f1) +staged=$(md5sum "$SRC" 2>/dev/null | cut -d' ' -f1) +echo "PRE daemons=$total keyed=$keyed diskmd5=$disk staged=$staged" +[ "$MODE" = preflight ] && exit 0 + +if [ "$keyed" != "$total" ] || [ "$disk" != "$OLD" ] || [ "$staged" != "$NEW" ]; then + echo "APPLY refused: pre-flight is not clean on this host" + exit 2 +fi + +# shellcheck disable=SC2001 # DIRS is a multi-line list +IDX=$(echo "$DIRS" | sed 's#.*/mn##' | sort -n) +STAMP=$(date +%Y%m%d-%H%M) +T0=$(date -u +%s) + +for i in $IDX; do $S systemctl stop "defcon-devnet-mn@$i"; done +$S cp -a $B/defcond "$B/defcond.bak-$STAMP" || { echo "APPLY backup failed"; exit 3; } +$S install -m 0755 "$SRC" $B/defcond || { echo "APPLY install failed"; exit 3; } +for i in $IDX; do $S systemctl start "defcon-devnet-mn@$i"; done +sleep 12 + +up=0; ran=0; keyed=0; reread=0 +for d in $DIRS; do + i=${d##*/mn} + [ "$($S systemctl is-active "defcon-devnet-mn@$i")" = active ] && up=$((up + 1)) + pid=$($S systemctl show -p MainPID --value "defcon-devnet-mn@$i") + [ "$($S md5sum "/proc/$pid/exe" 2>/dev/null | cut -d' ' -f1)" = "$NEW" ] && ran=$((ran + 1)) + $S grep -qx "$KEYLINE" "$d/defcon.conf" && keyed=$((keyed + 1)) + ts=$($S grep "$LOGLINE" "$d/devnet-defcon-q60/debug.log" 2>/dev/null | tail -1 | cut -d' ' -f1) + if [ -n "$ts" ] && [ "$(date -u -d "$ts" +%s 2>/dev/null || echo 0)" -ge "$T0" ]; then + reread=$((reread + 1)) + else + echo " NOREREAD mn$i last=$ts" + fi +done +$S rm -f "$SRC" +echo "POST up=$up/$total running-new=$ran/$total keyed=$keyed/$total reread=$reread/$total diskmd5=$($S md5sum $B/defcond | cut -d' ' -f1)" +[ "$up" = "$total" ] && [ "$ran" = "$total" ] && [ "$keyed" = "$total" ] && [ "$reread" = "$total" ] || exit 4 +exit 0 diff --git a/ops/c6a2-fleet-roll.sh b/ops/c6a2-fleet-roll.sh new file mode 100644 index 0000000..b1727f9 --- /dev/null +++ b/ops/c6a2-fleet-roll.sh @@ -0,0 +1,61 @@ +#!/bin/bash +# +# The C6a end-height move (#244, end 13488) on the 16 fleet hosts. +# Runs ON THE JUMP HOST, after c6a2-fleet-stage.sh has put the binary on every +# host. The binary moves and nothing else: no conf key. #243 activates by itself +# at height 13488, so every daemon must be on this binary before that height. +# +# Host by host: preflight (the deployed binary is the expected one, the #231 key +# is in every conf, the staged copy is intact), then apply +# (c6a2-fleet-roll-remote.sh). The roll STOPS at the first host that does not come +# back whole -- a half-understood failure repeated on fifteen more hosts is the +# worst outcome available. +# +# ./c6a2-fleet-roll.sh --check # preflight every host, change nothing +# ./c6a2-fleet-roll.sh # preflight, then roll +set -uo pipefail + +NEW_MD5=14583e7c2b40e8de2afb265d5bc1dfea +OLD_MD5=7bf411b0eb08731b6e050868b1d2ff72 +STAGED=/tmp/defcond-c6a2 +KEY=/root/.ssh/defcon_nodes +INVENTORY=${FLEET_INVENTORY:-/root/fleet-nodes-all.txt} +HERE=$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd) +REMOTE=$HERE/c6a2-fleet-roll-remote.sh +[ -f "$REMOTE" ] || { echo "missing helper: $REMOTE"; exit 1; } + +CHECK_ONLY=0 +[ "${1:-}" = "--check" ] && CHECK_ONLY=1 + +SSH="ssh -n -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" +SCP="scp -q -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" + +echo "==> $(date -u +%FT%TZ) new $NEW_MD5, replacing $OLD_MD5, inventory $INVENTORY, check-only=$CHECK_ONLY" +ok=0; failed=0 +while read -r entry; do + case "$entry" in ""|\#*) continue ;; esac + case "$entry" in *@*) target="$entry" ;; *) target="root@$entry" ;; esac + label=$(echo "$entry" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1.x/') + echo "=== $label $(date -u +%T)" + $SCP "$REMOTE" "$target:/tmp/c6a2-roll.sh" || { echo " FAIL: helper copy"; failed=$((failed+1)); break; } + pre=$($SSH "$target" "sh /tmp/c6a2-roll.sh $STAGED $NEW_MD5 $OLD_MD5 preflight" 2>&1) + echo " $pre" + case "$pre" in + *"diskmd5=$OLD_MD5 staged=$NEW_MD5"*) ;; + *) echo " FAIL: preflight not clean"; failed=$((failed+1)); break ;; + esac + if [ "$CHECK_ONLY" = 1 ]; then ok=$((ok+1)); continue; fi + + out=$($SSH "$target" "sh /tmp/c6a2-roll.sh $STAGED $NEW_MD5 $OLD_MD5 apply; echo rc=\$?; rm -f /tmp/c6a2-roll.sh" 2>&1) + echo "$out" | awk '{print " " $0}' + if echo "$out" | grep -q '^rc=0$' && echo "$out" | grep -q "diskmd5=$NEW_MD5"; then + ok=$((ok+1)) + else + echo " FAIL: this host did not come back whole -- stopping the roll here" + failed=$((failed+1)); break + fi +done < "$INVENTORY" + +echo +echo "==> $(date -u +%FT%TZ) hosts ok=$ok failed=$failed" +[ "$failed" = 0 ] || exit 1 diff --git a/ops/c6a2-fleet-stage.sh b/ops/c6a2-fleet-stage.sh new file mode 100644 index 0000000..74e5708 --- /dev/null +++ b/ops/c6a2-fleet-stage.sh @@ -0,0 +1,54 @@ +#!/bin/bash +# +# The C6a end-height move (#244, end 13488): put the fleet artefact on +# every fleet host AHEAD of the roll window, and prove on each host that it is +# the right file and that it can run there. Runs ON THE JUMP HOST. +# +# It writes one file to /tmp on each host and nothing else: no daemon is +# stopped, no conf is touched, nothing is installed. On each host it checks the +# md5 of the copy AS THAT HOST SEES IT (scp exiting 0 proves nothing), runs ldd +# on it (a build that links a library the target lacks has already been +# installed once on this project and would not start), and asks the binary for +# its version. +# +# usage: ./c6a2-fleet-stage.sh +set -uo pipefail + +EXPECT_MD5=14583e7c2b40e8de2afb265d5bc1dfea +DEST=/tmp/defcond-c6a2 +KEY=/root/.ssh/defcon_nodes +INVENTORY=${FLEET_INVENTORY:-/root/fleet-nodes-all.txt} + +DAEMON=${1:?usage: c6a2-fleet-stage.sh } +[ -f "$DAEMON" ] || { echo "no such file: $DAEMON"; exit 1; } +LOCAL_MD5=$(md5sum "$DAEMON" | cut -d' ' -f1) +echo "==> staging md5 $LOCAL_MD5 to $DEST on every host in $INVENTORY" +[ "$LOCAL_MD5" = "$EXPECT_MD5" ] || { echo " REFUSING: expected $EXPECT_MD5"; exit 1; } + +SSH="ssh -n -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" +SCP="scp -q -i $KEY -o ConnectTimeout=20 -o BatchMode=yes -o StrictHostKeyChecking=no" + +ok=0; failed=0 +while read -r entry; do + case "$entry" in ""|\#*) continue ;; esac + case "$entry" in *@*) target="$entry" ;; *) target="root@$entry" ;; esac + label=$(echo "$entry" | sed -E 's/([0-9]+\.[0-9]+\.[0-9]+)\.[0-9]+/\1.x/') + printf '%-30s ' "$label" + + have=$($SSH "$target" "md5sum $DEST 2>/dev/null | cut -d' ' -f1" 2>/dev/null) + if [ "$have" != "$LOCAL_MD5" ]; then + # a single transport failure to one of the production hosts has been transient before + $SCP "$DAEMON" "$target:$DEST" || $SCP "$DAEMON" "$target:$DEST" || { echo "COPY FAILED"; failed=$((failed+1)); continue; } + fi + out=$($SSH "$target" "chmod 0755 $DEST; echo md5=\$(md5sum $DEST | cut -d' ' -f1) missing=\$(ldd $DEST 2>/dev/null | grep -c 'not found') version=\$($DEST -version 2>/dev/null | head -1 | tr ' ' '_')" 2>/dev/null) + echo "$out" + case "$out" in + *"md5=$LOCAL_MD5 missing=0 version=DeFCoN"*) ok=$((ok+1)) ;; + *) failed=$((failed+1)) ;; + esac +done < "$INVENTORY" + +echo +echo "==> staged-and-runnable=$ok failed=$failed" +echo " Nothing was installed and no daemon was touched." +[ "$failed" = 0 ] || exit 1 diff --git a/ops/c6a2-roll-at-height.sh b/ops/c6a2-roll-at-height.sh new file mode 100644 index 0000000..dbecef7 --- /dev/null +++ b/ops/c6a2-roll-at-height.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Start the C6a fleet roll at exactly START (a Q60 base + 10, the base not a +# multiple of 72). Runs ON THE JUMP HOST under nohup, from /root/roll-c6a2. +# Height is read from two fleet hosts' mn1 (the higher answer wins). If the first +# height seen at or above START is already past LATEST, it does NOT roll: the +# window +10..+19 would not hold a 16-host roll. It also refuses to start at or +# past HARD_STOP: every daemon must be on the new binary before the end height +# 13488, and a roll begun too close to it is a fork, not a roll. +# +# usage: nohup ./c6a-roll-at-height.sh > at-height.log 2>&1 & +START=${1:?start height} +LATEST=${2:?latest acceptable first-seen height} +HARD_STOP=13470 +[ "$LATEST" -lt "$HARD_STOP" ] || { echo "REFUSING: latest $LATEST is not below the hard stop $HARD_STOP"; exit 4; } +DEADLINE=$(( $(date +%s) + 4*3600 )) +K=/root/.ssh/defcon_nodes +INV=/root/fleet-nodes-all.txt +SSH="ssh -n -i $K -o ConnectTimeout=10 -o BatchMode=yes -o StrictHostKeyChecking=no" +cd /root/roll-c6a2 || exit 1 +height_of() { # $1 = inventory entry + local t=$1 u=root S=""; case "$t" in *@*) u=${t%%@*};; *) t="root@$t";; esac + [ "$u" != root ] && S="sudo -n" + $SSH "$t" "$S /opt/defcon-devnet/bin/defcon-cli -datadir=/opt/defcon-devnet/mn1 -conf=/opt/defcon-devnet/mn1/defcon.conf getblockcount" 2>/dev/null +} +A=$(grep -v '^[[:space:]]*\(#\|$\)' "$INV" | sed -n 2p) +B=$(grep -v '^[[:space:]]*\(#\|$\)' "$INV" | sed -n 3p) +echo "armed $(date -u +%FT%TZ) start=$START latest=$LATEST hard-stop=$HARD_STOP" +last="" +while [ "$(date +%s)" -lt "$DEADLINE" ]; do + ha=$(height_of "$A"); hb=$(height_of "$B") + h=0 + [[ "$ha" =~ ^[0-9]+$ ]] && h=$ha + [[ "$hb" =~ ^[0-9]+$ ]] && [ "$hb" -gt "$h" ] && h=$hb + if [ "$h" != "$last" ]; then echo "$(date -u +%T) height $h"; last=$h; fi + if [ "$h" -ge "$START" ]; then + if [ "$h" -gt "$LATEST" ]; then echo "ABORT $(date -u +%T): first seen $h > $LATEST, not rolling"; exit 2; fi + echo "ROLL START $(date -u +%FT%TZ) at height $h" + ./c6a2-fleet-roll.sh > roll.log 2>&1 + rc=$? + echo "ROLL END $(date -u +%FT%TZ) rc=$rc height $(height_of "$B")" + exit $rc + fi + sleep 8 +done +echo "DEADLINE reached without height $START"; exit 3 diff --git a/ops/c6a2-s-check.sh b/ops/c6a2-s-check.sh new file mode 100644 index 0000000..8518349 --- /dev/null +++ b/ops/c6a2-s-check.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# C6a end-height check, run on the VPS (stdin), read-only. Before 13535 it is +# the control (both retired types still listed and mining); from 13536 on it is +# the measurement for the run's expectations and the closing conditions. +# llmqType: 1 llmq_50_60, 2 llmq_400_60, 3 llmq_400_85, 4 llmq_100_67, 5 llmq_60_75, 7 llmq_defcon +S=13488 +C="sudo -n -u defcon /usr/local/bin/defcon-cli -datadir=/home/defcon/.defcon -conf=/home/defcon/.defcon/defcon.conf" +C2="sudo -n -u defcon /usr/local/bin/defcon-cli -datadir=/home/defcon/.defcon2 -conf=/home/defcon/.defcon2/defcon.conf" +tip=$($C getblockcount) +# shellcheck disable=SC2086 # $tip is always the numeric getblockcount result; no word-splitting/globbing risk +echo "UTC $(date -u '+%F %T') tip $tip (end height $S, tip - S = $((tip - S))) devnet2 tip $($C2 getblockcount) same hash: $([ "$($C getblockhash $tip)" = "$($C2 getblockhash $tip 2>/dev/null)" ] && echo yes || echo no)" +echo "quorum list types at tip: $($C quorum list | grep -oE '"llmq_[a-z0-9_]+"' | tr -d '"' | tr '\n' ' ')" +echo "best chainlock: $($C getbestchainlock | grep -oE '"height": [0-9]+|"llmqType": "[a-z0-9_]+"' | tr '\n' ' ')" + +# Commitments (type 6) by llmqType and quorum height, from S-96 to the tip. +from=$((S - 96)) +# shellcheck disable=SC2086 # $tip is always numeric (getblockcount result) +[ $tip -lt $S ] && from=$((tip - 96)); [ $from -lt 1 ] && from=1 +$C getblockhash $from > /dev/null || exit 1 +tmp=$(mktemp -d) +# shellcheck disable=SC2086 # $tip is always numeric (getblockcount result) +for h in $(seq $from $tip); do + # shellcheck disable=SC2086 # $h is always numeric (seq output) + $C getblock "$($C getblockhash $h)" 2 > "$tmp/$h.json" 2>/dev/null +done +python3 - "$tmp" "$from" "$tip" "$S" <<'PY' +import json, os, sys +d, lo, hi, S = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4]) +names = {1: 'llmq_50_60', 2: 'llmq_400_60', 3: 'llmq_400_85', 4: 'llmq_100_67', 5: 'llmq_60_75', 7: 'llmq_defcon'} +rows = [] +blocks = {} +for h in range(lo, hi + 1): + try: + blocks[h] = json.load(open(os.path.join(d, f'{h}.json'))) + except Exception: + print(f' block {h}: unreadable') +hash2height = {b['hash']: h for h, b in blocks.items()} +for h, b in sorted(blocks.items()): + for tx in b.get('tx', []): + if tx.get('type') != 6: continue + c = (tx.get('qcTx') or {}).get('commitment') or {} + t = c.get('llmqType') + base = hash2height.get(c.get('quorumHash'), 'base<' + str(lo)) + null = (c.get('signersCount', 0) == 0 and c.get('validMembersCount', 0) == 0) + rows.append((h, names.get(t, t), base, 'null' if null else f"valid {c.get('validMembersCount')}")) +retired = [r for r in rows if r[1] in ('llmq_50_60', 'llmq_60_75')] +print(f'commitments in blocks {lo}..{hi}: {len(rows)} total') +for r in rows: + if (r[1] in ('llmq_50_60', 'llmq_60_75') and r[3] != 'null') or (r[0] >= S - 30 and r[3] != 'null'): + print(f' block {r[0]}: {r[1]} base {r[2]} {r[3]}') +nulls = {} +for r in rows: + if r[3] == 'null': nulls.setdefault(r[1], []).append(r[0]) +for k, v in nulls.items(): + print(f' null commitments {k}: {len(v)} in blocks {min(v)}..{max(v)}') +after = [r for r in retired if r[0] >= S] +print(f'RETIRED-TYPE commitments in blocks >= {S}: {len(after)}' + ('' if after else ' (expected 0)')) +PY +rm -rf "$tmp" + +# PoSe and Sentinel +$C protx list registered true | python3 -c " +import json,sys +l=json.load(sys.stdin) +pen=[x for x in l if x['state'].get('PoSePenalty',0)>0]; ban=[x for x in l if x['state'].get('PoSeBanHeight',-1) not in (-1,None)] +print(f'PoSe: registered {len(l)} penalty>0 {len(pen)} banned {len(ban)}')" +curl -s 'http://localhost:4100/api/v1/dsl/epochs?limit=4' | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',d if isinstance(d,list) else []) +for e in items[:4]: print(' epoch', e.get('epoch'), 'boundary', e.get('boundaryHeight', e.get('height')), 'status', e.get('status'), 'missed', e.get('missedCount'))" 2>/dev/null +# Explorer: rounds of the retired types at or above the end +curl -s "http://localhost:4100/api/v1/quorum-rounds?llmqName=llmq_50_60&limit=5" | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',[]) +print('explorer llmq_50_60 latest rounds:', [(i.get('expectedHeight'), i.get('status')) for i in items[:5]])" 2>/dev/null +curl -s "http://localhost:4100/api/v1/quorum-rounds?llmqName=llmq_60_75&limit=5" | python3 -c " +import json,sys +d=json.load(sys.stdin).get('data',{}); items=d.get('items',[]) +print('explorer llmq_60_75 latest rounds:', [(i.get('expectedHeight'), i.get('status')) for i in items[:5]])" 2>/dev/null +curl -s http://localhost:4100/api/v1/health | python3 -c "import sys,json; d=json.load(sys.stdin)['data']; print('health', d['status'], d['failing'], 'behind', d['behind'], d['rounds'])" diff --git a/ops/c6a2-seed-roll.sh b/ops/c6a2-seed-roll.sh new file mode 100644 index 0000000..c44a801 --- /dev/null +++ b/ops/c6a2-seed-roll.sh @@ -0,0 +1,72 @@ +#!/bin/bash +# +# The C6a end-height move (#244, end 13488) on the explorer VPS: the +# seed and devnet2, the canary before the fleet. Runs ON THE VPS as root. +# +# #243 is a devnet consensus change at height 13488: llmq_50_60 and llmq_60_75 +# form no new quorum from there. Every devnet daemon must run this binary before +# 13488; an old one past it forks off. No conf key changes: the #231 key +# (dslcommitmentv2height=12744) stays in both confs, and this roll moves only the +# binary. +# +# devnet2 executes /usr/local/bin/defcond-nobdb -- the FLEET artefact under +# another name -- and every roll before #231 missed it. +# +# ./c6a2-seed-roll.sh --check +# ./c6a2-seed-roll.sh +# +# The explorer reads the seed's RPC, so its sync ticks fail for the few seconds +# this takes and recover on their own. +set -uo pipefail + +NEW_SEED=a7822e1b912f232fd2c5e85b90023afa +NEW_FLEET=14583e7c2b40e8de2afb265d5bc1dfea +OLD_SEED=d050b97df48a74f57a09dadca5985c84 +OLD_FLEET=7bf411b0eb08731b6e050868b1d2ff72 +KEYLINE='dslcommitmentv2height=12744' +LOGLINE='Setting dslcommitmentv2height to 12744' + +CHECK_ONLY=0 +if [ "${1:-}" = "--check" ]; then CHECK_ONLY=1; shift; fi +SEED_BIN=${1:?usage: c6a2-seed-roll.sh [--check] } +FLEET_BIN=${2:?usage: c6a2-seed-roll.sh [--check] } + +# unit | installed path | datadir | source | new md5 | old md5 +NODES=( + "defcond-devnet|/usr/local/bin/defcond|/home/defcon/.defcon|$SEED_BIN|$NEW_SEED|$OLD_SEED" + "defcond-devnet2|/usr/local/bin/defcond-nobdb|/home/defcon/.defcon2|$FLEET_BIN|$NEW_FLEET|$OLD_FLEET" +) + +rc=0 +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + exec_path=$(grep -h '^ExecStart=' "/etc/systemd/system/$unit.service" | head -1 | sed 's/^ExecStart=//' | awk '{print $1}') + disk=$(md5sum "$bin" | cut -d' ' -f1); srcmd5=$(md5sum "$src" | cut -d' ' -f1) + keyed=$(grep -cx "$KEYLINE" "$dir/defcon.conf") + echo "PRE $unit executes=$exec_path disk=$disk (want old $old) source=$srcmd5 (want new $new) keyed=$keyed" + { [ "$exec_path" = "$bin" ] && [ "$disk" = "$old" ] && [ "$srcmd5" = "$new" ] && [ "$keyed" = 1 ]; } || { echo " REFUSING: preflight not clean"; rc=1; } +done +[ "$rc" = 0 ] || exit 1 +if [ "$CHECK_ONLY" = 1 ]; then echo "==> --check given, nothing changed"; exit 0; fi + +stamp=$(date +%Y%m%d-%H%M) +T0=$(date -u +%s) +for row in "${NODES[@]}"; do IFS='|' read -r unit bin dir src new old <<<"$row"; systemctl stop "$unit"; done +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + cp -a "$bin" "$bin.bak-$stamp" + install -m 0755 "$src" "$bin" +done +for row in "${NODES[@]}"; do IFS='|' read -r unit bin dir src new old <<<"$row"; systemctl start "$unit"; done +sleep 15 + +for row in "${NODES[@]}"; do + IFS='|' read -r unit bin dir src new old <<<"$row" + active=$(systemctl is-active "$unit") + running=$(md5sum "/proc/$(systemctl show -p MainPID --value "$unit")/exe" 2>/dev/null | cut -d' ' -f1) + ts=$(grep "$LOGLINE" "$dir/devnet-defcon-q60/debug.log" | tail -1 | cut -d' ' -f1) + reread=no; [ -n "$ts" ] && [ "$(date -u -d "$ts" +%s)" -ge "$T0" ] && reread=yes + echo "POST $unit active=$active running=$running (want $new) keyed=$(grep -cx "$KEYLINE" "$dir/defcon.conf") reread=$reread" + { [ "$active" = active ] && [ "$running" = "$new" ] && [ "$reread" = yes ]; } || rc=1 +done +exit $rc