Physrep source log range

[markhannum]

physrep: LSN-aware source selection to avoid no-overlap reconnect storms (DRQS 185276960)

Problem

A physical replicant that has fallen far behind (DRQS 185276960: ~2.1 GB behind)
can be handed "sources" by register_replicant that have already logdelete'd
the logs the replicant still needs. The replicant connects, discovers the
non-overlap only late — via the sentinel → truncation → sleep path — silently
retries, and repeats. In the reported incident this produced 300+ reconnects and
looked like a hang, with no operator-visible signal that no source could actually
serve the node.

Fix

Three complementary levers, designed so an upgraded binary is a behavior-neutral
drop-in and the rest can be turned on gradually.

1. Connect-time coverage check (primary fix, ships ON).
   New physrep_source_covers_me() (bdb/phys_rep_lsn.c) probes a candidate
   source's live log range and compares it to ours before find_new_repl_db()
   commits, skipping a provably non-covering source instead of discovering it the
   slow way. It reuses the existing log-range primitives, is a purely local
   decision, and fails safe (proceeds on any probe error), so it needs neither
   the firstfile column nor a uniform fleet. Gated by
   physrep_verify_source_range (default on).

2. Registry-side range filtering (ships OFF, via keepalive_v2).
   keepalive_v2 now reports each node's oldest log file into
   comdb2_physreps.firstfile, and register_replicant does strict
   [firstfile, file] range filtering when that column exists — with a permissive
   fallback pass so a not-yet-reporting fleet is never left with zero candidates.
   keepalive_v2 tolerates a missing firstfile column (checked via
   sys.physrep_tunables), so the binary can be deployed before the column is
   added. physrep_keepalive_v2 stays off by default so a new sender never
   invokes the v2 SP against an older metadb that cannot handle a missing column.

3. No-viable-source alarm + observability.
   After physrep_no_source_alarm_threshold (default 10) consecutive cycles in
   which every reachable candidate was probed and none covered our LSN, the node
   latches gbl_physrep_no_viable_source, logs PHYSREP NO VIABLE SOURCE, and
   re-registers rather than spinning on a stale list. The latched state is
   exposed as the physrep_no_viable_source metric in comdb2_metrics and is
   cleared as soon as the replicant connects to a viable source (including via a
   reverse connection). The alarm only counts a cycle when no candidate was
   merely unreachable/penalized, so a transiently-down covering source does not
   misfire it.

New tunables

Tunable	Default	Notes
physrep_verify_source_range	on	Local, fail-safe connect-time check; no compatibility concerns.
physrep_keepalive_v2	off	Enable fleet-wide only after the firstfile column exists.
physrep_no_source_alarm_threshold	10	Consecutive no-viable-source cycles before latching the alarm.

Recommended rollout ordering

1. Deploy the new binary across the fleet (sources, replicants, metadbs).
   Connect-time source verification is active immediately.
2. ALTER TABLE comdb2_physreps ADD COLUMN firstfile INT on the metadb cluster(s).
3. Enable physrep_keepalive_v2 fleet-wide so nodes begin reporting firstfile
   and register_replicant also range-filters during registration.

Enabling physrep_keepalive_v2 before steps 1–2 are complete risks a node
invoking the v2 keepalive SP on an older metadb that cannot tolerate a missing
firstfile column. See docs/pages/operating/physical_replication.md.

Compatibility / safety

• Behavior-neutral binary drop-in: the only default-on change is the local,
  fail-safe coverage check; all cross-node/metadb-observable behavior
  (keepalive_v2) ships off.
• Works with or without the comdb2_physreps.firstfile column, so the column can
  be added after the binary is deployed.
• Physrep SPs are embedded in the binary, so deploying updates them.

Testing

• tests/phys_rep_tiered.test — added two cases (in a separate commit):
  • phys_rep_verify_tunable_defaults: asserts the tunable defaults per variant.
  • phys_rep_firstfile_rollout (FIRSTFILE=0 variants): exercises the full
    production rollout end to end — healthy with no column and keepalive_v2 off;
    enable keepalive_v2 while the column is still absent and confirm the SP
    tolerates it and keeps refreshing last_keepalive; live ALTER add the
    column on the metadbs; confirm firstfile auto-populates with no restart.
• tests/tunables.test golden file updated for the new/changed tunable
  descriptions.
• Built and ran the new phys_rep_tiered.test cases in isolation on a Linux
  build host.

Commits

• physrep: LSN-aware source selection to avoid no-overlap retries (DRQS 185276960)
• test: physrep source-range tunable defaults and firstfile rollout (DRQS 185276960)

🤖 Generated with Claude Code

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**