physrep: skip rtcpu'd / decommissioned hosts when establishing reverse connections

[markhannum]

Summary

The physrep reverse-connection path did not consult rtcpu at all. This change makes a node decline to source reverse connections while it is itself rtcpu'd down, enforced on both the sending and receiving sides. A machine that rtcpu reports as down is being routed out of service and should not be handing replication sockets to replicants; replicants can pick up another source. The target replicant's rtcpu state is deliberately left alone.

Background

For cross-cloud physrep (PPCLD -> PDCLD), the replicant often cannot dial out to its source, so the source dials in to the replicant and hands it a socket (a "reverse connection"), which the replicant then uses to pull logs. There are two sides: the sender of reverse connections is the source/intermediary (db/reverse_conn.c), which runs one worker thread per target host and periodically calls send_reversesql_request(); the receiver is the replicant accepting the inbound reversesql appsock (plugins/reversesql/reversesql.c). Neither side consulted rtcpu before this change.

Changes

Sender (db/reverse_conn.c): before initiating a connection, the reverse-connection worker now checks this node's own rtcpu state (gbl_myhostname) and skips initiating while the node is rtcpu'd down, resuming automatically once it is back up. This sits alongside the existing bdb_am_i_coherent() self-gate.

Receiver (plugins/reversesql/reversesql.c): an inbound reversesql request from a source that rtcpu reports as down is rejected — the receiver-side mirror of the sender's self-gate.

The target host's rtcpu state is intentionally not consulted. A replicant that is merely rtcpu'd out of client service but still alive should keep replicating, exactly as it does on the normal (pull) path where the replicant dials out itself. The shared predicate physrep_revconn_host_is_up() is a simple is_node_up(host) == 1 and is now only applied to the sourcing node (self-gate) and the inbound source (receiver).

The predicate is exposed via a new revconn_host_is_up <host> message command for testing.

Testing

Adds physrep_revconn_rtcpu to tests/phys_rep_tiered.test, modeled on the existing physrep_rtcpu_source case. Using the fakedr trap to force rtcpu to report a host as down, it asserts that revconn_host_is_up returns "up" for a healthy host, "down" once the host is marked rtcpu'd, and "up" again after the mark is cleared, and that a node reports itself ineligible to source reverse connections while it is rtcpu'd down (the sender self-gate). Connections use --admin, since the node under test can itself be the one marked rtcpu'd (drtest) and would otherwise refuse normal client traffic.

Notes

This is hardening rather than the root-cause fix. The reverse-connect storm that prompted this was traced to an increase in physrep-fanout on the physrep metadb: each source then services a larger set of reverse-hosts and fans out many more simultaneous outbound connect attempts. That has been addressed separately. This change is defense-in-depth so that a node which has been rtcpu'd down no longer sources reverse connections, regardless of fanout.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
queuedb_rollover_noroll1_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
ssl_san
sc_parallel_logicalsc_generated
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
truncatesc_offline_generated [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
ssl_san
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
sc_resume_logicalsc_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**