TCP endpoints silently queue output after terminal send failure

[kentbull]

Problem

Client.tx() and Remoter.tx() unconditionally append output to txbs, even after the socket’s transmit direction has permanently failed:

def tx(self, data):
    self.txbs.extend(data)

Current send handling also does not retain the terminal exception on the endpoint. Some connection failures set cutoff, while BrokenPipeError is re-raised without recording durable transmit state:

except OSError as ex:
    if ex.args[0] in (errno.EAGAIN, errno.EWOULDBLOCK):
        count = 0
    elif ex.args[0] in (...):
        self.cutoff = True
        count = 0
    else:
        raise

After the initial failure, callers can continue queueing bytes that cannot be transmitted. The dead queue grows silently, and higher-level owners cannot inspect the endpoint to determine:

• whether sending is permanently closed;
• why it closed;
• which bytes remain unsent; or
• whether receiving may still continue.

Reproduction

• Create a connected Client with a socket whose send() raises BrokenPipeError.
• Queue output and service sends.
• Confirm that the original output remains queued.
• Queue more output after the terminal failure.
• Observe that the later output is silently appended.

This test fails on current main because client.tx(b"late") mutates txbs:

import errno
import socket
from unittest.mock import Mock

import pytest

from hio.core import tcp


def test_client_rejects_output_after_broken_send():
    client = tcp.Client(ha=("127.0.0.1", 6101))
    client.cs = Mock(spec=socket.socket)
    client.cs.send.side_effect = BrokenPipeError(
        errno.EPIPE, "broken pipe"
    )
    client.accepted = True

    original = b"unsent"
    client.tx(original)

    with pytest.raises(BrokenPipeError):
        client.serviceSends()

    assert bytes(client.txbs) == original

    client.tx(b"late")

    # Expected: terminal send state prevents the dead queue from growing.
    assert bytes(client.txbs) == original

Remoter has the same unconditional enqueue behavior.

Expected behavior

After a terminal send failure:

• the transmit direction is marked closed;
• the original exception remains available on the endpoint;
• already accepted but unsent bytes remain in txbs;
• later enqueue attempts fail observably without modifying txbs; and
• a transmit-only failure does not automatically suppress remaining receive service.

Retryable conditions such as EAGAIN and EWOULDBLOCK must not enter terminal state.

Proposed solution

Track terminal transmit state and its originating cause independently:

except BrokenPipeError as ex:
    self.txCutoff = True
    self.error = ex
    count = 0

Preserve the existing queue by deleting only bytes actually accepted by the socket:

while self.txbs and not self.txCutoff:
    count = self.send(self.txbs)
    del self.txbs[:count]
    break

Reject output submitted after transmit closure and chain the retained cause:

def tx(self, data):
    if self.txCutoff:
        ex = TransmitClosedError(
            "connection send direction is closed"
        )
        if self.error is None:
            self.error = ex
            raise ex
        raise ex from self.error

    self.txbs.extend(data)

The exact attribute and exception names may differ, but the endpoint must retain the terminal cause, preserve unsent bytes, and prevent later output from entering a permanently dead queue.

Apply the same contract to both Client and Remoter.

Related work

Upstream PR #163 partially addresses the Remoter side by adding directional transmit state, preserving queued bytes after BrokenPipeError, and stopping further send attempts.

It does not:

• retain the originating terminal exception;
• reject later enqueue attempts;
• preserve the cause through exception chaining; or
• provide the equivalent Client behavior.

This issue therefore remains applicable whether #163 merges as written or is replaced by another directional-state design.

Acceptance criteria

• Client and Remoter retain the original terminal send exception.
• Unsent bytes remain in txbs.
• Later tx() calls fail observably and do not mutate txbs.
• The enqueue failure exposes or chains the original transport cause.
• Retryable send conditions remain retryable.
• A transmit-only failure leaves receive service available.
• Connection-wide failures close both directions.
• A newly allocated connection resets terminal state and its retained cause.

[kentbull]

Downstream implementations:

• fix: preserve Client sends after peer write EOF GLEIF-IT/hio#9 partially implements the Client contract: it retains terminal send causes and unsent bytes and rejects later output, but connection-wide Client send errors still close only transmit.
• fix: retain Remoter terminal causes and queued output GLEIF-IT/hio#10 implements the corresponding Remoter contract, including connection-wide failure state.

Both PRs are merged into the fork's release/v0.6.20 branch. They must be adapted to current upstream main; they are not clean cherry-picks.