pyln-testing: wait for listen port release before starting a node
What the developer wrote
Authored by Ken Sedgwick
pyln-testing: wait for listen port release before starting a node
A node's connectd is a separate process holding the listen socket,
and on shutdown it exits on its own schedule after lightningd itself
is gone -- under valgrind, its teardown can lag by tens of seconds.
If a test restarts the node in that window, the new connectd fails
with 'Address already in use', lightningd exits, and the test times
out waiting for 'Server started with public key'. Seen in CI in
test_emergencyrecoverpenaltytxn, where the port was still held 37
seconds after the old connectd began shutting down.
The filesystem port locks don't cover this: they keep other workers
from reserving the port, but the restarting node owns its reservation
both times; nothing waits for the old connectd to actually release
the socket.
Before launching lightningd, bind-probe the node's port (with
SO_REUSEADDR, matching connectd, so TIME_WAIT sockets don't count as
in use) and only proceed once the bind succeeds.
Fixes: #9354
Changelog-None
What changed, and why it matters
This is a testing-framework fix, not a fix in the Core Lightning node software itself. It stops automated tests from failing when a background process (connectd) is slow to release a network port after a node restart. There is no direct security vulnerability being patched; it improves test reliability.
No security action required. Treat as a normal reliability improvement for the test suite. Reviewers can verify the bind-probe uses SO_REUSEADDR and only raises on unexpected errors.
Security signals we found
No security-relevant code change in production daemon
Fixes a test-framework race condition, not an exploit primitive
No input validation, cryptography, or authorization changes
No memory safety, parsing, or protocol changes
Evidence from the diff
The patch adds wait_for_port_released() in pyln-testing’s LightningD.start(). Before launching a new lightningd instance during tests, it repeatedly tries to bind the node’s listen port with SO_REUSEADDR until it succeeds or times out. This prevents ‘Address already in use’ failures when the previous connectd process is still shutting down (especially under valgrind). The change is purely in the Python test harness and does not alter Core Lightning daemon behavior.
Changed components
contrib/pyln-testing/pyln/testing/utils.pyInspect captured patch +38 / −0
diff --git a/contrib/pyln-testing/pyln/testing/utils.py b/contrib/pyln-testing/pyln/testing/utils.py
index 16c63ed9..a2ab613d 100644
--- a/contrib/pyln-testing/pyln/testing/utils.py
+++ b/contrib/pyln-testing/pyln/testing/utils.py
@@ -15,6 +15,7 @@ from pyln.client import Plugin
import ephemeral_port_reserve # type: ignore
import tempfile
+import errno
import json
import logging
import lzma
@@ -215,6 +216,40 @@ def cleanup_stale_port_locks():
pass # best-effort, never crash the test run over cleanup
+def wait_for_port_released(port, timeout=TIMEOUT):
+ """Wait until 127.0.0.1:port can be bound again.
+
+ A stopped node's connectd holds the listen socket until it exits,
+ which can be several seconds after lightningd itself is gone
+ (subdaemons are separate processes, and die slowly under valgrind).
+ Restarting the node before the port is released makes the new
+ connectd fail with 'Address already in use'.
+ """
+ start_time = time.time()
+ while True:
+ s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ # Match connectd's SO_REUSEADDR, so sockets lingering in
+ # TIME_WAIT don't count as "still in use".
+ s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ try:
+ s.bind(('127.0.0.1', port))
+ break
+ except OSError as e:
+ if e.errno != errno.EADDRINUSE:
+ raise
+ if time.time() - start_time > timeout:
+ raise TimeoutError(
+ "Port {} was not released within {} seconds"
+ .format(port, timeout))
+ finally:
+ s.close()
+ time.sleep(0.1)
+
+ waited = time.time() - start_time
+ if waited >= 1:
+ logging.info("Port %d took %.1fs to be released", port, waited)
+
+
class TailableProc(object):
"""A monitorable process that we can start, stop and tail.
@@ -848,6 +883,9 @@ class LightningD(TailableProc):
def start(self, stdin=None, wait_for_initialized=True, stderr_redir=False):
self.opts['bitcoin-rpcport'] = self.rpcproxy.rpcport
+ # On restart, the previous incarnation's connectd may still be
+ # dying and holding our listen port: don't launch until it's free.
+ wait_for_port_released(self.port)
TailableProc.start(self, stdin, stdout_redir=False, stderr_redir=stderr_redir)
if wait_for_initialized:
self.wait_for_log("Server started with public key")
Why this scored 17/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.