scripts: fix backwards compatibility test timing issues
What the developer wrote
Authored by ziggie
scripts: fix backwards compatibility test timing issues
The backwards compatibility test was failing intermittently due to two
related timing issues in the test setup.
The issue was that Dave's `wait_graph_sync dave 3` was hanging
for up to 60 minutes. Dave's initial gossip sync with Charlie could
complete before Charlie had forwarded the alice-bob channel
announcement, leaving Dave stuck at 2 channels until lnd's historical
syncer fired at its default interval of 1 hour. After this 1-hour idle,
some routing state had become stale, causing the subsequent payment from
alice to dave to fail with FAILURE_REASON_NO_ROUTE.
This issues is now addressed by setting `--historicalsyncinterval=10s` on
all nodes. This causes nodes to periodically re-sync the full gossip
state from their peers every 10 seconds instead of every hour. Dave
therefore picks up any missed channel announcements and routing policies
within seconds, and alice's routing graph stays up-to-date throughout
the test.
Additionally, lnd debug logs from all containers are now collected
before teardown on failure and uploaded as a CI artifact, making future
failures easier to diagnose.
What changed, and why it matters
This commit fixes a flaky test in the LND project's backwards-compatibility test suite. It changes how often test nodes re-sync gossip information from once per hour to every 10 seconds, preventing a race condition where one node would miss channel announcements and get stuck. It also adds collection of debug logs when the test fails so developers can diagnose future failures more easily. There is no security vulnerability being fixed here.
No security action required. This is a test reliability improvement. Reviewers may verify that the 10-second historical sync interval is acceptable for CI resource usage and that log artifact retention (7 days) meets project needs.
Security signals we found
No strong security signals were identified.
Evidence from the diff
The patch modifies the backwards-compatibility test harness only. It adds --historicalsyncinterval=10s to all lnd node configurations in docker-compose files, reducing the historical gossip sync interval from the default 1 hour to 10 seconds. This resolves an intermittent test failure where Dave’s wait_graph_sync dave 3 could hang for up to 60 minutes because Dave completed initial gossip sync before Charlie forwarded the alice-bob channel announcement, leaving Dave with only 2 channels until the next historical sync cycle. After the hour-long wait, routing state became stale, causing subsequent payments to fail with FAILURE_REASON_NO_ROUTE. The patch also adds a collect_logs() helper in network.sh and updates the EXIT trap in test.sh to collect lnd logs before tearing down the Docker cluster, and uploads them as a CI artifact in .github/workflows/main.yml on failure.
Changed components
scripts/bw-compatibility-test/docker-compose.yamlscripts/bw-compatibility-test/docker-compose.override.yamlscripts/bw-compatibility-test/network.shscripts/bw-compatibility-test/test.sh.github/workflows/main.ymlInspect captured patch +35 / −2
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 8d5bba9..4d54c1d 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -602,6 +602,14 @@ jobs:
- name: 🛡️ Backwards compatibility test
run: make backwards-compat-test
+ - name: 📋 Upload node logs on failure
+ if: failure()
+ uses: actions/upload-artifact@v4
+ with:
+ name: bw-compat-logs
+ path: scripts/bw-compatibility-test/logs/
+ retention-days: 7
+
#########################################
# Auto Cache Cleanup on Pull Requests
#########################################
diff --git a/scripts/bw-compatibility-test/docker-compose.override.yaml b/scripts/bw-compatibility-test/docker-compose.override.yaml
index 51ca758..59dc983 100644
--- a/scripts/bw-compatibility-test/docker-compose.override.yaml
+++ b/scripts/bw-compatibility-test/docker-compose.override.yaml
@@ -42,6 +42,7 @@ services:
--protocol.zero-conf
--protocol.simple-taproot-chans
--trickledelay=50
+ --historicalsyncinterval=10s
dave-pr:
image: lnd-dev:backward-compat-test-build
@@ -83,5 +84,6 @@ services:
--protocol.zero-conf
--protocol.simple-taproot-chans
--trickledelay=50
+ --historicalsyncinterval=10s
--db.backend=sqlite
--db.use-native-sql
diff --git a/scripts/bw-compatibility-test/docker-compose.yaml b/scripts/bw-compatibility-test/docker-compose.yaml
index 7860b44..ea901e6 100644
--- a/scripts/bw-compatibility-test/docker-compose.yaml
+++ b/scripts/bw-compatibility-test/docker-compose.yaml
@@ -71,6 +71,7 @@ services:
- "--protocol.zero-conf"
- "--protocol.simple-taproot-chans"
- "--trickledelay=50"
+ - "--historicalsyncinterval=10s"
bob:
image: lightninglabs/lnd:${LND_LATEST_VERSION}
@@ -111,6 +112,7 @@ services:
- "--protocol.zero-conf"
- "--protocol.simple-taproot-chans"
- "--trickledelay=50"
+ - "--historicalsyncinterval=10s"
charlie:
image: lightninglabs/lnd:${LND_LATEST_VERSION}
@@ -148,6 +150,7 @@ services:
- "--tlsextradomain=charlie"
- "--accept-keysend"
- "--trickledelay=50"
+ - "--historicalsyncinterval=10s"
dave:
image: lightninglabs/lnd:${LND_LATEST_VERSION}
@@ -185,6 +188,7 @@ services:
- "--tlsextradomain=dave"
- "--accept-keysend"
- "--trickledelay=50"
+ - "--historicalsyncinterval=10s"
- "--db.backend=sqlite"
- "--db.use-native-sql"
diff --git a/scripts/bw-compatibility-test/network.sh b/scripts/bw-compatibility-test/network.sh
index 98eca67..c712381 100644
--- a/scripts/bw-compatibility-test/network.sh
+++ b/scripts/bw-compatibility-test/network.sh
@@ -298,6 +298,24 @@ wait_for_active_chans() {
echo "🟢 $node now has exactly $expected_channels active channels!"
}
+# collect_logs copies the lnd log file from each running container into a
+# local ./logs directory. Call this before compose_down so logs are available
+# for CI artifact upload even after the cluster is torn down.
+function collect_logs() {
+ local log_dir="$DIR/logs"
+ mkdir -p "$log_dir"
+
+ for node in alice bob charlie dave bob-pr dave-pr; do
+ if docker ps -a --format '{{.Names}}' | grep -q "^${node}$"; then
+ mkdir -p "$log_dir/$node"
+ docker cp "$node:/root/.lnd/logs/bitcoin/regtest/lnd.log" \
+ "$log_dir/$node/lnd.log" 2>/dev/null || true
+ fi
+ done
+
+ echo "📋 Logs collected in $log_dir"
+}
+
# mine mines a number of blocks on the regtest network. If no
# argument is provided, it defaults to 6 blocks.
function mine() {
diff --git a/scripts/bw-compatibility-test/test.sh b/scripts/bw-compatibility-test/test.sh
index d8bb666..7b9f2f7 100755
--- a/scripts/bw-compatibility-test/test.sh
+++ b/scripts/bw-compatibility-test/test.sh
@@ -16,8 +16,9 @@ cd $DIR
compose_up
# Ensure that the cluster is shut down when the script exits
-# regardless of success
-trap compose_down EXIT
+# regardless of success. Logs are collected first so they are
+# available for CI artifact upload after the cluster is torn down.
+trap 'collect_logs; compose_down' EXIT
# Set up the network.
setup_network
Why this scored 15/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.