Keep unsigned settlements when rejecting commands (#3401)
What changed, and why it matters
This commit fixes a bug in Eclair, a Bitcoin Lightning Network node implementation. When the node received a duplicate command to settle a payment (fulfill or fail), it would mistakenly delete the still-pending settlement from its database. If the node disconnected before that settlement was signed, it would not be replayed on reconnection. For a payment being forwarded through the node, this could mean the downstream side got paid but the upstream side never received the proof it needed, causing the upstream payment to time out and potentially costing the node money. The fix keeps the pending settlement in the database until it has actually been signed.
Upgrade to a release containing this commit. Operators running routing nodes with relayed HTLCs should prioritize the update to avoid potential loss of funds from upstream HTLC timeouts. No immediate workaround is described; the fix is in the settlement acknowledgement logic.
Security signals we found
Loss of pending HTLC settlement state on disconnect/reconnect
Upstream HTLC timeout risk for relayed fulfilled downstream HTLCs
Duplicate/conflicting command handling caused premature DB acknowledgement
Fix adds explicit guard based on alreadyProposed local changes
Regression test covers duplicate CMD_FULFILL_HTLC and conflicting CMD_FAIL_HTLC
Evidence from the diff
In CommonHandlers.acking(), settlement commands were acknowledged (and thus scheduled for deletion from PendingCommandsDb) 10 seconds after being processed, regardless of whether the command had actually been included in a signed commitment. If a duplicate or conflicting settlement command arrived for an HTLC whose settlement was already proposed but not yet signed, the command was rejected, but the acking logic still scheduled deletion of the pending command. On disconnect/reconnect, the unsigned settlement would not be replayed, risking an upstream HTLC timeout for a relayed fulfilled HTLC. The patch checks CommitmentChanges.alreadyProposed() and skips acknowledgement/deletion when an unsigned settlement is already proposed; it is then acknowledged once signed.
Changed components
eclair-core/src/main/scala/fr/acinq/eclair/channel/fsm/CommonHandlers.scalaPendingCommandsDb settlement command acknowledgement logicLightning channel state machine (CommonHandlers.acking)HTLC settlement replay on reconnectionInspect captured patch +38 / −3
### eclair-core/src/main/scala/fr/acinq/eclair/channel/fsm/CommonHandlers.scala
@@ -106,9 +106,18 @@ trait CommonHandlers {
* @param cmd fail/fulfill command that has been processed
*/
def acking(channelId: ByteVector32, cmd: HtlcSettlementCommand): FSM.State[ChannelState, ChannelData] = {
- log.debug("scheduling acknowledgement of cmd id={}", cmd.id)
- context.system.scheduler.scheduleOnce(10 seconds)(PendingCommandsDb.ackSettlementCommand(nodeParams.db.pendingCommands, channelId, cmd))(context.system.dispatcher)
- state
+ state.stateData match {
+ case d: ChannelDataWithCommitments if CommitmentChanges.alreadyProposed(d.commitments.changes.localChanges.proposed, cmd.id) =>
+ // This command was rejected because we've already proposed a settlement for that HTLC (e.g. a duplicate of the
+ // same command), which hasn't been signed yet: it will be discarded if we disconnect, so we must keep it in our
+ // DB to replay it on reconnection. It will be acknowledged once signed.
+ log.debug("not acknowledging cmd id={}: settlement is pending signature", cmd.id)
+ state
+ case _ =>
+ log.debug("scheduling acknowledgement of cmd id={}", cmd.id)
+ context.system.scheduler.scheduleOnce(10 seconds)(PendingCommandsDb.ackSettlementCommand(nodeParams.db.pendingCommands, channelId, cmd))(context.system.dispatcher)
+ state
+ }
}
def acking(updates: List[UpdateMessage]): FSM.State[ChannelState, ChannelData] = {
### eclair-core/src/test/scala/fr/acinq/eclair/channel/states/e/NormalStateSpec.scala
@@ -1798,6 +1798,32 @@ class NormalStateSpec extends TestKitBaseClass with FixtureAnyFunSuiteLike with
awaitCond(bob.underlyingActor.nodeParams.db.pendingCommands.listSettlementCommands(initialState.channelId).isEmpty)
}
+ test("recv CMD_FULFILL_HTLC (don't acknowledge unsigned fulfill when rejecting another command)") { f =>
+ import f._
+ val sender = TestProbe()
+ val (r, htlc) = addHtlc(50000000 msat, alice, bob, alice2bob, bob2alice)
+ crossSign(alice, bob, alice2bob, bob2alice)
+
+ // actual test begins
+ val c = CMD_FULFILL_HTLC(htlc.id, r, None, None, replyTo_opt = Some(sender.ref))
+ // this would be done automatically when the relayer calls safeSend
+ bob.underlyingActor.nodeParams.db.pendingCommands.addSettlementCommand(channelId(bob), c)
+ sender.send(bob, c)
+ sender.expectMsgType[RES_SUCCESS[CMD_FULFILL_HTLC]]
+ bob2alice.expectMsgType[UpdateFulfillHtlc]
+ // We receive a duplicate of that command (e.g. replayed from the DB on reconnection), and a conflicting command.
+ sender.send(bob, c)
+ sender.expectMsg(RES_FAILURE(c, UnknownHtlcId(channelId(bob), htlc.id)))
+ val fail = CMD_FAIL_HTLC(htlc.id, FailureReason.LocalFailure(TemporaryNodeFailure()), None, replyTo_opt = Some(sender.ref))
+ sender.send(bob, fail)
+ sender.expectMsg(RES_FAILURE(fail, UnknownHtlcId(channelId(bob), htlc.id)))
+ // Our fulfill hasn't been signed yet, so we must not remove it from the DB: it would be lost if we disconnected.
+ sender.expectNoMessage(11 seconds)
+ val pending = bob.underlyingActor.nodeParams.db.pendingCommands.listSettlementCommands(channelId(bob))
+ assert(pending.size == 1)
+ assert(pending.collect { case cmd: CMD_FULFILL_HTLC => (cmd.id, cmd.r) } == Seq((htlc.id, r)))
+ }
+
private def testUpdateFulfillHtlc(f: FixtureParam, commitmentFormat: CommitmentFormat): Unit = {
import f._
Why this scored 66/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.