actor: add AskThen to receive an Ask reply as a later message
What changed, and why it matters
This commit adds a new helper called AskThen to LND's internal actor framework. It lets one actor send a request to another and receive the reply later as a normal message, instead of blocking and waiting. The main goal is to prevent deadlocks when two actors are waiting on each other. The change is defensive and includes tests, but it introduces new concurrency behavior that could affect reliability if misused.
Review the AskThen implementation for correct context handling, ensure the retry goroutine cannot leak or spin under edge cases, and verify integration tests cover cross-actor timeout and mailbox closure scenarios before relying on it in production code paths.
Security signals we found
New concurrency primitive added to actor framework
Helper goroutine performs retry delivery with backoff
Drop handler and DLO integration for undeliverable replies
Timeout handling with context cancellation
Designed to prevent transitive deadlock between actors
Evidence from the diff
The commit introduces AskThen, a pipe-to-self asynchronous Ask variant for the actor package. It sends a request via Ask (in-turn) or TryAsk (outside turn), immediately returns, and later delivers wrap(fn.Result[R]) to the caller’s own mailbox via a helper goroutine. It retries delivery on ErrMailboxFull with exponential backoff, logs and optionally reports drops when self has stopped, and supports a timeout. The implementation uses context.WithoutCancel and WithoutTurn to decouple delivery context from the original request context. Tests cover normal delivery, timeout, non-blocking behavior from plain goroutines, deadlock avoidance in cyclic asks, retention while self is full, and observable drops.
Changed components
actor/ask_then.goactor/ask_then_test.goInspect captured patch +475 / −0
### actor/ask_then.go
@@ -0,0 +1,176 @@
+package actor
+
+import (
+ "context"
+ "errors"
+ "time"
+
+ "github.com/lightningnetwork/lnd/fn/v2"
+)
+
+const (
+ // askThenMinRetry and askThenMaxRetry bound the pause between delivery
+ // attempts when self's mailbox is full. The pause doubles from the
+ // first to the second.
+ askThenMinRetry = time.Millisecond
+ askThenMaxRetry = 50 * time.Millisecond
+)
+
+// AskThenOption configures optional AskThen behavior.
+type AskThenOption func(*askThenOptions)
+
+// askThenOptions collects the optional settings of an AskThen.
+type askThenOptions struct {
+ onDrop func(err error)
+}
+
+// WithDropHandler registers a function AskThen calls when the wrapped result
+// can't be delivered to self because self has stopped or its mailbox is
+// closed, with the delivery error. It runs on a helper goroutine, so like the
+// wrap function it must not touch the caller's state directly; a typical use
+// is to log, count, or Tell a reconcile message elsewhere. Without a handler
+// the drop is still logged at warning level.
+func WithDropHandler(f func(err error)) AskThenOption {
+ return func(o *askThenOptions) {
+ o.onDrop = f
+ }
+}
+
+// AskThen sends msg to ref as an Ask bounded by timeout, returns immediately,
+// and later delivers wrap(result) to self as an ordinary message. The caller
+// handles the reply in a later turn, so its state is only ever touched inside
+// turns and its goroutine never waits on another actor. This is the
+// pipe-to-self pattern: where Future.Await inside Receive can deadlock when
+// the callee is, transitively, waiting on the caller, AskThen leaves the
+// caller free to serve the callee in the meantime.
+//
+// AskThen never blocks the calling goroutine, whichever goroutine that is.
+// Called from inside a receive turn, the request goes through the in-turn
+// send path and spills into the target's overflow queue if its mailbox is
+// full. Called from any other goroutine, the request is a TryAsk, so a full
+// mailbox is reported as a failed result rather than waited on.
+//
+// Whatever happens to the Ask, wrap is applied to exactly one result: the
+// callee's reply, the error the callee returned, a send failure such as
+// ErrMailboxFull, ErrMailboxOverflow or ErrActorTerminated, or
+// context.DeadlineExceeded once timeout elapses without a reply.
+//
+// Completion is best effort and detached, but never silently lost. The
+// wrapped result is delivered to self at most once. While self is alive the
+// result is retained and delivery is retried until self's mailbox has room,
+// so a full mailbox delays it rather than dropping it. The only way it is
+// dropped is that self has stopped (or its mailbox is closed), and that is
+// explicit: the drop is logged, the message goes to self's dead letter office
+// if it has one, and any WithDropHandler is called. A timeout abandons the
+// wait, not the request: the callee may still process the message, and its
+// late reply is discarded. A caller that needs a durable outcome must not
+// rely on the reply, or its error, always arriving. A behavior that records a
+// pending request should give that entry its own expiry and reconcile
+// against the callee's state if it is never completed.
+//
+// A timeout that is zero or negative is already expired. No message is sent
+// and wrap receives context.DeadlineExceeded, delivered like any other
+// result, because there is no sensible unbounded default and an Ask that may
+// never be answered is the failure this helper exists to avoid.
+//
+// A typical use takes two turns. In the first, the behavior sends the request
+// and records that it is pending; in the second, it handles the wrapped
+// result:
+//
+// case *startMsg:
+// // Turn one: ask, and remember we are waiting.
+// b.pending = true
+// actor.AskThen(
+// ctx, b.wallet, &queryBalance{}, b.self, 5*time.Second,
+// func(r fn.Result[Balance]) Msg {
+// return &balanceReply{res: r}
+// },
+// )
+//
+// case *balanceReply:
+// // Turn two: the reply arrives as a message.
+// b.pending = false
+// b.apply(m.res)
+//
+// The wrap function must be pure. It runs on a helper goroutine, not on the
+// actor's, so it may only map the result into a message and must not read or
+// write the caller's state.
+//
+// The Ask's lifetime is detached from ctx and bounded only by timeout, so it
+// outlives the turn. The reply is delivered to self with a context that
+// carries ctx's values but neither its cancellation nor its turn marker, as
+// the delivery runs on the helper goroutine and not on the actor's. It is
+// therefore never an in-turn send: it polls TryTell from the helper
+// goroutine, which never stalls the actors involved.
+func AskThen[M Message, R any, S Message](ctx context.Context,
+ ref ActorRef[M, R], msg M, self TellOnlyRef[S], timeout time.Duration,
+ wrap func(fn.Result[R]) S, opts ...AskThenOption) {
+
+ var o askThenOptions
+ for _, opt := range opts {
+ opt(&o)
+ }
+
+ // The delivery context must not be the Ask context, which is cancelled
+ // on a timeout and would fail the delivery at once. It is derived from
+ // the same base so the logger and other values survive.
+ base := context.WithoutCancel(ctx)
+ deliverCtx := WithoutTurn(base)
+
+ deliver := func(res fn.Result[R]) {
+ out := wrap(res)
+
+ wait := askThenMinRetry
+ for {
+ err := self.TryTell(deliverCtx, out)
+ switch {
+ case err == nil:
+ return
+
+ // Self is alive but busy, so keep the result and try
+ // again once it has had time to make room.
+ case errors.Is(err, ErrMailboxFull):
+ time.Sleep(wait)
+ wait = min(wait*2, askThenMaxRetry)
+
+ // Self is gone and nothing can handle the result. Say
+ // so rather than discarding it quietly.
+ default:
+ log.Warnf("AskThen: result for %s dropped: %v",
+ self.ID(), err)
+
+ if o.onDrop != nil {
+ o.onDrop(err)
+ }
+
+ return
+ }
+ }
+ }
+
+ if timeout <= 0 {
+ go deliver(fn.Err[R](context.DeadlineExceeded))
+
+ return
+ }
+
+ askCtx, cancel := context.WithTimeout(base, timeout)
+
+ // From inside a turn the Ask can't park, as it takes the in-turn path
+ // of the target's mailbox. From any other goroutine it would, so use
+ // the non-blocking form instead.
+ var fut Future[R]
+ if _, inTurn := activeTurn(ctx); inTurn {
+ fut = ref.Ask(askCtx, msg)
+ } else {
+ fut = ref.TryAsk(askCtx, msg)
+ }
+
+ fut.OnComplete(askCtx, func(res fn.Result[R]) {
+ // The wait is over, so release the timer before a possibly slow
+ // delivery.
+ cancel()
+
+ deliver(res)
+ })
+}
### actor/ask_then_test.go
@@ -0,0 +1,299 @@
+package actor
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "sync"
+ "sync/atomic"
+ "testing"
+ "testing/synctest"
+ "time"
+
+ "github.com/lightningnetwork/lnd/fn/v2"
+ "github.com/stretchr/testify/require"
+)
+
+// wrapReply maps an AskThen result into a message the collector can inspect.
+func wrapReply(res fn.Result[string]) *testMsg {
+ val, err := res.Unpack()
+ if err != nil {
+ return newTestMsg("err:" + err.Error())
+ }
+
+ return newTestMsg("reply:" + val)
+}
+
+// collectBehavior records every message it receives on a channel.
+type collectBehavior struct {
+ got chan string
+}
+
+// newCollectBehavior creates a collector with room for n messages.
+func newCollectBehavior(n int) *collectBehavior {
+ return &collectBehavior{got: make(chan string, n)}
+}
+
+// Receive implements ActorBehavior.
+func (c *collectBehavior) Receive(_ context.Context,
+ msg *testMsg) fn.Result[string] {
+
+ c.got <- msg.data
+
+ return fn.Ok("collected")
+}
+
+// slowBehavior never answers until the actor stops.
+type slowBehavior struct{}
+
+// Receive implements ActorBehavior.
+func (slowBehavior) Receive(actorCtx context.Context,
+ _ *testMsg) fn.Result[string] {
+
+ <-actorCtx.Done()
+
+ return fn.Ok("late")
+}
+
+// TestAskThenDeliversReply checks that the callee's reply arrives at self as
+// a wrapped message.
+func TestAskThenDeliversReply(t *testing.T) {
+ h := newActorTestHarness(t)
+ callee := h.newActor("ask-then-callee", newEchoBehavior(t, 0), 4)
+ collector := newCollectBehavior(1)
+ self := h.newActor("ask-then-self", collector, 4)
+
+ AskThen(
+ t.Context(), callee.Ref(), newTestMsg("hi"),
+ self.Ref(), time.Second, wrapReply,
+ )
+
+ select {
+ case got := <-collector.got:
+ require.Equal(t, "reply:echo: hi", got)
+ case <-time.After(5 * time.Second):
+ t.Fatal("no reply delivered")
+ }
+}
+
+// TestAskThenTimeout checks that a callee that never answers yields a wrapped
+// deadline error, and that a non-positive timeout is already expired.
+func TestAskThenTimeout(t *testing.T) {
+ h := newActorTestHarness(t)
+ callee := h.newActor("ask-then-slow", slowBehavior{}, 4)
+ collector := newCollectBehavior(2)
+ self := h.newActor("ask-then-timeout-self", collector, 4)
+
+ ctx := t.Context()
+ AskThen(
+ ctx, callee.Ref(), newTestMsg("a"), self.Ref(),
+ 20*time.Millisecond, wrapReply,
+ )
+ AskThen(
+ ctx, callee.Ref(), newTestMsg("b"), self.Ref(), 0, wrapReply,
+ )
+
+ for range 2 {
+ select {
+ case got := <-collector.got:
+ require.Equal(
+ t, "err:"+context.DeadlineExceeded.Error(), got,
+ )
+ case <-time.After(5 * time.Second):
+ t.Fatal("no timeout delivered")
+ }
+ }
+}
+
+// TestAskThenNeverBlocksFromPlainGoroutine checks that against a full
+// mailbox, AskThen from a goroutine that isn't an actor returns at once and
+// reports ErrMailboxFull as the wrapped result.
+func TestAskThenNeverBlocksFromPlainGoroutine(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ callee, _ := newWedgedActor(t)
+ defer callee.Stop()
+
+ collector := newCollectBehavior(1)
+ self, err := NewActor(ActorConfig[*testMsg, string]{
+ ID: "ask-then-plain-self", Behavior: collector,
+ MailboxSize: 4,
+ })
+ require.NoError(t, err)
+ self.Start()
+ defer self.Stop()
+
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+
+ AskThen(
+ t.Context(), callee.Ref(),
+ newTestMsg("x"), self.Ref(), time.Hour,
+ wrapReply,
+ )
+ }()
+
+ synctest.Wait()
+
+ select {
+ case <-done:
+ default:
+ t.Fatal("AskThen blocked on a full mailbox")
+ }
+
+ got := <-collector.got
+ require.True(t, strings.HasSuffix(got, ErrMailboxFull.Error()))
+ })
+}
+
+// askThenCycle asks peer count questions from its receive turn on "start",
+// and counts the replies that come back to it as messages.
+type askThenCycle struct {
+ peer ActorRef[*testMsg, string]
+ self TellOnlyRef[*testMsg]
+ count int
+
+ mu sync.Mutex
+ replies int
+ done chan struct{}
+}
+
+// Receive implements ActorBehavior.
+func (c *askThenCycle) Receive(ctx context.Context,
+ msg *testMsg) fn.Result[string] {
+
+ switch {
+ case msg.data == "start":
+ for range c.count {
+ AskThen(
+ ctx, c.peer, newTestMsg("q"), c.self,
+ 10*time.Second, wrapReply,
+ )
+ }
+
+ case strings.HasPrefix(msg.data, "reply:"):
+ c.mu.Lock()
+ c.replies++
+ if c.replies == c.count {
+ close(c.done)
+ }
+ c.mu.Unlock()
+ }
+
+ return fn.Ok("ok")
+}
+
+// TestAskThenInTurnDoesNotDeadlock has two actors with one slot mailboxes
+// ask each other a burst of questions from their turns. Blocking Asks and
+// Awaits would wedge both. AskThen lets every question and reply through.
+func TestAskThenInTurnDoesNotDeadlock(t *testing.T) {
+ const count = 100
+
+ a := &askThenCycle{count: count, done: make(chan struct{})}
+ b := &askThenCycle{count: count, done: make(chan struct{})}
+
+ h := newActorTestHarness(t)
+ actorA := h.newActor("ask-then-cycle-a", a, 1)
+ actorB := h.newActor("ask-then-cycle-b", b, 1)
+ a.peer, a.self = actorB.Ref(), actorA.Ref()
+ b.peer, b.self = actorA.Ref(), actorB.Ref()
+
+ ctx := t.Context()
+ actorA.Ref().Tell(ctx, newTestMsg("start"))
+ actorB.Ref().Tell(ctx, newTestMsg("start"))
+
+ for _, beh := range []*askThenCycle{a, b} {
+ select {
+ case <-beh.done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("actors deadlocked")
+ }
+ }
+}
+
+// TestAskThenRetainsResultWhileSelfFull checks that a result for a busy self
+// is delayed rather than dropped, and delivered once there is room.
+func TestAskThenRetainsResultWhileSelfFull(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ h := newActorTestHarness(t)
+ callee := h.newActor("retain-callee", newEchoBehavior(t, 0), 4)
+
+ selfBeh := &countWedge{wedgeBehavior: newWedgeBehavior()}
+ self, err := NewActor(ActorConfig[*testMsg, string]{
+ ID: "retain-self", Behavior: selfBeh, MailboxSize: 1,
+ })
+ require.NoError(t, err)
+ self.Start()
+
+ ctx := t.Context()
+ self.Ref().Tell(ctx, newTestMsg("a"))
+ <-selfBeh.started
+ require.NoError(t, self.Ref().TryTell(ctx, newTestMsg("b")))
+
+ AskThen(
+ ctx, callee.Ref(), newTestMsg("q"), self.Ref(),
+ time.Hour, wrapReply,
+ )
+
+ // Self stays full for a long while. The result must survive.
+ time.Sleep(10 * time.Second)
+ synctest.Wait()
+ require.Equal(t, 1, len(chanMailbox(t, self).ch))
+
+ close(selfBeh.release)
+
+ // Give the retry loop time to see the room that opened up.
+ time.Sleep(time.Second)
+ synctest.Wait()
+
+ // The retained result arrived as the third message.
+ require.EqualValues(t, 3, selfBeh.n.Load())
+
+ self.Stop()
+ h.dlo.Stop()
+ callee.Stop()
+ })
+}
+
+// TestAskThenDropIsObservable checks that when self has stopped, the dropped
+// result is reported through the drop handler instead of vanishing.
+func TestAskThenDropIsObservable(t *testing.T) {
+ h := newActorTestHarness(t)
+ callee := h.newActor("drop-callee", newEchoBehavior(t, 0), 4)
+ self := h.newActor("drop-self", newCollectBehavior(1), 4)
+
+ ctx := t.Context()
+ self.Stop()
+
+ dropped := make(chan error, 1)
+ AskThen(
+ ctx, callee.Ref(), newTestMsg("q"), self.Ref(), time.Second,
+ wrapReply, WithDropHandler(func(err error) {
+ dropped <- err
+ }),
+ )
+
+ select {
+ case err := <-dropped:
+ require.True(t, errors.Is(err, ErrActorTerminated))
+ case <-time.After(5 * time.Second):
+ t.Fatal("drop was not reported")
+ }
+}
+
+// countWedge is a wedgeBehavior that counts the messages it has handled.
+type countWedge struct {
+ *wedgeBehavior
+
+ n atomic.Int32
+}
+
+// Receive implements ActorBehavior.
+func (c *countWedge) Receive(actorCtx context.Context,
+ msg *testMsg) fn.Result[string] {
+
+ res := c.wedgeBehavior.Receive(actorCtx, msg)
+ c.n.Add(1)
+
+ return res
+}Why this scored 24/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.