Merge pull request #11465 from Roasbeef/actor-drain-overflow-race
What changed, and why it matters
This patch fixes a concurrency bug in LND's new actor mailbox system. When an actor was shutting down, the cleanup routine that drains leftover messages could read the internal overflow queue at the same time another goroutine was still trying to send a message or check the queue length. The fix makes the drain routine hold the same lock as senders while it pops messages, preventing a data race. The patch also adds regression tests and a release note describing the race.
Apply the patch and run the actor package tests with the Go race detector enabled (-race) to confirm the regression test passes. Review other mailbox methods for similar assumptions about Close serializing access without explicit locks.
Security signals we found
Data race on shared queue during shutdown drain
Missing lock while mutating overflow slice in Drain
Concurrent Send/OverflowLen still active after Close
Potential memory corruption or message loss due to unsynchronized slice access
Regression tests added for race and stranded-sender scenarios
Evidence from the diff
In actor/mailbox.go, ChannelMailbox.Drain() previously iterated over m.overflow without holding overflowMu, relying on the assumption that after Close() no other goroutine would touch the queue. However, Send’s wait loop and OverflowLen acquire overflowMu and inspect the queue before observing the closed flag, so Drain could pop the slice while another goroutine read it, causing a Go race-detector violation and potential memory corruption or lost messages. The patch changes Drain to lock overflowMu, check length, pop one envelope, unlock, yield it, and repeat. A new test, TestChannelMailboxDrainRacesSend, reproduces the race with concurrent OverflowLen/Send calls after Close. Another test, TestTellParkedBehindOverflowReturnsOnStop, verifies that a Tell parked behind a non-empty overflow is not stranded by Stop and that post-termination Tell returns promptly.
Changed components
actor/mailbox.goactor/overflow_test.godocs/release-notes/unreleased/11431.mdInspect captured patch +162 / −11
### actor/mailbox.go
@@ -174,10 +174,9 @@ func (m *ChannelMailbox[M, R]) OverflowLen() int {
}
// popOverflowLocked removes and returns the oldest overflow envelope. The
-// caller must hold overflowMu, or be the single draining goroutine once the
-// mailbox is closed, and must have checked the queue is non-empty. Resetting
-// the episode state when the queue empties lives here so every consumer of
-// the queue shares it.
+// caller must hold overflowMu and must have checked the queue is non-empty.
+// Resetting the episode state when the queue empties lives here so every
+// consumer of the queue shares it.
func (m *ChannelMailbox[M, R]) popOverflowLocked() envelope[M, R] {
env := m.overflow[0]
@@ -467,12 +466,20 @@ func (m *ChannelMailbox[M, R]) Drain() iter.Seq[envelope[M, R]] {
}
// Then the overflow, which always trails the channel, so send
- // order is preserved. Close set closed under the write lock
- // and every writer checks it under the read lock, so nothing
- // touches the queue now. Drain runs on the actor loop right
- // after Close, which makes it the queue's only user.
- for len(m.overflow) > 0 {
+ // order is preserved. Close does not stop other goroutines from
+ // reading the queue: Send's wait loop and OverflowLen take
+ // overflowMu and look at it before they see the closed flag. So
+ // each pop holds overflowMu, but yield runs outside it, as the
+ // consumer may block or call back into the mailbox.
+ for {
+ m.overflowMu.Lock()
+ if len(m.overflow) == 0 {
+ m.overflowMu.Unlock()
+ return
+ }
+
env := m.popOverflowLocked()
+ m.overflowMu.Unlock()
if !yield(env) {
return
### actor/overflow_test.go
@@ -293,3 +293,145 @@ func TestExternalSendWaitHonorsContext(t *testing.T) {
require.False(t, mbox.Send(ctx, overflowEnv(2)))
require.Equal(t, 1, mbox.OverflowLen())
}
+
+// TestChannelMailboxDrainRacesSend checks that Drain pops the overflow under
+// overflowMu. A Send or OverflowLen call can still arrive after Close, and
+// both read the queue under that lock, so an unlocked pop in Drain is a data
+// race that the race detector reports.
+func TestChannelMailboxDrainRacesSend(t *testing.T) {
+ const count = 2000
+
+ mbox := NewChannelMailbox[TestMessage, int](t.Context(), 1)
+
+ // One envelope sits in the channel and the rest overflow.
+ for i := range count {
+ require.NoError(t, mbox.SendInTurn(overflowEnv(i)))
+ }
+ require.Equal(t, count-1, mbox.OverflowLen())
+
+ mbox.Close()
+
+ var (
+ drained = make(chan struct{})
+ finished = make(chan struct{})
+ )
+ go func() {
+ defer close(finished)
+
+ for {
+ select {
+ case <-drained:
+ return
+ default:
+ }
+
+ _ = mbox.OverflowLen()
+
+ ctx, cancel := context.WithTimeout(
+ t.Context(), time.Millisecond,
+ )
+ _ = mbox.Send(ctx, overflowEnv(-1))
+ cancel()
+ }
+ }()
+
+ var got []int
+ for env := range mbox.Drain() {
+ got = append(got, env.message.Value)
+ }
+ close(drained)
+ <-finished
+
+ want := make([]int, count)
+ for i := range want {
+ want[i] = i
+ }
+ require.Equal(t, want, got)
+ require.Zero(t, mbox.OverflowLen())
+}
+
+// gateBehavior floods its own mailbox from its receive turn on a "start"
+// message, which fills the overflow queue, and then blocks on gate.
+type gateBehavior struct {
+ self ActorRef[*testMsg, string]
+ count int
+ gate chan struct{}
+ started chan struct{}
+}
+
+// Receive implements ActorBehavior.
+func (b *gateBehavior) Receive(ctx context.Context,
+ msg *testMsg) fn.Result[string] {
+
+ if msg.data == "start" {
+ for range b.count {
+ b.self.Tell(ctx, newTestMsg("work"))
+ }
+
+ close(b.started)
+ <-b.gate
+ }
+
+ return fn.Ok("ok")
+}
+
+// TestTellParkedBehindOverflowReturnsOnStop checks that shutdown cannot
+// strand a sender that is parked behind a non-empty overflow, and that a Tell
+// after the actor has terminated returns promptly.
+func TestTellParkedBehindOverflowReturnsOnStop(t *testing.T) {
+ beh := &gateBehavior{
+ count: 50,
+ gate: make(chan struct{}),
+ started: make(chan struct{}),
+ }
+
+ h := newActorTestHarness(t)
+ a := h.newActor("parked-tell", beh, 1)
+ beh.self = a.Ref()
+
+ a.Ref().Tell(t.Context(), newTestMsg("start"))
+ <-beh.started
+
+ // The overflow is non-empty, so an external Tell parks.
+ returned := make(chan struct{})
+ go func() {
+ defer close(returned)
+
+ a.Ref().Tell(t.Context(), newTestMsg("external"))
+ }()
+
+ select {
+ case <-returned:
+ t.Fatal("Tell returned while the overflow was non-empty")
+ case <-time.After(50 * time.Millisecond):
+ }
+
+ a.Stop()
+ close(beh.gate)
+
+ select {
+ case <-returned:
+ case <-time.After(2 * time.Second):
+ t.Fatal("parked Tell was stranded by Stop")
+ }
+
+ select {
+ case <-a.Done():
+ case <-time.After(2 * time.Second):
+ t.Fatal("actor did not terminate")
+ }
+
+ // A Tell after termination must not block.
+ late := make(chan struct{})
+ go func() {
+ defer close(late)
+
+ a.Ref().Tell(t.Context(), newTestMsg("late"))
+ }()
+
+ select {
+ case <-late:
+ case <-time.After(2 * time.Second):
+ t.Fatal("Tell after Done blocked")
+ }
+}
### docs/release-notes/unreleased/11431.md
@@ -10,8 +10,10 @@
full mailbox: they spill into a bounded overflow queue, configurable with
`ActorConfig.MailboxOverflowLimit`, and fail with `ErrMailboxOverflow` at the
cap. A blocking `Send` from outside the actor now waits for that overflow to
- drain so it can't overtake accepted messages. `AskThen` delivers an `Ask`
- reply to the caller as a later message, and `Actor.Done`,
+ drain so it can't overtake accepted messages, and draining the overflow on
+ shutdown [no longer races with such a
+ `Send`](https://github.com/lightningnetwork/lnd/pull/11465). `AskThen`
+ delivers an `Ask` reply to the caller as a later message, and `Actor.Done`,
`Actor.StopAndWait` and `ActorSystem.ShutdownAndWait` let an owner wait for
the last `Receive` to return before closing resources. The `actor` module
now requires `lnd/queue` v1.2.1.Why this scored 44/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.