actor: never park sends made from inside a receive turn
What changed, and why it matters
This commit fixes a design flaw in LND's internal actor framework where an actor could freeze itself by sending a message to another actor whose mailbox was full. If several actors did this in a circle, the whole group could deadlock. The fix marks messages sent during an actor's own processing turn as 'in-turn' and routes them through a special non-blocking overflow queue, so the actor never stops moving. It also preserves message order and adds a hard cap so runaway message floods fail loudly instead of consuming unlimited memory.
Review the new lock ordering (mu then overflowMu) and the Close path to ensure no inversion or missed wakeups. Audit existing behaviors that spawn goroutines and pass the receive context, confirming they use WithoutTurn where appropriate. Run the new overflow and cycle tests under the race detector. Consider whether 100,000 default overflow limit is appropriate for production node memory constraints.
Security signals we found
Deadlock elimination in actor message-passing core
Bounded overflow queue prevents unbounded memory growth under overload
New error path ErrMailboxOverflow for explicit failure instead of silent blocking
Context turn marker could be misused if behavior passes turn context to a long-lived goroutine without WithoutTurn
Lock ordering and close-path changes in mailbox.go require careful concurrency review
Evidence from the diff
The patch modifies the actor package’s ChannelMailbox to add an in-turn send path. The actor runtime now stamps the context passed to a behavior’s Receive handler with a turn marker (beginTurn/endTurn). Sends made with that active turn context use SendInTurn, which writes to the channel if space is available and the overflow is empty, otherwise appends to a bounded overflow slice. The receive loop refills the channel from overflow oldest-first. External blocking Send waits for overflowEmpty before writing, preserving acceptance-order delivery. TrySend reports full when overflow is non-empty. Overflow is capped by ActorConfig.MailboxOverflowLimit (default 100,000); exceeding it returns ErrMailboxOverflow. A new WithoutTurn helper lets spawned goroutines opt out of in-turn treatment. Tests cover deadlock cycles, ordering, cap behavior, drain, and context expiration.
Changed components
actor/actor.goactor/mailbox.goactor/turn.goactor/overflow_test.goInspect captured patch +748 / −6
### actor/actor.go
@@ -35,6 +35,13 @@ type ActorConfig[M Message, R any] struct {
// MailboxFactory is an optional factory for creating the actor's
// mailbox. If nil, a default ChannelMailbox will be used.
MailboxFactory MailboxFactory[M, R]
+
+ // MailboxOverflowLimit is the hard cap on the default mailbox's
+ // overflow queue, which absorbs sends made from inside a receive turn
+ // when the mailbox is full. A send that would exceed it fails with
+ // ErrMailboxOverflow. Zero selects DefaultMailboxOverflowLimit. It has
+ // no effect when MailboxFactory is set.
+ MailboxOverflowLimit int
}
// envelope wraps a message with its associated promise. This allows the sender
@@ -117,7 +124,11 @@ func NewActor[M Message, R any](cfg ActorConfig[M, R]) (*Actor[M, R],
if cfg.MailboxFactory != nil {
mailbox = cfg.MailboxFactory(ctx, mailboxCapacity)
} else {
- mailbox = NewChannelMailbox[M, R](ctx, mailboxCapacity)
+ mailbox = NewChannelMailbox[M, R](
+ ctx, mailboxCapacity,
+ WithMailboxID(cfg.ID),
+ WithOverflowLimit(cfg.MailboxOverflowLimit),
+ )
}
actor := &Actor[M, R]{
@@ -152,7 +163,11 @@ func (a *Actor[M, R]) Start() {
func (a *Actor[M, R]) process() {
// Use the new iterator pattern for receiving messages.
for env := range a.mailbox.Receive(a.ctx) {
- result := a.behavior.Receive(a.ctx, env.message)
+ // Mark the context as this actor's receive turn, so sends the
+ // behavior makes with it never park on a full mailbox.
+ turnCtx, endTurn := beginTurn(a.ctx, a.id)
+ result := a.behavior.Receive(turnCtx, env.message)
+ endTurn()
// If a promise was provided (i.e., it was an "ask"
// operation), complete the promise with the result from
@@ -210,6 +225,24 @@ func (ref *actorRefImpl[M, R]) Tell(ctx context.Context, msg M) {
env := envelope[M, R]{message: msg, promise: nil}
+ // A send made from inside a receive turn must not park the sending
+ // actor, so it takes the mailbox's non-blocking in-turn path.
+ if sender, ok := ref.inTurnSender(ctx); ok {
+ if err := sender.SendInTurn(env); err != nil {
+ if ref.actor.ctx.Err() != nil {
+ ref.trySendToDLO(msg)
+ return
+ }
+
+ // Tell has no way to report the failure, so make it
+ // loud. A sender that needs the signal can use TryTell.
+ log.Errorf("Actor %s: in-turn Tell of %s dropped: %v",
+ ref.actor.id, msg.MessageType(), err)
+ }
+
+ return
+ }
+
// Use mailbox Send method which internally checks both contexts.
if !ref.actor.mailbox.Send(ctx, env) {
// Failed to send - check if actor terminated.
@@ -249,6 +282,19 @@ func (ref *actorRefImpl[M, R]) Ask(ctx context.Context, msg M) Future[R] {
env := envelope[M, R]{message: msg, promise: promise}
+ // As with Tell, a send from inside a receive turn never parks.
+ if sender, ok := ref.inTurnSender(ctx); ok {
+ if err := sender.SendInTurn(env); err != nil {
+ if ref.actor.ctx.Err() != nil {
+ err = ErrActorTerminated
+ }
+
+ promise.Complete(fn.Err[R](err))
+ }
+
+ return promise.Future()
+ }
+
// Use mailbox Send method which internally checks both contexts.
if !ref.actor.mailbox.Send(ctx, env) {
// Determine the error based on what failed.
@@ -269,6 +315,20 @@ func (ref *actorRefImpl[M, R]) Ask(ctx context.Context, msg M) Future[R] {
return promise.Future()
}
+// inTurnSender returns the mailbox's non-blocking send path if ctx belongs to
+// an active receive turn and the mailbox supports one.
+func (ref *actorRefImpl[M, R]) inTurnSender(
+ ctx context.Context) (inTurnSender[M, R], bool) {
+
+ if _, ok := activeTurn(ctx); !ok {
+ return nil, false
+ }
+
+ sender, ok := ref.actor.mailbox.(inTurnSender[M, R])
+
+ return sender, ok
+}
+
// TryTell enqueues a message only if the mailbox can take it immediately,
// returning ErrMailboxFull rather than waiting for room. Unlike Tell it never
// blocks, whichever goroutine calls it.
### actor/mailbox.go
@@ -2,6 +2,8 @@ package actor
import (
"context"
+ "errors"
+ "fmt"
"iter"
"sync"
"sync/atomic"
@@ -36,7 +38,59 @@ type Mailbox[M Message, R any] interface {
Drain() iter.Seq[envelope[M, R]]
}
+// ErrMailboxOverflow is returned by a send made from inside a receive turn
+// when the mailbox's overflow queue is already at its hard cap. The send had
+// no effect: nothing was enqueued and the mailbox state is unchanged, so the
+// caller may retry once the receiver has drained below the cap. The error
+// returned by the mailbox wraps this value with the mailbox ID, so test for it
+// with errors.Is.
+var ErrMailboxOverflow = errors.New("mailbox overflow limit reached")
+
+// DefaultMailboxOverflowLimit is the default hard cap on envelopes queued in a
+// ChannelMailbox's overflow queue by sends made from inside a receive turn.
+const DefaultMailboxOverflowLimit = 100_000
+
+// ChannelMailboxOption configures optional ChannelMailbox behavior.
+type ChannelMailboxOption func(*mailboxOptions)
+
+// mailboxOptions collects the optional settings of a ChannelMailbox.
+type mailboxOptions struct {
+ overflowLimit int
+ id string
+}
+
+// WithOverflowLimit sets the hard cap on the in-turn overflow queue. A send
+// that would exceed it fails with ErrMailboxOverflow. A value of zero or less
+// selects DefaultMailboxOverflowLimit.
+func WithOverflowLimit(limit int) ChannelMailboxOption {
+ return func(o *mailboxOptions) {
+ o.overflowLimit = limit
+ }
+}
+
+// WithMailboxID sets the identifier the mailbox includes in its logs and
+// errors. It is normally the owning actor's ID.
+func WithMailboxID(id string) ChannelMailboxOption {
+ return func(o *mailboxOptions) {
+ o.id = id
+ }
+}
+
+// inTurnSender is implemented by mailboxes that can accept a send from an
+// actor's own receive turn without ever blocking it. The actor ref prefers it
+// over Send when the send context belongs to an active turn.
+type inTurnSender[M Message, R any] interface {
+ // SendInTurn enqueues the envelope without blocking, or fails with an
+ // error wrapping ErrMailboxOverflow, or ErrMailboxClosed.
+ SendInTurn(env envelope[M, R]) error
+}
+
// ChannelMailbox is a channel-based implementation of the Mailbox interface.
+//
+// Sends made from inside an actor's receive turn never park. When the channel
+// is full they spill into a bounded overflow queue that the receive loop
+// refills the channel from, so a cycle of actors that send to each other from
+// their turns can't deadlock on full mailboxes.
type ChannelMailbox[M Message, R any] struct {
ch chan envelope[M, R]
closed atomic.Bool
@@ -52,27 +106,242 @@ type ChannelMailbox[M Message, R any] struct {
//
//nolint:containedctx
actorCtx context.Context
+
+ // id names the mailbox in logs and errors.
+ id string
+
+ // overflowMu guards the overflow queue and the episode bookkeeping
+ // below. Lock order is always mu (read side) first, then overflowMu.
+ overflowMu sync.Mutex
+
+ // overflow holds envelopes sent from inside a receive turn that did
+ // not fit in ch, oldest first.
+ overflow []envelope[M, R]
+
+ // overflowEmpty is closed when the overflow drains back to empty. It
+ // is replaced each time the overflow goes from empty to non-empty, and
+ // is only meaningful while the overflow is non-empty. Send waits on it
+ // so that it can't overtake envelopes already accepted into the
+ // overflow.
+ overflowEmpty chan struct{}
+
+ // overflowLimit is the hard cap on len(overflow).
+ overflowLimit int
+
+ // overflowCapLogged is set once the hard cap has been reported in the
+ // current overflow episode, so a sender that keeps retrying against a
+ // full queue doesn't log per message. An episode ends when the
+ // overflow drains back to empty.
+ overflowCapLogged bool
+
+ // overflowWarned is set once the soft threshold has been logged for
+ // the current overflow episode.
+ overflowWarned bool
}
// NewChannelMailbox creates a new channel-based mailbox with the specified
// buffer capacity and actor context.
func NewChannelMailbox[M Message, R any](actorCtx context.Context,
- capacity int) *ChannelMailbox[M, R] {
+ capacity int, opts ...ChannelMailboxOption) *ChannelMailbox[M, R] {
if capacity <= 0 {
capacity = 1
}
+ var o mailboxOptions
+ for _, opt := range opts {
+ opt(&o)
+ }
+ if o.overflowLimit <= 0 {
+ o.overflowLimit = DefaultMailboxOverflowLimit
+ }
+
return &ChannelMailbox[M, R]{
- ch: make(chan envelope[M, R], capacity),
- actorCtx: actorCtx,
+ ch: make(chan envelope[M, R], capacity),
+ actorCtx: actorCtx,
+ id: o.id,
+ overflowLimit: o.overflowLimit,
+ }
+}
+
+// OverflowLen returns the number of envelopes currently parked in the
+// overflow queue behind the channel.
+func (m *ChannelMailbox[M, R]) OverflowLen() int {
+ m.overflowMu.Lock()
+ defer m.overflowMu.Unlock()
+
+ return len(m.overflow)
+}
+
+// popOverflowLocked removes and returns the oldest overflow envelope. The
+// caller must hold overflowMu, or be the single draining goroutine once the
+// mailbox is closed, and must have checked the queue is non-empty. Resetting
+// the episode state when the queue empties lives here so every consumer of
+// the queue shares it.
+func (m *ChannelMailbox[M, R]) popOverflowLocked() envelope[M, R] {
+ env := m.overflow[0]
+
+ // Zero the slot so the envelope can be collected, then reslice past it.
+ m.overflow[0] = envelope[M, R]{}
+ m.overflow = m.overflow[1:]
+
+ if len(m.overflow) == 0 {
+ m.overflow = nil
+ close(m.overflowEmpty)
+
+ if m.overflowWarned {
+ log.Infof("Mailbox %q: overflow drained", m.id)
+ }
+ m.overflowWarned = false
+ m.overflowCapLogged = false
+ }
+
+ return env
+}
+
+// SendInTurn is the in-turn send path. It never blocks: the envelope goes
+// straight into the channel when there is room and nothing is queued ahead of
+// it, and is otherwise appended to the overflow queue.
+//
+// Appending whenever the overflow is non-empty, even if the channel has since
+// gained room, keeps a single sender's messages in order: a later message
+// must not jump ahead of an earlier one still sitting in the overflow.
+//
+// If the queue would exceed the hard cap the send is rejected with an error
+// wrapping ErrMailboxOverflow and nothing is enqueued. Returning an error
+// rather than parking is the point: parking an actor on a full mailbox is how
+// a cycle of actors deadlocks.
+func (m *ChannelMailbox[M, R]) SendInTurn(env envelope[M, R]) error {
+ // Lock order: mu (read side) then overflowMu. The read lock is the
+ // close guard that makes the channel send below safe.
+ m.mu.RLock()
+ defer m.mu.RUnlock()
+
+ m.overflowMu.Lock()
+ defer m.overflowMu.Unlock()
+
+ if m.IsClosed() {
+ return ErrMailboxClosed
+ }
+
+ queued := len(m.overflow)
+ if queued == 0 {
+ select {
+ case m.ch <- env:
+ return nil
+
+ default:
+ }
+ }
+
+ if queued+1 > m.overflowLimit {
+ if !m.overflowCapLogged {
+ m.overflowCapLogged = true
+
+ log.Errorf("Mailbox %q: overflow limit of %d reached, "+
+ "rejecting in-turn sends (msg_type=%s)", m.id,
+ m.overflowLimit, env.message.MessageType())
+ }
+
+ return fmt.Errorf("%w: mailbox %q at %d queued messages",
+ ErrMailboxOverflow, m.id, m.overflowLimit)
+ }
+
+ if queued == 0 {
+ m.overflowEmpty = make(chan struct{})
+ }
+
+ m.overflow = append(m.overflow, env)
+ queued++
+
+ softLimit := max(m.overflowLimit/10, 1)
+ if queued >= softLimit && !m.overflowWarned {
+ m.overflowWarned = true
+
+ log.Warnf("Mailbox %q: overflow growing (len=%d, limit=%d)",
+ m.id, queued, m.overflowLimit)
+ }
+
+ return nil
+}
+
+// refillFromOverflow moves queued overflow envelopes into the channel, oldest
+// first, for as long as the channel has room. Only the receive loop calls it,
+// right after taking an envelope off the channel.
+//
+// This maintains the invariant that a non-empty overflow implies a non-empty
+// channel, so the receiver never sleeps on an empty channel while overflow
+// envelopes wait. After a refill either the overflow is empty or a
+// non-blocking send just failed, meaning the channel was full. Nothing but
+// the receiver removes envelopes from the channel, so it stays non-empty until
+// the receiver's next receive, which refills again. If the receiver frees a
+// slot between an in-turn sender's failed channel send and its append, the
+// refill needs overflowMu, so it runs after the append and sees the new
+// envelope.
+//
+// Fairness caveat: Go hands a freed slot straight to a sender parked on the
+// full channel, as part of the receive itself. So while external producers
+// stay parked on this mailbox, refill finds the channel full and the overflow
+// doesn't drain. Under sustained external saturation in-turn traffic keeps
+// accumulating until the hard cap rejects it with ErrMailboxOverflow, which
+// trades a slow mailbox for a loud failure under overload.
+func (m *ChannelMailbox[M, R]) refillFromOverflow() {
+ // This deliberately doesn't take mu. A Send parked on a full channel
+ // holds mu's read side while it waits for this very receiver, and a
+ // pending Close queues behind it, so taking the read side here would
+ // block new readers behind that writer and deadlock the receiver.
+ // Close closes the channel while holding overflowMu, which makes the
+ // closed check below a sufficient guard.
+ m.overflowMu.Lock()
+ defer m.overflowMu.Unlock()
+
+ // After Close the channel is closed, so sending would panic. Drain
+ // picks up whatever is left in the overflow.
+ if m.IsClosed() {
+ return
+ }
+
+ for len(m.overflow) > 0 {
+ select {
+ case m.ch <- m.overflow[0]:
+ m.popOverflowLocked()
+
+ default:
+ return
+ }
}
}
// Send implements Mailbox.Send with context-aware blocking send.
+//
+// Delivery order is acceptance order: a Send that starts after another send
+// has been accepted is delivered after it. Envelopes in the overflow queue
+// have been accepted but trail the channel, so Send first waits for the
+// overflow to drain rather than writing to the channel ahead of them. This
+// is what lets an owner enqueue a shutdown or barrier message from outside
+// the actor and know it runs after everything already accepted. The wait is
+// bounded by ctx and the actor's lifetime, like the send itself.
func (m *ChannelMailbox[M, R]) Send(ctx context.Context,
env envelope[M, R]) bool {
+ for {
+ m.overflowMu.Lock()
+ if len(m.overflow) == 0 {
+ m.overflowMu.Unlock()
+ break
+ }
+ drained := m.overflowEmpty
+ m.overflowMu.Unlock()
+
+ select {
+ case <-drained:
+ case <-ctx.Done():
+ return false
+ case <-m.actorCtx.Done():
+ return false
+ }
+ }
+
m.mu.RLock()
defer m.mu.RUnlock()
@@ -100,6 +369,16 @@ func (m *ChannelMailbox[M, R]) TrySend(env envelope[M, R]) bool {
return false
}
+ // Hold overflowMu across the check and the send. Reporting full while
+ // the overflow is non-empty keeps a TrySend from overtaking an earlier
+ // in-turn send from the same sender, which is still queued there.
+ m.overflowMu.Lock()
+ defer m.overflowMu.Unlock()
+
+ if len(m.overflow) > 0 {
+ return false
+ }
+
select {
case m.ch <- env:
return true
@@ -120,6 +399,11 @@ func (m *ChannelMailbox[M, R]) Receive(
return
}
+ // Top the channel back up from the overflow
+ // before running the behavior, which may take
+ // arbitrarily long.
+ m.refillFromOverflow()
+
if !yield(env) {
return
}
@@ -140,6 +424,10 @@ func (m *ChannelMailbox[M, R]) Close() {
m.mu.Lock()
defer m.mu.Unlock()
+ // Close the channel under overflowMu, see refillFromOverflow.
+ m.overflowMu.Lock()
+ defer m.overflowMu.Unlock()
+
m.closed.Store(true)
close(m.ch)
@@ -160,19 +448,33 @@ func (m *ChannelMailbox[M, R]) Drain() iter.Seq[envelope[M, R]] {
}
// Drain all remaining messages from the channel.
+ chanLoop:
for {
select {
case env, ok := <-m.ch:
// Channel closed, nothing left to drain.
if !ok {
- return
+ break chanLoop
}
if !yield(env) {
return
}
default:
// Channel empty, done draining.
+ break chanLoop
+ }
+ }
+
+ // Then the overflow, which always trails the channel, so send
+ // order is preserved. Close set closed under the write lock
+ // and every writer checks it under the read lock, so nothing
+ // touches the queue now. Drain runs on the actor loop right
+ // after Close, which makes it the queue's only user.
+ for len(m.overflow) > 0 {
+ env := m.popOverflowLocked()
+
+ if !yield(env) {
return
}
}
### actor/overflow_test.go
@@ -0,0 +1,295 @@
+package actor
+
+import (
+ "context"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/lightningnetwork/lnd/fn/v2"
+ "github.com/stretchr/testify/require"
+)
+
+// overflowEnv builds a tell envelope carrying the given value.
+func overflowEnv(v int) envelope[TestMessage, int] {
+ return envelope[TestMessage, int]{message: TestMessage{Value: v}}
+}
+
+// TestChannelMailboxInTurnOverflow checks that in-turn sends spill into the
+// overflow queue when the channel is full, that the receive loop refills the
+// channel from it, and that order is preserved throughout.
+func TestChannelMailboxInTurnOverflow(t *testing.T) {
+ ctx := t.Context()
+ mbox := NewChannelMailbox[TestMessage, int](ctx, 2)
+
+ // Two fit in the channel, the other three overflow.
+ for i := range 5 {
+ require.NoError(t, mbox.SendInTurn(overflowEnv(i)))
+ }
+ require.Equal(t, 3, mbox.OverflowLen())
+
+ // Receiving yields every message in send order, and the overflow
+ // drains as the channel frees up.
+ recvCtx, cancel := context.WithCancel(ctx)
+ defer cancel()
+
+ var got []int
+ for env := range mbox.Receive(recvCtx) {
+ got = append(got, env.message.Value)
+ if len(got) == 5 {
+ break
+ }
+ }
+
+ require.Equal(t, []int{0, 1, 2, 3, 4}, got)
+ require.Zero(t, mbox.OverflowLen())
+}
+
+// TestChannelMailboxOverflowCap checks that a send past the cap fails with
+// ErrMailboxOverflow, leaves the mailbox unchanged, and succeeds again once
+// the receiver has made room.
+func TestChannelMailboxOverflowCap(t *testing.T) {
+ ctx := t.Context()
+ mbox := NewChannelMailbox[TestMessage, int](
+ ctx, 1, WithOverflowLimit(2), WithMailboxID("capped"),
+ )
+
+ // One in the channel, two in the overflow.
+ for i := range 3 {
+ require.NoError(t, mbox.SendInTurn(overflowEnv(i)))
+ }
+
+ err := mbox.SendInTurn(overflowEnv(3))
+ require.ErrorIs(t, err, ErrMailboxOverflow)
+ require.ErrorContains(t, err, "capped")
+ require.Equal(t, 2, mbox.OverflowLen())
+
+ // Take one message off, which refills the channel from the overflow
+ // and leaves room under the cap.
+ recvCtx, cancel := context.WithCancel(ctx)
+ defer cancel()
+ for env := range mbox.Receive(recvCtx) {
+ require.Equal(t, 0, env.message.Value)
+ break
+ }
+
+ require.NoError(t, mbox.SendInTurn(overflowEnv(3)))
+}
+
+// TestChannelMailboxTrySendBehindOverflow checks that TrySend reports full
+// while the overflow is non-empty, so it can't overtake a queued in-turn send.
+func TestChannelMailboxTrySendBehindOverflow(t *testing.T) {
+ mbox := NewChannelMailbox[TestMessage, int](t.Context(), 1)
+
+ require.NoError(t, mbox.SendInTurn(overflowEnv(0)))
+ require.NoError(t, mbox.SendInTurn(overflowEnv(1)))
+ require.Equal(t, 1, mbox.OverflowLen())
+
+ require.False(t, mbox.TrySend(overflowEnv(2)))
+}
+
+// TestChannelMailboxDrainOverflow checks that after Close, Drain yields the
+// channel's envelopes followed by the overflow, in send order.
+func TestChannelMailboxDrainOverflow(t *testing.T) {
+ mbox := NewChannelMailbox[TestMessage, int](t.Context(), 2)
+
+ for i := range 5 {
+ require.NoError(t, mbox.SendInTurn(overflowEnv(i)))
+ }
+
+ mbox.Close()
+ require.ErrorIs(t, mbox.SendInTurn(overflowEnv(9)), ErrMailboxClosed)
+
+ var got []int
+ for env := range mbox.Drain() {
+ got = append(got, env.message.Value)
+ }
+
+ require.Equal(t, []int{0, 1, 2, 3, 4}, got)
+ require.Zero(t, mbox.OverflowLen())
+}
+
+// turnSendBehavior sends count messages to peer from its receive turn on a
+// "start" message, and records the number of "work" messages it receives.
+type turnSendBehavior struct {
+ peer ActorRef[*testMsg, string]
+ count int
+
+ mu sync.Mutex
+ received int
+ done chan struct{}
+}
+
+// Receive implements ActorBehavior.
+func (b *turnSendBehavior) Receive(ctx context.Context,
+ msg *testMsg) fn.Result[string] {
+
+ if msg.data == "start" {
+ for range b.count {
+ b.peer.Tell(ctx, newTestMsg("work"))
+ }
+
+ return fn.Ok("started")
+ }
+
+ b.mu.Lock()
+ defer b.mu.Unlock()
+
+ b.received++
+ if b.received == b.count {
+ close(b.done)
+ }
+
+ return fn.Ok("worked")
+}
+
+// TestInTurnTellDoesNotDeadlockCycle sets up two actors with one slot
+// mailboxes that each flood the other from their receive turns. With parking
+// sends both would wedge on the other's full mailbox. Spilling to the
+// overflow lets every message through.
+func TestInTurnTellDoesNotDeadlockCycle(t *testing.T) {
+ const count = 200
+
+ a := &turnSendBehavior{count: count, done: make(chan struct{})}
+ b := &turnSendBehavior{count: count, done: make(chan struct{})}
+
+ h := newActorTestHarness(t)
+ actorA := h.newActor("cycle-a", a, 1)
+ actorB := h.newActor("cycle-b", b, 1)
+ a.peer = actorB.Ref()
+ b.peer = actorA.Ref()
+
+ ctx := t.Context()
+ actorA.Ref().Tell(ctx, newTestMsg("start"))
+ actorB.Ref().Tell(ctx, newTestMsg("start"))
+
+ for _, beh := range []*turnSendBehavior{a, b} {
+ select {
+ case <-beh.done:
+ case <-time.After(10 * time.Second):
+ t.Fatal("actors deadlocked on full mailboxes")
+ }
+ }
+}
+
+// TestInTurnAskOverflowCap checks that an Ask made from a turn past the
+// overflow cap completes its Future with ErrMailboxOverflow instead of
+// parking, and that the cap is configurable through ActorConfig.
+func TestInTurnAskOverflowCap(t *testing.T) {
+ beh := newWedgeBehavior()
+ a, err := NewActor(ActorConfig[*testMsg, string]{
+ ID: "overflow-ask",
+ Behavior: beh,
+ MailboxSize: 1,
+ MailboxOverflowLimit: 1,
+ })
+ require.NoError(t, err)
+ a.Start()
+ t.Cleanup(a.Stop)
+
+ ref := a.Ref()
+ bg := t.Context()
+
+ // Wedge the actor and fill its single slot.
+ ref.Tell(bg, newTestMsg("a"))
+ <-beh.started
+ require.NoError(t, ref.TryTell(bg, newTestMsg("b")))
+
+ turnCtx, end := WithTurnForTest(bg, "sender")
+ defer end()
+
+ // One overflow slot absorbs the first ask, the second hits the cap.
+ _ = ref.Ask(turnCtx, newTestMsg("c"))
+
+ res := ref.Ask(turnCtx, newTestMsg("d")).Await(bg)
+ _, err = res.Unpack()
+ require.ErrorIs(t, err, ErrMailboxOverflow)
+
+ // A TryTell never uses the overflow, so it reports full.
+ err = ref.TryTell(turnCtx, newTestMsg("e"))
+ require.ErrorIs(t, err, ErrMailboxFull)
+}
+
+// TestEndedTurnContextStillParks checks that a context whose turn has ended
+// is treated like any other producer's, so it blocks on a full mailbox and
+// honors its own deadline rather than spilling into the overflow.
+func TestEndedTurnContextStillParks(t *testing.T) {
+ beh := newWedgeBehavior()
+ a, err := NewActor(ActorConfig[*testMsg, string]{
+ ID: "ended-turn",
+ Behavior: beh,
+ MailboxSize: 1,
+ })
+ require.NoError(t, err)
+ a.Start()
+ t.Cleanup(a.Stop)
+
+ ref := a.Ref()
+ bg := t.Context()
+
+ ref.Tell(bg, newTestMsg("a"))
+ <-beh.started
+ require.NoError(t, ref.TryTell(bg, newTestMsg("b")))
+
+ turnCtx, end := WithTurnForTest(bg, "sender")
+ end()
+
+ ctx, cancel := context.WithTimeout(turnCtx, 50*time.Millisecond)
+ defer cancel()
+
+ _, err = ref.Ask(ctx, newTestMsg("c")).Await(bg).Unpack()
+ require.ErrorIs(t, err, context.DeadlineExceeded)
+
+ require.Zero(t, chanMailbox(t, a).OverflowLen())
+}
+
+// TestExternalSendKeepsOrderBehindOverflow checks that a barrier message
+// sent from outside the actor with an ordinary Tell is delivered after the
+// in-turn messages already accepted into the overflow queue.
+func TestExternalSendKeepsOrderBehindOverflow(t *testing.T) {
+ mbox := NewChannelMailbox[TestMessage, int](t.Context(), 1)
+
+ // Channel holds 0, the overflow holds 1 and 2.
+ for i := range 3 {
+ require.NoError(t, mbox.SendInTurn(overflowEnv(i)))
+ }
+
+ sent := make(chan bool, 1)
+ go func() {
+ sent <- mbox.Send(t.Context(), overflowEnv(99))
+ }()
+
+ // The barrier must not be accepted while earlier messages wait.
+ select {
+ case <-sent:
+ t.Fatal("barrier overtook queued messages")
+ case <-time.After(50 * time.Millisecond):
+ }
+
+ var got []int
+ recvCtx, cancel := context.WithCancel(t.Context())
+ defer cancel()
+ for env := range mbox.Receive(recvCtx) {
+ got = append(got, env.message.Value)
+ if len(got) == 4 {
+ break
+ }
+ }
+
+ require.True(t, <-sent)
+ require.Equal(t, []int{0, 1, 2, 99}, got)
+}
+
+// TestExternalSendWaitHonorsContext checks that a Send waiting behind the
+// overflow gives up when its context does, and enqueues nothing.
+func TestExternalSendWaitHonorsContext(t *testing.T) {
+ mbox := NewChannelMailbox[TestMessage, int](t.Context(), 1)
+ require.NoError(t, mbox.SendInTurn(overflowEnv(0)))
+ require.NoError(t, mbox.SendInTurn(overflowEnv(1)))
+
+ ctx, cancel := context.WithTimeout(t.Context(), 20*time.Millisecond)
+ defer cancel()
+
+ require.False(t, mbox.Send(ctx, overflowEnv(2)))
+ require.Equal(t, 1, mbox.OverflowLen())
+}
### actor/turn.go
@@ -0,0 +1,85 @@
+package actor
+
+import (
+ "context"
+ "sync/atomic"
+)
+
+// turnKey is the context key under which the runtime records the receive turn
+// a context belongs to.
+type turnKey struct{}
+
+// turn identifies one in-flight receive turn of an actor. The runtime stamps
+// it onto the context it hands to the behavior, so a send made with that
+// context can tell that it is running on an actor's own goroutine, where
+// parking would stall the actor's whole mailbox.
+type turn struct {
+ // actorID is the ID of the actor processing the turn.
+ actorID string
+
+ // active is cleared when the turn returns. A goroutine the behavior
+ // spawned keeps the context, and with it this marker, after the turn
+ // is over; such a goroutine is not the actor's own and may park like
+ // any other producer.
+ active atomic.Bool
+}
+
+// beginTurn returns a context marked as the receive turn of the given actor,
+// along with the function that ends the turn. The caller must invoke the end
+// function once the behavior returns. A context that already carries a turn
+// is shadowed, since only the innermost turn is the one running on this
+// goroutine.
+func beginTurn(ctx context.Context, actorID string) (context.Context, func()) {
+ t := &turn{actorID: actorID}
+ t.active.Store(true)
+
+ return context.WithValue(ctx, turnKey{}, t), func() {
+ t.active.Store(false)
+ }
+}
+
+// activeTurn returns the receive turn ctx belongs to, if that turn is still
+// running.
+func activeTurn(ctx context.Context) (*turn, bool) {
+ t, ok := ctx.Value(turnKey{}).(*turn)
+ if !ok || t == nil || !t.active.Load() {
+ return nil, false
+ }
+
+ return t, true
+}
+
+// TurnActor reports the ID of the actor whose receive turn ctx belongs to. The
+// bool is false outside a turn, and for a context that outlived the turn it
+// was created in.
+func TurnActor(ctx context.Context) (string, bool) {
+ t, ok := activeTurn(ctx)
+ if !ok {
+ return "", false
+ }
+
+ return t.actorID, true
+}
+
+// WithoutTurn returns ctx with its receive turn marker cleared, so a send made
+// with the result is no longer treated as running on the actor's own
+// goroutine. It is for handing a context to a goroutine that outlives the
+// turn and must not be treated as the actor. All other values and the
+// cancellation of ctx are kept. A context with no turn is returned unchanged.
+func WithoutTurn(ctx context.Context) context.Context {
+ if _, ok := ctx.Value(turnKey{}).(*turn); !ok {
+ return ctx
+ }
+
+ return context.WithValue(ctx, turnKey{}, (*turn)(nil))
+}
+
+// WithTurnForTest marks ctx as the receive turn of the given actor, as the
+// runtime does before invoking a behavior. It exists so tests outside this
+// package can exercise turn-sensitive paths without standing up an actor. The
+// returned function ends the turn.
+func WithTurnForTest(ctx context.Context,
+ actorID string) (context.Context, func()) {
+
+ return beginTurn(ctx, actorID)
+}Why this scored 53/100
Community notes
Notes can correct, qualify, or add evidence to the AI analysis. Every note shown here has been validated by a human moderator.
The AI analysis stands alone for now. Submit a note if you can add evidence or important context.