Worker Queue

The pattern is:

1. Encode the logical work key into the subject.
2. Set MaxMsgsPerSubject: 1.
3. Use DiscardNew.
4. Enable DiscardNewPerSubject.

For example:

work.reconcile.foo-123
work.reconcile.foo-456

Then configure the stream roughly like:

jetstream.StreamConfig{
    Name:                 "WORK",
    Subjects:             []string{"work.reconcile.*"},
    Retention:            jetstream.WorkQueuePolicy,
    MaxMsgsPerSubject:    1,
    Discard:              jetstream.DiscardNew,
    DiscardNewPerSubject: true,
}

The behavior becomes:

publish work.reconcile.foo-123
    → accepted

publish work.reconcile.foo-123 again
    → rejected because one message already exists for that subject

worker ACKs first message
    → WorkQueuePolicy removes it from the stream

publish work.reconcile.foo-123 again
    → accepted

This is much closer to:

“Only one outstanding job per logical key.”

than the duplicate-window mechanism.

DiscardNewPerSubject is important. It applies the discard-new behavior independently to each subject rather than treating the stream-wide message limit as the relevant boundary. NATS maintainers explicitly describe DiscardNewPerSubject with one message per subject as a way to implement key-based exclusion where removing the message releases the key. (GitHub)

The key difference

Nats-Msg-Id deduplication

Identity: Nats-Msg-Id header
Lifetime: fixed duplicate window
Removal: timer expiration

It protects primarily against uncertain publisher retries.

One message per subject

Identity: logical key encoded in subject
Lifetime: while the message exists in the stream
Removal: ACK/removal, expiration, or explicit deletion

This tracks the lifecycle you actually described.

The duplicate tracking table does not clear an ID when its corresponding message leaves a WorkQueue stream. NATS maintainers have confirmed that this separation is intentional: deduplication state expires according to the duplicate window, independently of ACKs or stream-message removal. (GitHub)

Important limitation: publish rejection must be observed

Use a JetStream publish call and inspect the returned error:

ack, err := js.Publish(
    ctx,
    "work.reconcile.foo-123",
    payload,
)
if err != nil {
    // Could mean this subject already has its one allowed message, so you need to check it before declaring this a true failure.
    return err
}

_ = ack

Do not use an unacknowledged Core NATS publish for this workflow, because the publisher needs the JetStream response to know whether the new message was accepted or rejected. JetStream publish calls provide that storage acknowledgement. (NATS Documentation)

One caveat

This guarantees only one stored message per subject. During processing, that message remains in the stream until it is acknowledged, so another publish for the same key remains rejected. That is useful here.

However, JetStream can still redeliver that same message if AckWait expires. Thus two workers could temporarily process the same stream message concurrently unless you use an adequate AckWait and InProgress() heartbeats. But this is a known problem with distributed systems since we can't know for sure if the remote worker process is dead, slow, etc (possibly in theory but not practical). Hence it is acceptable to have a timeout and just retry the job anyway.

So this solves:

two independently published jobs for foo-123

It does not fully eliminate:

one job redelivered while the original worker is still processing, but neither do other solutions like locking with timeouts