Reading paths: Start below for the plain-language explanation, or jump to the technical deep dive for implementation details and verification.
A queue holds work until something is ready to process it. Adding another queue can help connect systems, but it also creates another place where the state of a job can differ.
My Claude history includes a Redis-removal and queue-consolidation plan. The available index does not prove the migration outcome. This article explains the design questions that make that kind of change important.
Give each component a clear job
Imagine a restaurant with an order book and a bell. The book records the order. The bell tells the kitchen that work is waiting.
Trouble starts if the bell system and the order book both independently decide whether an order is complete.
Software can use a database to record durable job state and a message queue to notify workers. That is a valid design when their responsibilities and recovery rules are clear.
Messages can arrive more than once
A worker might complete an action, then lose its connection before acknowledging the message. The queue may deliver that message again.
The worker needs a way to recognize already-completed work or safely repeat it. This property is called idempotency: repeating a request has the same intended effect as performing it once.
Some actions need a unique request identifier enforced by the destination. Other actions need a check of the external result before retrying.
A simpler diagram can hide a difficult migration
Removing a queue means deciding what happens to messages already waiting or running.
A migration plan should answer:
- Where do new jobs enter during the transition?
- Who owns jobs already in progress?
- How are duplicates recognized?
- How are stranded jobs found and recovered?
- What conditions trigger a rollback?
Changing the architecture without settling those questions can leave work in neither system or execute it in both.
Separate the job from its message
A job is the logical work someone requested. A message is one way to tell a worker about that work. A job may outlive several messages or execution attempts.
This distinction helps when a notification is lost or delivered twice. The worker can look up the job's current state rather than assuming every received message represents new work.
It also gives the system a way to recover jobs that were recorded but never dispatched. A reconciliation process can find eligible work without a current owner and arrange another notification, following the system's recovery rules.
The notification mechanism and the durable record then have complementary responsibilities rather than competing definitions of completion.
Identify the dangerous gaps
Consider the sequence: save a job, publish its message, execute the action, record the result, acknowledge the message. A process can fail between any two steps.
If saving succeeds but publishing fails, the job exists without a worker knowing about it. If the action succeeds but recording fails, the next attempt may be uncertain whether to repeat the action.
These gaps are more useful to discuss than a vague promise of reliable delivery. Draw each one and define how the system detects and recovers from it.
Some designs record an outgoing notification alongside the job in the same database transaction, then publish it separately. That can address one gap, but it does not make every external side effect automatically safe to repeat.
Claim work before doing it
When several workers receive the same logical job, only the intended owner should proceed. A claim operation needs to check eligibility and assign ownership together according to the storage system's guarantees.
Reading pending and later writing running as unrelated operations can leave a race: two workers may both read pending before either writes its claim.
A temporary ownership lease can help detect abandoned work, but it needs rules for renewal, expiration, and stale workers. A late worker must not keep changing the job after a replacement has legitimately taken over.
These are design requirements, not features guaranteed merely by using a queue product.
Idempotency is about the actual effect
Suppose a worker sends an email and crashes before saving that fact. Marking the job pending again does not undo the email. The next attempt may send a duplicate.
A local check may help, but it cannot always prove the external effect did not happen. Where supported, a destination-enforced request key can let repeated requests refer to the same operation. Otherwise, recovery may need to inspect the destination or involve a person when the result is uncertain.
Define the duplicate behavior for each consequential action. A general instruction to "make the worker idempotent" is not enough unless the implementation addresses the place where the irreversible effect actually occurs.
Keep failed work inspectable
After a bounded number of unsuccessful attempts, a job may be set aside for review. That should preserve its input identity, attempts, useful errors, and current external-state uncertainty.
Setting work aside is not the same as resolving it. Operators need a way to see why it stopped and whether it can be retried safely after the underlying issue changes.
Avoid a retry button that forgets earlier effects or creates an unrelated duplicate job with no trace back to the original request. Recovery should preserve the logical history so the team can explain what happened and count the work accurately.
Migrate with an inventory
Before replacing a queue path, count work by state: waiting, owned, running, completed, and uncertain. Decide which system remains authoritative during each migration phase.
Stop or redirect new intake in a controlled way, then handle existing work according to its state. In-flight work may need to finish under its current owner rather than being copied into the new system immediately.
Record the mapping for moved jobs so a late message from the old system can be recognized. Afterward, reconcile the inventory rather than relying only on a successful deployment message. The deployment proves code changed; it does not prove every preexisting job arrived safely at its intended state.
Observe the user-visible backlog
A queue length can shrink because work completed, expired, failed, or moved elsewhere. The falling number alone does not prove improved delivery.
Track outcomes and the age of the oldest eligible item. Distinguish waiting work from work currently owned by a worker, and distinguish business requests from individual attempts. Those choices determine whether an operator can tell if the system is keeping up.
When several queues remain necessary, explain how their counts relate. Adding their lengths may double-count the same logical work if one queue is a notification channel for jobs represented in another store.
Practice interruption in a controlled environment
Tests should interrupt work at the handoff gaps, deliver a message twice, and let an old worker report after a new attempt begins. Check the external effect as well as the job status.
These scenarios make the architecture's promises concrete. They also reveal whether a simplification actually reduced the number of states the team must reason about, or merely moved the complexity behind a different component name.
Check useful invariants
An invariant is a rule that must remain true. For example: a completed job should not be restarted just because an old notification arrives.
Write a few such rules and test interruption around the handoff points. Observe both the message queue and the durable job record.
The goal is not automatically one technology or one queue. It is a clear answer to who owns the work and how the system recovers when delivery and execution disagree.
Technical deep dive
Plain-language bridge: A message tells a worker that work may exist. A durable job record decides what that work is and whether someone already owns it.
Diagram: A notification is not ownership
[accept request]
|
v
[transaction: job + outbox]
|
v
[publisher] ----> [queue message]
|
v
[durable job record] <---- atomic claim -- [worker]
^ |
| v
+--- fenced completion ------- [perform work]
The message points to a durable job. The claim grants ownership, and the generation check rejects stale completion writes; external effects need their own protection.
Separate job identity from delivery identity
An illustrative model uses one logical job ID and many possible message deliveries. The job carries its current state, claim generation, lease expiry, and result reference. The message carries the job ID and enough routing information to find the record.
An at-least-once delivery system can deliver the same message more than once. A worker must therefore check and claim the job rather than interpreting delivery as exclusive ownership. Message acknowledgments and durable job completion have different meanings.
Close the acceptance/publication gap
If the application writes a job and then publishes a message, a crash between those steps can leave a durable job that no worker hears about. Publishing first can produce a message for a job whose database transaction never committed.
One possible design commits the job and an outbox entry in the same database transaction. A publisher delivers outbox entries and marks them sent. It may publish twice if it crashes after sending but before recording success, so consumers still need deduplication.
The outbox solves a local atomicity gap; it does not create universal exactly-once execution across external systems.
Use leases with fencing where necessary
A lease allows recovery if a worker disappears. Expiry alone cannot stop a paused old worker from resuming after a new worker takes over. A fencing generation, a monotonically increasing ownership number, lets writes reject the stale worker.
worker A claims generation 4
A pauses; lease expires
worker B claims generation 5
A resumes and submits a result with generation 4
result store accepts only the currently owned generation
Every relevant write boundary must enforce the generation for this protection to hold. For an external effect that cannot enforce fencing, use provider-supported idempotency or another explicit reconciliation strategy. A local lease check immediately before the call does not eliminate a race during the call.
Migrate ownership without two active authorities
During queue consolidation, identify which component may claim new work at each phase. Shadow observation can compare state, but two independent claim authorities can both decide they own the same job. Freeze or gate the old claim path before handing over, with a defined policy for already-running work.
Test crashes after job commit, publication, claim, external effect, and completion recording. Verify that every accepted job remains discoverable and that stale workers cannot overwrite newer ownership. These are design checks for a migration; the available Claude index does not establish that such a migration completed.