When the Message Queue Became Our Enemy
Three years ago, our payment processing system was drowning in race conditions. Orders would complete successfully but inventory would still show as available. Payments would process twice for the same transaction. Our message queue, which we’d proudly built to handle “eventual consistency,” had become a breeding ground for edge cases that only surfaced during peak traffic.
The problem wasn’t our technology stack. It was our mental model. We were thinking about distributed systems as a collection of services passing messages, when we should have been thinking about them as state machines managing transitions. This shift in perspective led us to discover a pattern that most teams overlook: the Saga pattern with explicit state modeling.
Why Traditional Event Sourcing Falls Short
Event sourcing gets a lot of attention in distributed systems circles, and for good reason. It provides an audit trail, enables replay capabilities, and makes debugging easier. But pure event sourcing has a dirty secret: it pushes complexity into the event handlers. When you have dozens of microservices each maintaining their own event projections, you end up with a web of implicit dependencies that’s nearly impossible to reason about.
Consider a typical e-commerce flow: OrderCreated, PaymentProcessed, InventoryReserved, ShippingScheduled. Each service listens for relevant events and updates its state. But what happens when PaymentProcessed arrives before InventoryReserved? Or when the shipping service is down for maintenance? You need compensation logic, retry mechanisms, and careful ordering guarantees. The cognitive load becomes overwhelming.
The Saga pattern addresses this by making state transitions explicit and centralized. Instead of hoping that distributed event handlers will eventually reach consistency, you model the entire business process as a finite state machine with well-defined transitions and compensation actions.
The Choreography vs Orchestration Decision
Most teams gravitate toward choreographed sagas because they feel more “microservices native.” Each service publishes events when it completes its work, and other services react accordingly. No central coordinator, no single point of failure. It sounds elegant until you try to implement error handling.
We tried choreography first. When our payment service failed after inventory had been reserved, we needed the inventory service to listen for PaymentFailed events and automatically release the reservation. When shipping failed after payment succeeded, we needed compensation logic in both the payment and inventory services. The complexity exploded exponentially with each new service we added to the flow.
Orchestrated sagas, on the other hand, use a central coordinator that explicitly manages the workflow. Yes, it’s another moving part. But it’s also the only place where you need to understand the complete business process. When things go wrong, you have exactly one place to look. Our saga orchestrator became a 400-line state machine that replaced thousands of lines of distributed compensation logic scattered across services.
Implementation Patterns That Actually Work in Production
The devil is in the implementation details. Our saga orchestrator runs as a separate service that persists its state to a database with strong consistency guarantees. Each saga instance gets identified by a correlation ID that flows through all related messages. When a step completes successfully, the orchestrator transitions to the next state and sends the appropriate command. When a step fails, it executes the compensation sequence.
The key insight is treating the orchestrator itself as a stateful service, not just a message router. We use a simple state table with columns for saga_id, current_state, payload, and created_at. State transitions are atomic database operations. This gives us durability, exactly-once processing semantics, and the ability to resume sagas after orchestrator restarts.
For monitoring, we emit metrics at each state transition and maintain a dashboard showing active sagas by state. The longest-running saga in each state becomes an early warning system for service degradation. When payments start taking longer than usual, we see it in our saga metrics before customers start complaining.
The Patterns Hiding in Your Current Architecture
You’re probably already using distributed state machines without realizing it. Every retry policy is a state machine. Every circuit breaker is a state machine. Every deployment pipeline is a state machine. The difference is whether you’re modeling them explicitly or letting them emerge organically from your code.
Look at your current error handling. How many places in your codebase have logic that checks “if this fails, then do that, but only if we’re in this particular state”? Those are state transitions waiting to be formalized. The purchase order that can be pending, confirmed, shipped, or cancelled. The user onboarding flow that moves through email verification, profile setup, and initial configuration. The CI/CD pipeline that progresses through build, test, deploy, and verification stages.
Making these state machines explicit doesn’t require a complete rewrite. Start with your most painful distributed workflow. Model its states and transitions on a whiteboard. Identify the current implicit states hiding in your conditional logic. Then gradually migrate toward explicit state management, one transition at a time.
I’m not saying this to add complexity. It’s about making the complexity you already have visible and manageable. Once you start seeing your distributed systems as collections of communicating state machines, patterns that seemed impossible become obvious. And the next time someone suggests eventually consistent event sourcing as the solution to everything, you’ll know when to push back.