Concurrency, Coroutines & Resilience Patterns Masterclass: Asyncio, Trio, uvloop, Circuit Breaker & Idempotency
Total Questions: 16 Curriculum: Concurrency Primitives & Event Loop Engine (Q1–Q4), Modern Async Runtimes: asyncio, uvloop, Trio & AnyIO (Q5–Q8), Distributed Resilience: Backoff, Jitter & Circuit Breakers (Q9–Q12), Production Idempotency with FastAPI & Redis (Q13–Q16)
Overview
Modern backend engineering requires deep mastery over asynchronous programming and distributed fault tolerance. When building distributed systems, network partitions, intermittent service failures, and duplicate network retries are inevitable realities.
This masterclass dissects the mechanics of Python concurrency (coroutines, event loops, and structured concurrency), evaluates the four major runtime libraries (asyncio, uvloop, Trio, and AnyIO), details mission-critical resilience patterns (Exponential Backoff with Jitter, Circuit Breaker state machines), and provides a battle-tested FastAPI distributed idempotency engine backed by Redis.
Table of Contents
- Concurrency Primitives & Event Loop Engine (Q1–Q4)
- Modern Async Runtimes: asyncio, uvloop, Trio & AnyIO (Q5–Q8)
- Distributed Resilience: Backoff, Jitter & Circuit Breakers (Q9–Q12)
- Production Idempotency with FastAPI & Redis (Q13–Q16)
Concurrency Primitives & Event Loop Engine
Q1: What is the fundamental difference between Concurrency and Parallelism in operating systems and Python runtime?
Level: Intermediate / Senior
Technical Answer:
- Concurrency: Concurrency is about dealing with lots of things at once. It is an architectural structure where multiple tasks make interleaved progress on a single core by yielding control during idle or I/O wait periods (cooperative multitasking).
- Parallelism: Parallelism is about doing lots of things at once. It requires multiple physical CPU cores executing instructions simultaneously.
In CPython, the Global Interpreter Lock (GIL) prevents multiple native OS threads from executing Python bytecode in parallel. However, since most backend applications spend the majority of their time waiting for network packets (database queries, external APIs, disk I/O), concurrency allows a single CPU core to switch to other tasks while waiting for I/O completion.
Analogy & Workflow:
[Concurrency (1 Cook)]
Puts water on boil ──► Switches to chopping vegetables ──► Checks boiling water
(Single worker switching tasks smartly)
[Parallelism (2 Cooks)]
Cook A: Boils water
Cook B: Chops vegetables
(Multiple physical workers executing instructions at the exact same instant)
Q2: How does a Python Coroutine differ from a standard subroutine (function), and how does the await keyword pause and resume execution?
Level: Senior
Technical Answer: A standard function (subroutine) has a single entry point and exit point: once invoked, it executes to completion or until an unhandled exception occurs, blocking the thread.
A Coroutine is a specialized function that can pause execution at specific points without blocking the underlying thread, relinquishing control back to the scheduler, and resuming later with its full execution context (local variables, instruction pointer) preserved.
Mechanics of await:
- A coroutine must be declared with
async def. - When the execution reaches
await <awaitable>, the coroutine yields control back to the event loop. - The event loop registers a callback for when the awaited I/O operation (socket read, timer) completes.
- While the awaited operation is in-flight, the event loop runs other pending tasks.
- Once the OS signals that data is ready (via
epoll/kqueue), the event loop places the coroutine back on the runnable queue and resumes execution immediately after theawaitexpression.
Code:
import asyncio
import time
# Synchronous subroutine (Thread freezes during I/O)
def sync_worker():
time.sleep(2) # OS thread is blocked; CPU cannot do anything else
# Asynchronous coroutine (Yields control back to event loop)
async def async_worker(task_id: int):
print(f"Task {task_id}: Beginning I/O operation")
await asyncio.sleep(2) # Pauses here, event loop schedules other coroutines
print(f"Task {task_id}: Resumed and finished")
Q3: What is the Event Loop, and what is the internal lifecycle of task scheduling, I/O multiplexing, and callback execution?
Level: Senior / Principal
Technical Answer:
The Event Loop is the central orchestrator of an asynchronous runtime. It runs in a continuous loop, managing a ready queue of runnable callbacks and polling OS kernel notification systems (epoll on Linux, kqueue on macOS) for I/O readiness.
The 5-Phase Event Loop Iteration:
- Poll for I/O: Calls
epoll_waitwith a calculated timeout based on the earliest scheduled timer. - Process I/O Events: For every socket ready to read or write, find its associated Future and mark it as done, moving its callback to the ready queue.
- Process Timers: Check expired timers (
call_later,call_at) and enqueue their callbacks. - Execute Ready Callbacks: Dequeue and execute ready callbacks sequentially until the queue is empty.
- Repeat: Check if the loop should terminate or proceed to the next iteration.
Workflow Diagram:
┌────────────────────────────────────────────────────────┐
│ THE EVENT LOOP │
│ │
│ ┌──────────────────┐ ┌───────────────────┐ │
│ │ Ready Queue │ ◄──────── │ OS I/O Events │ │
│ │ (Runnable Tasks)│ │ (epoll / kqueue) │ │
│ └────────┬─────────┘ └─────────▲─────────┘ │
│ │ │ │
│ ▼ Execute │ Poll │
│ ┌──────────────────┐ ┌─────────┴─────────┐ │
│ │ Coroutine Frame │ ──await──►│ Register Socket / │ │
│ │ (Runs bytecode) │ │ Timer with OS │ │
│ └──────────────────┘ └───────────────────┘ │
└────────────────────────────────────────────────────────┘
Q4: Why can't __init__ be asynchronous in Python, and how does the Async Factory Pattern resolve asynchronous object initialization?
Level: Senior
Technical Answer:
In Python, __init__ must always return None (Q2 from Object Internals). If you define async def __init__(self):, calling MyClass() returns a coroutine object instead of the initialized instance, violating the CPython object creation protocol (type.__call__).
The Solution: Async Factory Pattern
Keep __init__ pure, synchronous, and focused strictly on storing pre-resolved dependencies. Provide a @classmethod async def create(...) factory method that executes asynchronous setup (establishing database connections, reading remote configuration) and returns a fully initialized instance.
Code:
import asyncio
class AsyncDatabasePool:
def __init__(self, raw_connection):
# Synchronous __init__: purely stores already-resolved resources
self.connection = raw_connection
self.is_connected = True
@classmethod
async def create(cls, dsn: str, pool_size: int = 10) -> "AsyncDatabasePool":
"""Asynchronous factory method that handles all network I/O."""
print(f"Connecting to {dsn} with pool_size={pool_size}...")
# Simulate asynchronous TCP handshake and authentication
await asyncio.sleep(0.5)
raw_conn = f"TCP_SOCKET({dsn})"
instance = cls(raw_conn)
return instance
async def execute(self, query: str):
if not self.is_connected:
raise RuntimeError("Database pool disconnected")
return f"Executed '{query}' on {self.connection}"
# Usage:
async def main():
# pool = AsyncDatabasePool("...") # WRONG: Cannot await here
pool = await AsyncDatabasePool.create("postgresql://prod-db:5432/main")
print(await pool.execute("SELECT 1;"))
asyncio.run(main())
Modern Async Runtimes: asyncio, uvloop, Trio & AnyIO
Q5: How does uvloop achieve a 2–4x performance boost over standard asyncio, and what are its trade-offs?
Level: Senior
Technical Answer:
Standard asyncio's default event loop is implemented entirely in pure Python (asyncio.base_events). Python bytecode interpretation for scheduling queues and callback management adds non-trivial CPU overhead.
How uvloop Works:
uvloop is a drop-in replacement written in Cython that binds directly to libuv—the battle-tested, high-performance C library that powers Node.js.
- It bypasses Python bytecode for the core scheduling loop, socket polling, and timer management.
- It delivers 2x–4x higher throughput for I/O-intensive network servers, often matching Go and Node.js in HTTP/TCP micro-benchmarks.
Trade-offs:
- Compilation Requirement: Must compile C extensions during installation (
pip install uvlooprequires a C compiler and does not run on Windows native). - Does Not Fix Paradigm Issues: Retains all of standard
asyncio's architectural shortcomings (orphaned background tasks, unhandled task cancellation).
Code:
import asyncio
import uvloop
# Global activation: 2 lines of code to accelerate all asyncio operations
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
async def main():
print(f"Active event loop: {asyncio.get_event_loop()}")
asyncio.run(main())
# Output: <uvloop.Loop running=False closed=False debug=False>
Q6: What is Structured Concurrency, what is the Task Leak problem in asyncio, and how does Trio's Nursery pattern solve it?
Level: Principal
Technical Answer:
In standard asyncio, tasks can be spawned using asyncio.create_task() in a "fire-and-forget" manner. If the parent coroutine exits or crashes, the background task continues running detached in the background.
The Task Leak Problem:
- Orphaned Resources: Background tasks hold database connections and memory long after their caller has finished.
- Silent Failures: If the detached task raises an exception, the error is often lost or logged as an unhandled warning.
- Cancellation Chaos: If a web request times out, cancelling the parent does not automatically cancel detached child tasks.
Trio's Solution: Nurseries Trio enforces Structured Concurrency: the lifetime of concurrent tasks is strictly bounded by code blocks.
- Tasks are spawned inside a
nursery(async with trio.open_nursery() as nursery:). - Execution cannot leave the
async withblock until all child tasks in that nursery complete. - If any child task raises an exception, Trio automatically cancels all sibling tasks in that nursery, waits for them to terminate cleanly, and then raises the exception.
Code:
# TRIO: Guaranteed Structured Concurrency
import trio
async def background_worker(name: str, delay: float):
await trio.sleep(delay)
print(f"Worker {name} finished successfully")
async def parent_service():
async with trio.open_nursery() as nursery:
# Spawn tasks inside the nursery boundary
nursery.start_soon(background_worker, "Task-A", 1.0)
nursery.start_soon(background_worker, "Task-B", 0.5)
print("Both tasks scheduled; nursery guarantees completion before exiting.")
# GUARANTEED: Reaching here means both Task-A and Task-B have fully terminated!
print("All tasks finished cleanly. Zero task leaks.")
trio.run(parent_service)
Workflow Diagram:
[asyncio Fire-and-Forget: Task Leak]
Parent Coroutine ──► Spawns Task-A via create_task()
Parent Returns ──► Task-A continues running indefinitely (LEAKED!)
[Trio Nursery: Structured Concurrency]
Parent enters `async with trio.open_nursery() as nursery:`
├── Spawns Task-A
└── Spawns Task-B
Parent waits at block boundary until [Task-A AND Task-B] finish
Clean exit: 100% resource reclamation
Q7: What is AnyIO, why is it adopted by FastAPI and HTTPX, and how does it bridge the gap between asyncio and Trio?
Level: Senior
Technical Answer:
Library authors faced an architectural dilemma: write async libraries for asyncio (large ecosystem, fire-and-forget flaws) or Trio (superior structured concurrency, smaller ecosystem).
AnyIO is an asynchronous compatibility layer that allows developers to write async code once and run it transparently on top of either asyncio or Trio.
Key Capabilities:
- Brings Structured Concurrency to
asyncio: Providesanyio.create_task_group(), offering Trio-style nursery semantics on standardasynciowithout requiring Python 3.11'sTaskGroup. - Unified API: Universal implementations of sockets, files, synchronization primitives, and cancellation scopes.
- Industry Standard Foundation: Both FastAPI (via Starlette) and HTTPX are built on top of AnyIO to maintain runtime flexibility.
Code:
import anyio
async def fetch_endpoint(url: str):
await anyio.sleep(0.5)
print(f"Fetched {url}")
async def main():
# AnyIO TaskGroup: Structured concurrency that works everywhere
async with anyio.create_task_group() as tg:
tg.start_soon(fetch_endpoint, "https://api.github.com")
tg.start_soon(fetch_endpoint, "https://api.stripe.com")
print("All requests completed safely.")
# Run on default asyncio backend
anyio.run(main, backend="asyncio")
# Run on Trio backend without changing any code!
# anyio.run(main, backend="trio")
Q8: Comprehensive Comparison: How do asyncio, uvloop, Trio, and AnyIO compare across performance, structured concurrency, and ecosystem?
Level: Senior
Technical Answer:
| Dimension | asyncio | uvloop | Trio | AnyIO |
|---|---|---|---|---|
| Runtime Engine | Pure Python + C | C (libuv via Cython) | Pure Python | Delegates to backend |
| I/O Throughput | Baseline (1x) | High (2x–4x) | Baseline (1x) | Matches backend |
| Structured Concurrency | No (Manual before 3.11) | No (Manual) | Native & Enforced | Native via TaskGroup |
| Task Leak Prevention | Prone to leaks | Prone to leaks | 100% Guaranteed | 100% Guaranteed |
| Ecosystem Size | Dominant standard | Same as asyncio | Smaller | Bridges both |
| Third-Party Dep | None (Built-in) | External (pip install) | External (pip install) | External (pip install) |
| Best Production Use | Standard backend apps | High-throughput APIs | Complex safety-critical systems | Modern libraries (FastAPI/HTTPX) |
Distributed Resilience: Backoff, Jitter & Circuit Breakers
Q9: What is Exponential Backoff with Jitter, why is Full Jitter critical, and how does it defeat the Thundering Herd retry storm?
Level: Senior
Technical Answer: When an external service experiences transient failures (network blip, database lock contention, rate limit 429), retrying immediately or with a fixed delay worsens the outage by hammering the already struggling service.
Exponential Backoff multiplies the delay exponentially: $$\text{delay} = \min(\text{base_delay} \times 2^{\text{attempt}}, \text{max_delay})$$
The Thundering Herd Problem without Jitter: If 1,000 clients fail at the exact same instant (e.g. during a network drop), exponential backoff causes all 1,000 clients to retry at the exact same subsequent moments (1s, 2s, 4s, 8s), creating devastating periodic spikes that knock the service back down!
Full Jitter Solution: Multiply the calculated exponential delay by a uniform random variable between 0 and 1: $$\text{actual_delay} = \text{random.uniform}(0, \text{delay})$$ This evenly disperses the retry traffic across the entire recovery timeline.
Code:
import asyncio
import random
from typing import Callable, Any
async def fetch_with_full_jitter_retry(
operation: Callable[[], Any],
max_retries: int = 5,
base_delay: float = 0.5,
max_delay: float = 30.0
) -> Any:
for attempt in range(max_retries):
try:
return await operation()
except Exception as exc:
if attempt == max_retries - 1:
print(f"[RETRY EXHAUSTED] Attempt {attempt + 1} failed: {exc}")
raise
# 1. Calculate capped exponential backoff
raw_delay = min(base_delay * (2 ** attempt), max_delay)
# 2. Apply Full Jitter (AWS recommended pattern)
sleep_duration = random.uniform(0.1, raw_delay)
print(f"[RETRY] Attempt {attempt + 1} failed. Backing off for {sleep_duration:.2f}s...")
await asyncio.sleep(sleep_duration)
Q10: How does a Circuit Breaker State Machine operate across CLOSED, OPEN, and HALF-OPEN states?
Level: Senior / Principal
Technical Answer: A Circuit Breaker detects persistent failures and temporarily prevents requests from hitting a failing downstream service, protecting both the downstream service (allowing it to recover) and the caller (failing fast rather than hanging on timeouts).
The Three States:
- CLOSED (Normal Operation):
- All requests pass through to the downstream service.
- Failures are tracked. If consecutive failures exceed
failure_threshold, the circuit trips to OPEN.
- OPEN (Failing / Downstream Down):
- All calls fail immediately without attempting any network connection (
CircuitBreakerOpenException). - No resources or network sockets are consumed.
- After
recovery_timeoutseconds elapse, the circuit transitions to HALF-OPEN.
- All calls fail immediately without attempting any network connection (
- HALF-OPEN (Canary Testing):
- Allows a single test request through to probe service health.
- If the test request succeeds: transitions back to CLOSED (service has recovered).
- If the test request fails: immediately resets back to OPEN for another timeout duration.
State Machine Diagram:
Consecutive Failures >= Threshold
┌─────────────────────────────────────────────────────┐
│ │
▼ │
┌───────────┐ ┌───────────┐
│ CLOSED │ │ OPEN │
│ (Healthy)│ │ (Tripped) │
└───────────┘ └───────────┘
▲ │
│ │
│ Test Request Succeeds Timeout Window Passes
│ │
│ ┌─────────────────┐ │
└─────────────┤ HALF-OPEN │ ◄───────────────────┘
│(Canary Testing) │
└────────┬────────┘
│
│ Test Request Fails
└──────────────────────────────► (Back to OPEN)
Q11: How do you implement a production-grade Async Circuit Breaker in pure Python?
Level: Senior / Principal
Technical Answer: Below is a thread-safe, coroutine-compatible Circuit Breaker implementation complete with failure counting, time-based state transitions, and canary test isolation.
Code:
import time
from enum import Enum
from typing import Callable, Any
class CircuitState(Enum):
CLOSED = "CLOSED"
OPEN = "OPEN"
HALF_OPEN = "HALF_OPEN"
class CircuitBreakerOpen(Exception):
"""Raised when downstream service is failing and circuit is open."""
pass
class AsyncCircuitBreaker:
def __init__(
self,
failure_threshold: int = 5,
recovery_timeout: float = 30.0,
name: str = "default_circuit"
):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.name = name
self.state = CircuitState.CLOSED
self.failure_count = 0
self.last_failure_time = 0.0
def _update_state(self):
"""Transitions from OPEN to HALF_OPEN when recovery window passes."""
if self.state == CircuitState.OPEN:
if time.time() - self.last_failure_time >= self.recovery_timeout:
print(f"[{self.name}] Recovery timeout reached; entering HALF-OPEN canary state.")
self.state = CircuitState.HALF_OPEN
async def call(self, async_func: Callable, *args, **kwargs) -> Any:
self._update_state()
if self.state == CircuitState.OPEN:
raise CircuitBreakerOpen(f"Circuit '{self.name}' is OPEN. Failing fast to protect system.")
try:
result = await async_func(*args, **kwargs)
self._on_success()
return result
except Exception as exc:
self._on_failure(exc)
raise
def _on_success(self):
if self.state == CircuitState.HALF_OPEN:
print(f"[{self.name}] Canary test succeeded. Restoring circuit to CLOSED state.")
self.state = CircuitState.CLOSED
self.failure_count = 0
def _on_failure(self, exc: Exception):
self.failure_count += 1
self.last_failure_time = time.time()
print(f"[{self.name}] Call failed ({self.failure_count}/{self.failure_threshold}): {exc}")
if self.state == CircuitState.HALF_OPEN:
print(f"[{self.name}] Canary test failed. Resetting circuit to OPEN.")
self.state = CircuitState.OPEN
elif self.failure_count >= self.failure_threshold:
print(f"[{self.name}] Threshold exceeded. TRIPPING circuit to OPEN.")
self.state = CircuitState.OPEN
Q12: How do you compose Circuit Breakers and Exponential Backoff together in enterprise microservices?
Level: Senior
Technical Answer: In resilient microservice architectures, Exponential Backoff and Circuit Breakers are not mutually exclusive—they serve complementary roles:
- Exponential Backoff is Patient: Retries transient glitches (network drops, packet loss, momentary lock waits).
- Circuit Breaker is Protective: Protects the infrastructure during prolonged systemic outages (database hard crash, downstream service down for minutes).
Composition Rule: The Circuit Breaker must wrap the Retry Mechanism. When a service call is made:
- If the circuit is
OPEN, fail fast immediately (0 network calls, 0 retry delay). - If the circuit is
CLOSED, execute the operation using Exponential Backoff. - If all retry attempts fail, the outer Circuit Breaker records the failure. Multiple consecutive retry exhaustion events trip the circuit breaker.
Code:
# Composing both resilience patterns into a unified client
breaker = AsyncCircuitBreaker(failure_threshold=3, recovery_timeout=60.0, name="payment_gateway")
async def resilient_downstream_call(endpoint: str, payload: dict) -> dict:
async def call_with_backoff():
return await fetch_with_full_jitter_retry(
operation=lambda: raw_http_post(endpoint, payload),
max_retries=3,
base_delay=0.5
)
# Circuit breaker guards the entire retry suite
return await breaker.call(call_with_backoff)
Production Idempotency with FastAPI & Redis
Q13: What is Idempotency, and why are POST endpoints in payment processing and order placement vulnerable to double execution?
Level: Senior
Technical Answer: An operation is Idempotent if performing it multiple times produces the exact same side-effect and result as performing it once ($f(f(x)) = f(x)$).
GET,PUT,DELETEare inherently idempotent by HTTP specification.POSTis non-idempotent: each request typically creates a new record or triggers a transaction.
The Double-Execution Vulnerability: In network communication, the Two Generals Problem applies:
- Client sends a payment request (
POST /payments, $100). - The server successfully processes the payment with the bank.
- Before the server can send back the HTTP 201 response, a network drop disconnects the connection.
- The client's HTTP library sees a
ConnectionErrorand automatically retries the POST request. - Without idempotency protection, the server processes the payment a second time, charging the customer $200!
Q14: How does the X-Idempotency-Key header work with Redis distributed locking and caching?
Level: Senior / Principal
Technical Answer:
Production idempotency systems use an X-Idempotency-Key (UUIDv4) supplied by the client in the request header, backed by a 4-phase state machine in Redis:
- Header Validation: Check that
X-Idempotency-Keyis provided on critical state-changing endpoints. - Cache Check & In-Flight Lock:
- Query Redis for
idempotency:<key>. - If completed: Return the previously cached response immediately (short-circuiting the business logic).
- If state == 'PROCESSING': A concurrent request is currently in flight. Reject immediately with HTTP 409 Conflict to prevent parallel double-execution race conditions.
- Query Redis for
- Lock Acquisition:
- Write
idempotency:<key> = "PROCESSING"with a short TTL (e.g. 120 seconds) using atomicSETNX.
- Write
- Execution & Completion Caching:
- Execute critical database transaction / payment processing.
- On success: Overwrite key with the final JSON response payload and a long TTL (e.g. 24 hours).
- On exception: Delete the Redis lock key so the client is free to retry safely.
Q15: Complete Production Implementation: How do you build an Idempotency Dependency for FastAPI and Redis?
Level: Senior / Principal
Technical Answer: Below is the battle-tested, production-ready FastAPI dependency and route implementation.
Code:
import json
from fastapi import FastAPI, Header, HTTPException, Depends, status
import redis
app = FastAPI(title="Payment Service")
# Connect to Redis cluster
redis_client = redis.Redis(host="localhost", port=6379, db=0, decode_responses=True)
def verify_idempotency_key(
x_idempotency_key: str = Header(None, alias="X-Idempotency-Key")
):
"""
FastAPI Dependency to enforce and coordinate request idempotency.
"""
# 1. Enforce presence of key on critical mutations
if not x_idempotency_key:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="Missing required 'X-Idempotency-Key' header."
)
redis_key = f"idempotency:{x_idempotency_key}"
cached_entry = redis_client.get(redis_key)
if cached_entry:
# 2. Check if a concurrent request is currently processing this key
if cached_entry == "PROCESSING":
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="A concurrent request with this Idempotency-Key is already in progress. Please retry shortly."
)
# 3. Request already completed in the past: Return cached payload
return {
"key": x_idempotency_key,
"cached": True,
"data": json.loads(cached_entry)
}
# 4. Acquire in-flight processing lock with a 2-minute safety TTL
# Prevents concurrent requests while ensuring the lock expires if server crashes
lock_acquired = redis_client.set(redis_key, "PROCESSING", ex=120, nx=True)
if not lock_acquired:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail="Conflict acquiring idempotency lock."
)
return {"key": x_idempotency_key, "cached": False, "data": None}
@app.post("/api/v1/payments", status_code=status.HTTP_201_CREATED)
async def process_payment(
payload: dict,
idempotency: dict = Depends(verify_idempotency_key)
):
key = idempotency["key"]
redis_key = f"idempotency:{key}"
# Short-circuit: Return past result without re-executing payment gateway
if idempotency["cached"]:
return {
"info": "Duplicate request detected and safely deduplicated",
"response": idempotency["data"]
}
try:
# --- EXECUTE CRITICAL TRANSACTION LOGIC ---
amount = payload.get("amount")
customer_id = payload.get("customer_id")
transaction_record = {
"transaction_id": f"tx_{key[:8]}",
"customer_id": customer_id,
"amount_charged": amount,
"status": "COMPLETED"
}
# 5. Overwrite the 'PROCESSING' lock with the final result and 24-hour TTL
redis_client.set(redis_key, json.dumps(transaction_record), ex=86400)
return {
"info": "Original request processed successfully",
"response": transaction_record
}
except Exception as exc:
# 6. Safety Rollback: Delete the in-flight lock on failure so the user can retry
redis_client.delete(redis_key)
raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail=f"Transaction failed: {str(exc)}"
)
Q16: What happens if the server crashes while an idempotency key is in PROCESSING state, and how do TTLs and Dead-Letter recovery handle it?
Level: Senior / Principal
Technical Answer:
If an application container crashes mid-flight (e.g. OOM killed or power loss) after setting idempotency:key = "PROCESSING" but before finalizing:
- The 120-second TTL Safeguard: Because the
PROCESSINGlock was set withex=120, Redis automatically expires the lock after 2 minutes. The user is not blocked forever. - Reconciliation Worker: For mission-critical banking transactions, background workers query the external payment gateway using the idempotency key to determine if the charge actually succeeded at the provider. If it did, the worker writes the completed record to Redis; if not, it clears the key, allowing the user to safely retry.