The problem: one slow dependency takes down everything

An order service calls a recommendation service to decorate the order page. The recommendation service becomes slow, not dead: each call takes thirty seconds instead of fifty milliseconds. The order service has a pool of 200 request threads. Every incoming request waits on the slow call, the pool fills up, and soon the order page, which does not even need recommendations, cannot serve anyone. A non-critical dependency has turned into a full outage, and it spread because the failure was slow rather than loud.

In a distributed system the network can drop, delay or duplicate messages, and a remote process can be slow, dead or half-working, and from the caller's side these look the same: no answer yet. Failure is the normal state of some component at any given time. The design question is how far one failure is allowed to travel.

Timeouts everywhere

Every call that leaves your process needs a timeout: HTTP clients, database drivers, message producers, lock acquisition, connection pool borrowing. Many libraries default to "wait forever" or to values far too long for a user-facing request.

  • Separate connect timeout (can I reach the server?) from read/response timeout (how long do I wait for the answer?).
  • Make timeouts consistent with the caller's own budget. If the user-facing request must finish within two seconds, the three downstream calls inside it cannot each be allowed ten seconds. Propagating a deadline down the call chain is a good practice.
  • Base values on observed latency percentiles of the dependency plus some margin, not on guesses. See observability for how to measure it.
RestClient client = RestClient.builder()    .requestFactory(new JdkClientHttpRequestFactory(        HttpClient.newBuilder()            .connectTimeout(Duration.ofMillis(300)).build()))    .build();// plus a read timeout on the request factory, set per dependency

A timeout does not tell you whether the remote operation happened. The server may have processed the request and the response was lost. That is why retries need idempotency, covered in Idempotency patterns.

Retries with exponential backoff and jitter

Retrying is useful for transient errors such as a dropped connection or a brief overload. It is dangerous when done carelessly: if a dependency is struggling and every client immediately retries three times, you have quadrupled the load on a system that was already failing. This is called a retry storm.

Good retry rules:

  1. Retry only errors that are likely transient (timeouts, connection resets, 503) and operations that are safe to repeat. Do not retry a 400 or a validation failure.
  2. Limit attempts to a small number.
  3. Use exponential backoff: wait longer after each failure.
  4. Add jitter: randomise the wait so thousands of clients do not retry in lockstep.
  5. Retry at one layer only. If three layers each retry three times, one failure becomes up to twenty-seven calls.
resilience4j:  retry:    instances:      inventory:        maxAttempts: 3        waitDuration:         enableExponentialBackoff: true        exponentialBackoffMultiplier: 2        enableRandomizedWait: true        retryExceptions:          - java.io.IOException          - java.util.concurrent.TimeoutException

Check the property names for the library version you use. The concepts matter more than the exact keys.

Circuit breakers

A circuit breaker watches calls to a dependency and, when failures cross a threshold, opens: it fails calls immediately without trying the remote service. After a wait period it goes half-open, lets a few trial calls through, and closes again if they succeed.

  CLOSED --failure rate too high--> OPEN
     ^                               |
     |                          wait elapsed
     +--trial calls succeed-- HALF-OPEN
                  |--trial fails--> OPEN

The benefits are two: callers fail fast instead of tying up threads on a dependency that cannot answer, and the struggling service gets breathing room to recover. Configure a minimum number of calls before judging, a failure-rate threshold, a slow-call threshold (slowness counts as failure), and a sensible wait duration. Expose the breaker state as a metric and alert when it opens.

Bulkheads

Ships have watertight compartments so one leak does not sink the vessel. A bulkhead limits the resources any one dependency can consume, so the failure of one cannot starve the others.

  • Semaphore bulkhead: limits concurrent calls to a dependency. When the limit is reached, further calls are rejected immediately.
  • Thread pool bulkhead: runs calls to a dependency on a dedicated pool with its own queue.
  • Separate connection pools for critical and non-critical database workloads follow the same idea.

In the opening story, a limit of, say, twenty concurrent recommendation calls would have left 180 threads free for the order page. With virtual threads, threads are cheap, but downstream capacity, connection pools and memory are still finite, so limits remain important.

Fallbacks and graceful degradation

When a call fails, decide what the user gets instead:

  • A default: an empty recommendations list.
  • Cached or stale data: the last known price, labelled appropriately in the interface if accuracy matters.
  • A simplified path: skip the optional enrichment and continue.
  • A clear error: for operations where a wrong answer is worse than none, such as checking an account balance before a payment. A fallback that returns "approved" when the fraud service is unreachable is a business decision and needs an explicit owner, not a silent default.

Fallbacks must be tested; they are code paths that run only during incidents, so they rot quickly.

Partial failure examples

  • Payment succeeded, response lost. The client times out and retries; without an idempotency key the customer is charged twice.
  • Database write committed, event not published. Another service never learns of the order. This dual-write problem is addressed by the outbox pattern in Kafka delivery semantics.
  • One slow replica. Load balancers that only check "is it up?" keep sending traffic to it; health checks and outlier detection help.
  • Clock differences. Using wall-clock comparisons between machines for ordering or lock expiry gives wrong results under skew.

Composing the patterns with Resilience4j

Resilience4j is a Java library that offers these ideas as separate modules: time limiter, retry, circuit breaker, bulkhead and rate limiter. Order matters when you combine them. A typical order from outermost to innermost is: bulkhead, time limiter, circuit breaker, retry, then the actual call, or a variation of this, but you should decide it consciously. A retry inside a circuit breaker lets the breaker see the final outcome of each logical call; reversed, each attempt counts separately. Use annotations such as @CircuitBreaker and @Retry on a Spring bean for simplicity, remembering that, like @Transactional, they work through proxies and are bypassed by calls from within the same class.

Common mistakes

  • No timeout, or the same large timeout everywhere.
  • Retrying non-idempotent operations.
  • Retrying at every layer.
  • A circuit breaker with no metrics, so nobody knows it opened.
  • Fallbacks that hide a real problem forever. A degraded mode should be visible on a dashboard.
  • Testing only the happy path. Inject latency and errors in a test environment to see what really happens.

Trade-offs

Every mechanism adds configuration, states to understand and ways to misconfigure. A call between two modules of a monolith does not need a circuit breaker. Apply these patterns at the boundaries that cross a network and to dependencies whose failure you cannot tolerate; start with timeouts, because they are cheap and prevent the worst cascades. Resilience also costs consistency or functionality: a degraded answer is by definition not the best answer.

Checklist

  • Every outbound call has connect and read timeouts that fit the caller's budget.
  • Retries: few, backed off, jittered, on idempotent operations, at one layer.
  • Circuit breakers on critical remote dependencies, with metrics and alerts.
  • Bulkheads separate critical from optional work.
  • Every fallback is deliberate, tested and visible.
  • Failure injection is part of testing.

Where to go next