Building Fault-Tolerant Microservices: Resilience4j Circuit Breaker & Retry with Spring Boot 4 and Java 25 Virtual Threads



1. Introduction: The Cascading Failure Problem

Modern cloud systems connect many independent microservices together. A single user request often touches several downstream services. For example, an order request calls payment services, inventory systems, and shipping gateways. When one downstream service slows down or crashes, serious problems begin across the platform. In traditional systems, incoming HTTP threads wait for slow remote calls. These threads stay blocked for seconds or minutes.

Eventually, all application threads become blocked. The host service runs out of memory and crashes. Then, the services calling that host also run out of threads and crash. Engineers call this disaster a cascading failure. A minor outage in one third-party service brings down the entire business platform. Distributed architectures must stop cascading failures before they spread across the network.

In this comprehensive engineering guide, we build a resilient, production-ready system. We combine Spring Boot 4.0.0, Java 25 Virtual Threads, and Resilience4j. You will learn how to configure circuit breakers, retries, and semaphore bulkheads. You will also see how to build graceful degraded fallbacks that keep your applications fast and reliable.

2. Understanding the Circuit Breaker Pattern

The Circuit Breaker pattern takes its name from electrical safety switches in buildings. In your home, an electrical breaker stops electrical flow when current spikes. This action prevents overheating and fires. In software engineering, a circuit breaker wraps calls to remote services. It monitors the rate of failures and response times over a sliding window.

When the remote service is healthy, the circuit breaker allows requests to pass normally. But when the error rate crosses a chosen threshold, the circuit breaker trips. It cuts off communication immediately. Future requests fail fast or jump directly to a fallback method. The system does not wait for remote timeouts. This protects both your service and the struggling downstream provider.

To understand how this fits into clean system design, review our guides on Clean Architecture in Spring Boot 4 and Distributed API Rate Limiting with Redis. Combining rate limiting at the API edge with circuit breakers in internal services gives you true defense-in-depth.

3. The Circuit Breaker Finite State Machine

A circuit breaker works as a finite state machine. It moves between three main operational states: CLOSED, OPEN, and HALF-OPEN. Each state handles incoming traffic differently based on statistical thresholds.

Here is how the three states operate in production:

• CLOSED State: In this state, everything is normal. The circuit breaker forwards every call to the downstream service. It measures outcomes in a sliding window. This window can track the last N calls (count-based) or calls within the last N seconds (time-based). If the failure percentage stays below the threshold, the circuit stays CLOSED.
• OPEN State: When the failure rate or slow call rate exceeds your threshold, the circuit trips to OPEN. The circuit breaker immediately rejects all subsequent requests. It throws a CallNotPermittedException without touching the network. This gives the failing downstream service time to recover.
• HALF-OPEN State: After a configurable wait duration passes, the circuit enters HALF-OPEN. It permits a limited number of probe calls to test the remote service. If these probe calls succeed, the circuit resets to CLOSED. If any probe call fails, the circuit returns to OPEN for another waiting period.

4. Why Java 25 Virtual Threads Change Bulkheading

Before Java 21 and Java 25, thread management in Java was expensive. Traditional platform threads mapped directly to operating system threads. Each platform thread consumed approximately one megabyte of operating system memory. Under heavy traffic, creating thousands of threads caused operating system thrashing and out-of-memory errors.

To protect systems, engineers used the Bulkhead pattern. In older libraries like Netflix Hystrix, developers used ThreadPoolBulkhead. They created isolated thread pools with fixed queues for each external dependency. If service A slowed down, only its private thread pool filled up. However, managing dozens of thread pools wasted CPU cycles on context switching.

Java 25 Virtual Threads completely eliminate the need for thread pool bulkheads. Virtual threads are lightweight user-mode threads managed directly by the Java Virtual Machine runtime. A single application can easily run one million virtual threads simultaneously without running out of memory. They require only a few kilobytes of RAM each. When a virtual thread performs blocking network I/O, the JVM unmounts it from its operating system carrier thread.

The carrier thread becomes free to run other tasks immediately. When the network response arrives from the socket, the JVM scheduler remounts the virtual thread onto an available carrier thread. Therefore, with Java 25 and Spring Boot 4, you should always use SemaphoreBulkhead instead of thread pools. A semaphore bulkhead uses an atomic permit counter. It limits concurrency without thread handoffs or queue overhead. For a deeper look at virtual thread performance, read our analysis on Virtual Threads in Spring Boot 4 and gRPC vs REST Inter-Service Performance.

5. Sliding Window Algorithms: Count-Based vs. Time-Based

Resilience4j calculates failure rates using one of two sliding window algorithms: Count-Based or Time-Based. Understanding the mathematical difference between these two algorithms helps you choose the right configuration for your microservices.

In a Count-Based Sliding Window, the circuit breaker stores execution results in a circular ring buffer of fixed size N (for example, N = 100). Each slot records whether a call succeeded or failed. When a new call finishes, it overwrites the oldest measurement in the ring buffer. The failure rate is simply the count of failed calls divided by the total calls currently recorded. This algorithm works best for steady, predictable traffic streams.

In a Time-Based Sliding Window, the circuit breaker divides a fixed time duration (such as 60 seconds) into N discrete epoch buckets (for example, 10 buckets of 6 seconds each). Each bucket aggregates success, failure, and slow call totals. When a time epoch expires, the oldest bucket drops out of the calculation. Time-based windows work best for bursty or seasonal traffic where call volume varies dramatically between peak and off-peak hours.

6. Resilience4j Interceptor Pipeline Architecture

Resilience4j provides several fault-tolerance patterns. In enterprise applications, you often combine multiple patterns on a single method. The execution order of these decorators is critical for proper operation.

Resilience4j executes decorators in this specific order from outside to inside:

1. Bulkhead: Limits the maximum number of concurrent executions. It protects system resources first.
2. TimeLimiter: Enforces an absolute maximum duration for asynchronous execution.
3. RateLimiter: Restricts the number of calls within a rolling time window.
4. CircuitBreaker: Checks if the remote dependency is healthy. If the circuit is OPEN, it fast-fails immediately.
5. Retry: Executes the target method. If a transient error occurs, it retries with exponential backoff before the circuit records a failure.

Putting the Retry decorator inside the CircuitBreaker is essential. If Retry were placed outside, a single failed call with three retries would count as three separate circuit breaker failures. Placing Retry inside ensures that only final, unrecoverable failures trigger circuit breaker state changes.

7. Step-by-Step Implementation with Spring Boot 4

Let us implement a complete fault-tolerant payment processing service. First, create your project using Spring Boot 4.0.0 and Java 25. Add the necessary dependencies to your pom.xml file.

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-validation</artifactId>
    </dependency>

    <!-- Resilience4j Modules -->
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-circuitbreaker</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-retry</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-bulkhead</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-micrometer</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>

Enable Java 25 Virtual Threads in your application.yml file:

spring:
  application:
    name: resilience4j-circuit-breaker-demo
  threads:
    virtual:
      enabled: true

server:
  port: 8080

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

8. Configuring Resilience Registries in Java 25

Modern enterprise applications favor explicit, type-safe programmatic configuration. Programmatic setup prevents hidden reflection bugs and allows dynamic runtime tuning. Create a configuration class named ResilienceConfig.java.

package com.webcodein.workshop.resilience.config;

import io.github.resilience4j.bulkhead.BulkheadConfig;
import io.github.resilience4j.bulkhead.BulkheadRegistry;
import io.github.resilience4j.circuitbreaker.CircuitBreakerConfig;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;
import io.github.resilience4j.retry.RetryConfig;
import io.github.resilience4j.retry.RetryRegistry;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.time.Duration;

@Configuration
public class ResilienceConfig {

    public static final String PAYMENT_CB = "paymentService";
    public static final String PAYMENT_RETRY = "paymentRetry";
    public static final String PAYMENT_BULKHEAD = "paymentBulkhead";

    @Bean
    public CircuitBreakerRegistry circuitBreakerRegistry() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
                .slidingWindowType(CircuitBreakerConfig.SlidingWindowType.COUNT_BASED)
                .slidingWindowSize(5)
                .minimumNumberOfCalls(3)
                .failureRateThreshold(50.0f)
                .waitDurationInOpenState(Duration.ofMillis(2000))
                .permittedNumberOfCallsInHalfOpenState(2)
                .automaticTransitionFromOpenToHalfOpenEnabled(true)
                .build();
        return CircuitBreakerRegistry.of(config);
    }

    @Bean
    public RetryRegistry retryRegistry() {
        RetryConfig config = RetryConfig.custom()
                .maxAttempts(2)
                .waitDuration(Duration.ofMillis(100))
                .build();
        return RetryRegistry.of(config);
    }

    @Bean
    public BulkheadRegistry bulkheadRegistry() {
        BulkheadConfig config = BulkheadConfig.custom()
                .maxConcurrentCalls(25)
                .maxWaitDuration(Duration.ofMillis(50))
                .build();
        return BulkheadRegistry.of(config);
    }
}

9. Building the Resilient Service and Graceful Fallback

Next, define the domain records and the payment service. Notice how Java 25 records provide compact constructors with validation checks. This prevents invalid domain states from ever reaching your business logic.

package com.webcodein.workshop.resilience.model;

import jakarta.validation.constraints.DecimalMin;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;
import java.math.BigDecimal;
import java.time.Instant;

public record PaymentRequest(
        @NotBlank String transactionId,
        @NotBlank String accountNumber,
        @NotNull @DecimalMin("0.01") BigDecimal amount,
        @NotBlank String currency
) {
    public PaymentRequest {
        if (transactionId == null || transactionId.isBlank()) {
            throw new IllegalArgumentException("Transaction ID cannot be blank");
        }
        if (amount == null || amount.compareTo(BigDecimal.ZERO) <= 0) {
            throw new IllegalArgumentException("Amount must be strictly positive");
        }
    }
}

Now, create ResilientPaymentService.java. The service wraps the external gateway call with Bulkhead, CircuitBreaker, and Retry decorators. When the circuit is OPEN or fails, it returns a degraded fallback response.

package com.webcodein.workshop.resilience.service;

import com.webcodein.workshop.resilience.config.ResilienceConfig;
import com.webcodein.workshop.resilience.model.PaymentRequest;
import com.webcodein.workshop.resilience.model.PaymentResponse;
import io.github.resilience4j.bulkhead.Bulkhead;
import io.github.resilience4j.bulkhead.BulkheadRegistry;
import io.github.resilience4j.circuitbreaker.CallNotPermittedException;
import io.github.resilience4j.circuitbreaker.CircuitBreaker;
import io.github.resilience4j.circuitbreaker.CircuitBreakerRegistry;
import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryRegistry;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Service;

import java.util.function.Supplier;

@Service
public class ResilientPaymentService {

    private static final Logger log = LoggerFactory.getLogger(ResilientPaymentService.class);

    private final ExternalPaymentGateway externalGateway;
    private final CircuitBreaker circuitBreaker;
    private final Retry retry;
    private final Bulkhead bulkhead;

    public ResilientPaymentService(
            ExternalPaymentGateway externalGateway,
            CircuitBreakerRegistry cbRegistry,
            RetryRegistry retryRegistry,
            BulkheadRegistry bulkheadRegistry
    ) {
        this.externalGateway = externalGateway;
        this.circuitBreaker = cbRegistry.circuitBreaker(ResilienceConfig.PAYMENT_CB);
        this.retry = retryRegistry.retry(ResilienceConfig.PAYMENT_RETRY);
        this.bulkhead = bulkheadRegistry.bulkhead(ResilienceConfig.PAYMENT_BULKHEAD);
    }

    public PaymentResponse processPayment(PaymentRequest request) {
        Supplier<PaymentResponse> gatewayCall = () -> externalGateway.processRemotePayment(request);

        // Chain decorators: Bulkhead -> CircuitBreaker -> Retry
        Supplier<PaymentResponse> resilientPipeline = Bulkhead.decorateSupplier(bulkhead,
                CircuitBreaker.decorateSupplier(circuitBreaker,
                        Retry.decorateSupplier(retry, gatewayCall)));

        try {
            return resilientPipeline.get();
        } catch (CallNotPermittedException ex) {
            log.warn("Circuit OPEN for payment {}. Fast fallback triggered.", request.transactionId());
            return PaymentResponse.fallback(request.transactionId(), request.amount(), request.currency(), "Circuit breaker is OPEN");
        } catch (Exception ex) {
            log.error("Payment failed for {}. Executing fallback.", request.transactionId(), ex);
            return PaymentResponse.fallback(request.transactionId(), request.amount(), request.currency(), ex.getMessage());
        }
    }
}

10. Exposing REST Endpoints and Actuator Metrics

The REST controller exposes endpoints to process transactions, check circuit breaker states, and simulate downstream network failures.

package com.webcodein.workshop.resilience.controller;

import com.webcodein.workshop.resilience.model.PaymentRequest;
import com.webcodein.workshop.resilience.model.PaymentResponse;
import com.webcodein.workshop.resilience.service.ResilientPaymentService;
import jakarta.validation.Valid;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.Map;

@RestController
@RequestMapping("/api/payments")
public class PaymentController {

    private final ResilientPaymentService paymentService;

    public PaymentController(ResilientPaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @PostMapping("/process")
    public ResponseEntity<PaymentResponse> processPayment(@Valid @RequestBody PaymentRequest request) {
        PaymentResponse response = paymentService.processPayment(request);
        return ResponseEntity.ok(response);
    }

    @GetMapping("/circuit-breaker-status")
    public ResponseEntity<Map<String, Object>> getStatus() {
        return ResponseEntity.ok(paymentService.getCircuitBreakerMetrics());
    }

    @PostMapping("/simulate-failure")
    public ResponseEntity<Map<String, Object>> toggleFailure(@RequestParam boolean enabled) {
        paymentService.setSimulateFailure(enabled);
        return ResponseEntity.ok(Map.of("failureSimulationEnabled", enabled));
    }
}

11. Comparative Fault Tolerance Matrix

The table below compares different resilience patterns used across modern microservices.

12. Real-World Enterprise Case Studies

Case Study A: Global FinTech Payment Gateway

A global financial platform processed thousands of card transactions per minute. During a major cloud networking glitch, their primary bank gateway started timing out after 30 seconds. Because their legacy Java 17 service used traditional platform threads, all 400 worker threads blocked waiting for sockets. The entire checkout API went down worldwide, costing approximately $85,000 per hour in lost revenue.

The engineering team upgraded to Spring Boot 4, Java 25 Virtual Threads, and Resilience4j. They configured a circuit breaker with a 50% failure threshold and a 2-second timeout window. When the bank gateway slowed down again during a subsequent regional outage, the circuit breaker tripped within 6 seconds. Subsequent transactions immediately fell back to an asynchronous processing queue. Customers received confirmed transaction receipts, and orders processed successfully when the bank recovered, reducing incident recovery time from two hours to zero user-facing downtime.

Case Study B: High-Volume E-Commerce Inventory Service

During Black Friday flash sales, a major online retailer experienced severe database connection pool exhaustion. Thousands of shoppers refreshed product pages simultaneously. Our companion article on HikariCP Connection Pool Tuning explains how connection starvation happens under sudden load spikes. The inventory query service slowed down, causing the main catalog service to crash.

By implementing a SemaphoreBulkhead limited to 100 concurrent permits and a CircuitBreaker on the inventory service, the catalog service isolated the slow queries. When inventory queries backed up, the circuit breaker tripped and served cached inventory counts. The site stayed online, and shoppers continued purchasing without noticing the background degradation, maintaining 99.99% availability throughout the holiday weekend.

Case Study C: Healthcare EHR Patient Record Ingestion

A hospital network ingested medical telemetry from thousands of bedside monitors. The ingestion engine sent biometric data to a centralized analytics server. When the analytics server underwent rolling maintenance, connection resets flooded the ingestion service.

The engineering team added Resilience4j with exponential backoff retries and a local disk-backed fallback. Transient connection drops recovered within two retry attempts without tripping the circuit breaker. When longer outages occurred, the circuit opened cleanly and streamed telemetry to encrypted local buffers, achieving zero patient data loss and 100% compliance with clinical safety standards.

13. Common Production Anti-Patterns and Fixes

Anti-Pattern 1: Retrying Non-Idempotent HTTP POST Requests

Retrying a failed credit card charge or inventory reservation can cause double charges and duplicate orders. Retries must only run on idempotent operations.

// WRONG: Blindly retrying non-idempotent operations
@Retry(name = "paymentRetry")
public PaymentResponse chargeCreditCard(PaymentRequest request) {
    return restClient.post().uri("/charges").body(request).retrieve().body(PaymentResponse.class);
}

// CORRECT: Only retry safe network failures with an Idempotency-Key
@Retry(name = "paymentRetry", fallbackMethod = "chargeFallback")
public PaymentResponse chargeCreditCardSafely(PaymentRequest request) {
    return restClient.post()
            .uri("/charges")
            .header("Idempotency-Key", request.transactionId())
            .body(request)
            .retrieve()
            .body(PaymentResponse.class);
}

Anti-Pattern 2: Wrapping Everything in ThreadPoolBulkhead on Virtual Threads

Creating thread pool bulkheads when running on Java 25 Virtual Threads wastes CPU and introduces thread-pinning hazards.

// WRONG: Allocating dedicated OS thread pools in Java 25
ThreadPoolBulkheadConfig badConfig = ThreadPoolBulkheadConfig.custom()
        .maxThreadPoolSize(20)
        .coreThreadPoolSize(10)
        .queueCapacity(100)
        .build();

// CORRECT: Use lightweight SemaphoreBulkhead with Virtual Threads
BulkheadConfig modernConfig = BulkheadConfig.custom()
        .maxConcurrentCalls(50)
        .maxWaitDuration(Duration.ofMillis(20))
        .build();

Anti-Pattern 3: Discarding the Original Root Cause in Fallback Methods

Writing a fallback method that swallows the original exception makes debugging in production impossible. Always capture the exception parameter.

// WRONG: Swallowing exception context
public PaymentResponse paymentFallback(PaymentRequest request) {
    return PaymentResponse.fallback(request.transactionId(), request.amount(), request.currency(), "Unknown error");
}

// CORRECT: Logging and recording the exact root cause
public PaymentResponse paymentFallback(PaymentRequest request, Throwable ex) {
    log.warn("Payment fallback for {}. Reason: {}", request.transactionId(), ex.getMessage());
    return PaymentResponse.fallback(request.transactionId(), request.amount(), request.currency(), ex.getClass().getSimpleName() + ": " + ex.getMessage());
}

Anti-Pattern 4: Ignoring Slow Call Rate Thresholds

Many developers configure failure rate thresholds but ignore slow call rate thresholds. In production, a service that takes 15 seconds to reply does more damage than one that fails in 5 milliseconds. Always configure slowCallRateThreshold alongside slowCallDurationThreshold.

// CORRECT: Protect against slow responses, not just hard exceptions
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
        .slidingWindowSize(10)
        .slowCallDurationThreshold(Duration.ofSeconds(2))
        .slowCallRateThreshold(60.0f)
        .failureRateThreshold(50.0f)
        .build();

14. Automated Testing and ArchUnit Verification

A fault-tolerant system must have automated tests verifying state transitions. Below is an integration test showing how the circuit breaker trips from CLOSED to OPEN when the failure rate hits 50%.

@Test
@DisplayName("Should trip circuit breaker from CLOSED to OPEN after failures")
void shouldTripCircuitBreakerToOpen() {
    mockGateway.setSimulateFailure(true);

    // Send 5 failing requests
    for (int i = 1; i <= 5; i++) {
        PaymentRequest req = new PaymentRequest("TX-FAIL-" + i, "ACC-" + i, new BigDecimal("100.00"), "USD");
        PaymentResponse res = paymentService.processPayment(req);
        assertThat(res.degradedFallback()).isTrue();
    }

    Map<String, Object> metrics = paymentService.getCircuitBreakerMetrics();
    assertThat(metrics.get("state")).isEqualTo("OPEN");

    // Subsequent calls fail fast without touching the gateway
    PaymentRequest blocked = new PaymentRequest("TX-BLOCKED", "ACC-BLOCKED", new BigDecimal("50.00"), "USD");
    PaymentResponse blockedRes = paymentService.processPayment(blocked);
    assertThat(blockedRes.message()).contains("Circuit breaker is OPEN");
}

We also enforce architectural rules using ArchUnit 1.4.1. Controllers must never bypass the service layer to talk directly to external gateways.

@ArchTest
static final ArchRule controllers_must_not_access_external_gateways_directly =
        noClasses().that().resideInAPackage("..controller..")
                .should().dependOnClassesThat().haveSimpleNameEndingWith("Gateway")
                .as("Controllers must communicate through Service layer, not external gateways");

15. Companion Workshop Repository

The complete, runnable code for this article is available in our public GitHub repository. It includes all unit tests, MockMvc controller slice tests, and ArchUnit rules running on Java 25 and Spring Boot 4.0.0.

Access the companion code here: https://github.com/salahtobok/learn.webcodein.workshop/tree/main/architecture/resilience4j-circuit-breaker-spring-boot-4-virtual-threads

16. Frequently Asked Questions (FAQ)