Event-Driven Microservices in Java 25: Apache Kafka & Spring Boot 4

In modern cloud-native architectures, synchronous REST-based communication between microservices often introduces significant operational friction. Tight temporal coupling, cascading network timeouts, thread starvation, and brittle distributed transactions turn simple system requests into fragile domino chains. When a downstream inventory service or billing gateway slows down, upstream services stall, consuming memory and connection pools until the entire application cluster exhausts its resources.

To build distributed systems that scale reliably to millions of operations per second, enterprise engineering teams rely on Event-Driven Architecture (EDA) powered by Apache Kafka. With the arrival of Spring Boot 4 and Java 25, integrating with Kafka has evolved dramatically. Modern features such as the declarative @ServiceConnection Testcontainers SPI, native JSON serialization, virtual-thread concurrency, and robust Dead Letter Topic (DLT) error handling provide unprecedented developer ergonomics and runtime resilience. This comprehensive technical guide walks you through designing, implementing, testing, and optimizing production-grade event-driven microservices in Java 25 and Spring Boot 4.

Table of Contents

• Why Event-Driven Architecture? Moving Beyond Synchronous REST
• Comparative Analysis: Messaging and Inter-Service Communication Strategies
• Kafka Architecture Fundamentals: Topics, Partitions, and Offsets
• Event Payloads, JSON Serialization, and Partition Key Routing
• Hands-On Workshop: Building Producers and Consumers with Spring Boot 4
• Consumer Group Scaling, Rebalances, and Offset Management
• Integration Testing with Ephemeral Kafka Testcontainers
• Enterprise Use Cases: Real-World Event-Driven Topologies
• CI/CD Pipeline Automation: GitHub Actions and Docker Optimization
• Production Readiness and Enterprise Guardrails
• Common Pitfalls and How to Fix Them
• Frequently Asked Questions (FAQ)
• Conclusion and Next Steps
• References and Companion Code

Why Event-Driven Architecture? Moving Beyond Synchronous REST

Traditional REST architectures demand that both the client and the server be operational at the exact same instant. If Service A makes a synchronous HTTP POST call to Service B, Service A must block a thread while waiting for Service B to parse the payload, query its database, and return an HTTP status code. If Service B calls Service C, the latency compounds exponentially.

Event-Driven Architecture replaces this synchronous coupling with asynchronous, immutable facts known as events. In an event-driven system:

• Producers and Consumers are Completely Decoupled: An order microservice records that an order was created (OrderCreatedEvent) and publishes it to a Kafka topic. It does not know or care how many downstream systems consume the event, nor does it wait for them to finish.
• Temporal Decoupling and Resilience: If the billing service or inventory service is temporarily offline for a rolling deployment or experiencing a transient network partition, messages buffer safely inside Kafka’s durable commit log. When the consumer comes back online, it resumes reading from its last committed offset with zero data loss.
• Horizontal Scalability via Partitioning: By distributing topic data across multiple partitions, Kafka allows consumer groups to scale horizontally. Adding more consumer instances dynamically distributes partition workloads across physical cluster nodes.

Comparative Analysis: Messaging and Inter-Service Communication Strategies

Choosing the appropriate communication backbone depends on latency requirements, message persistence, ordering guarantees, and throughput demands. The following matrix compares the leading enterprise paradigms:

Kafka Architecture Fundamentals: Topics, Partitions, and Offsets

To master Kafka with Spring Boot, you must understand how data is organized under the hood. Unlike traditional message queues where messages are pushed and then destroyed upon consumption, Kafka is an append-only distributed commit log.

Topics and Partitions

A Topic is a logical category or feed name to which records are published. Topics in Kafka are subdivided into Partitions. Each partition is an ordered, immutable sequence of records that is continually appended to. Partitions allow a topic to scale beyond the storage and processing limits of a single server. Kafka stores each partition across one or more cluster brokers according to the configured replication factor.

Offsets and Immutability

Every record written to a partition is assigned a sequential, monotonically increasing integer called an Offset. The offset uniquely identifies each record within its partition. Because records are immutable, consumers maintain their own position in the log by committing their current offset. This allows multiple consumer groups to read the same topic at completely different paces without interfering with each other.

Event Payloads, JSON Serialization, and Partition Key Routing

In high-throughput event-driven microservices, data serialization and partition routing dictate system performance and ordering correctness.

Deterministic Partition Key Routing

When producing an event, the producer can specify a partition key (such as orderId or customerId). Kafka uses a Murmur2 hashing function on the serialized key bytes to calculate the target partition:

partition = murmur2(keyBytes) % numPartitions

This mathematical guarantee ensures that all events sharing the exact same key (for example, OrderCreated, OrderPaid, OrderShipped) land in the exact same partition. Because Kafka strictly preserves order within a partition, consumers are guaranteed to process state transitions in their exact chronological sequence.

Designing Clean Event Envelopes

Enterprise systems wrap domain payloads inside a standard metadata envelope. This ensures traceability across microservice boundaries:

package com.webcodein.kafka.model;

import java.math.BigDecimal;
import java.time.Instant;
import java.util.UUID;

public record OrderEvent(
    UUID eventId,
    String orderId,
    BigDecimal amount,
    String status,
    Instant timestamp,
    String traceId
) {
    public OrderEvent(String orderId, BigDecimal amount, String status) {
        this(UUID.randomUUID(), orderId, amount, status, Instant.now(), UUID.randomUUID().toString());
    }
}

Hands-On Workshop: Building Producers and Consumers with Spring Boot 4

Let us explore the companion workshop project located in workshop/architecture/event-driven-kafka-spring-boot-4. The application demonstrates an enterprise order processing pipeline using Spring Boot 4, Java 25, and Spring Kafka.

1. Maven Build Dependencies

In Spring Boot 4, import the spring-boot-starter-kafka module alongside jackson-databind and spring-boot-docker-compose:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-kafka</artifactId>
</dependency>

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-docker-compose</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>

2. Application Configuration

Configure the producer and consumer serializers in application.yml. To prevent remote execution vulnerabilities, always specify trusted packages for JSON deserialization:

spring:
  application:
    name: event-driven-kafka-demo
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      acks: all
      retries: 3
    consumer:
      group-id: order-group
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "com.webcodein.kafka.model"

3. Declarative Topic Creation

Spring Kafka provides a NewTopic bean to automatically provision Kafka topics on application startup if they do not exist:

package com.webcodein.kafka.config;

import org.apache.kafka.clients.admin.NewTopic;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.TopicBuilder;

@Configuration
public class KafkaTopicConfig {

    public static final String ORDERS_TOPIC = "orders-topic";

    @Bean
    public NewTopic ordersTopic() {
        return TopicBuilder.name(ORDERS_TOPIC)
                .partitions(3)
                .replicas(1)
                .build();
    }
}

4. Implementing the Producer Service

The OrderProducerService leverages Spring’s KafkaTemplate to publish events asynchronously. We pass orderId as the partition key to guarantee ordering:

package com.webcodein.kafka.service;

import com.webcodein.kafka.config.KafkaTopicConfig;
import com.webcodein.kafka.model.OrderEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

import java.math.BigDecimal;
import java.util.UUID;

@Service
public class OrderProducerService {

    private static final Logger log = LoggerFactory.getLogger(OrderProducerService.class);
    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    public OrderProducerService(KafkaTemplate<String, OrderEvent> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public OrderEvent sendOrder(double amount) {
        String orderId = "ORD-" + UUID.randomUUID().toString().substring(0, 8);
        OrderEvent event = new OrderEvent(orderId, BigDecimal.valueOf(amount), "CREATED");

        log.info("Publishing order event with key [{}]: {}", orderId, event);
        kafkaTemplate.send(KafkaTopicConfig.ORDERS_TOPIC, orderId, event);
        return event;
    }
}

5. Implementing the Consumer Service

Spring’s @KafkaListener annotation connects a worker method to one or more topics. The framework manages partition assignment, polling loops, and offset commits behind the scenes:

package com.webcodein.kafka.service;

import com.webcodein.kafka.config.KafkaTopicConfig;
import com.webcodein.kafka.model.OrderEvent;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;

@Service
public class OrderConsumerService {

    private static final Logger log = LoggerFactory.getLogger(OrderConsumerService.class);
    private final List<OrderEvent> processedOrders = new CopyOnWriteArrayList<>();

    @KafkaListener(topics = KafkaTopicConfig.ORDERS_TOPIC, groupId = "order-group")
    public void consume(OrderEvent event) {
        log.info("Successfully received order event: {}", event);
        processedOrders.add(event);
    }

    public List<OrderEvent> getProcessedOrders() {
        return processedOrders;
    }
}

Consumer Group Scaling, Rebalances, and Offset Management

In distributed systems, individual microservice instances will scale up, crash, or restart. Kafka manages this elasticity through Consumer Groups and dynamic Rebalances.

The Cooperative Sticky Assignor

In earlier Kafka versions, adding or removing a consumer triggered an “eager rebalance”. All consumers dropped their partitions, paused message consumption, and waited for the Group Coordinator to reassign partitions from scratch. This caused global latency spikes known as “stop-the-world rebalances”.

Modern Spring Boot applications use the Cooperative Sticky Assignor (CooperativeStickyAssignor). Under this protocol, consumers continue processing unaffected partitions while only the reassigned partitions are migrated. This ensures near-zero downtime during rolling updates and auto-scaling events.

Committed Offset Storage

Consumers track their processing progress by committing offsets to an internal, compacted Kafka topic called __consumer_offsets. By pairing sensible commit intervals with idempotent message handling or the Transactional Outbox Pattern, applications achieve true at-least-once or exactly-once semantics.

Integration Testing with Ephemeral Kafka Testcontainers

Unit tests that mock Kafka produce false positives because they fail to test real network polling, consumer lag, JSON serialization, and partition routing. For robust verification, use Testcontainers.

Spring Boot 4 @ServiceConnection with Kafka

In Spring Boot 4, the @ServiceConnection annotation inspects the running KafkaContainer, discovers its dynamically assigned broker port (e.g. localhost:54321), and automatically injects spring.kafka.bootstrap-servers into the Spring Environment. This completely eliminates manual @DynamicPropertySource boilerplate:

package com.webcodein.kafka;

import com.webcodein.kafka.model.OrderEvent;
import com.webcodein.kafka.service.OrderConsumerService;
import com.webcodein.kafka.service.OrderProducerService;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.testcontainers.containers.KafkaContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.utility.DockerImageName;

import java.time.Duration;

import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;

@SpringBootTest
@Testcontainers
class KafkaIntegrationTest {

    @Container
    @ServiceConnection
    static KafkaContainer kafka = new KafkaContainer(
            DockerImageName.parse("confluentinc/cp-kafka:7.7.1")
    );

    @Autowired
    private OrderProducerService producerService;

    @Autowired
    private OrderConsumerService consumerService;

    @Test
    @DisplayName("Should successfully publish order event and consume asynchronously")
    void shouldProduceAndConsumeOrderEvent() {
        // Arrange & Act
        OrderEvent published = producerService.sendOrder(149.99);

        // Assert asynchronous consumption using Awaitility
        await().atMost(Duration.ofSeconds(10)).untilAsserted(() -> {
            assertThat(consumerService.getProcessedOrders())
                    .extracting(OrderEvent::orderId)
                    .contains(published.orderId());
        });
    }
}

For more details on container testing patterns, see our dedicated guide on TDD with Spring Boot 4 and Testcontainers Zero-Mocking.

Enterprise Use Cases: Real-World Event-Driven Topologies

Use Case 1: Order Fulfillment Choreography

In an e-commerce platform, placing an order requires inventory reservation, payment authorization, and shipment scheduling. Rather than using an orchestrator that synchronously calls each microservice, the order service emits an OrderCreatedEvent. The inventory service reserves stock, the payment service charges the credit card, and the notification service emails the user—all operating independently in parallel.

Use Case 2: Real-Time Fraud Detection with Partition Key Routing

A banking application inspects credit card swipes in real-time. By publishing transactions partitioned by cardNumber, all swipes for a specific card arrive in a single partition. A fraud detection consumer evaluates sliding time-window metrics (such as multiple transactions in different cities within 10 minutes) without needing distributed locks across multiple cluster nodes.

Use Case 3: Change Data Capture (CDC) and Database Outbox Sync

To avoid dual-write distributed transaction failures between relational databases and Kafka, enterprises use the Transactional Outbox Pattern. The business service writes to an outbox table within the local database transaction, and a CDC engine (such as Debezium) streams change events to Kafka, ensuring zero message loss even during database failovers.

CI/CD Pipeline Automation: Fast GitHub Actions Execution

Running Kafka integration tests in CI pipelines requires fast Docker provisioning. Modern GitHub Actions runners execute Testcontainers seamlessly when pre-pulling container images:

name: CI Build & Kafka Integration Tests

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 15

    steps:
      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up JDK 25
        uses: actions/setup-java@v4
        with:
          java-version: '25'
          distribution: 'temurin'
          cache: maven

      - name: Pre-pull Kafka Image
        run: docker pull confluentinc/cp-kafka:7.7.1

      - name: Run Maven Tests
        run: mvn -B -ntp clean test

Production Readiness and Enterprise Guardrails

Follow these architectural guardrails when running Spring Boot Kafka services in production:

• Configure Idempotent Producers: Always enable spring.kafka.producer.properties.enable.idempotence=true (default in Kafka 3.0+). This guarantees that network retries never produce duplicate messages in the broker.
• Pair with Dead Letter Topics (DLT): When a consumer encounters a malformed payload or unrecoverable business exception, it should not crash or stall partition consumption. Route bad records to a DLT topic (e.g. orders-topic.DLT) for manual review.
• Tune Max Poll Interval: If your consumer executes long-running tasks, configure max.poll.interval.ms appropriately. If the processing duration exceeds this threshold, the Kafka coordinator assumes the consumer died and triggers an unnecessary rebalance storm.
• Leverage Java 25 Virtual Threads: For consumers performing downstream blocking I/O (such as writing to a relational database or calling an external HTTP API), enable Java 25 Virtual Threads via spring.threads.virtual.enabled=true to process incoming records concurrently without carrier thread exhaustion.

Common Pitfalls and How to Fix Them

Pitfall 1: Poison Pill Deserialization Loop

Symptom: Consumer logs rapidly flood with SerializationException, CPU spikes to 100%, and consumption completely halts.

Root Cause: A producer published a corrupted or incompatible JSON payload that fails to deserialize into the target Java class.

Solution: Wrap deserializers inside Spring’s ErrorHandlingDeserializer. It intercepts serialization errors, delegates the record to an error handler, and routes it to a dead letter topic rather than repeatedly retrying.

Pitfall 2: Keyless Messages Breaking Chronological Order

Symptom: Consumers process OrderShipped events before OrderCreated events.

Root Cause: The producer sent records with a null key. Kafka distributes keyless messages in a round-robin fashion across partitions, eliminating causal ordering.

Solution: Always provide a deterministic partition key (e.g., orderId) when calling kafkaTemplate.send().

Pitfall 3: Rebalance Storms on Slow Downstream Calls

Symptom: Consumers continuously disconnect, partitions flap, and consumer lag grows uncontrollably.

Root Cause: Heavy downstream operations cause the consumer poll loop to exceed max.poll.interval.ms.

Solution: Lower max.poll.records (e.g., from 500 to 50), increase max.poll.interval.ms, or offload processing to asynchronous virtual thread worker pools.

Frequently Asked Questions (FAQ)

When should I choose Apache Kafka over RabbitMQ?

Choose Kafka when you need high-throughput distributed commit logging, message replayability, stream processing, or event sourcing where consumers must read events at different speeds. Choose RabbitMQ when you need complex broker-side routing (AMQP topic/fanout exchanges) or strict per-message acknowledgement for transient task queues.

Can Spring Kafka consumers use Java 25 Virtual Threads?

Yes. Setting spring.threads.virtual.enabled=true in Spring Boot 4 configures the underlying ConcurrentMessageListenerContainer task executor to dispatch consumer worker tasks onto Project Loom virtual threads, allowing massive concurrency for I/O-heavy processing.

How does @ServiceConnection work with Testcontainers Kafka?

Spring Boot 4’s @ServiceConnection automatically inspects the KafkaContainer instance in your JUnit 5 test, extracts the dynamically mapped bootstrap server address, and binds it to the Spring environment, completely eliminating manual property mappings.

Does Kafka guarantee Exactly-Once Processing (EOS)?

Kafka provides Exactly-Once Semantics (EOS) across Kafka-to-Kafka stream pipelines using transactional producers and consumer read-committed isolation. When writing to external non-Kafka databases, pair your consumer with idempotent operations or the Transactional Outbox pattern to achieve end-to-end exactly-once guarantees.

How does Spring Boot 4 Docker Compose simplify Kafka local development?

With spring-boot-docker-compose on your classpath, running mvn spring-boot:run automatically boots Kafka in Docker, registers the bootstrap broker address, and shuts down the container when your application exits. Learn more in our Modern Local Development Guide.

Conclusion and Next Steps

Event-Driven Architecture is essential for building decoupled, horizontally scalable, and resilient microservices. By harnessing Spring Boot 4 and Apache Kafka with Java 25, developers can build reactive systems with unmatched developer productivity.

From deterministic partition key routing and cooperative rebalancing to zero-boilerplate testing with Testcontainers and @ServiceConnection, the modern Java ecosystem provides a bulletproof foundation for event streaming.

To deepen your event-driven expertise, explore our companion architectural guides on the Transactional Outbox Pattern in Spring Boot 4 and Zero-Mocking TDD with Testcontainers.

References and Companion Code

• Companion Workshop Code on GitHub (Spring Boot 4, Apache Kafka, Testcontainers)
• Official Spring for Apache Kafka Documentation
• Apache Kafka Official Documentation
• Testcontainers for Kafka Guide
• Confluent Kafka Architecture Best Practices