How to Build Truly Idempotent Kafka Consumers: A Mathematical Approach to Handling Duplicates
Every Kafka engineer has faced the dreaded duplicate message scenario. You’ve configured your producers for exactly-once semantics, implemented careful error handling, and yet somehow duplicate emails are still reaching customers or duplicate transactions are appearing in your database. The frustration is real, and the solutions often feel like an endless maze of distributed caches, complex transaction patterns, and infrastructure overhead.
The OSO engineers discovered something fascinating whilst working with a major European banking institution: most teams fundamentally misunderstand what idempotency means in distributed systems. This misunderstanding leads to over-engineered solutions that increase complexity whilst failing to solve the core problem.
The breakthrough came when they shifted from trying to prevent duplicates to embracing the mathematical definition of idempotency—where the same input always produces the same output. This approach enabled them to build a system processing over 1 million messages per 5 minutes with minimal resources, using nothing more than standard Kafka libraries and clever database design.
The Idempotency Misconception That’s Costing Engineering Teams
When Kafka introduced the idempotent producer in version 0.11, the documentation stated it “strengthens the semantics of delivery from at least once to exactly once.” This definition created a dangerous conflation between delivery guarantees and true idempotency that continues to mislead engineering teams today.
Kafka’s “exactly-once” semantics create a false sense of security for downstream consumers. Teams assume that if they configure enable.idempotence=true on their producers, duplicate handling becomes someone else’s problem. The reality is more nuanced—whilst the broker won’t store duplicate messages from the same producer session, consumer-side duplicates can still occur through rebalancing, processing timeouts, and application-level failures.
The focus on preventing duplicates rather than handling them leads to architectural complexity. Teams implement distributed caches to track processed message IDs, add transactional outbox patterns to coordinate database writes with Kafka commits, and build complex retry mechanisms—all whilst missing the fundamental insight that duplicates are inevitable in distributed systems.
The mathematical definition of idempotency offers a different approach: a function is idempotent if applying it multiple times with the same input produces the same output. In practical terms, this means designing your consumers so that processing the same message twice results in identical system state, rather than trying to ensure messages are never processed twice.
Real-World Corner Cases Where “Exactly Once” Fails
The OSO engineers catalogued every corner case that could lead to duplicates in their banking system, discovering that even the most carefully configured Kafka setup cannot eliminate all scenarios.
Producer retries create duplicates despite idempotent configuration when network partitions occur during the acknowledgement phase. The producer receives no response and retries, but the original message may have been successfully committed. Whilst the broker can detect duplicates within a single producer session, if the producer restarts or the session expires, duplicate detection fails.
Consumer rebalancing scenarios duplicate message processing when consumers take too long to process batches. If a consumer processes messages but fails to commit offsets before the session timeout, Kafka triggers a rebalance and assigns those partitions to another consumer. The new consumer will reprocess the same messages, potentially duplicating side effects in downstream systems.
Database transaction failures after successful Kafka acknowledgements create the most insidious duplicates. A consumer might successfully publish to an output topic but fail to update its database due to connection issues or constraint violations. When the consumer retries, it will reprocess the original message and potentially publish duplicates to the output topic.
// Example of the problematic pattern that leads to duplicates
@KafkaListener(topics = "input-topic")
public void processMessage(CustomerEvent event) {
try {
// Process and store in database
customerRepository.save(buildCustomerRecord(event));
// Publish to output topic
kafkaTemplate.send("output-topic", buildOutputMessage(event));
// If this fails, we'll reprocess and potentially duplicate the output
} catch (Exception e) {
// Retry logic that can cause duplicates
throw new RetryableException("Processing failed", e);
}
}These corner cases taught the OSO engineers that perfect duplicate prevention is impossible in practice. Instead, they needed to design systems that could gracefully handle duplicates when they inevitably occurred.
The OSO Combined Inbox-Outbox Pattern
Rather than fighting against the reality of duplicates, the OSO engineers designed a micro-batch processing system that assumes duplicates will occur and handles them elegantly. Their combined inbox-outbox pattern leverages database constraints and micro-batch processing to achieve both high throughput and idempotent behaviour.
The system processes messages in batches, validating each event against the expected schema and storing them in a database table with a unique constraint on the message ID. The database handles duplicate detection automatically—if the same message ID appears twice, the database ignores the duplicate insert without throwing an error.
Database-level duplicate detection using unique constraints and batch upserts proved more reliable than application-level caching. The OSO engineers discovered that their database could efficiently handle batch operations with automatic duplicate filtering, eliminating the need for distributed caches or complex coordination mechanisms.
Maintaining streaming semantics whilst ensuring message persistence for audit requirements required careful orchestration. The system stores messages with a “pending” status, processes the entire batch, updates the status to “sent” only after successful publication to the output topic, and commits the Kafka offsets. If any step fails, the batch is retried, but the database constraints ensure no duplicates are created.
// OSO's idempotent micro-batch processing approach
@KafkaListener(topics = "customer-events")
public void processBatch(List<CustomerEvent> events) {
List<CommunicationRecord> records = new ArrayList<>();
// Validate and prepare all messages in the batch
for (CustomerEvent event : events) {
CommunicationRecord record = validateAndBuildRecord(event);
record.setStatus(ProcessingStatus.PENDING);
record.setMessageId(event.getMessageId()); // Unique constraint on this field
records.add(record);
}
try {
// Database handles duplicates via unique constraint
communicationRepository.batchInsertIgnoreDuplicates(records);
// Generate output messages deterministically
List<OutputMessage> outputMessages = records.stream()
.map(this::buildIdempotentOutputMessage)
.collect(toList());
// Publish batch to output topic
kafkaTemplate.send("communication-gateway", outputMessages);
// Mark as processed only after successful publication
communicationRepository.markAsProcessed(
records.stream().map(CommunicationRecord::getMessageId).collect(toList())
);
} catch (Exception e) {
// Batch will be retried, but duplicates are handled by DB constraints
throw new RetryableException("Batch processing failed", e);
}
}This pattern eliminates the need for complex distributed coordination whilst providing strong consistency guarantees through the database’s ACID properties.
Performance Engineering: The Art of Batch Size Optimisation
Achieving 1 million messages per 5 minutes required careful tuning of batch sizes and database interactions. The OSO engineers discovered that batch size optimization is more art than science, requiring empirical testing with real workloads.
Through extensive performance testing, they determined that batch sizes between 20-35 messages provided optimal throughput for their use case. Smaller batches increased the overhead of database round trips, whilst larger batches risked timeout issues and increased memory pressure during retries.
Balancing database capacity, memory consumption, and latency requirements meant understanding the database’s batch processing capabilities. Their database could efficiently process batches of up to 50 records, but the optimal size also depended on message size, validation complexity, and the speed of the output topic producers.
Resource utilisation strategies enabled impressive efficiency gains. By processing messages in carefully sized batches and leveraging database-level duplicate detection, they eliminated the need for additional caching infrastructure or complex schedulers. The entire system ran on minimal resources whilst dramatically outperforming their previous message-by-message processing approach.
// Configuration for optimal batch processing
@Component
public class BatchProcessingConfig {
@Value("${kafka.consumer.batch-size:25}")
private int batchSize;
@Value("${kafka.consumer.timeout:5000}")
private int batchTimeout;
@Bean
public ConsumerFactory<String, CustomerEvent> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, batchSize);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, batchTimeout);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
// Process at least once - we handle duplicates at application level
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
return new DefaultKafkaConsumerFactory<>(props);
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, CustomerEvent>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, CustomerEvent> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setBatchListener(true);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
return factory;
}
}The key insight was that batch size optimization requires understanding the entire pipeline—from Kafka consumer configuration through database performance characteristics to downstream producer capacity.
Practical Implementation Strategies
Building truly idempotent consumers requires attention to several implementation details that ensure consistent behaviour under all conditions.
Validation patterns that ensure message integrity before database insertion prevent corrupt data from causing inconsistent system state. The OSO engineers implemented comprehensive validation that normalises message formats, handles missing fields gracefully, and marks invalid messages for monitoring rather than failing the entire batch.
Status management techniques maintain streaming flow without complex schedulers by keeping all state transitions within the same database transaction. Rather than implementing separate scheduler processes to handle failed messages, the system retries entire batches and relies on the database constraints to maintain consistency.
Testing methodologies for corner cases and performance characteristics proved crucial for building confidence in the system. The engineers created comprehensive test suites that simulated network partitions, database failures, consumer rebalances, and various timing issues to ensure the system behaved correctly under adverse conditions.
The validation approach ensures that duplicate messages not only produce the same output but also store identical data structures, making the system truly idempotent rather than just duplicate-resistant.
Architectural Simplification and Long-Term Maintenance
Embracing mathematical idempotency rather than fighting duplicates led to dramatic architectural simplification. The OSO engineers eliminated distributed caches, complex schedulers, and coordination mechanisms that had plagued their previous implementations.
How embracing idempotency reduces dependencies on distributed caches and external schedulers becomes clear when you consider the maintenance overhead of these components. Distributed caches require cluster management, data expiration policies, and consistency mechanisms. External schedulers need failure handling, state persistence, and coordination logic. The database-centric approach eliminates these dependencies whilst providing stronger consistency guarantees.
Scalability patterns that grow linearly with partition count make capacity planning straightforward. Since each consumer instance processes its assigned partitions independently and the database handles all coordination, scaling simply means adding more consumer instances and Kafka partitions. There are no shared state stores or coordination bottlenecks to complicate horizontal scaling.
Maintenance benefits of using standard Kafka libraries without custom extensions reduce the operational burden significantly. The system uses only well-documented Kafka features and relies on battle-tested database capabilities. This approach minimises the risk of subtle bugs in custom code and makes the system accessible to engineers who understand standard Kafka patterns.
The long-term stability of this approach became evident over five years of production operation. Despite processing millions of messages and encountering every conceivable failure scenario, the system required no architectural changes or resource increases. The simplicity of the design made debugging straightforward and performance tuning predictable.
Conclusion
The journey from complex duplicate-prevention mechanisms to simple idempotent design reveals a fundamental truth about distributed systems: fighting against their nature leads to brittleness, whilst embracing their characteristics enables elegance.
True idempotency—mathematical consistency rather than duplicate prevention—enables simpler, more maintainable architectures. By designing systems where the same input always produces the same output, regardless of how many times it’s processed, engineers can build robust systems that handle the inevitable failures and edge cases of distributed computing.
The OSO engineers’ success demonstrates that focusing on use-case-specific solutions rather than universal “exactly-once” implementations leads to better engineering outcomes. Their system processes millions of messages with minimal resources, maintains strong consistency guarantees, and operates reliably in production—all whilst using standard Kafka libraries and simple database operations.
For teams struggling with duplicate handling in their Kafka architectures, the path forward involves auditing current duplicate-handling strategies and asking a fundamental question: are you solving the right problem? Instead of preventing duplicates, consider designing systems that handle them gracefully. The mathematical definition of idempotency provides a powerful framework for building systems that are both simpler and more robust than their duplicate-prevention counterparts.
The next time you encounter duplicate messages in your Kafka system, remember that the solution might not involve more complexity—it might involve embracing the duplicates and designing your system to handle them with mathematical precision.
This post first appeared on Read More