Making Kafka Consumers Idempotent: A Spring Boot Playbook

Every Kafka tutorial starts the same way: "Kafka guarantees at-least-once delivery." Every production incident that involves Kafka ends the same way: someone stares at a duplicate charge, a duplicate email, or two rows where there should be one, and asks how it happened.

The answer is always the same. At-least-once means at-least-once. Your consumer will see the same message more than once. If your handler isn't idempotent, that's a bug in your code — not Kafka's.

Why duplicates happen

The most common cause isn't dramatic. A consumer processes a message, does the work, and dies before committing the offset. The rebalance hands the partition to another instance, which reads the same message again. Same story if the broker times out waiting for a commit, or if the consumer's poll loop lags past max.poll.interval.ms.

Retries make it worse. Every retry is another chance to double-process. And any time you're calling an external API from inside the handler — a payment provider, a webhook, an email service — the network can lie to you: the request succeeded, the response got lost, you retry.

The pattern I keep reaching for

Every consumer I ship has the same first ten lines: check a dedupe key in Redis, exit early if it's already been processed, otherwise take the lock and do the work. Here's what it looks like in Spring Boot:

@KafkaListener(topics = "orders.placed", groupId = "billing-service")
public void onOrderPlaced(ConsumerRecord<String, OrderPlacedEvent> record,
                          Acknowledgment ack) {
    var event = record.value();
    var dedupeKey = "orders.placed:%s:%s".formatted(event.orderId(), event.eventId());

    // SET key value NX EX 86400 — set only if absent, expire in 24h
    var acquired = redis.opsForValue()
        .setIfAbsent(dedupeKey, "processing", Duration.ofHours(24));

    if (Boolean.FALSE.equals(acquired)) {
        log.debug("Skipping duplicate {}", dedupeKey);
        ack.acknowledge();
        return;
    }

    try {
        billing.charge(event);
        ack.acknowledge();
    } catch (Exception e) {
        // Release the lock so a retry can re-process
        redis.delete(dedupeKey);
        throw e;
    }
}

The dedupe key is (topic, entity-id, event-id). Topic scopes it so an "orders.placed" event and an "orders.updated" event with the same order id don't collide. The event-id is the producer's unique id per emission — not the Kafka offset, which changes across topics.

The 24-hour TTL is a bet: if a duplicate arrives more than a day late, either something is very wrong or the operation is safe to re-run. Tune it to your consumer lag SLO.

When Redis isn't enough

Redis dedupe is fast and cheap, but it's a soft guarantee: the key can be evicted under memory pressure, the cluster can lose a shard, or a race between two consumers can slip through if you get the ordering wrong. For anything financial, that's not good enough.

For those, I push the dedupe key into the same database transaction as the write. A unique constraint on (event_id) in your idempotency table means the second insert throws a constraint violation — you catch it, treat it as "already processed", and ack. The database is the single source of truth for "did this event's side-effect land".

The one thing not to do

Don't rely on the Kafka offset alone. Committed offsets tell you what your consumer thinks it processed, not what actually landed downstream. If the crash is between "I called the payment API" and "I committed the offset," the offset tells you nothing useful — the money already moved.

The dedupe key belongs to the business event, not to Kafka. Treat it that way and the rest of the failure modes stop mattering.