Dead-Letter Queue
When configuring error handling for your event processors, you might want to consider a Dead-Letter Queue to park events that you were unable to handle.
Instead of either logging the error and continuing, or infinitely retrying the current event, a Dead-Letter Queue will park the event in the queue so you can decide to try and handle it again later. In addition, it will prevent handling of later events in the same sequence until the failed event is successfully processed, preserving correct event ordering.
|
Insight and management
Axoniq Platform provides insight into the Dead-Letter Queue and tools for its management.
It’s straightforward to see the dead letters in the queue and decide to retry them or remove them from the queue.
You can find more information on the Dead-Letter Queue page of Axoniq Platform.
|
Queue scoping and naming
Each Event Handling Component within a processor receives its own dead-letter queue instance.
This ensures dead letters are scoped to a single component, preventing interference between different event handlers.
Each queue is identified by a name following the pattern DeadLetterQueue[processorName][componentName], where processorName is the name of the event processor and componentName is the name of the Event Handling Component within that processor.
This naming convention is also used as the processingGroup when creating queue instances through the SequencedDeadLetterQueueFactory.
The componentName is the name you provide when registering Event Handling Components.
With the Configuration API, this is the name passed to declarative("myHandler", …) or autodetected("myHandler", …).
With Spring Boot, it is the Spring bean name of the event handler.
It is important to keep component names stable, as they are part of the queue identifier used to persist dead letters.
Changing a component name would effectively orphan any previously dead-lettered events.
For example, given a processor named order-processor with a component registered as orderProjection, the queue name would be DeadLetterQueue[order-processor][orderProjection].
Event ordering
Axon Framework’s event processors maintain the ordering of events within the same sequence, even when you configure parallel processing. A typical example is handling events for the same entity in publishing order: if one event fails, applying subsequent events to that entity would produce inconsistent state.
Therefore, a Dead-Letter Queue must enqueue both the failed event and all subsequent events in the same sequence.
To that end, the supported dead-letter queue is a SequencedDeadLetterQueue.
It tracks sequences by associating each event with a sequence identifier, which is determined by the sequencing policy.
Implementations
The following dead-letter queue implementations are available:
-
InMemorySequencedDeadLetterQueue- In-memory variant of the dead-letter queue. Useful for testing purposes, but as it does not persist dead letters, it is unsuited for production environments. -
JpaSequencedDeadLetterQueue- JPA variant of the dead-letter queue. It persists failed events in adead_letter_entrytable. The JPA dead-letter queue is a suitable option for production environments. -
JdbcSequencedDeadLetterQueue- JDBC variant of the dead-letter queue. It persists failed events in adead_letter_entrytable. The JDBC dead-letter queue is a suitable option for production environments and allows customization of the SQL dialect via aDeadLetterStatementFactory.
Idempotency
Before configuring a SequencedDeadLetterQueue it is vital to validate whether your event handling functions are idempotent.
As a processor consists of several Event Handling Components (as explained in the intro of this chapter), some handlers may succeed in event handling while others will not.
A configured dead-letter queue does not stall event handling, so a failure in one Event Handling Component does not cause a rollback for other event handlers.
Furthermore, when the failed event is retried later via dead-letter processing, only the Event Handling Component that originally failed is invoked again. However, all event handlers within that component that match the event are invoked.
If your event handlers are not idempotent, retrying dead letters may produce undesired side effects. We strongly recommend making your event handlers idempotent when using the dead-letter queue.
The principle of exactly once delivery is no longer guaranteed; at-least-once delivery is the reality to cope with.
Configuration
Dead-letter queue configuration happens through the DeadLetterQueueConfiguration class, which provides a fluent API for enabling and customizing the queue.
The configuration is applied on the PooledStreamingEventProcessorConfiguration using the deadLetterQueue method.
-
Configuration API
-
Spring Boot
import org.axonframework.messaging.core.configuration.MessagingConfigurer;
import org.axonframework.messaging.eventhandling.configuration.EventProcessorModule;
public class AxonConfig {
public void configureDeadLetterQueue() {
var configurer = MessagingConfigurer.create();
configurer.eventProcessing(ep -> ep.pooledStreaming(ps -> ps
.processor(
EventProcessorModule
.pooledStreaming("my-processor")
.eventHandlingComponents(components -> components
.declarative("myHandler", cfg -> myHandlerComponent))
.customized((cfg, processorConfig) -> processorConfig
.deadLetterQueue(dlq -> dlq.enabled()))
)
));
}
}In Spring Boot, enabling the dead-letter queue for a specific processor only requires a property:
axon.eventhandling.processors.my-processor.dlq.enabled=trueSpring Boot auto-configuration will automatically detect whether JPA or JDBC is available and create the appropriate SequencedDeadLetterQueueFactory bean.
If both JPA and JDBC are available, the auto-configuration creates a JPA-based queue by default.
No additional bean definitions are needed for the default setup.
|
Enabling the DLQ via properties enables it for all Event Handling Components assigned to that processor.
Each component gets its own separate queue instance (see Queue scoping and naming), but the enabled/disabled setting applies uniformly to the entire processor.
To selectively enable or disable the DLQ for individual components, use the Configuration API or an |
You can set the maximum number of saved sequences (defaults to 1024) and the maximum number of dead letters in a sequence (also defaults to 1024).
If either of these thresholds is exceeded, the queue will throw a DeadLetterQueueOverflowException.
This exception means the processor will stop processing new events altogether.
Thus, the processor moves back to the behavior described at the start of the Error Handling section.
Using a custom factory
To customize how the dead-letter queue is created (for example, to set maxSequences or maxSequenceSize), you can provide a custom SequencedDeadLetterQueueFactory.
The Configuration API example below uses InMemorySequencedDeadLetterQueue to keep it concise; for production, replace it with JpaSequencedDeadLetterQueue or JdbcSequencedDeadLetterQueue as shown in the Spring Boot tab:
-
Configuration API
-
Spring Boot
import org.axonframework.messaging.core.configuration.MessagingConfigurer;
import org.axonframework.messaging.deadletter.InMemorySequencedDeadLetterQueue;
import org.axonframework.messaging.eventhandling.configuration.EventProcessorModule;
public class AxonConfig {
public void configureDeadLetterQueue() {
var configurer = MessagingConfigurer.create();
configurer.eventProcessing(ep -> ep.pooledStreaming(ps -> ps
.processor(
EventProcessorModule
.pooledStreaming("my-processor")
.eventHandlingComponents(components -> components
.declarative("myHandler", cfg -> myHandlerComponent))
.customized((cfg, processorConfig) -> processorConfig
.deadLetterQueue(dlq -> dlq
.enabled()
.factory((name, config) ->
InMemorySequencedDeadLetterQueue.builder()
.maxSequences(256)
.maxSequenceSize(256)
.build())))
)
));
}
}Define a custom SequencedDeadLetterQueueFactory bean and enable the DLQ via properties.
The factory receives the queue name (following the DeadLetterQueue[processorName][beanName] pattern from Queue scoping and naming, where beanName is the Spring bean name of the event handler) as its first parameter, and is called once per Event Handling Component.
When this bean is present, it replaces the Spring Boot auto-configured factory for all processors where the DLQ is enabled:
import org.axonframework.conversion.Converter;
import org.axonframework.messaging.eventhandling.conversion.EventConverter;
import org.axonframework.messaging.eventhandling.deadletter.SequencedDeadLetterQueueFactory;
import org.axonframework.messaging.eventhandling.deadletter.jpa.JpaSequencedDeadLetterQueue;
import org.axonframework.messaging.core.unitofwork.transaction.jpa.JpaTransactionalExecutorProvider;
import jakarta.persistence.EntityManagerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class DlqConfig {
@Bean
SequencedDeadLetterQueueFactory myDeadLetterQueueFactory(
EntityManagerFactory entityManagerFactory,
EventConverter eventConverter,
Converter converter
) {
return (processingGroup, config) ->
JpaSequencedDeadLetterQueue.builder()
.processingGroup(processingGroup)
.maxSequences(256)
.maxSequenceSize(256)
.transactionalExecutorProvider(
new JpaTransactionalExecutorProvider(entityManagerFactory))
.eventConverter(eventConverter)
.genericConverter(converter)
.build();
}
}axon.eventhandling.processors.my-processor.dlq.enabled=trueSequence identifier caching
To speed up event processing, the framework caches the sequence identifiers of sequences that are currently in the dead-letter queue.
Before processing each event, it checks this cache to determine whether the event’s sequence is already dead-lettered. If it is, the event must be enqueued rather than processed normally.
This avoids a database round-trip for every healthy event that is not dead-lettered.
Internally, the dead-letter queue is wrapped with a CachingSequencedDeadLetterQueue to provide this optimization.
Size the cache to match the maximum number of distinct failing sequences you expect at any given time. Increase it if you regularly have more concurrent failing sequences than the default of 1024. Set it to 0 to disable caching entirely. This is useful when memory is constrained or when failing sequences are highly dynamic and cache hits would be rare:
-
Configuration API
-
Spring Boot
import org.axonframework.messaging.core.configuration.MessagingConfigurer;
import org.axonframework.messaging.eventhandling.configuration.EventProcessorModule;
public class AxonConfig {
public void configureDeadLetterQueue() {
var configurer = MessagingConfigurer.create();
configurer.eventProcessing(ep -> ep.pooledStreaming(ps -> ps
.processor(
EventProcessorModule
.pooledStreaming("my-processor")
.eventHandlingComponents(/* ... */)
.customized((cfg, processorConfig) -> processorConfig
.deadLetterQueue(dlq -> dlq
.enabled()
.cacheMaxSize(4096)))) // Set a larger cache
));
}
}axon.eventhandling.processors.my-processor.dlq.enabled=true
axon.eventhandling.processors.my-processor.dlq.cache.size=4096To disable caching entirely:
axon.eventhandling.processors.my-processor.dlq.cache.size=0Applying defaults across processors
|
This feature is available through the Configuration API only.
In Spring Boot, configure each processor individually using |
You can set dead-letter queue defaults that apply to all pooled streaming processors, which individual processors can then override:
import org.axonframework.messaging.core.configuration.MessagingConfigurer;
import org.axonframework.messaging.eventhandling.configuration.EventProcessorModule;
public class AxonConfig {
public void configureDeadLetterQueue() {
var configurer = MessagingConfigurer.create();
configurer.eventProcessing(ep -> ep.pooledStreaming(ps -> ps
// Enable DLQ for all pooled streaming processors by default
.defaults(d -> d.deadLetterQueue(dlq -> dlq
.enabled()
.cacheMaxSize(2048)))
// This processor inherits the defaults
.processor(EventProcessorModule
.pooledStreaming("processor-with-dlq")
.eventHandlingComponents(/* ... */)
.notCustomized())
// This processor explicitly disables DLQ
.processor(EventProcessorModule
.pooledStreaming("processor-without-dlq")
.eventHandlingComponents(/* ... */)
.customized((cfg, c) -> c
.deadLetterQueue(dlq -> dlq.disabled())))
));
}
}Configuring DLQ with an EventProcessorDefinition bean
When using Spring Boot with EventProcessorDefinition beans, you can apply custom DLQ configuration through the customized step.
When enabled, the framework creates a separate dead-letter queue for each Spring bean matched by assigningHandlers, using the bean name as the component name.
For example, if assigningHandlers matches beans orderProjection and orderNotifier, the factory will create queues named DeadLetterQueue[order-processor][orderProjection] and DeadLetterQueue[order-processor][orderNotifier].
import org.axonframework.extension.spring.config.EventProcessorDefinition;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class ProcessorConfig {
@Bean
EventProcessorDefinition orderProcessor() {
return EventProcessorDefinition.pooledStreaming("order-processor")
.assigningHandlers(descriptor ->
descriptor.beanName().startsWith("order"))
.customized(config -> config
.deadLetterQueue(dlq -> dlq
.enabled()
.cacheMaxSize(2048)));
}
}Properties-based DLQ configuration (axon.eventhandling.processors.<name>.dlq.enabled=true) is applied before the EventProcessorDefinition customization.
The EventProcessorDefinition always takes precedence: calling .deadLetterQueue(dlq → dlq.enabled()) or .deadLetterQueue(dlq → dlq.disabled()) explicitly overrides the property in either direction.
Processing sequences
Once you resolve the problem that led to dead lettering events, you can start processing the dead letters.
The SequencedDeadLetterProcessor interface provides two operations:
-
CompletableFuture<Boolean> processAny()- Processes the oldest dead-letter sequence. Returnstrueif a sequence was processed successfully. -
CompletableFuture<Boolean> process(Predicate<DeadLetter<? extends EventMessage>>)- Processes the oldest dead-letter sequence whose first entry matches the predicate. The predicate only tests the first entry because sequences are processed as a unit: matching the first entry selects the entire sequence. Returnstrueif a matching sequence was processed successfully.
If the processing of a dead letter fails, the event will be offered to the dead-letter queue again. How the dead-lettering process reacts to this depends on the enqueue policy.
You can retrieve SequencedDeadLetterProcessor instances from the Configuration based on the processor name and the component name.
Each Event Handling Component with DLQ enabled is registered as a SequencedDeadLetterProcessor.
Below are examples of how to process dead-letter sequences for a specific component:
-
Process the oldest dead-letter sequence matching a message name
-
Process the oldest dead-letter sequence in the queue
-
Process all dead-letter sequences in the queue
import org.axonframework.common.configuration.Configuration;
import org.axonframework.messaging.core.QualifiedName;
import org.axonframework.messaging.deadletter.DeadLetter;
import org.axonframework.messaging.deadletter.SequencedDeadLetterProcessor;
import org.axonframework.messaging.eventhandling.EventMessage;
import java.util.concurrent.TimeUnit;
public class DeadLetterProcessor {
private static final QualifiedName ERROR_EVENT_NAME =
new QualifiedName("YourApplicationEvent");
private final Configuration configuration;
public void retryErrorEventSequence(String processorName, String componentName) {
configuration.getModuleConfiguration(processorName)
.map(m -> m.getComponents(SequencedDeadLetterProcessor.class))
.map(processors -> processors.get(componentName))
.ifPresent(dlp ->
dlp.process(letter -> letter.message()
.type()
.qualifiedName()
.equals(ERROR_EVENT_TYPE))
.orTimeout(30, TimeUnit.SECONDS)
.join());
}
}import org.axonframework.common.configuration.Configuration;
import org.axonframework.messaging.deadletter.SequencedDeadLetterProcessor;
import java.util.concurrent.TimeUnit;
public class DeadLetterProcessor {
private final Configuration configuration;
public void retryAnySequence(String processorName, String componentName) {
configuration.getModuleConfiguration(processorName)
.map(m -> m.getComponents(SequencedDeadLetterProcessor.class))
.map(processors -> processors.get(componentName))
.ifPresent(dlp ->
dlp.processAny()
.orTimeout(30, TimeUnit.SECONDS)
.join());
}
}import org.axonframework.common.configuration.Configuration;
import org.axonframework.messaging.deadletter.SequencedDeadLetterProcessor;
import java.util.concurrent.TimeUnit;
public class DeadLetterProcessor {
private final Configuration configuration;
public void retryAllSequences(String processorName, String componentName) {
configuration.getModuleConfiguration(processorName)
.map(m -> m.getComponents(SequencedDeadLetterProcessor.class))
.map(processors -> processors.get(componentName))
.ifPresent(this::processUntilEmpty);
}
private void processUntilEmpty(SequencedDeadLetterProcessor<?> dlp) {
boolean processed = true;
while (processed) {
processed = dlp.processAny()
.orTimeout(30, TimeUnit.SECONDS)
.join();
}
}
}Detecting dead-letter processing in handlers
Some event handlers benefit from knowing whether the event being processed is a dead letter, for example to adjust logging or apply different recovery logic.
Inject a DeadLetter parameter into the event handling method to get this information.
The parameter exposes several attributes, such as cause() and diagnostics().
The DeadLetter parameter is nullable: it is null during normal event handling, and non-null when the event is being retried as part of dead-letter processing.
import org.axonframework.messaging.deadletter.DeadLetter;
import org.axonframework.messaging.eventhandling.EventMessage;
import org.axonframework.messaging.eventhandling.annotation.EventHandler;
class MyEventHandler {
// SomeEvent is your application-defined event class.
@EventHandler
public void on(SomeEvent event, DeadLetter<EventMessage> deadLetter) {
if (deadLetter != null) {
// Retrying a dead letter: inspect why it was parked and how many times it was tried
int retries = (int) deadLetter.diagnostics().getOrDefault("retries", 0);
deadLetter.cause().ifPresent(cause ->
log.warn("Retrying dead-lettered {} (attempt {}): {} - {}",
event.getClass().getSimpleName(),
retries + 1,
cause.type(),
cause.message()));
}
// The same handler logic runs for both initial processing and retries.
// Ensure this method is idempotent.
updateProjection(event);
}
}Attributes
A dead letter contains the following attributes:
| Attribute | Type | Description |
|---|---|---|
|
|
The |
|
|
The cause for the message to be dead lettered.
Empty if the letter was enqueued because it belongs to a failing sequence (the failure occurred in an earlier letter in that sequence, not in this one).
Updated to reflect the most recent failure on each retry attempt.
|
|
|
The moment in time when the event was enqueued in a dead-letter queue. |
|
|
The moment in time when this letter was last touched.
Will equal the |
|
|
The diagnostic |
|
|
The captured processing context from the moment of failure. Contains resources like the tracking token and sequence identifier. |
Enqueue policy
By default, when you configure a dead-letter queue and event handling fails, the event is dead-lettered. However, you might not want all event failures to result in dead-lettered entries. Similarly, when letter processing fails, you might want to reconsider whether you want to enqueue the letter again.
Configure an EnqueuePolicy to control this behavior.
The policy receives a DeadLetter and the failure cause (Throwable), and returns an EnqueueDecision.
The EnqueueDecision tells the framework whether to enqueue, re-queue, or evict the letter.
You can also modify the stored exception, for example, to truncate a cause that would otherwise exceed a database column limit.
You can customize the policy to exclude some events when handling fails. As a consequence, these events will be skipped.
Axon Framework evaluates the policy at two points: when initial event handling fails, and again when dead-letter processing fails. This means a letter that keeps failing can be evicted or enriched with additional diagnostic metadata (such as a retry count) rather than staying in the queue forever.
See the sample EnqueuePolicy below for an example of retry-count-based eviction:
import org.axonframework.messaging.core.QualifiedName;
import org.axonframework.messaging.deadletter.DeadLetter;
import org.axonframework.messaging.deadletter.Decisions;
import org.axonframework.messaging.deadletter.EnqueueDecision;
import org.axonframework.messaging.deadletter.EnqueuePolicy;
import org.axonframework.messaging.eventhandling.EventMessage;
// ErrorEvent is your application-defined event class.
public class CustomEnqueuePolicy implements EnqueuePolicy<EventMessage> {
private static final QualifiedName ERROR_EVENT_TYPE =
new QualifiedName(ErrorEvent.class);
@Override
public EnqueueDecision<EventMessage> decide(DeadLetter<? extends EventMessage> letter,
Throwable cause) {
if (cause instanceof NullPointerException) {
// It's pointless:
return Decisions.doNotEnqueue();
}
int retries = (int) letter.diagnostics().getOrDefault("retries", -1);
if (letter.message().type().qualifiedName().equals(ERROR_EVENT_TYPE)) {
// Important and new entry:
return Decisions.enqueue(cause);
}
if (retries < 10) {
// Let's continue and increase retries:
return Decisions.requeue(cause, l -> l.diagnostics().and("retries", retries + 1));
}
// Exhausted all retries:
return Decisions.evict();
}
}The Decisions utility class provides factory methods for the five possible outcomes:
-
Decisions.enqueue(cause):Create a new dead letter for an event that failed during initial handling. Pass the failureThrowableso it is stored as the letter’scausefor inspection during retries. -
Decisions.requeue(cause, diagnosticsUpdater):Re-queue a letter that failed again during retry processing. ThediagnosticsUpdaterfunction transforms the letter’s current diagnostics metadata, making it the right place to track retry counts or other state. The key difference fromenqueue:requeueupdates an existing letter, whereasenqueuecreates a new one. -
Decisions.ignore():Leave an existing letter in the queue unchanged. Use this when you want to keep the letter for future retry without updating its cause or diagnostics. -
Decisions.doNotEnqueue():Skip the failed event entirely during initial handling. The event is never added to the queue; handlers that already succeeded keep their results. -
Decisions.evict():Remove an existing letter from the queue permanently during retry processing. Use this when a letter has exhausted its retries or is otherwise no longer relevant.
doNotEnqueue() and evict() are semantically distinct: doNotEnqueue() prevents a letter from ever being created, while evict() removes a letter that already exists.
You are free to construct your own EnqueueDecision implementation when none of these fits.
See the following example for configuring a custom policy:
-
Configuration API
-
Spring Boot
import org.axonframework.messaging.core.configuration.MessagingConfigurer;
import org.axonframework.messaging.eventhandling.configuration.EventProcessorModule;
public class AxonConfig {
public void configureDeadLetterQueue() {
var configurer = MessagingConfigurer.create();
configurer.eventProcessing(ep -> ep.pooledStreaming(ps -> ps
.processor(
EventProcessorModule
.pooledStreaming("my-processor")
.eventHandlingComponents(/* ... */)
.customized((cfg, processorConfig) -> processorConfig
.deadLetterQueue(dlq -> dlq
.enabled()
.enqueuePolicy(new CustomEnqueuePolicy()))))
));
}
}import org.axonframework.extension.spring.config.EventProcessorDefinition;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class ProcessorConfig {
@Bean
EventProcessorDefinition myProcessor() {
return EventProcessorDefinition.pooledStreaming("my-processor")
.assigningHandlers(descriptor ->
descriptor.beanName().startsWith("my"))
.customized(config -> config
.deadLetterQueue(dlq -> dlq
.enabled()
.enqueuePolicy(new CustomEnqueuePolicy())));
}
}