Distributed Tracing

Distributed Tracing tracks a message’s path through an application and across service boundaries. Axon Framework provides tracing decorators for commands, events, queries, repositories, state managers, event storage, snapshots, distributed connectors, and annotated handler methods.

Tracing decorators wrap framework components by composition. SpanFactory is the public abstraction for span creation. Registering a SpanFactory, or the Micrometer Tracer and Propagator used to create one, enables the decorators discovered through ConfigurationEnhancer services.

A trace is a collection of related spans. Each span records one timed operation and its relationship to other operations. The following capture shows an HTTP request that dispatches a command through Axon Server and continues through command handling, event publication, event processing, a subscription-query update, and a follow-up command:

A command dispatched through Axon Server rendered in Jaeger

The capture uses axon.tracing.event-processor.distributed-in-same-trace=true. The default streaming topology uses a separate batch trace linked to the publication trace.

Sagas, process managers and deadlines are not available in Axon Framework 5.

Modules

Module Contents

axon-messaging

The tracing API, built-in message attribute providers, logging implementation, and decorators for CommandBus, EventSink, event handling, QueryBus, and annotated handlers.

axon-modelling

Decorators for Repository and StateManager.

axon-eventsourcing

Decorators for EventStorageEngine and SnapshotStore, plus event-tag attributes.

axoniq-distributed-messaging

Decorators for distributed CommandBusConnector and QueryBusConnector transport legs.

axoniq-tracing-micrometer

The Micrometer Tracing implementation of SpanFactory, backed by an application-provided Tracer and Propagator.

Configuration

The declarative configuration API and Spring Boot properties configure the same tracing components. The Spring properties use axon.tracing.* because the distributed connector and thread-local bridge settings extend the existing shared tracing namespace.

  • Configuration API

  • Spring Boot

Register a Micrometer Tracer and Propagator as framework components:

public MessagingConfigurer registerTracing(MessagingConfigurer configurer,
                                            Tracer tracer,
                                            Propagator propagator) {
    return configurer.componentRegistry(registry -> registry
            .registerComponent(Tracer.class, configuration -> tracer)
            .registerComponent(Propagator.class, configuration -> propagator));
}

The service-loaded Micrometer enhancer creates the MicrometerSpanFactory. The component enhancers then decorate the components present in the application.

Add the Micrometer binding, Actuator, the OpenTelemetry bridge, and an OTLP exporter:

<dependency>
    <groupId>io.axoniq.framework</groupId>
    <artifactId>axoniq-tracing-micrometer</artifactId>
    <version>${axoniq-framework.version}</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
management:
  tracing:
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://localhost:4318/v1/traces

Spring Boot provides the Micrometer Tracer and Propagator beans. The Axoniq Framework binding uses them to create the active MicrometerSpanFactory.

Tracing settings

Spring Boot exposes the following settings under axon.tracing.*. Declarative applications register the corresponding MessagingTracingSettings, ModellingTracingSettings, EventSourcingTracingSettings, and DistributedTracingSettings components.

Property Default Effect

axon.tracing.enabled

true

Controls the complete tracing autoconfiguration.

axon.tracing.command-bus.enabled

true

Controls CommandBus tracing.

axon.tracing.event-sink.enabled

true

Controls EventSink tracing.

axon.tracing.event-store.enabled

true

Controls EventStorageEngine tracing.

axon.tracing.event-processor.enabled

true

Controls event-handling component tracing.

axon.tracing.event-processor.batch-trace-enabled

true

Controls the streaming batch span.

axon.tracing.event-processor.distributed-in-same-trace

false

Selects whether eligible event-processing spans continue the publisher trace.

axon.tracing.event-processor.distributed-in-same-trace-time-limit

PT2M

Limits which events are eligible for the same-trace topology.

axon.tracing.event-sourcing-handlers-enabled

false

Controls per-method spans for @EventSourcingHandler methods.

axon.tracing.query-bus.enabled

true

Controls QueryBus tracing.

axon.tracing.repository.enabled

true

Controls Repository tracing.

axon.tracing.state-manager.enabled

true

Controls StateManager tracing.

axon.tracing.snapshot-store.enabled

true

Controls SnapshotStore tracing.

axon.tracing.command-bus-connector.enabled

true

Controls distributed command connector tracing.

axon.tracing.query-bus-connector.enabled

true

Controls distributed query connector tracing.

axon.tracing.thread-local-context-propagation.enabled

true

Controls the worker-thread bridge without disabling span creation or message metadata propagation.

axon.tracing.attribute-providers.*

true

Controls the message-id, message-type, metadata, aggregate-identifier, and event-tags providers.

Multiple tracing destinations

Axon Framework uses one active SpanFactory. Exporter fan-out belongs to the OpenTelemetry SDK configuration.

  • Configuration API

  • Spring Boot

Compose exporters before building the SDK:

private SpanExporter compositeExporter(String primaryEndpoint, String secondaryEndpoint) {
    SpanExporter primary = OtlpGrpcSpanExporter.builder()
                                               .setEndpoint(primaryEndpoint)
                                               .build();
    SpanExporter secondary = OtlpGrpcSpanExporter.builder()
                                                 .setEndpoint(secondaryEndpoint)
                                                 .build();
    return SpanExporter.composite(primary, secondary);
}

Expose another SpanExporter bean. Spring Boot composes the exporter beans into its OpenTelemetry SDK:

@Bean
SpanExporter secondarySpanExporter() {
    return OtlpGrpcSpanExporter.builder()
                               .setEndpoint("http://secondary-collector:4317")
                               .build();
}

Span attribute providers

SpanAttributesProvider contributes searchable attributes to message-carrying spans. The built-in providers are:

Provider Default key Description

MessageIdSpanAttributesProvider

axoniq.message.id

Message identifier.

MessageTypeSpanAttributesProvider

axoniq.message.type

Qualified message type and version.

MetadataSpanAttributesProvider

axoniq.metadata.{key}

Message metadata entries.

AggregateIdentifierSpanAttributesProvider

axoniq.aggregate.identifier

Aggregate identifier when available.

EventTagsSpanAttributesProvider

axoniq.event_tag.{key}

Event tags resolved by the configured TagResolver.

Event tags are resolved when the payload representation needed by the configured TagResolver is available. If no tag can be resolved, no event-tag attributes are added.

public final class TenantSpanAttributesProvider implements SpanAttributesProvider {

    @Override
    public Map<String, String> provideForMessage(Message message, @Nullable ProcessingContext context) {
        String tenant = message.metadata().get("tenant");
        return tenant == null ? Map.of() : Map.of("company.tenant", tenant);
    }
}

  • Configuration API

  • Spring Boot

public MessagingConfigurer registerProvider(MessagingConfigurer configurer) {
    return configurer.componentRegistry(registry -> SpanAttributesProviderRegistry.register(
            registry,
            configuration -> new TenantSpanAttributesProvider()
    ));
}
@Bean
SpanAttributesProvider tenantSpanAttributesProvider() {
    return new TenantSpanAttributesProvider();
}

Every SpanAttributesProvider bean is contributed to the provider registry.

Traced components

Span names use <Component>.<operation> <messageName>. Dispatch spans use PRODUCER, handler spans use CONSUMER, and local operations use INTERNAL.

Commands

Span name Kind Description

CommandBus.dispatch <commandName>

PRODUCER

Command dispatch.

CommandBus.handle <commandName>

CONSUMER

Command handling.

<HandlerClass>.<method>(<PayloadClass>)

INTERNAL

Annotated @CommandHandler invocation.

Events

Span name Kind Description

EventSink.publish <eventName>

PRODUCER

Event publication and trace-context propagation.

EventProcessor.process <eventName>

CONSUMER

Event processing. Carries axoniq.event_processor.name.

StreamingEventProcessor.batch

INTERNAL

One streaming batch. Carries axoniq.event_processor.name.

<HandlerClass>.<method>(<EventClass>)

INTERNAL

Annotated @EventHandler invocation.

Streaming consumer trace topology

The streaming settings select relationships between the publisher, batch, and process spans. They do not describe scheduling, threads, processing location, or the context in which handling runs.

Streaming settings EventProcessor.process parent Publisher relationship

batch-trace-enabled=true, distributed-in-same-trace=false (default)

StreamingEventProcessor.batch

Span link

batch-trace-enabled=false, distributed-in-same-trace=false

No parent

Span link

distributed-in-same-trace=true, event within the time limit

Publisher span

Parent

distributed-in-same-trace=true, event outside the time limit

No parent

Span link

Same-trace mode omits the batch span. An event without propagated trace metadata has no publisher relationship, while its process span is still created.

Queries

Span name Kind Description

QueryBus.query <queryName>

PRODUCER

Direct-query dispatch.

QueryBus.respond <queryName>

INTERNAL

Finite direct-query response stream.

QueryBus.subscriptionQuery <queryName>

PRODUCER

Subscription setup. Carries messaging.message.conversation_id.

QueryBus.initialResponse <queryName>

INTERNAL

Finite initial response. Carries messaging.message.conversation_id.

QueryBus.handle <queryName>

CONSUMER

Query handling.

QueryBus.emitUpdate

INTERNAL

Subscription-query update emission.

QueryBus.completeSubscriptions

INTERNAL

Matching subscription completion.

QueryBus.completeSubscriptionsExceptionally

INTERNAL

Exceptional matching subscription completion.

<HandlerClass>.<method>(<QueryClass>)

INTERNAL

Annotated @QueryHandler invocation.

All spans belonging to one distributed subscription carry the subscription query identifier as messaging.message.conversation_id. This includes subscription setup, the initial response, and distributed update transport spans.

Domain modelling and event sourcing

Span name Kind Description

Repository.load <EntityType>

INTERNAL

Entity load.

Repository.loadOrCreate <EntityType>

INTERNAL

Entity load or creation.

Repository.persist <EntityType>

INTERNAL

Entity persistence.

Repository.attach <EntityType>

INTERNAL

Lifecycle attachment.

StateManager.loadManagedEntity <EntityType>

INTERNAL

Managed-entity resolution.

EventStorageEngine.appendTransaction

INTERNAL

Storage append transaction.

SnapshotStore.store <name>

INTERNAL

Snapshot persistence.

SnapshotStore.load <name>

INTERNAL

Snapshot loading.

Distributed connectors

Span name Kind Description

CommandBusConnector.dispatch <commandName>

PRODUCER

Command transport send leg.

CommandBusConnector.handle <commandName>

CONSUMER

Command transport receive leg.

QueryBusConnector.query <queryName>

PRODUCER

Direct-query transport leg.

QueryBusConnector.subscriptionQuery <queryName>

PRODUCER

Subscription setup transport leg.

QueryBusConnector.handle <queryName>

CONSUMER

Query transport receive leg.

QueryBusConnector.queryUpdate <updateType>

PRODUCER / CONSUMER

Update send and delivery legs. The consumer links to the originating subscription.

Context propagation

Axon Framework carries trace context through ProcessingContext and message metadata. The Micrometer binding also makes the active span visible to instrumented libraries and logging during a scoped operation.

Parent selection uses the processing context first where applicable, then propagated message metadata, and finally the current tracer context. The current tracer context is the lowest-priority context installed by external instrumentation. Missing or invalid propagation metadata does not prevent message handling.

For Reactor and OpenTelemetry SDK configuration, see OpenTelemetry.