= Kubernetes

Kubernetes has become an industry standard when it comes to deploying Software. While we recommend installing Axon Server on virtual machines due to their better stability, this guide will help you install Axon Server Cluster on Kubernetes.

AxonIQ does not provide a HELM chart, nor any installation instructions for service meshes such as istio. You can adapt the instruction in this guide for your specific situation.

Overview

The Kubernetes setup works with a variety of individual parts:

  1. Stateful set: The stateful set for each node runs an Axon Server with a stable hostname, which is necessary for headless services.

  2. Persistent Volume Claims: As Axon Server needs to store data, there are several PVCs necessary. These are included in the StatefulSet configuration.

  3. Network Service: For applications to reach Axon Server and communication between nodes, Kubernetes requires a network service that resolves to the pods created by the stateful sets.

  4. Config Map: A config map with the properties for Axon Server’s configuration.

Step 1: Create a namespace

We recommend running Axon Server in its separate namespace. During this guide, we will use the axonserver namespace. Because of Kubernetes networking, the namespace name selected affects some of the properties in your property file, such as axoniq.axonserver.domain and axoniq.axonserver.internal-domain.

Create it using the following command:

kubectl create namespace axonserver

You can name this namespace any way you like. However, if you modify this, make sure you modify commands and properties later in this tutorial as well.

Step 1: Choose your image (optional)

AxonIQ provides ready-to-use Axon Server images. In this guide, we will use the latest-jdk-17 image. For a guide on all images, and how to make your own, see choosing and building your own Docker image. If you choose to do this, replace the image in the commands below.

Step 2: AxonIQ Console

In order to run a cluster of Professional nodes, you need to buy a license on AxonIQ Console. The nodes will connect to AxonIQ Console, automatically retrieve the license and form the cluster.

By connecting your Axon Server Professional to AxonIQ Console you will unlock additional functionality.

Why connect to AxonIQ Console?

By connecting your Axon Server to AxonIQ Console, you will be able to:

  • Automatic initialization and clustering of Axon Server instances

  • Create multiple contexts in Axon Server to organize your applications

  • Inspect the status of your Axon Server through AxonIQ Console

  • Get deep insights into the status and performance of your applications when also connected to AxonIQ Console

  • Create multiple users and Access token in Axon Server

In contrast, without connecting to AxonIQ Console, you can:

  • Only have one context in Axon Server

  • Only have one user and one access key in Axon Server

Obtaining AxonIQ Console access token

You can obtain the console access token by following these steps:

  1. Log into AxonIQ Console.

  2. If this is your first time logging in, you will be prompted to enter your name. After this, your own Workspace will be created automatically

  3. Go to the Axon Server page via the left menu

  4. In the sidebar on the right, click "Manage access tokens" to open access token management

  5. Click the button under "Show Token" to reveal your unique access token

Step 3: Configure properties

When connecting to AxonIQ Console, your server will automatically receive information to form the cluster. This does require the hostname information to be set correctly; otherwise the nodes won’t be able to find each other.

So, in addition to the access token to AxonIQ Console from Step 2, you need to set the proper information. For a normal setup, we recommend the following property files for the three nodes:

  • axonserver-1.properties

  • axonserver-2.properties

  • axonserver-3.properties

server.servlet.context-path=/axonserver-1
axoniq.axonserver.name=axonserver-1
axoniq.axonserver.hostname=axonserver-1
axoniq.axonserver.domain=axonserver.svc.cluster.local
axoniq.axonserver.internal-domain=axonserver.svc.cluster.local
axoniq.console.authentication=YOUR_ACCESS_TOKEN_FROM_STEP_2
axoniq.axonserver.enterprise.licenseDirectory=/axonserver/data/license
server.servlet.context-path=/axonserver-2
axoniq.axonserver.name=axonserver-2
axoniq.axonserver.hostname=axonserver-2
axoniq.axonserver.domain=axonserver.svc.cluster.local
axoniq.axonserver.internal-domain=axonserver.svc.cluster.local
axoniq.console.authentication=YOUR_ACCESS_TOKEN_FROM_STEP_2
axoniq.axonserver.enterprise.licenseDirectory=/axonserver/data/license
server.servlet.context-path=/axonserver-3
axoniq.axonserver.name=axonserver-3
axoniq.axonserver.hostname=axonserver-3
axoniq.axonserver.domain=axonserver.svc.cluster.local
axoniq.axonserver.internal-domain=axonserver.svc.cluster.local
axoniq.console.authentication=YOUR_ACCESS_TOKEN_FROM_STEP_2
axoniq.axonserver.enterprise.licenseDirectory=/axonserver/data/license

A full list of properties can be found in the configuration reference. You can always customize these later on.

We can now create the config map:

kubectl create configmap axonserver-1-properties --from-file=./axonserver-1.properties -n axonserver
kubectl create configmap axonserver-2-properties --from-file=./axonserver-2.properties -n axonserver
kubectl create configmap axonserver-3-properties --from-file=./axonserver-3.properties -n axonserver

This config has been kept as simple as possible for the purpose of this guide. We recommend securing your Axon Server through Access Control. See securing Axon Server for more information.

Step 4: Network services

We want Axon Server to be reachable on port 8024 for the Axon Server Dashboard and port 8124 for your Axon Framework applications.

First, we will need to create the proper services. These provide stable DNS names in Kubernetes. Create the following YAMLs and save it to axonserver-1-service.yaml axonserver-2-service.yaml axonserver-3-service.yaml:

  • axonserver-1-service.yaml

  • axonserver-2-service.yaml

  • axonserver-3-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: axonserver-1-dashboard
  labels:
    app: axonserver-1
spec:
  ports:
  - name: gui
    port: 8024
    targetPort: 8024
  selector:
    app: axonserver-1
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: axonserver-1
  labels:
    app: axonserver-1
spec:
  ports:
  - name: grpc
    port: 8124
    targetPort: 8124
  - name: grpc-internal
    port: 8224
    targetPort: 8224
  clusterIP: None
  selector:
    app: axonserver-1
apiVersion: v1
kind: Service
metadata:
  name: axonserver-2-dashboard
  labels:
    app: axonserver-2
spec:
  ports:
  - name: gui
    port: 8024
    targetPort: 8024
  selector:
    app: axonserver-2
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: axonserver-2
  labels:
    app: axonserver-2
spec:
  ports:
  - name: grpc
    port: 8124
    targetPort: 8124
  - name: grpc-internal
    port: 8224
    targetPort: 8224
  clusterIP: None
  selector:
    app: axonserver-2
apiVersion: v1
kind: Service
metadata:
  name: axonserver-3-dashboard
  labels:
    app: axonserver-3
spec:
  ports:
  - name: gui
    port: 8024
    targetPort: 8024
  selector:
    app: axonserver-3
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: axonserver-3
  labels:
    app: axonserver-3
spec:
  ports:
  - name: grpc
    port: 8124
    targetPort: 8124
  - name: grpc-internal
    port: 8224
    targetPort: 8224
  clusterIP: None
  selector:
    app: axonserver-3

Now, we can create the network services:

kubectl apply -f axonserver-1-service.yaml -n axonserver
kubectl apply -f axonserver-2-service.yaml -n axonserver
kubectl apply -f axonserver-3-service.yaml -n axonserver

Step 5: Creating the StatefulSet

Now it’s time to run the StatefulSet. The YAML below contains everything needed to run your Axon Server instances' cluster nodes. Please save it to axonserver-sts-1.yaml axonserver-sts-2.yaml axonserver-sts-3.yaml.

  • axonserver-sts-1.yaml

  • axonserver-sts-2.yaml

  • axonserver-sts-3.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: axonserver-1
  labels:
    app: axonserver-1
spec:
  serviceName: axonserver-1
  replicas: 1
  selector:
    matchLabels:
      app: axonserver-1
  template:
    metadata:
      labels:
        app: axonserver-1
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: axonserver-1
        image: axoniq/axonserver:latest-jdk-17
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "4Gi"
            cpu: "1"
          limits:
            memory: "16Gi"
            cpu: "2"
        ports:
        - name: gui
          containerPort: 8024
          protocol: TCP
        - name: grpc
          containerPort: 8124
          protocol: TCP
        - name: grpc-internal
          containerPort: 8224
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /axonserver/data
        - name: events
          mountPath: /axonserver/events
        - name: log
          mountPath: /axonserver/log
        - name: config 
          mountPath: /axonserver/config
          readOnly: true
        startupProbe:
          httpGet:
            path: /axonserver-1/actuator/info
            port: 8024
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        readinessProbe:
          httpGet:
            path: /axonserver-1/actuator/health/readiness
            port: 8024
          initialDelaySeconds: 60
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        livenessProbe:
          httpGet:
            path: /axonserver-1/actuator/health/liveness
            port: 8024
          initialDelaySeconds: 90
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 10
          timeoutSeconds: 10
      volumes:
        - name: config 
          configMap:
            name: axonserver-1-properties
            items: 
            - key: axonserver-1.properties
              path: axonserver.properties
  volumeClaimTemplates:
    - metadata:
        name: log
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: data
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 2Gi
    - metadata:
        name: events
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 20Gi
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: axonserver-2
  labels:
    app: axonserver-2
spec:
  serviceName: axonserver-2
  replicas: 1
  selector:
    matchLabels:
      app: axonserver-2
  template:
    metadata:
      labels:
        app: axonserver-2
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: axonserver-2
        image: axoniq/axonserver:latest-jdk-17
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "4Gi"
            cpu: "1"
          limits:
            memory: "16Gi"
            cpu: "2"
        ports:
        - name: gui
          containerPort: 8024
          protocol: TCP
        - name: grpc
          containerPort: 8124
          protocol: TCP
        - name: grpc-internal
          containerPort: 8224
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /axonserver/data
        - name: events
          mountPath: /axonserver/events
        - name: log
          mountPath: /axonserver/log
        - name: config 
          mountPath: /axonserver/config
          readOnly: true
        startupProbe:
          httpGet:
            path: /axonserver-2/actuator/info
            port: 8024
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        readinessProbe:
          httpGet:
            path: /axonserver-2/actuator/health/readiness
            port: 8024
          initialDelaySeconds: 60
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        livenessProbe:
          httpGet:
            path: /axonserver-2/actuator/health/liveness
            port: 8024
          initialDelaySeconds: 90
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 10
          timeoutSeconds: 10
      volumes:
        - name: config 
          configMap:
            name: axonserver-2-properties
            items: 
            - key: axonserver-2.properties
              path: axonserver.properties
  volumeClaimTemplates:
    - metadata:
        name: log
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: data
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 2Gi
    - metadata:
        name: events
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 20Gi
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: axonserver-3
  labels:
    app: axonserver-3
spec:
  serviceName: axonserver-3
  replicas: 1
  selector:
    matchLabels:
      app: axonserver-3
  template:
    metadata:
      labels:
        app: axonserver-3
    spec:
      terminationGracePeriodSeconds: 120
      containers:
      - name: axonserver-3
        image: axoniq/axonserver:latest-jdk-17
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "4Gi"
            cpu: "1"
          limits:
            memory: "16Gi"
            cpu: "2"
        ports:
        - name: gui
          containerPort: 8024
          protocol: TCP
        - name: grpc
          containerPort: 8124
          protocol: TCP
        - name: grpc-internal
          containerPort: 8224
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /axonserver/data
        - name: events
          mountPath: /axonserver/events
        - name: log
          mountPath: /axonserver/log
        - name: config 
          mountPath: /axonserver/config
          readOnly: true
        startupProbe:
          httpGet:
            path: /axonserver-3/actuator/info
            port: 8024
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        readinessProbe:
          httpGet:
            path: /axonserver-3/actuator/health/readiness
            port: 8024
          initialDelaySeconds: 60
          periodSeconds: 5
          timeoutSeconds: 10
          failureThreshold: 110
        livenessProbe:
          httpGet:
            path: /axonserver-3/actuator/health/liveness
            port: 8024
          initialDelaySeconds: 90
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 10
          timeoutSeconds: 10
      volumes:
        - name: config 
          configMap:
            name: axonserver-3-properties
            items: 
            - key: axonserver-3.properties
              path: axonserver.properties
  volumeClaimTemplates:
    - metadata:
        name: log
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 10Gi
    - metadata:
        name: data
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 2Gi
    - metadata:
        name: events
      spec:
        accessModes: [ ReadWriteOnce ]
        resources:
          requests:
            storage: 20Gi

Now we can apply the StatefulSet:

kubectl apply -f axonserver-sts-1.yaml -n axonserver
kubectl apply -f axonserver-sts-2.yaml -n axonserver
kubectl apply -f axonserver-sts-3.yaml -n axonserver

Kubernetes takes some time after a Pod is started to update internal DNS entries, depending on the k8s cluster resources and current status. In our experience this can goes up to ~5 minutes. You will observe some initial warning with Unable to resolve host: this is normal, and it will solve as soon as DNS is ready.

You can now wait until Axon Server has started up by following the logging using the following command:

kubectl logs -f axonserver-1-0 -n axonserver
kubectl logs -f axonserver-2-0 -n axonserver
kubectl logs -f axonserver-3-0 -n axonserver

Your server will be operational once you see logging similar to:

io.axoniq.axonserver.AxonServer: Started AxonServer in 29.056 seconds

Step 6: Ingresses (optional)

The network services are only reachable from within the Kubernetes cluster. From here, there are two ways to access the defined ports: port-forwarding or setting up an Ingress.

Port-forwarding

Kubernetes provides a way to forward ports from the cluster to your local machine. This is useful for development purposes. To forward the Axon Server dashboard to your local machine, you can use the following command:

kubectl port-forward service/axonserver-1-dashboard 8024:8024 -n axonserver

To forward the gRPC port for your Axon Framework applications, you can use the following command:

kubectl port-forward service/axonserver-1 8124:8124 -n axonserver

Ingresses

If you want a more permanent way to access the Axon Server dashboard, you can define an ingress. This requires an Ingress Controller to be installed in your Kubernetes cluster. How to do this is beyond the scope of this guide.

To create an ingress for the Axon Server dashboard, save the following YAML to axonserver-1-ingress.yaml axonserver-2-ingress.yaml axonserver-3-ingress.yaml:

  • axonserver-1

  • axonserver-2

  • axonserver-3

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: axonserver-1
  annotations:
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: axonserver-1-dashboard
            port:
              number: 8024
        pathType: Prefix
        path: /axonserver-1
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: axonserver-2
  annotations:
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: axonserver-2-dashboard
            port:
              number: 8024
        pathType: Prefix
        path: /axonserver-2
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: axonserver-3
  annotations:
    kubernetes.io/ingress.class: public
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: axonserver-3-dashboard
            port:
              number: 8024
        pathType: Prefix
        path: /axonserver-3

You can now apply the ingress:

kubectl apply -f axonserver-1-ingress.yaml -n axonserver
kubectl apply -f axonserver-2-ingress.yaml -n axonserver
kubectl apply -f axonserver-3-ingress.yaml -n axonserver

If you want to change the name of the Axon Server from axonserver-1 to any other name, be sure to synchronize the following:

  1. In the axonserver-1.properties file (both context-path and hostname)

  2. In the StatefulSet (name, app labels, hostname, and probe urls)

  3. In the network services

  4. In the Ingress

This is necessary to ensure that all parts of the Axon Server setup are in sync.

Next

Now that you have Axon Server running in Kubernetes, you can start using it with your Axon Framework applications.

How you connect your applications depends on where you run them. If you run them in the same Kubernetes cluster, you can use the network service to connect. For the complete example above, this means to set the following property for your Axon Framework applications:

axon.axonserver.servers=axonserver-1.axonserver.svc.cluster.local:8124

If you want to connect remotely, you need to use port-forwarding to your local computer. This is not recommended for production use.

You can do this as follows:

kubectl port-forward service/axonserver-1 8124:8124 -n axonserver

Now your Axon Framework applications will automatically to this port 8124 on your local machine, as it’s the default settings.

Going to production

When you go to production, you need to consider the following:

  1. Security: Ensure your Axon Server is secure using TLS and authentication. See securing Axon Server for more information.