Axon Server EE

This section is split into 3 sub-sections.

• Construction of the Axon Server EE Docker Image

• Deployment using Docker Compose and

• Deployment using Kubernetes

Construction of the Image

Axon does not provide a public image for Axon Server EE. A starter Dockerfile is included below which can be tailored as per your requirements. This will work for OpenShift, Kubernetes, as well as Docker and Docker Compose.

The starter file helps create the image in multiple stages,

• The image will be based on a compact image from Google’s “distroless” base images at the gcr.io repository, in this case “gcr.io/distroless/java:11”.

• The first stage creates a user and group named “axonserveree” so that we can run Axon Server EE as a non-root user. It also creates the directories that will become our volumes ( /axonservee), and finally sets the ownership.

• The second stage begins by copying the account (in the form of the “passwd” and “group” files) and the home directory with its volume mount points, carefully keeping ownership set to the new user.

• The last steps copy the executable jar (axonserver.jar -> the enterprise version) and a common set of properties. It marks the volume mounting points and exposed ports and finally specifies the command to start Axon Server EE.

FROM busybox as source
RUN addgroup -S axonserveree \
    && adduser -S -h /axonserveree -D axonserveree \
    && mkdir -p /axonserveree/config /axonserveree/data \
                /axonserveree/events /axonserveree/log \
    && chown -R axonserveree:axonserveree /axonserveree

FROM gcr.io/distroless/java:11

COPY --from=source /etc/passwd /etc/group /etc/
COPY --from=source --chown=axonserveree /axonserveree /axonserveree

COPY --chown=axonserveree axonserver.jar axonserver-cli.jar axonserver.properties /axonserveree/

USER axonserveree
WORKDIR /axonserveree

VOLUME [ "/axonserveree/config", "/axonserveree/data", "/axonserveree/events", "/axonserveree/log" ]
EXPOSE 8024/tcp 8124/tcp 8224/tcp

ENTRYPOINT [ "java", "-jar", "axonserver.jar" ]

For the common properties (axonserver.properties), the minimum set can be added to ensure that the volumes get mounted and logs generated. Again these can be tailored as per the deployment requirements.

axoniq.axonserver.event.storage=/axonserveree/events
axoniq.axonserver.snapshot.storage=/axonserveree/events
axoniq.axonserver.replication.log-storage-folder=/axonserveree/log
axoniq.axonserver.controldb-path=/axonserveree/data
axoniq.axonserver.pid-file-location=/axonserveree/data

logging.file=/axonserveree/data/axonserver.log
logging.file.max-history=10
logging.file.max-size=10MB

Place the Dockerfile, the Axon Server EE jar file (axonserver.jar), the Axon Server EE client jar file (axonserver-cli.jar) and the axonserver.properties. The image can be constructed using the following command.

$ docker build --tag ${TAG} .

The ${TAG} could be any tag that you would like to give to the Axon Server EE Docker image. This completes the construction of the Docker image. The image can pushed to your local repository or you could keep it local if you only want to run it on your development machine. The next step is to run it either using Docker Compose or Kubernetes.

Docker Compose

Axon Server EE is meant to be run in a distributed manner i.e. as a cluster where there will be multiple instances of Axon Server EE nodes running all interconnected to each other.

The installation process assumes that Docker Compose will be used to run a 3-node Axon Server EE cluster i.e. running 3 services of the same container image we built above. Let us designate these services as "axonserver-1", "axonserver-2" and "axonserver-3". We will also give a tag to the image that we constructed above as "axonserver-ee:running".

Each container instance will use separate volumes for “data”, “events”, and “log”. We will use "secrets" to inject the license file, tokens as well as the cluster/context definitions using the autocluster mechanism. An environment variable is added to tell Axon Server about the location of the license file.

The complete docker-compose file is depicted below.

version: '3.3'
services:
  axonserver-1:
    image: axonserver-ee:running
    hostname: axonserver-1
    volumes:
      - axonserver-data1:/axonserver/data
      - axonserver-events1:/axonserver/events
      - axonserver-log1:/axonserver/log
    secrets:
      - source: axoniq-license
        target: /axonserver/config/axoniq.license
      - source: axonserver-properties
        target: /axonserver/config/axonserver.properties
      - source: axonserver-token
        target: /axonserver/config/axonserver.token
    environment:
      - AXONIQ_LICENSE=/axonserver/config/axoniq.license
    ports:
      - '8024:8024'
      - '8124:8124'
      - '8224:8224'
    networks:
      - axon-demo

  axonserver-2:
    image: axonserver-ee:running
    hostname: axonserver-2
    volumes:
      - axonserver-data2:/axonserver/data
      - axonserver-events2:/axonserver/events
      - axonserver-log2:/axonserver/log
    secrets:
      - source: axoniq-license
        target: /axonserver/config/axoniq.license
      - source: axonserver-properties
        target: /axonserver/config/axonserver.properties
      - source: axonserver-token
        target: /axonserver/config/axonserver.token
    environment:
      - AXONIQ_LICENSE=/axonserver/config/axoniq.license
    ports:
      - '8025:8024'
      - '8125:8124'
      - '8225:8224'
    networks:
      - axon-demo

  axonserver-3:
    image: axonserver-ee:running
    hostname: axonserver-3
    volumes:
      - axonserver-data3:/axonserver/data
      - axonserver-events3:/axonserver/events
      - axonserver-log3:/axonserver/log
    secrets:
      - source: axoniq-license
        target: /axonserver/config/axoniq.license
      - source: axonserver-properties
        target: /axonserver/config/axonserver.properties
      - source: axonserver-token
        target: /axonserver/config/axonserver.token
    environment:
      - AXONIQ_LICENSE=/axonserver/config/axoniq.license
    ports:
      - '8026:8024'
      - '8126:8124'
      - '8226:8224'
    networks:
      - axon-demo

volumes:
  axonserver-data1:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/data1
      o: bind
  axonserver-events1:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/events1
      o: bind
  axonserver-log1:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/log1
      o: bind
  axonserver-data2:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/data2
      o: bind
  axonserver-events2:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/events2
      o: bind
  axonserver-log2:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/log2
      o: bind
  axonserver-data3:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/data3
      o: bind
  axonserver-events3:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/events3
      o: bind
  axonserver-log3:
    driver: local
    driver_opts:
      type: none
      device: ${PWD}/log3
      o: bind

networks:
  axon-demo:

secrets:
  axonserver-properties:
    file: ./axonserver.properties
  axoniq-license:
    file: ./axoniq.license
  axonserver-token:
    file: ./axonserver.token

The “axonserver-token” secret is used to allow the CLI to talk with nodes. The access control section details the generation of these tokens. A similar approach can be used to configure more secrets for the certificates, and so enable SSL.

The "axonserver.properties" properties file referred to in the secrets’ definition section is depicted below.

axoniq.axonserver.autocluster.first=axonserver-1
axoniq.axonserver.autocluster.contexts=_admin,default
axoniq.axonserver.accesscontrol.enabled=true
axoniq.axonserver.accesscontrol.internal-token=${generated_token}
axoniq.axonserver.accesscontrol.systemtokenfile=/axonserver/config/axonserver.tok

Starting Axon Server EE using the docker-compose command is depicted below.

$ docker-compose up

Kubernetes

The deployment of Axon Server EE on Kubernetes essentially follows the same principle as we have seen for Axon Server SE i.e. using Stateful Sets. However to cater to the distributed deployment topology of Axon Server EE, there will be some changes that would need to be done.

The Dockerfile that we built above would need a change. This is due to the fact that volumes are mounted as owned by the mount location’s owner in Docker, while Kubernetes uses a special security context, defaulting to root. Since our EE image runs Axon Server under its own user (axonserveree), it has no rights on the mounted volume other than “read”. The context can be specified, but only through the user or group’s ID, and not using their name as we did in the image, because that name does not exist in the k8s management context. So we have to adjust the first stage to specify a specific numeric value (here we have given 1001) , and then use that value in the security context of the Stateful set which we shall see below.

The change is depicted below. As before, create the image using docker build and give it a tag (e.g. axonserveree-running)

FROM busybox as source
RUN addgroup -S -g 1001 axonserveree \
    && adduser -S -u 1001 -h /axonserveree -D axonserveree \
    && mkdir -p /axonserveree/config /axonserveree/data \
                /axonserveree/events /axonserveree/log \
    && chown -R axonserveree:axonserveree /axonserveree

We would need to supply a licence/token file (for client applications) and cluster/context definitions via an axonserver.properties file. Unlike Docker Compose, Kubernetes mounts Secrets and ConfigMaps as directories rather than files, so we need to split license and configuration to two separate locations. For the license secret we can use a new location “/axonserver/license/axoniq.license” and adjust the environment variable to match. For the system token we’ll use “/axonserver/security/token.txt”, and for the properties file we’ll use a ConfigMap that we mount on top of the “/axonserver/config” directory.

These can be created using "kubectl" directly from their respective file as depicted below. It is recommended to create a dedicated namespace before creating the secrets and the config maps.

$ kubectl create secret generic axonserveree-license --from-file=./axoniq.license -n ${axonserveree-ns}
secret/axonserver-license created
$ kubectl create secret generic axonserveree-token --from-file=./axoniq.token -n ${axonserveree-ns}
secret/axonserver-token created
$ kubectl create configmap axonserveree-properties --from-file=./axonserver.properties -n ${axonserveree-ns}
configmap/axonserver-properties created
$

In the descriptor we now have to declare the secret, add a volume for it, and mount the secret on the volume. Then a list of volumes has to be added to link the actual license and properties.

The complete spec for the Axon Server EE Stateful set is given below. This includes the security context, the volume mounts, the readiness and liveness probes and finally the volumes.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: axonserveree
  labels:
    app: axonserveree
spec:
  serviceName: axonserveree
  replicas: 1
  selector:
    matchLabels:
      app: axonserveree
  template:
    metadata:
      labels:
        app: axonserveree
    spec:
      securityContext:
        runAsUser: 1001
        fsGroup: 1001
      containers:
      - name: axonserveree
        image: axonserver-ee:running
        imagePullPolicy: IfNotPresent
        ports:
        - name: grpc
          containerPort: 8124
          protocol: TCP
        - name: gui
          containerPort: 8024
          protocol: TCP
        env:
        - name: AXONIQ_LICENSE
          value: "/axonserveree/license/axoniq.license"
        volumeMounts:
        - name: data
          mountPath: /axonserveree/data
        - name: events
          mountPath: /axonserveree/events
        - name: log
          mountPath: /axonserveree/log
        - name: config
          mountPath: /axonserveree/config
          readOnly: true
        - name: system-token
          mountPath: /axonserveree/security
          readOnly: true
        - name: license
          mountPath: /axonserveree/license
          readOnly: true
        readinessProbe:
          httpGet:
            path: /actuator/info
            port: 8024
          initialDelaySeconds: 5
          periodSeconds: 5
          timeoutSeconds: 1
          failureThreshold: 30
        livenessProbe:
          httpGet:
            path: /actuator/info
            port: 8024
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
      volumes:
        - name: config
          configMap:
            name: axonserveree-properties
        - name: system-token
          secret:
            secretName: axonserveree-token
        - name: license
          secret:
            secretName: axonserveree-license
  volumeClaimTemplates:
    - metadata:
        name: events
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 5Gi
    - metadata:
        name: log
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 1Gi

The StatefulSet can be applied using the following command (assuming that the StatefulSet spec is stored in the file "axonserveree-sts.yml").

$ kubectl apply -f axonserver-sts.yml -n ${axonserveree-ns}
statefulset.apps/axonserveree created

The next step would be to create the two services required for Axon Server EE i.e. axonserver-gui on 8024 (HTTP) and axonserver on 8124 (gRPC).

---
apiVersion: v1
kind: Service
metadata:
  name: axonserveree-gui
  labels:
    app: axonserveree
spec:
  ports:
  - name: gui
    port: 8024
    targetPort: 8024
  selector:
    app: axonserveree
  type: ClusterIP
---
apiVersion: v1
kind: Service
metadata:
  name: axonserveree
  labels:
    app: axonserveree
spec:
  ports:
  - name: grpc
    port: 8124
    targetPort: 8124
  clusterIP: None
  selector:
    app: axonserveree
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: axonserveree
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/affinity-mode: persistent
spec:
  rules:
  - host: axonserveree
    http:
      paths:
      - backend:
          serviceName: axonserveree-gui
          servicePort: 8024
---

The services use an Ingress to allow incoming traffic and can be deployed with the following command (assuming that the Service(s) spec is stored in the file "axonserveree-ing.yml").

$ kubectl apply -f axonserveree-ing.yml -n ${axonserveree-ns}
service/axonserveree-gui created
service/axonserveree created
ingress.networking.k8s.io/axonserveree created

The final step is to scale out the cluster. The simplest approach, and most often correct one, is to use a scaling factor other than 1, letting Kubernetes take care of deploying several instances. This means we will get several nodes that Kubernetes can dynamically manage and migrate as needed, while at the same time fixing the name and storage. We will get a number suffixed to the name starting at 0, so a scaling factor of 3 gives us “axonserver-0” through “axonserver-2”.

$ kubectl scale sts axonserveree -n ${axonserveree-ns} --replicas=3
statefulset.apps/axonserveree scaled

This completes a basic setup to help install Axon Server EE on Kubernetes. The customer can choose to tailor the entire setup based on their requirements and usage of Kubernetes.