The disconnect between Tech and Client

I had a talk with one of my clients which went like this:

Tekton and Kubernetes are nice and all, our devs seem to be quite happy, but where do I see Test results? Jenkins has a feature like this, but now all developers use this tekton stuff and all these fancy console commands.
I just want to look at my reports and get a glimpse of how the softwaretests went. Those terminals don't really do that for me.
Can you get me something like that?

That's a spot-on observation by my client. A lot of Kubernetes tools are heavily focused on CLI-Usage, or in the "early stages of development", when features are higher up on the priority-list than shiny UIs.

One of these examples would be the tekton ecosystem. There are rarely any UIs, and if there are, like the Tekton Dashboard for example, they are very barebones.

Since I am a very CLI-Centric guy and by habit don't really think about using UIs that often, the request of my client ran me over like a truck.

So we dug around a bit and found a very promising tool.
The Allure Framework.

What is allure and why should I use it?

The allure framework does a few things:

Generating Results in json-Format (thank god no more xml)
Report creation
History-Data Collection

All of this is then slapped together in a pretty static-html report, ready to be served on your favorite server.
If you don't have one on hand, the optionally downloadable cli can also spin up a Webserver via the allure serve pathToReportDir command and host the reports itself. Magical!

To give you a little tour of how a report looks, have a gif!

I tried to get a gif to work, but 80% of the time I get errors from dev, and the ones that render turn into potato-quality.
No idea what this cloudinary thing does, but it's not great..

So if this is too small/ugly for you, which I can probably only blame on myself, don't worry! I'll show you how to spin up a demo yourself :)

How does it work?

First off, Allure has tons of connectors for different frameworks and languages, which are documented here.

But let me warn you: The documentation seems to be very minimal right now. There is a lot of trial and error involved to get what you want
To be fair, it seems to be a very powerful tool with many supported languages and Frameworks. Documenting everything to the smallest detail would probably be an enormous task.

My client was especially interested in the Cucumber and JUnit5 support. As of now we roughly finished implementing a continous integration pipeline with Report generation for JUnit5 Testsuites. Cucumber is probably following as soon as I find the time.

Let's test stuff!

To get allure to generate a report for you, all you need is the following additions to your pom.xml:



<properties>
  <surefire-plugin.version>2.22.2</surefire-plugin.version>
  <allure.version>2.13.3</allure.version>
  <aspectj.version>1.9.5</aspectj.version>
</properties>

<dependency>
  <groupId>io.qameta.allure</groupId>
  <artifactId>allure-junit5</artifactId>
  <version>${allure.version}</version>
  <scope>test</scope>
</dependency>

<build>
  <plugins>
    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${surefire-plugin.version}</version>
      <configuration>
        <testFailureIgnore>true</testFailureIgnore>
        <argLine>
          -javaagent:"${settings.localRepository}/org/aspectj/aspectjweaver/${aspectj.version}/aspectjweaver-${aspectj.version}.jar"
        </argLine>
        <systemPropertyVariables>
          <allure.results.directory>${project.build.directory}/allure-results</allure.results.directory
          <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
        </systemPropertyVariables>
      </configuration>
      <dependencies>
        <dependency>
          <groupId>org.aspectj</groupId>
          <artifactId>aspectjweaver</artifactId>
          <version>${aspectj.version}</version>
        </dependency>
      </dependencies>
    </plugin>
    <plugin>
      <groupId>io.qameta.allure</groupId>
      <artifactId>allure-maven</artifactId>
      <version>2.10.0</version>
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>report</goal>
          </goals>
          <configuration>
            <reportVersion>${allure.version}</reportVersion>
            <allureDownloadUrl>https://repo.maven.apache.org/maven2/io/qameta/allure/allure-commandline/${allure.version}/allure-commandline-${allure.version}.zip</allureDownloadUrl>
          </configuration>    
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>

Allure also brings its own set of annotations for Testcases.
Here are a few I like to use:



@Feature("Some Name")
@Epic("Some Name")
@Story("Some Name")
@Severity(SeverityLevel.CRITICAL)
@Description("non-dev readable description")
@Link("link to JIRA-Tickets or some cool resources)

Want to give it a go?
Here is a simple Dockerfile I slapped together, that needs minimal work from your side.
Just create a simple maven project, configure your pom.xml like I've shown above, add some tests, annotate them and give it a run for it's money with a mvn package!



FROM maven:3.6.3-jdk-11 as builder

WORKDIR /opt/service

# This is for caching purposes in local builds
# That way you don't have to download the whole internet every time your source changes
COPY pom.xml . 
RUN mvn dependency:go-offline

COPY src/ src/

# Run the Build and generate a report
RUN mvn clean install && mvn allure:report


FROM nginx:1.19.0-alpine as server

# Copy the report and serve it via nginx
COPY --from=builder /opt/service/target/site/allure-maven-plugin/ /usr/share/nginx/html/

The challenge of CIs

Now we have a nice tool, but what now?
I quickly noticed that those reports may be nice, but if I am going to deploy these reports in a container, I need a little bit more than what I showed you earlier.
Allures history-feature can't do anything this way. Allure can't create a history if there is no data of previous runs to be found.

So let's create a solution for this!

I thought I had a nifty idea by designing the report generation process like this:

As you can see, I introduced a new component. MinIO as S3 Storage Provider. Why minIO? Because it's commandline client has a great --newer-than/--older-than Feature. So minio could do the sorting work for me, I don't have to do that myself!

If I wanted to create a report with Data up to 7 Days old, it would be as easy as



mc cp --recursive --newer-than 7d minio/allure/results/ results/
allure generate results/ -o report/
mc cp --recursive report/ minio/allure/reports-history/$(date +%y-%m-%d-%H%M)/

And I'd get a report with Data for 7 Days. It would get stored in minio and would be ready to be put into a container.
Honestly, I loved that idea and was kind of proud of that solution. If you have a better idea, I am all ears!

Let's containerize it!

So how would this look inside a container?



FROM maven:3.6.3-jdk-11 as builder

WORKDIR /opt/service

# This is for caching purposes in local builds
# That way you don't have to download the whole internet every time your source changes
COPY pom.xml . 
RUN mvn dependency:go-offline

COPY src/ src/
RUN mvn clean install && mvn allure:report

FROM openjdk:14-alpine as generator

WORKDIR /tmp
COPY --from=builder /opt/service/target/allure-results/* results/
COPY --from=builder /opt/service/target/site/allure-maven-plugin/* reports/

RUN apk add curl tar gzip \
    && curl -L https://bintray.com/qameta/maven/download_file?file_path=io%2Fqameta%2Fallure%2Fallure-commandline%2F2.13.4%2Fallure-commandline-2.13.4.tgz | tar xvz \
    && curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
    && chmod +x mc \
    && ln -s $PWD/mc /usr/bin/mc \
    && mc config host add minio http://ipofminio:9000 minioadmin minioadmin \
    && mc mb -p minio/allure \
    && mc cp --recursive results/* minio/allure/results/ \
    && mc cp --recursive reports/* minio/allure/reports/$(date +'%Y-%m-%d-%H%M')/ \
    && mc cp --recursive minio/allure/results/ results/  \
    && allure-2.13.4/bin/allure generate results/ -o reports-with-history-data/ \
    && mc cp --recursive reports-with-history-data/ minio/allure/reports-history/$(date +'%Y-%m-%d-%H%M')/

FROM nginx:1.19.0-alpine as server

COPY --from=generator /tmp/reports-with-history-data/ /usr/share/nginx/html/

What we did is, we introduced another stage with FROM openjdk:14-alpine as generator
in which we do all the upload/fetching/sorting stuff for our history-reports.
All you would need to build this is a minio instance, which is, for local testing, easily deployed via:



docker run -p 9000:9000 --name minio --rm minio/minio server /data

But what about tekton and kubernetes?

You may ask this, rightfully, since I teased it in the beginning.
Making this run in tekton was pretty easy, even though it needed a little bit more prep-work.

First off would be to create an Image, that would be buildable outside of Kubernetes, and still has all the connections to minio we need.
The Dockerfile up there would probably crash the build, since a minio running in kubernetes wouldn't be reachable by your CI, which normally is running outside of kubernetes. So let's modify our Container a little bit to work with those requirements.

What do we have to do?
First off, I want to split up my container into two.

a reporting container that builds the reports and pushes them to minio
a nginx container that pulls the report and serves it

Why do I want this?
Having to build an image that contains your report every time the report changes, doesn't seem viable to me. It takes unnecessary space in your docker registry and doesn't bring that much benefit. IMO it would be better to have a dynamic container that can be parameterized and does the whole result-pulling, generating and archiving in it's entrypoint. So let's do just that!

Allure-Reporter Container

Dockerfile.allure



FROM openjdk:14-alpine

WORKDIR /tmp

RUN apk add curl tar gzip \
    && curl -L https://bintray.com/qameta/maven/download_file?file_path=io%2Fqameta%2Fallure%2Fallure-commandline%2F2.13.4%2Fallure-commandline-2.13.4.tgz | tar xvz \
    && curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
    && chmod +x mc \
    && ln -s $PWD/mc /usr/bin/mc \
    && ln -s $PWD/allure-2.13.4/bin/allure /usr/bin/allure \
    && allure --version \
    && mc --version

WORKDIR /opt/myproject
VOLUME /opt/myproject/results

COPY entrypoint-allure.sh /usr/local/bin/entrypoint
RUN chmod +x /usr/local/bin/entrypoint 

USER nobody
ENTRYPOINT bash -c entrypoint

entrypoint-allure.sh



#!/bin/bash

set -e

# Capture the name of the project, so we can differentiate results in minio
CURRENT_DATE=$(date '+%Y%m%d%H%M') 

# Gather all results you can find
# Should be Emulated via Volume Mount locally
mkdir results
for resultDir in $(find . -type d -iname allure-results);do
  cp $resultDir/* results
done
# Generate the actual allure report of this testrun and publish it
allure generate results/ -o allure-report/

# Configure Minio and make sure the bucket allure exists
mc config host add minio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY 
mc mb -p minio/allure 

# Publish the results
mc cp --recursive results/ minio/allure/$PROJECTNAME/allure-results/
# Publish the report
mc cp --recursive allure-report/ minio/allure/$PROJECTNAME/allure-report-independent/$CURRENT_DATE/

# Get all results, including those from past runs, and generate a history report to publish
mc cp --recursive minio/allure/$PROJECTNAME/allure-results/ results/ 
allure generate results -o allure-report-history/ 
mc cp --recursive allure-report-history/ minio/allure/$PROJECTNAME/allure-report-history/$CURRENT_DATE/

Allure-Server Container

Dockerfile.nginx:



FROM nginx:1.19.0-alpine as server

ARG MINIO_ENDPOINT=http://minio.default.svc.cluster.local:9000
ARG MINIO_SECRET_KEY=minioadmin
ARG MINIO_SECRET_PASS=minioadmin
ARG PROJECTNAME=myproject

RUN curl -LO https://dl.min.io/client/mc/release/linux-amd64/mc \
    && chmod +x mc \
    && ln -s $PWD/mc /usr/bin/mc \
    && ln -s $PWD/allure-2.13.4/bin/allure /usr/bin/allure \
    && mc --version

COPY entrypoint-nginx.sh /usr/local/bin/entrypoint
RUN chmod +x /usr/local/bin/entrypoint 

USER nobody
ENTRYPOINT bash -c entrypoint

entrypoint-nginx.sh:



#!/bin/bash

set -e

# Capture the name of the project, so we can differentiate results in minio
CURRENT_DATE=$(date '+%Y%m%d%H%M') 

# Configure Minio and make sure the bucket allure exists
mc config host add minio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
mc mb -p minio/allure 

NEWEST_REPORT=$(mc find minio/allure/$PROJECTNAME/allure-report-history/ --maxdepth 1 --newer-than 1d | cut -d'/' -f5- | sort -r | head -n1)
NEWEST_REPORT=${NEWEST_REPORT%/}
mc cp --recursive minio/allure/$PROJECTNAME/allure-report-history/$NEWEST_REPORT report
rm /opt/nginx/html/index.html
cp -r report/$NEWEST_REPORT/* /opt/nginx/html

nginx

With these two containers and four files, you should be pretty much set to go.
We split our all-in-one-container into two separate containers, that are buildable without a connection to minio.
Our reporter and it's nginx-server.

Let's go k8s

I am assuming, you have a running kubernetes setup, if not, you can take a look at my "lazy k8s solution" to running kubernetes, which can take care of that for you.

As final part of the tekton integration, you can use this task to run the report-generator (The first Container up there) and deploy the nginx-server via the file that is referenced in the kubectl step.
allure-reporter-task.yaml:



apiVersion: tekton.dev/v1beta1
kind: Task
metadata: 
  name: allure-reporter-task
  namespace: tekton
spec:
  workspaces: 
    - name: git
  params: 
   - name: minioEndpoint
     default: http://minio.default.svc.cluster.local:9000
   - name: minioAccessKey
     default: minioadmin
   - name: minioSecretKey
     default: minioadmin
   - name: pathToDeploymentYaml
     default: k8s/allure-server.yaml
  steps:
    - name: generate-and-publish-allure-report
      image: allure-report-generator:1.0.0
      command: 
        - /bin/bash
      args:
        - -c
        - |
          cd /workspace/git/
          entrypoint
      env:
        - name: MINIO_ENDPOINT
          value: $(params.minioEndpoint)
        - name: MINIO_ACCESS_KEY
          value: $(params.minioAccessKey)
        - name: MINIO_SECRET_KEY
          value: $(params.minioSecretKey)
    - name: serve-allure-report
      image: lachlanevenson/k8s-kubectl
      command: ["kubectl"]
      args:
        - "apply"
        - "-f"
        - "$(workspaces.git.path)/$(params.pathToDeploymentYaml)"

To get your allure-server deployed, the following file should be in your apps git-repository, so tekton can find it during a taskrun

Example of k8s/allure-server.yaml:



---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: allure-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: allure-server
  template:
    metadata:
      labels:
        app: allure-server
    spec:
      containers:
        - name: allure-server
          image: your-registry/allure-server:1.0.0
          imagePullPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: allure-server
  namespace: default
spec:
  selector:
    app: allure-server
  type: ClusterIP 
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: allure-server-ingress-route
  namespace: default
spec:
  entryPoints:
    - http
  routes:
  - match: Host(`localhost`) && PathPrefix(`/allure`)
    kind: Rule
    services:
    - name: allure-server
      port: 80
    middlewares:
    - name: allure-stripprefix
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: allure-stripprefix
  namespace: default
spec:
  stripPrefix:
    prefixes:
      - /allure

The difference in this approach, in comparison to the Dockerfile I demonstrated up there, I put all the logic of the Dockerfile into entrypoint scripts.
You know, all the copying, report generation, sorting, everything.

That way I don't have to push the same image over and over into my registry, just because the report was updated. I can just have one reporter image that dynamically pulls the data it needs from storage.

Also, the images you need can be built by a CI, which normally shouldn't have any access to storage solutions like minio. So a build of the Dockerfiles with a minio-connection would break your build. That's exactly why we put all the networking stuff into the entrypoints, as they will not be executed by your CI.

The future

I am already working on a practical demo of allure and tekton with zero-to-k8s. So you don't have to do all the copying and pasting from here.

End of story

But for now, I leave you with this.
A random assortment of stuff, just to get some nice shiny UI for your testresults :D

If you have questions, suggestions, or other feedback, feel free to hammer away at your keyboard :)

Blog

Continuous Test-Reporting with Allure

Raphael Habereder