TheLazy.Dev
Portfolio GitHub

Blog

webdev |testing |cicd |tutorial

Distributed Traces for Testing with Tekton Pipelines and Tracetest

adnanrahic

Adnan Rahić

Posted on September 8, 2023

Distributed Traces for Testing with Tekton Pipelines and Tracetest

Tekton is an open-source framework for creating efficient CI/CD systems. This empowers developers to seamlessly construct, test, and deploy applications across various cloud environments and on-premise setups.

Tracetest, an open-source testing tool that uses OpenTelemetry traces for testing, offers a sophisticated test harness for distributed cloud-native apps. It empowers users to test their apps by harnessing data from distributed traces produced by OpenTelemetry. This enables creating test specs and assertions that validate whether an application aligns with the intended behavior, as defined by pre-established test parameters.

Why use distributed traces for testing?

The rationale behind integrating Tracetest with Tekton is compelling. Tracetest uses pre-existing OpenTelemetry instrumentation to execute assertions against every part of an HTTP transaction.

Combining Tracetest and Tekton enables adding trace-based testing into CI/CD pipelines within your Kubernetes cluster. This integration not only furnishes the capability to initiate planned test cycles and synthetic testing but also preserves the core tenets of trace-based testing. It empowers exhaustive and in-depth assertions with trace data.

Infrastructure Overview

The following is a high level sequence diagram on how Tekton and Tracetest interact with the different pieces of the system.

tekton and tracetest architecture

1. Install Tekton Pipelines, Triggers, and Dashboard 

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/triggers/latest/interceptors.yaml

kubectl apply --filename \
https://storage.googleapis.com/tekton-releases/dashboard/latest/release.yaml
Enter fullscreen mode Exit fullscreen mode

2. Install Tracetest CLI

Install Tracetest CLI by following these instructions for your OS.

brew install kubeshop/tracetest/tracetest
Enter fullscreen mode Exit fullscreen mode

Sample for MacOS.

3. Install Tracetest in Your Kubernetes Cluster 

tracetest server install

[Output]
How do you want to run TraceTest? [type to search]:
  Using Docker Compose
> Using Kubernetes
Enter fullscreen mode Exit fullscreen mode

Select Using Kubernetes.

[Output]
Do you have OpenTelemetry based tracing already set up, or would you like us to install a demo tracing environment and app? [type to search]:
  I have a tracing environment already. Just install Tracetest
> Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app.
Enter fullscreen mode Exit fullscreen mode

Select Just learning tracing! Install Tracetest, OpenTelemetry Collector and the sample app..

Confirm that Tracetest is running:

kubectl get all -n tracetest

[Output]
NAME                                  READY   STATUS    RESTARTS        AGE
pod/otel-collector-7f4d87489f-vp6zn   1/1     Running   0               5m41s
pod/tracetest-78b9c84c57-t4prx        1/1     Running   3 (4m15s ago)   5m29s
pod/tracetest-postgresql-0            1/1     Running   0               5m42s

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
service/otel-collector            ClusterIP   10.96.173.226   <none>        4317/TCP              5m46s
service/tracetest                 ClusterIP   10.96.248.146   <none>        11633/TCP,4317/TCP   5m42s
service/tracetest-postgresql      ClusterIP   10.96.155.147   <none>        5432/TCP              5m42s
service/tracetest-postgresql-hl   ClusterIP   None            <none>        5432/TCP              5m42s

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/otel-collector   1/1     1            1           5m46s
deployment.apps/tracetest        1/1     1            1           5m42s

NAME                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/otel-collector-7f4d87489f   1         1         1       5m46s
replicaset.apps/tracetest-78b9c84c57        1         1         1       5m42s

NAME                                    READY   AGE
statefulset.apps/tracetest-postgresql   1/1     5m42s
Enter fullscreen mode Exit fullscreen mode

By default, Tracetest is installed in the tracetest namespace.

To explore the Tracetest Web UI, run the command:

kubectl --kubeconfig <path-to-your-home>/.kube/config --context <your-cluster-context> --namespace tracetest port-forward svc/tracetest 11633
Enter fullscreen mode Exit fullscreen mode

Tracetest Web UI

4. Create a Test in Tracetest

Start by clicking Create > Create New Test > HTTP Request > Next > Choose Example (dropdown) > Pokeshop - List (generates a sample test from the Tracetest demo) > Next > URL is prefilled with http://demo-pokemon-api.demo/pokemon?take=20&skip=0 > Create & Run.

This will trigger the test and display a distributed trace in the Trace tab to run assertions against.

Tracetest distributed trace test run view

Proceed to add a test spec to assert all database queries return within 500 ms. Click the Test tab and proceed to click the Add Test Spec button.

In the span selector make sure to add this selector:

span[tracetest.span.type="database"]
Enter fullscreen mode Exit fullscreen mode

In the assertion field add:

attr:tracetest.span.duration < 500ms
Enter fullscreen mode Exit fullscreen mode

Save the test spec and publish the test.

Assertion for database queries

The database spans that are returning in less than 500ms are labeled in green.

Assertions failing

This is an example of a trace-based test that asserts against every single part of an HTTP transaction, including all interactions with the database.

Let's introduce how Tekton makes it possible to run this test as part of your CI/CD pipeline.

5. Create a Task in Tekton

Click the Automate tab.

test automate

This contains both a YAML definition for the test run and a guide how to run the test with the Tracetest CLI.

Save this into a file called test-api.yaml:

# test-api.yaml

type: Test
spec:
  id: L7wr5xeVR
  name: Pokeshop - List
  description: Get a Pokemon
  trigger:
    type: http
    httpRequest:
      method: GET
      url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - selector: span[tracetest.span.type="database"]
    name: Database queries less than 500ms
    assertions:
    - attr:tracetest.span.duration < 500ms
Enter fullscreen mode Exit fullscreen mode

Note that you'll use this CLI command to run the test:

tracetest run test --file test-api.yaml --required-gates test-specs --output pretty
Enter fullscreen mode Exit fullscreen mode

It set's the required gates to pass the test by only validating the test specs.

Create another YAML file, name it install-and-run-tracetest.yaml.
This contains the Tekton Task definition.

# install-and-run-tracetest.yaml

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: install-and-run-tracetest
spec:
  steps:
    - name: create-test-files
      image: ubuntu
      script: |
        #!/usr/bin/env bash
        cat <<EOF >/workspace/test-api.yaml
        type: Test
        spec:
          id: L7wr5xeVR
          name: Pokeshop - List
          description: Get a Pokemon
          trigger:
            type: http
            httpRequest:
              method: GET
              url: http://demo-pokemon-api.demo/pokemon?take=20&skip=0
              headers:
              - key: Content-Type
                value: application/json
          specs:
          - selector: span[tracetest.span.type="database"]
            name: Database queries less than 500ms
            assertions:
            - attr:tracetest.span.duration < 500ms
        EOF
      volumeMounts:
      - name: custom
        mountPath: /workspace
    - name: install-and-run-tracetest
      image: kubeshop/tracetest:v0.13.3
      # The official Tracetest image comes with the Tracetest CLI installed
      script: |
        # Configure and Run Tracetest CLI
        tracetest configure -g --endpoint http://tracetest.tracetest.svc.cluster.local:11633/
        tracetest run test --file /workspace/test-api.yaml --required-gates test-specs --output pretty
      volumeMounts:
      - name: custom
        mountPath: /workspace
  volumes:
  - name: custom
    emptyDir: {}
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f ./install-and-run-tracetest.yaml
Enter fullscreen mode Exit fullscreen mode

Make sure to use the Tracetest service as the endpoint for your tracetest configure command. This may vary depending on your installation.

http://tracetest.tracetest.svc.cluster.local:11633/
Enter fullscreen mode Exit fullscreen mode

6. Run the Tracetest Trace-based Test in Tekton with a TaskRun

Finally, to run the test, create a TaskRun.

Create a file called install-and-run-tracetest-run.yaml.

# install-and-run-tracetest-run.yaml

apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: install-and-run-tracetest-run
spec:
  taskRef:
    name: install-and-run-tracetest
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f ./install-and-run-tracetest-run.yaml
Enter fullscreen mode Exit fullscreen mode

Here's how to check the logs:

kubectl logs --selector=tekton.dev/taskRun=install-and-run-tracetest-run
Enter fullscreen mode Exit fullscreen mode

You can also trigger a Task with the Tekton CLI.

tkn task start install-and-run-tracetest

[Output]
TaskRun started: install-and-run-tracetest-run-xmhfg
Enter fullscreen mode Exit fullscreen mode

In order to track the TaskRun progress run:

tkn taskrun logs install-and-run-tracetest-run-gccjk -f -n default

[Output]
[install-and-run-tracetest] ✔ Pokeshop - List (http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/3/test) - trace id: 0549641531d3221ded696f2fd3b20ce6
[install-and-run-tracetest]     ✔ Database queries less than 500 ms
[install-and-run-tracetest]
Enter fullscreen mode Exit fullscreen mode

To preview which tasks failed or succeeded, use this command:

tkn taskrun list

[Output]
NAME                                  STARTED          DURATION   STATUS
install-and-run-tracetest-run         3 minutes ago    23s        Succeeded
install-and-run-tracetest-run-nmptn   7 minutes ago    33s        Failed
install-and-run-tracetest-run-bhf7v   20 minutes ago   23s        Succeeded
install-and-run-tracetest-run-wm8bj   21 minutes ago   22s        Succeeded
install-and-run-tracetest-run-dbrbt   23 minutes ago   24s        Failed
Enter fullscreen mode Exit fullscreen mode

7. Trigger Trace-based Tests with an EventListener

By using Tektons's triggers, you can trigger tests via an eventlistener.

Create a TriggerTemplate and TriggerBinding 

# install-and-run-tracetest-trigger-binding.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerTemplate
metadata:
  name: install-and-run-tracetest-template
spec:
  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: TaskRun
    metadata:
      generateName: install-and-run-tracetest-run-
    spec:
      taskRef:
        name: install-and-run-tracetest
Enter fullscreen mode Exit fullscreen mode 
# install-and-run-tracetest-trigger-template.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: TriggerBinding
metadata:
  name: install-and-run-tracetest-binding
spec:
  params:
  - name: run
    value: $(body.run)
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f install-and-run-tracetest-trigger-binding.yaml
kubectl apply -f install-and-run-tracetest-trigger-template.yaml
Enter fullscreen mode Exit fullscreen mode

Create an EventListener 

# install-and-run-tracetest-event-listener.yaml

apiVersion: triggers.tekton.dev/v1beta1
kind: EventListener
metadata:
  name: install-and-run-tracetest-event-listener
spec:
  serviceAccountName: tekton-robot
  triggers:
    - name: install-and-run-tracetest-trigger 
      bindings:
      - ref: install-and-run-tracetest-binding
      template:
        ref: install-and-run-tracetest-template
Enter fullscreen mode Exit fullscreen mode

The EventListener requires a service account to run. To create the service account for this example create a file named tekton-robot-rbac.yaml and add the following:

# tekton-robot-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tekton-robot
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: triggers-example-eventlistener-binding
subjects:
- kind: ServiceAccount
  name: tekton-robot
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tekton-triggers-eventlistener-roles
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: triggers-example-eventlistener-clusterbinding
subjects:
- kind: ServiceAccount
  name: tekton-robot
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tekton-triggers-eventlistener-clusterroles
Enter fullscreen mode Exit fullscreen mode 
kubectl apply -f tekton-robot-rbac.yaml
kubectl apply -f install-and-run-tracetest-event-listener.yaml
Enter fullscreen mode Exit fullscreen mode

Enable port forwarding.

kubectl port-forward service/el-install-and-run-tracetest-event-listener 8080
Enter fullscreen mode Exit fullscreen mode

Hitting the port forwarded endpoint will trigger the task.

curl -v \
   -H 'content-Type: application/json' \
   -d '{"run":true}' \
   http://localhost:8080
Enter fullscreen mode Exit fullscreen mode

Checking the taskruns will confirm this.

tkn taskrun list

[Output]
NAME                                  STARTED          DURATION   STATUS
install-and-run-tracetest-run-69zrz   4 seconds ago    ---        Running(Pending)
Enter fullscreen mode Exit fullscreen mode

Finally, check the logs:

tkn taskrun logs -f install-and-run-tracetest-run-69zrz

[Output]
[install-and-run-tracetest] ✔ Pokeshop - List (http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/5/test)
[install-and-run-tracetest]     ✔ Database queries less than 500 ms
Enter fullscreen mode Exit fullscreen mode

8. Preview Trace-based Tests with Tekton Dashboard

Start by port forwarding the Tekton Dashboard.

kubectl --namespace tekton-pipelines port-forward svc/tekton-dashboard 9097:9097
Enter fullscreen mode Exit fullscreen mode

Open it up in your browser at http://localhost:9097. Navigate to the TaskRuns. Open the TaskRun you ran above.

Tekton Dashboard TaskRuns

This lets you easily preview the Tracetest test runs by copying the link to the Tracetest instance. However, because you're using an internal service for Tracetest, make sure to use the port forwarded link to access the Tracetest Dashboard.

Replace:

http://tracetest.tracetest.svc.cluster.local:11633/test/RUkKQ_aVR/run/4/test
Enter fullscreen mode Exit fullscreen mode

With:

http://localhost:11633/test/RUkKQ_aVR/run/4/test
Enter fullscreen mode Exit fullscreen mode

Tracetest Dashboard test run preview

Alternatively, you can use the Deep Link feature to trigger a new test run directly via the browser like this:

http://localhost:11633/test/RUkKQ_aVR/run?
Enter fullscreen mode Exit fullscreen mode

Conclusion

To sum up, the partnership between Tekton and Tracetest offers a powerful approach to testing Kubernetes with distributed tracing. Tekton provides a framework for building efficient CI/CD pipelines. Tracetest utilizes OpenTelemetry traces for testing, making it a valuable tool for testing cloud-native apps.

By combining Tracetest and Tekton, developers gain the ability to integrate trace-based testing into Kubernetes clusters. This integration allows for scheduled test cycles, synthetic tests, and thorough assertions using trace data. This cohesive approach ensures that applications align with intended behavior, improving overall reliability and functionality. As cloud-native apps continue to evolve, this integration showcases how to enhance testing as well.

Next Steps

To explore more options that Tracetest gives you, check out the docs to learn more!

If you like what the Tekton and Tracetest communities are doing, please leave a ⭐️ in GitHub.

💖 💪 🙅 🚩
adnanrahic
Adnan Rahić

Posted on September 8, 2023

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

Distributed Traces for Testing with Tekton Pipelines and Tracetest
webdev Distributed Traces for Testing with Tekton Pipelines and Tracetest

September 8, 2023