Synthetic Monitoring with the Tracetest GitHub Action

It’s official! You can now use synthetic monitoring to run trace-based tests with Tracetest’s new GitHub Action. I’ve already implemented dogfooding and it’s currently running in production as health checks, running hourly, here.

In this article, I’ll show how I used GitHub Actions to enable synthetic monitoring. You’ll learn how to define a GitHub Action and create a schedule to run tests against the OpenTelemetry Demo.

What is Synthetic Monitoring?

Synthetic monitoring is a technique used to assess system health, functionality, and responsiveness. Instead of relying on actual user data, synthetic monitoring involves the creation of simulated transactions within the system to mimic user behavior. These synthetic transactions aim to replicate common user paths and activities, enabling you to proactively identify and address potential issues before they impact real users.

Designing a User Interaction with Synthetic Monitoring

The public beta of Tracetest is now available, offering a simplified solution for developers to efficiently integrate Tracetest into their infrastructure. Live demos are available to simplify onboarding with trace-based tests, allowing users to explore, execute, and create tests for instrumented systems such as Pokeshop and OpenTelemetry Demo.

I’ll walk you through designing tests for the OpenTelemetry Demo. It’s an astronomy shop that sells telescopes. One flow I wanted to map is a user purchasing a product in the shop, with the following steps:

To do that, I’ve created a test suite based on Open Telemetry Frontend trace-based tests that run six trace-based tests in sequence:

1. See ads on the shop.
2. Get product recommendations.
3. Browse product.
4. Add product to cart.
5. View shopping cart.
6. Perform a checkout and buy products.

In the TestSuite specification, you can see each one of the steps chained:

# test suite based on https://github.com/open-telemetry/opentelemetry-demo/tree/main/test/tracetesting/frontend-service

type: TestSuite
spec:
  id: otel-demo-test-suite
  name: OTel Demo Synthetic tests
  description: Run all Frontend tests enabled in sequence, simulating a process of a user purchasing products on Astronomy store
  steps:
  - ./01-see-ads.yaml
  - ./02-get-product-recommendation.yaml
  - ./03-browse-product.yaml
  - ./04-add-product-to-cart.yaml
  - ./05-view-cart.yaml
  - ./06-checking-out-cart.yaml

In each test, you see the action that’s triggered and which assertions are performed against the API response and trace, like on the View shopping cart test:

# test suite based on https://github.com/open-telemetry/opentelemetry-demo/tree/main/test/tracetesting/frontend-service

type: Test
spec:
  id: frontend-view-cart
  name: 'Frontend: View cart'
  description: Simulate a user viewing the shopping cart
  trigger:
    type: http
    httpRequest:
      url: http://${var:FRONTEND_ADDR}/api/cart?userId=2491f868-88f1-4345-8836-d5d8511a9f83
      method: GET
      headers:
      - key: Content-Type
        value: application/json
  specs:
  - name: It called the frontend with success
    selector: span[tracetest.span.type="general" name="Tracetest trigger"]
    assertions:
    - attr:tracetest.response.status = 200
  - name: It retrieved the cart items correctly
    selector: span[name="oteldemo.CartService/GetCart"]
    assertions:
    - attr:rpc.grpc.status_code = 0

To run these tests locally and validate them, add your user in the Tracetest demo organization, and then configure our CLI to that environment:

# run tracetest configure and choose a organization and environment for the CLI
➜  ~ tracetest configure

What Organization do you want to use?:
  > tracetest-demo (ttorg_2179a9cd8ba8dfa5)
What Environment do you want to use?:
  > opentelemetry-demo (ttenv_597cf82c754c2a01)
 SUCCESS  Successfully configured Tracetest CLI

After that, download the files from the Tracetest GitHub repository in the otel-demo folder and run the following command:

# run test suite with Tracetest CLI
tracetest run testsuite --file ./_testsuite.yaml --vars ./_variableset.yaml

After running this test suite, you’ll see an output like this:

Creating a GitHub Action to Run Synthetic Monitoring

After structuring the test suite to run as part of the synthetic monitoring, define a GitHub Workflow to run these tests with a schedule trigger. Set a cron expression to define the periodicity of the tests. I’ve set it to run each hour.

Additionally, set a workflow_dispatch to enable running the trigger manually and check if the action is working fine.

The .github/workflows/scheduled-jobs.yml file was defined initially as:

name: Synthetic monitoring with Tracetest

on:
  # allows the manual trigger
  workflow_dispatch:

  schedule:
  - cron: '0 */1 * * *' # every hour

jobs:
    otel-demo-trace-based-tests:
    name: Run trace based tests for Open Telemetry demo
    runs-on: ubuntu-latest

    steps:
            # To be defined ...

In this file, you can see a job called otel-demo-trace-based-tests that runs in an ubuntu-latest worker with the steps to run the test.

These steps are:

1. Checking out the test code.
2. Installing and configuring Tracetest CLI to point to the OpenTelemetry Demo environment.
3. Running the test suite.

In the first step of checking out the code, use the actions/checkout that encapsulates all the Git actions needed:

#...

otel-demo-trace-based-tests:
    name: Run trace based tests for Open Telemetry demo
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

        # more steps to be added...

Then, to configure Tracetest you use the brand new GitHub Action, kubeshop/tracetest-github-action, that installs the CLI and automatically configures the CLI using an environment token that points to the demo:

#...

otel-demo-trace-based-tests:
    name: Run trace based tests for Open Telemetry demo
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Configure Tracetest CLI
        uses: kubeshop/tracetest-github-action@v1
        with:
          token: {{add-your-environment-token-here}}

To generate an environment token, you can go to app.tracetest.io, log in, and choose an organization and environment:

Then, click on Settings (on the left bottom corner of the screen), choose the Token tab, and click on Create a new token to generate a new token for your action.

Once you have finished and created the token and added it to the Configure Tracetest CLI step, in the following steps, you can run Tracetest CLI commands to trigger tests, like tracetest run testsuite that you used to test the synthetic monitoring test suite:

#...

otel-demo-trace-based-tests:
    name: Run trace based tests for Open Telemetry demo
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Configure Tracetest CLI
        uses: kubeshop/tracetest-github-action@v1
        with:
          token: ${{secrets.TRACETEST_OTELDEMO_TOKEN}}

      - name: Run synthetic monitoring tests
        run: |
          tracetest run testsuite --file ./_testsuite.yaml --vars ./_variableset.yaml

As a final step, you can add a notification step that tell you if the tests failed. To do that, I'm using slackapi/slack-github-action that allows you to send a Slack message about the test conclusion. In my case, I've set it to send messages only in case of failures:

#...

otel-demo-trace-based-tests:
    name: Run trace based tests for Open Telemetry demo
    runs-on: ubuntu-latest

    steps:
      - # steps defined on previous snippets...

      - name: Send message on Slack in case of failure
        if: ${{ failure() }}
        uses: slackapi/slack-github-action@v1.24.0
        with:
          # check the block kit builder docs to understand how it works
          # and how to modify it: https://api.slack.com/block-kit
          payload: |
            {
              "blocks": [
                {
                  "type": "header",
                  "text": {
                    "type": "plain_text",
                    "text": ":warning: Synthetic Monitoring Alert - OTel Demo :warning:",
                    "emoji": true
                  }
                },
                {
                  "type": "section",
                  "fields": [
                    {
                      "type": "mrkdwn",
                      "text": "*Status:*\nFailed"
                    },
                    {
                      "type": "mrkdwn",
                      "text": "*Pipeline:*\n<${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|View workflow>"
                    }
                  ]
                }
              ]
            }
        env:
          SLACK_WEBHOOK_URL: http://path.to.my.slack.webhook
          SLACK_WEBHOOK_TYPE: INCOMING_WEBHOOK

Finally, you can run the workflow manually by going to the Actions under your GitHub repository or wait until the schedule trigger runs your test. After running the workflow, you can see each step being executed by the GitHub workers:

For more details about this implementation, you can see the entire definition of .github/workflows/scheduled-jobs.yml on GitHub and the open history of the running tests here.

Build Your Own Synthetic Monitoring

In this article, I’ve explained how the Tracetest team uses synthetic monitoring to evaluate the Tracetest open beta using GitHub Actions using Tracetest’s new GitHub Action to configure GitHub Workflows. With these tools, our team can add more monitoring to Tracetest and validate that our systems are working as expected. You’re now fully equipped to implement this knowledge in your own GitHub Actions to trigger trace-based tests easily!

Would you like to learn more about Tracetest and what it brings to the table? Visit the Tracetest docs and try it out by downloading it today!

Also, please feel free to join our Discord community, give Tracetest a star on GitHub, or schedule a time to chat 1:1.