TheLazy.Dev
Portfolio GitHub

Blog

go |opentelemetry |monitoring |observability

How to Get Started with OpenTelemetry Go

habmic

Michael Haberman

Posted on August 31, 2022

How to Get Started with OpenTelemetry Go

OpenTelemetry Go The Mandalorian

In this guide, you will learn hands-on how to create and visualize traces with OpenTelemetry Go without prior knowledge.

We will start with creating a simple to-do app that uses Mongo and the Gin framework. Then, we will send tracing data to Jaeger Tracing and to Aspecto for visualization. You can find all the relevant files in this Github repository.

What to Expect

  1. Intro to OpenTelemetry
  2. Hello world: OpenTelemetry GO example
    1. Create main.go file with Gin and Mongo
    2. Install OpenTelemetry GO
    3. Gin instrumentation: gin.Context
  3. Visualization with Jaeger and Aspecto
    1. OpenTelemetry Go and Jaeger Tracing
    2. OpenTelemetry Go and Aspecto

Intro to OpenTelemetry

OpenTelemetry is a collection of APIs and SDKs that allows us to collect, export, and generate traces, logs, and metrics (also known as the three pillars of observability).

It is a CNCF community-driven open-source project (Cloud Native Computing Foundation, the folks in charge of Kubernetes).

In a cloud-native environment, we use OpenTelemetry (OTel for short) to gather data from our system operations and events. In other words, to instrument our distributed services. This data enables us to understand and investigate our software’s behavior and troubleshoot performance issues and errors.

OpenTelemetry serves as a standard observability framework that captures all data under a single specification. It provides several components, including:

  1. APIs and SDKs per programming language for generating telemetry
  2. The OpenTelemetry Collector; receives, processes, and exports telemetry data to different destinations.
  3. OTLP protocol for shipping telemetry data

To get a deeper understanding of this technology, including its structure and the primary motivation, visit this guide.

For this OpenTelemetry Golang guide, here are the terms you need to know:

  • Span: The most basic unit. A span represents an event in our system (e.g., an HTTP request or a database operation that spans over time). A span would usually be the parent of another span, its child, or both.

  • Trace: ‘Call-stacks’ for distributed services. Traces represent a tree of spans connected in a child/parent relationship. Traces specify the progression of requests across different services and components in our app (DB, data sources, queues, etc.). For example, sending an API call to user-service resulted in a DB query to users-db.

  • Exporter: Once we create a span, the exporter handles sending the data to our backend (e.g., in memory, Jaeger Tracing, or console output)

  • Context propagation – The mechanism that allows us to correlate events across distributed services. Context is referred to as the metadata we collect and transfer*. Propagation* is how the context is packaged and transferred across services, often via HTTP headers. Context propagation is one of the areas where OpenTelemetry shines.

  • Instrumentation – instrumentation libraries gather the data and generate spans based on different libraries in our applications (Kafka, Mongo, Gin, etc.).

There are two ways to instrument our app – manually and automatically:

  • Auto instrumentation: Automatically create spans from the application libraries we use with ready-to-use OpenTelemetry libraries.
  • Manual instrumentation: Manually add code to your application to define the beginning and end of each span and the payload.

To understand more of the OpenTelemetry jargon, visit the official documentation.

Hello world: OpenTelemetry Go example

We will start by creating our to-do service and installing two libraries (Gin and Mongo) to understand how instrumentations work.

Step 1: Create main.go file for our to-do app

1) Install Gin and Mongo-driver

go get -u github.com/gin-gonic/gin
go get go.mongodb.org/mongo-driver/mongo
Enter fullscreen mode Exit fullscreen mode

2) Set up gin and mongo to listen on “/todo”

3) Create some to-do’s to seed Mongo

package main
import (
  "context"
  "net/http"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
)
var client *mongo.Client
func main() {
  connectMongo()
  setupWebServer()
}
func connectMongo() {
  opts := options.Client()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func setupWebServer() {
  r := gin.Default()
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Important: Make sure to pass c.Request.Context() as the context and not c itself - TBD
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}
Enter fullscreen mode Exit fullscreen mode

Now that our small todo app is ready, let’s introduce OpenTelemetry.

Step 2: Install OpenTelemetry Go

We will be configuring OpenTelemetry to instrument our Go app.

1) To install the OTel SDK, run:

go get go.opentelemetry.io/otel /
go.opentelemetry.io/otel/sdk /
Enter fullscreen mode Exit fullscreen mode

2) Instrument our Gin and Mongo libraries to generate traces.

3) Gin instrumentation: Install otelgin

go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
Enter fullscreen mode Exit fullscreen mode

4) Mongo instrumentation: Install otelmongo

go get go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo
Enter fullscreen mode Exit fullscreen mode

Gin instrumentation: gin.Context

We previously discussed the idea of context propagation – the way to transfer metadata between distributed services to correlate events in our system.

The Gin framework has its own type gin.Context which gets passed as a parameter to an HTTP handler. However, the context that should be passed down to the mongo operations is the standard Go library Context object, available in gin.Context.Request.Context.

//Make sure to pass c.Request.Context() as the context and not c itself
cur, findErr := collection.Find(c.Request.Context(), bson.D{})
Enter fullscreen mode Exit fullscreen mode

So make sure that you pass the Context to the mongodb operation. Check out this issue for more info.

We now have our todo app ready and instrumented. It’s time to utilize OpenTelemetry to its full potential. Our ability to visualize traces is where the true troubleshooting power of this technology comes into play.

For visualization, we’ll be using the open-source Jaeger Tracing and Aspecto.

Visualization with Jaeger and Aspecto

The setup to export traces to Jaeger or Aspecto is relatively similar. Follow along with the Jaeger setup, then switch to Aspecto by changing a few lines of code.

OpenTelemetry Go and Jaeger Tracing: Export traces to Jaeger

Jaeger Tracing is a suite of open source projects managing the entire distributed tracing “stack”: client, collector, and UI. Jaeger UI is the most commonly used open-source to visualize traces.

Here’s what the setup looks like:

1) Install the Jaeger exporter

go get go.opentelemetry.io/otel/exporters/jaeger
Enter fullscreen mode Exit fullscreen mode

2) Create a tracing folder and a jaeger.go file

3) Add the following code to the file

package tracing
import (
  "go.opentelemetry.io/otel/exporters/jaeger"
  "go.opentelemetry.io/otel/sdk/resource"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)
func JaegerTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint("http://localhost:14268/api/traces")))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}
Enter fullscreen mode Exit fullscreen mode

4) Go back to the main.go file and modify our code to use the JaegerTraceProvider function we just created

func main() {
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  setupWebServer()
}
Enter fullscreen mode Exit fullscreen mode

Next, we are going to hook up the instrumentations we installed.

5) Add the Mongo instrumentation. In our connectMongo function by adding this line

opts.Monitor = otelmongo.NewMonitor()
Enter fullscreen mode Exit fullscreen mode

The function shold look like this

func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
Enter fullscreen mode Exit fullscreen mode

Now, add the Gin instrumentation.

6) Go to the startWebServer function and add this line right after we create the gin instance

r.Use(otelgin.Middleware("todo-service"))
Enter fullscreen mode Exit fullscreen mode

The function should look like this

func startWebServer() {
  r := gin.Default()
  //Gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}
Enter fullscreen mode Exit fullscreen mode

Below is the complete main.go file. Now we’re finally ready to export to Jaeger.

package main
import (
  "context"
  "log"
  "net/http"
  "github.com/aspecto-io/opentelemerty-examples/tracing"
  "github.com/gin-gonic/gin"
  "go.mongodb.org/mongo-driver/bson"
  "go.mongodb.org/mongo-driver/mongo"
  "go.mongodb.org/mongo-driver/mongo/options"
  "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
  "go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/propagation"
)
var client *mongo.Client
func main() {   //Export traces to Jaeger
  tp, tpErr := tracing.JaegerTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()
}
func connectMongo() {
  opts := options.Client()
  //Mongo OpenTelemetry instrumentation
  opts.Monitor = otelmongo.NewMonitor()
  opts.ApplyURI("mongodb://localhost:27017")
  client, _ = mongo.Connect(context.Background(), opts)
  //Seed the database with some todo's
  docs := []interface{}{
      bson.D{{"id", "1"}, {"title", "Buy groceries"}},
      bson.D{{"id", "2"}, {"title", "install Aspecto.io"}},
      bson.D{{"id", "3"}, {"title", "Buy dogz.io domain"}},
  }
  client.Database("todo").Collection("todos").InsertMany(context.Background(), docs)
}
func startWebServer() {
  r := gin.Default()
  //gin OpenTelemetry instrumentation
  r.Use(otelgin.Middleware("todo-service"))
  r.GET("/todo", func(c *gin.Context) {
      collection := client.Database("todo").Collection("todos")
      //Make sure to pass c.Request.Context() as the context and not c itself
      cur, findErr := collection.Find(c.Request.Context(), bson.D{})
      if findErr != nil {
          c.AbortWithError(500, findErr)
          return
      }
      results := make([]interface{}, 0)
      curErr := cur.All(c, &results)
      if curErr != nil {
          c.AbortWithError(500, curErr)
          return
      }
      c.JSON(http.StatusOK, results)
  })
  _ = r.Run(":8080")
}
Enter fullscreen mode Exit fullscreen mode

Export traces to Jaeger

  1. Run the todo-service with go run main.go
  2. Make an HTTP GET request to localhost:8080/todo to generate some traces in Go
  3. Open Jaeger at http://localhost:16686/search to view those traces

You can now see the Jaeger UI. Select todo-service and click on Find traces. You should see your trace on the right:

Jaeger UI displays opentelemetry traces in go for our todo-service

By clicking the trace, you can drill down and see more details about it that allow you to further investigate on your own:

Jaeger UI. To-do service drill down.

Visualization with OpenTelemetry Go and Aspecto

So now you know the basic concepts of OpenTelemetry. You used it on your own to instrument your Go libraries, create traces and export the data to Jaeger for visualization.

Let’s take our visualization capabilities to the next level with Aspecto.

Exporting and visualizing our data in Aspecto is easy (we will see that in a moment). You can try it yourself with the free-forever plan that has no limited features. Give this Live Playground a try to get a better idea.

This is how we do it:

  1. Create a new free account at www.aspecto.io
  2. Install the otlp exporter by running 
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
Enter fullscreen mode Exit fullscreen mode
  1. Create a file called aspecto.go in your tracing folder and add the code below
  2. Replace the {ASPECTO_AUTH} with your unique Aspecto token ID – https://app.aspecto.io/app/integration/token (Settings > Integrations > Tokens) 
func AspectoTraceProvider() (*sdktrace.TracerProvider, error) {
  exp, err := otlptracegrpc.New(context.Background(),
      otlptracegrpc.WithEndpoint("collector.aspecto.io:4317"),
      otlptracegrpc.WithHeaders(map[string]string{
          "Authorization": "<ADD YOUR TOKEN HERE>",
      }))
  if err != nil {
      return nil, err
  }
  tp := sdktrace.NewTracerProvider(
      sdktrace.WithBatcher(exp),
      sdktrace.WithResource(resource.NewWithAttributes(
          semconv.SchemaURL,
          semconv.ServiceNameKey.String("todo-service"),
          semconv.DeploymentEnvironmentKey.String("production"),
      )),
  )
  return tp, nil
}
Enter fullscreen mode Exit fullscreen mode
  1. Back in the main.go file, replace the JaegerTraceProvider with AspectoTraceProvider.

The main function should now look like this

func main() {   //Export traces to Aspecto
  tp, tpErr := tracing.AspectoTraceProvider()
  if tpErr != nil {
      log.Fatal(tpErr)
  }
  otel.SetTracerProvider(tp)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  connectMongo()
  startWebServer()}
Enter fullscreen mode Exit fullscreen mode

Exporting traces to Aspecto

  1. Run the todo-service with go run main.go
  2. Make an HTTP GET request to localhost:8080/todo to generate some traces
  3. Navigate to app.aspecto.io to view the traces

We also sent a few traces with errors to see what it looks like. Our traces should look something like this:

In the Trace Search view, using the left filters pane, simply filter out by the service name (you can also filter by error, HTTP method, and much more).

Aspecto Trace Search. Displays OpenTelemetry traces for our todo app in Go.

If we drill down into one of these traces, we can see in more detail how long each request took and clear visualization of the entire workflow.

Aspecto Trace Viewer UI. Drill down into specific trace.

In more complex workflows, traces would look something like the below:

Aspecto Trace Viewer UI. Drill down into specific trace with a lot of services.

That’s all folks! We hope this guide was informative and easy to follow. You can find all files ready to use in our Github repository.

If you have any questions or issues, feel free to reach out to us via our chat.

To learn more about it, check out our OpenTelemetry Bootcamp. It’s a 6 episode YouTube series that covers OpenTelemetry from zero to 100 including the implementation of opentelemetry in production, security, sampling, and more. Completely free and vendor-neutral.

Follow these guides below, if you want to install OpenTelemetry in more officially supported languages:

💖 💪 🙅 🚩
habmic
Michael Haberman

Posted on August 31, 2022

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

How to Get Started with OpenTelemetry Go
go How to Get Started with OpenTelemetry Go

August 31, 2022