OpenTelemetry aims to standardize the extraction, transformation and export of data through a set of SDKs and APIs avoiding Vendor Lock-in. Instrumentation is “automatic” and allows you to define what you want to instrument, e.g. Sql Server, HttpClient, Postgre, gRPC, etc. Opentelemetry meets the observability pillars (Trace, Metric and Log).

For data visualization, you can choose between free solutions or large players like Dynatrace, Microsoft, NewRelic, Datadog, Amazon, etc.

Many OpenTelemetry examples use an Agent Collector that can be inside the same POD as your application or somewhere in your cluster. In Figure 1 , the agent is inside the Web API POD. The system sends the data to Collector which then exports it somewhere.

To make the POC (Proof Of Concept) simpler I used Signoz. It is important to note that this example does not mean that it is the best or the worst way. It's one more way to do it.

Signoz is an open source tool that has the collector and dashboard to display the data. It has a community version and can be found on Github: https://github.com/SigNoz/signoz

Figure 2 shows how the components were, the collector is in Signoz and the application exports to using the OTLP protocol (OpenTelemetry Protocol). Several solutions support this protocol.

Let's comment on the code snippets necessary for the POC to work. The code is on Github:
https://github.com/hugobritobh/poc_pentelemetry

Figure 3 below shows the instrumentation of the Metrics. Added metrics for HttpClient (line 30), AspNetCore (line 31) and EventCounter (line 32). In debug mode I visualize the data through the console. Finally, I export the data to Signoz (using OTLP).

Figure 4 below shows the Trace instrumentation. Added for AWS (line 58), AspNetCore (line 59) and HttpClient (line 60). Exporting the data to Signoz (using OTLP).

It is important to note that there are several instrumentations, but for this example I used only a few. Another point is that I didn't use the OpenTelemetry Log function, but it is possible to do and view it in Signoz.

Figure 5 below shows a trace of the C# Web API sample project displayed in the Console.

Figure 6 below shows a tracing of the C# Background Service (Console) project. In this example code, an Application Exception occurs (for example).

The next step is to install Signoz to visualize the data. You can choose a tutorial on the Signoz page itself. In this POC I used it with docker:https://signoz.io/docs/install/docker/

The figure below shows a Signoz Dashboard from the Web API application. You can check response time, error percentage and other details.

Figure 8 shows an overview of the behavior of an application, which in this case was the Test API.

It is possible to see details of Trace, that is, to go through the path that was taken by the application and even the values of the variables (depending on the instrumentation that was made) as shown in the figure below.

There is a menu on the left where we can view all the exceptions (Figure 10):

Figure 11 presents details of an exception.

To visualize metrics, you can create Dashboards. I created one just for testing (for C#). You can import Json from this link: https://github.com/hugobritobh/poc_pentelemetry/blob/main/Metric/Signoz/Dashboard/Metrics.json

I consider OpenTelemetry to be the future (one to two years from now). Its use and instrumentation is quite simple and you can choose a few ways to do it. For an environment that does not have any observability, I would use OpenTelemetry. However, if you have an APM tool, do not replace it with OpenTelemetry at this time.

Portuguese version of the article: https://medium.com/@hugobritobh/opentelemetry-signoz-b3c204afa20b

Code on Github: https://github.com/hugobritobh/poc_opentelemetry

Github: https://github.com/hugobritobh