TheLazy.Dev
Portfolio GitHub

Blog

aws |newrelic |kubernetes |sre

OTEL Demo with EKS and New Relic

networkandcode

Shakir

Posted on July 18, 2024

OTEL Demo with EKS and New Relic

Hi👋, In this post we shall deploy the OTEL demo app on EKS, and send the metrics, logs and traces to New Relic.

Prerequisites

It's assumed you know how to install and use tools such as eksctl, kubectl, helm, helmfile and know some observability basics.

Cluster

Create a cluster on EKS with this command.

eksctl create cluster --name newrelic-otel-demo --zones=us-east-1a,us-east-1b
Enter fullscreen mode Exit fullscreen mode

Once the cluster is created, the kubeconfig's current context will be automatically changed to it.

$ kubectl config current-context
nc@newrelic-otel-demo.us-east-1.eksctl.io
Enter fullscreen mode Exit fullscreen mode

License

We need a license key🔑 first from New Relic, log in to the New Relic website and headerover to Administration > API keys and copy the license key and set it as a variable. Note that the free account is sufficient for this demo.

export NEW_RELIC_LICENSE_KEY='<NEW_RELIC_LICENSE_KEY>'
Enter fullscreen mode Exit fullscreen mode

We can create a namespace for our demo and keep a secret in it for the license.

$ kubectl create ns otel-demo-newrelic
namespace/newrelic-otel-demo created

$ kubectl create secret generic newrelic-license-key --from-literal=license-key="$NEW_RELIC_LICENSE_KEY" -n otel-demo-newrelic
secret/newrelic-license-key created
Enter fullscreen mode Exit fullscreen mode

Helm

We would be using the following helm values. We have disabled extra components that come along with the helm chart which we do not need for this demo.

$ cat otel-demo-newrelic-values.yaml
prometheus:
  enabled: false
opensearch:
  enabled: false
grafana:
  enabled: false
jaeger:
  enabled: false

opentelemetry-collector:
  config:
    exporters:
      otlphttp/newrelic:
        endpoint: https://otlp.nr-data.net:4317
        headers:
          api-key: ${NEW_RELIC_LICENSE_KEY}
        tls:
          insecure: false
    service:
      pipelines:
        logs:
          exporters: [otlphttp/newrelic, debug]
        metrics:
          exporters: [otlphttp/newrelic, debug]
          receivers: [httpcheck/frontendproxy, redis, otlp]
        traces:
          exporters: [otlphttp/newrelic, debug]
  extraEnvs:
    - name: NEW_RELIC_LICENSE_KEY
      valueFrom:
        secretKeyRef:
          key: license-key
          name: newrelic-license-key
    - name: OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE
      value: delta
Enter fullscreen mode Exit fullscreen mode

And the helmfile would refer to the public helm chart and the helm values file we just defined above.

$ cat otel-demo-newrelic-helmfile.yaml
repositories:
- name: open-telemetry
  url: https://open-telemetry.github.io/opentelemetry-helm-charts

releases:
- name: otel-demo-newrelic
  chart: open-telemetry/opentelemetry-demo
  namespace: otel-demo-newrelic
  values:
  - otel-demo-newrelic-values.yaml
Enter fullscreen mode Exit fullscreen mode

It's time to deploy the helm release.

helmfile sync -f otel-demo-newrelic-helmfile.yaml
Enter fullscreen mode Exit fullscreen mode

We can check the helm release.

$ helm ls -n otel-demo-newrelic
NAME                NAMESPACE           REVISION    UPDATED                                 STATUS      CHART                       APP VERSION
otel-demo-newrelic  otel-demo-newrelic  1           2024-07-17 22:02:13.440621 +0530 IST    deployed    opentelemetry-demo-0.32.1   1.11.0
Enter fullscreen mode Exit fullscreen mode

Pods

In some time, all the pods should be running. I had seen image pull back off for sometime, however it fixed itself in a while.

$ kubectl get po -n otel-demo-newrelic
NAME                                                        READY   STATUS    RESTARTS   AGE
otel-demo-newrelic-accountingservice-76fc85f7f9-vxps4       1/1     Running   0          25m
otel-demo-newrelic-adservice-7f8964f98-z2nqd                1/1     Running   0          25m
otel-demo-newrelic-cartservice-bbdfdc59c-w4cqg              1/1     Running   0          25m
otel-demo-newrelic-checkoutservice-f6fd788db-khgq6          1/1     Running   0          25m
otel-demo-newrelic-currencyservice-f9f77f79d-8n946          1/1     Running   0          25m
otel-demo-newrelic-emailservice-66c84d6b64-frs8b            1/1     Running   0          25m
otel-demo-newrelic-flagd-7444ccd98d-cknxl                   1/1     Running   0          25m
otel-demo-newrelic-frauddetectionservice-7c85f7c6bf-mpbw9   1/1     Running   0          25m
otel-demo-newrelic-frontend-c995cfcdd-dsgvc                 1/1     Running   0          25m
otel-demo-newrelic-frontendproxy-76bb8f75cf-kzmn6           1/1     Running   0          25m
otel-demo-newrelic-imageprovider-66db4b77cf-7rg97           1/1     Running   0          25m
otel-demo-newrelic-kafka-74fbd4c749-dspqm                   1/1     Running   0          25m
otel-demo-newrelic-loadgenerator-864475d886-wtnxs           1/1     Running   0          25m
otel-demo-newrelic-otelcol-6b89d8dd84-l5mx4                 1/1     Running   0          25m
otel-demo-newrelic-paymentservice-59677d8c8f-bds4f          1/1     Running   0          25m
otel-demo-newrelic-productcatalogservice-845d79f865-6hs7c   1/1     Running   0          25m
otel-demo-newrelic-quoteservice-76b5f85685-5zz86            1/1     Running   0          25m
otel-demo-newrelic-recommendationservice-8d6bb74c-6zfm4     1/1     Running   0          25m
otel-demo-newrelic-shippingservice-6d44b76489-n4tlw         1/1     Running   0          25m
otel-demo-newrelic-valkey-7f5ffc95ff-wz9zg                  1/1     Running   0          25m
Enter fullscreen mode Exit fullscreen mode

New Relic

We can now go to New Relic and see if the data is flowing. Check All Entities > Services-OpenTelemetry, we should start seeing the data for all our services.
OTEL services in New Relic

We could either click on one of the service names on the pic above, or choose a service from the APM & Services section to see quite a lot of information about a particular service including built-in dashboards📈.
Service overview

We could also check the traces page for a particular service, as we are sending traces too, based on our otel collector configuration.
Traces page

Already a lot of information, forget about creating custom panels or alerts 😃, understading and managing what we have already given by New Relic is in itself a great administration task😉.

Yet we will try to run a simple query using the query builder.
Query builder

We could make custom dashboards with such queries. Alright so that's the end of this post. We just saw how we can use the open telemetry collector to send observability data to newrelic and just explored some of the UI functionalities on New Relic. See below for the housekeeping check.

Cleanup

Note that free account comes with a data ingestion limit of 100GB per month. I think it ingested around 17GB approximately for about 10 hours(I left my cluster ON and more than 85% of data was coming from traces). So it's a good idea to stop sending traffic once your demo is done, so you can leverage the free limit later when required.

Delete the namespace to delete all the objects we created in the kubernetes cluster for this demo. This would stop sending data to newrelic.

kubectl delete ns otel-demo-newrelic
namespace "otel-demo-newrelic" deleted
Enter fullscreen mode Exit fullscreen mode

Also you may unset the license key variable.

unset NEW_RELIC_LICENSE_KEY
Enter fullscreen mode Exit fullscreen mode

Finally delete the cluster.

eksctl delete cluster --name newrelic-otel-demo --region=us-east-1
Enter fullscreen mode Exit fullscreen mode

Thank you for reading !!!.

💖 💪 🙅 🚩
networkandcode
Shakir

Posted on July 18, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related

OTEL Demo with EKS and New Relic
aws OTEL Demo with EKS and New Relic

July 18, 2024