🤓 Introduction

Datadog is a cloud monitoring service provider which helps to track and analyse your cloud services from a health and operations perspective and providing support for debugging issues.

Datadog provides different integrations with multiple cloud providers like AWS, GCP, and Azure. In this blog we will be seeing Datadog integration with AWS using one of the AWS Cloudformation service with the help of Serverless framework.

📄 Pre Requisites

You will need AWS and Datadog account and knowledge of AWS services like IAMRole, Cloudformation and Serverless framework.

Note: Here we are using Serverless framework with TypeScript and Serverless Compose to deploy multiple different stack.

🛠 Datadog Cloudformation Setup

In order to create Datadog Resources using Cloudformation, Datadog needs Cloudformation Extension Plugins to be activated, so our first step is to start with adding an activation stack to our Serverless application. These extension plugins needs IAM roles to be attached in order to execute the Cloudformation events. You can find more details regarding this in Datadog Amazon Cloudformation Documentation

Note: Here you can find a basic setup guide for Datadog-Cloudformation Integration

We'll create two service stack in the Serverless application, one for the Extensions Activation which contains Cloudformation extension activation (as we'll only deploy it once because we can not deploy it multiple times to activate the same resources) and one for the Infra Resources (To create different Datadog monitors and dashboards).

🧩 Setting Up the Extension Activation Stack

Let's create a IAM Role for the Role Delegation and Extensions Executions Role.

🧮 Setting Up the IAM Role

We are providing access to Datadog AWS account (464622532012) using Role Delegation to our IAMRole with necessary permissions.

DatadogExecutionRole: {
    Type: "AWS::IAM::Role",
    Properties: {
      Path: "/",
      RoleName: `DatadogAWSIntegrationRole`,
      Description: "IAM Role to integrate Datadog using Cloudformation",
      MaxSessionDuration: 8400,
      AssumeRolePolicyDocument: {
        Version: "2012-10-17",
        Statement: [
          {
            Effect: "Allow",
            Principal: {
              AWS: "arn:aws:iam::464622532012:root",
            },
            Action: "sts:AssumeRole",
            Condition: {
              StringEquals: {
                "sts:ExternalId": ["${ssm:/datadog/integration/externalId}"],
              },
            },
          },
          {
            Effect: "Allow",
            Principal: {
              Service: [
                "hooks.cloudformation.amazonaws.com",
                "resources.cloudformation.amazonaws.com",
              ],
            },
            Action: "sts:AssumeRole",
          },
        ],
      },
      Policies: [
        {
          PolicyName: `DatadogIntegrationRolePolicy`,
          PolicyDocument: {
            Version: "2012-10-17",
            Statement: [
              {
                Sid: "VisualEditor0",
                Effect: "Allow",
                Action: [
                  "health:DescribeAffectedEntities",
                  "budgets:ViewBudget",
                  "logs:DeleteSubscriptionFilter",
                  "states:ListStateMachines",
                  "states:DescribeStateMachine",
                  "tag:GetTagValues",
                  "logs:DescribeSubscriptionFilters",
                  "cloudtrail:GetTrailStatus",
                  "codedeploy:List*",
                  "es:ListTags",
                  "cloudwatch:Describe*",
                  "elasticmapreduce:Describe*",
                  "dynamodb:Describe*",
                  "rds:Describe*",
                  "route53:List*",
                  "elasticloadbalancing:Describe*",
                  "logs:FilterLogEvents",
                  "cloudfront:GetDistributionConfig",
                  "apigateway:GET",
                  "ses:Get*",
                  "cloudtrail:LookupEvents",
                  "fsx:ListTagsForResource",
                  "support:DescribeTrustedAdvisor*",
                  "events:CreateEventBus",
                  "lambda:List*",
                  "dynamodb:List*",
                  "s3:GetBucketNotification",
                  "cloudtrail:DescribeTrails",
                  "s3:PutBucketNotification",
                  "organizations:Describe*",
                  "organizations:List*",
                  "logs:PutSubscriptionFilter",
                  "fsx:DescribeFileSystems",
                  "kinesis:Describe*",
                  "codedeploy:BatchGet*",
                  "autoscaling:Describe*",
                  "s3:GetBucketTagging",
                  "tag:GetResources",
                  "logs:DescribeLogStreams",
                  "rds:List*",
                  "s3:GetBucketLogging",
                  "tag:GetTagKeys",
                  "elasticmapreduce:List*",
                  "backup:List*",
                  "sns:Publish",
                  "elasticache:List*",
                  "redshift:DescribeLoggingStatus",
                  "cloudwatch:Get*",
                  "support:RefreshTrustedAdvisorCheck",
                  "elasticfilesystem:DescribeFileSystems",
                  "es:DescribeElasticsearchDomains",
                  "xray:GetTraceSummaries",
                  "ecs:List*",
                  "health:DescribeEvents",
                  "sqs:ListQueues",
                  "logs:DescribeLogGroups",
                  "kinesis:List*",
                  "ecs:Describe*",
                  "health:DescribeEventDetails",
                  "cloudwatch:List*",
                  "logs:TestMetricFilter",
                  "elasticfilesystem:DescribeAccessPoints",
                  "elasticache:Describe*",
                  "sns:List*",
                  "xray:BatchGetTraces",
                  "ec2:Describe*",
                  "directconnect:Describe*",
                  "es:ListDomainNames",
                  "elasticfilesystem:DescribeTags",
                  "s3:ListAllMyBuckets",
                  "cloudfront:ListDistributions",
                  "redshift:DescribeClusters",
                  "s3:GetBucketLocation",
                  "lambda:GetPolicy",
                  "secretsmanager:CreateSecret",
                  "secretsmanager:DeleteSecret",
                  "secretsmanager:DescribeSecret",
                ],
                Resource: "*",
              },
            ],
          },
        },
      ],
    },
  }

Note: This role will work as extension execution role as well, if you need to have separate roles than you can create separate roles as well.

Now we'll create four AWS::CloudFormation::TypeActivation resources to attach to Activation Stack which will include.

Datadog::Dashboards::Dashboard - to create and manage Datadog Dashboards
Datadog::Monitors::Monitor - to create and manage Datadog Monitors
Datadog::Monitors::Downtime - to create and manage Monitors Downtime
Datadog::Integrations::AWS - to create integration between AWS and Datadog

Please refer to the Cloudformation Extension - Activation Guide on how to activate the Cloudformation extensions

If we deploy this Activation Stack we will be able to create the IAMRole along with the Cloudformation Extension Activation. (Note: We'll need to deploy this stack only once)

To use the activated extensions, you will need to set the Type Configurations for each extensions plugins using below code block. Also helpful if you are creating different stages and account for deployment with same AWS account.

aws cloudformation set-type-configuration --type-name "Datadog::Monitors::Monitor" --type RESOURCE --configuration-alias ConfigurationName --configuration '{"DatadogCredentials": {"ApiKey": "DATADOG_API_KEY_PROD", "ApplicationKey": "DATADOG_APPLICATION_KEY_PROD", "ApiURL": "https://api.datadoghq.eu" }}'

Now we move on to creating the Datadog Resources such as Integration, Monitors and Dashboards.

🏗 Creating Infra Stack

We'll create a different Serverless service stack to deploy the datadog resources.

⚙️ Datadog AWS Integration

Here you will need to provide your AWS Account ID so that Datadog account can do the integration between the two AWS accounts.

RoleName would be the same as we have created in the Activation Stack before.

Note:- ExternalIDSecretName should be unique in your account Secret Manager and will be used to store the external ID by the extension.

DatadogAWSIntegration: {
    Type: "Datadog::Integrations::AWS",
    Properties: {
      AccountID: "AWS_ACCOUNT_ID", // your AWS account ID
      ExternalIDSecretName: "DatadogIntegrationExternalID",
      RoleName: "DatadogAWSIntegrationRole",
    },
},

📊 Datadog Dashboards

Dashboards are helpful to monitor the services and different widgets for any services. Find more about Dashboards in Datadog Dashboards Documentation

Dashboard: {
    Type: "Datadog::Dashboards::Dashboard",
    Properties: {
      DashboardDefinition: "", // stringified object with  dashboard configurations and definitions.
    },
},

⏱ Datadog Monitor

Monitors are to create and capture events for any particular services. You can find more about monitors in Datadog Monitors Documentation

Monitor: {
  Type: "Datadog::Monitors::Monitor",
  Properties: {
    Type: "event-v2 alert", // type of the monitor
    Query: "defines the query of the monitor to capture specific triggered events",
    Name: "monitor-name",
    Message: "message body which will be sent to the communication channel",
    Tags: [], // array of tags assigned to the monitor
    Options: {
      NotifyAudit: false,
      Locked: false,
      EnableLogsSample: false,
      Thresholds: {
        Critical: 1,
        CriticalRecovery: 0.5,
        Warning: 0.75,
      },
      NewHostDelay: 300,
      NotifyNoData: false,
      IncludeTags: true,
      EscalationMessage: "",
    },
    Priority: 1, // priority level - critical
  }

Hope this has been a worthwhile reading! ✨

Furthermore you can explore many more solutions and services that Datadog provides like Logs Forwarding, Cloud Security, Tracing, etc.

Blog

Datadog-Cloudformation Integration using Serverless Framework

Jay Motka