Cloud Custodian is a open-source tool. It uses a stateless rules engine for policy definition and enforcement, with metrics, structured outputs and detailed reporting for clouds infrastructure. It integrates tightly with server less runtime to provide real time remediation/response with low operational overhead.
Cloud Custodian offers opensource cloud security, governance, and management of public cloud environments (AWS, Azure and GCP).

Benefits of using Cloud Custodian:

• Consolidate security findings from multiple accounts and regions into a single report or central view.
• Cost-effective and easily manageable services, such as Lambda and EventBridge.
• Custodian is a policy-as-code solution that is easy to implement and maintain.
• By incorporating Actions into policies, it ensures that appropriate measures are taken immediately upon identifying errors, maintaining consistent and suitable configuration aligned with business requirements.
• Cloud Custodian is a lightweight serverless solution that leverages Infrastructure as Code and EC2/Container technologies for deployment and automation support.

Use Case: Deploy Cloud Custodian policies on AWS Organization accounts from central account.

AWS Organization and Organization Unit

This post will demonstrate how to utilize Cloud Custodian to deploy custodian policies from a centrally managed account and govern all AWS accounts under organization.
Additionally, a non-compliance resource report also can be generated for all accounts and stored in a central S3 bucket to review.

Pre-Requisites:

• CloudCustodian Management IAM role should be created on shared-services account with required permissions.

• CloudCustodian Remote IAM role should be created on all member accounts with required resource & assume role permissions to allow execution from central account.

Multi-Account Cloud Custodian Policy Deployment

The diagram below illustrates that a Shared-services account is used to fulfill this requirement, and Cloud Custodian policy deployments executed from this account to all other accounts.

We have set up this deployment on a daily basis to meet our current requirements. Depending on the needs, we may automate the deployment process to occur with every commit by triggering the deployment pipeline when policy updates are made to the S3 bucket via codebuild CI/CD pipeline.

1. Validate Custodian Policies and Copying to Central S3 Bucket (CI/CD)

The Custodian policy validation codebuild runs for each git pull request and the custodian policies are only copied to the central S3 bucket if the Custodian policy validation step is successful.

Cloud Custodian Git Repository Folder Structure:

- policies/
  |-- all-accounts-policies/
  |-- shared-ou-accounts-policies/
  |-- security-ou-accounts-policies/
  |-- development-ou-accounts-policies/
  |-- production-ou-accounts-policies/
- scripts/
  |-- deploy-custodian-policies.sh
custodian-buildspec.yml

Cloud Custodian Central S3 Bucket Folder Structure:

- policies/
  |-- all-accounts-policies/
  |-- shared-ou-accounts-policies/
  |-- security-ou-accounts-policies/
  |-- development-ou-accounts-policies/
  |-- production-ou-accounts-policies/
- configs/
  |-- all_accounts.yml
  |-- shared_ou_accounts.yml
  |-- security_ou_accounts.yml
  |-- development_ou_accounts.yml
  |-- production_ou_accounts.yml
- scripts/
  |-- deploy-custodian-policies.sh
custodian-buildspec.yml

Sample codebuild buildspec.yml file to validate custodian policy files

In addition to the above validation steps, it is also possible to write a custom script with additional validations to validate the policies and include it in the buildspec.

2. Generate custodian account yaml config files

By configuring the following Python script as a Lambda function in the management account and setting it to run with an EventBridge rule, the script will be triggered whenever a new account is created. This Lambda function will populate the new custodian accounts YAML files. 

Sample production-ou-accounts.yml file

3. Deploy Custodian Policies on Multi Accounts

a. Custodian Policy deployment using c7n-org

The c7n-org tool enables the management of cloud resources across multiple AWS accounts and regions by utilizing the open-source cloud custodian (c7n) policy engine. With this tool, policies can be defined in a central location and then applied to multiple AWS accounts and regions.

With c7n-org, you can:

• Define policies for multiple AWS accounts and regions in a single configuration file.

• Use the same policy definitions across multiple AWS accounts and regions.

• Manage policies in a centralized manner and enforce them across AWS accounts and regions as required.

• Generate reports regarding policy compliance across various AWS accounts and regions.

Below, you'll find script lines demonstrating how I use c7n-org to deploy custodian policies with different custodian account files.

deploy-custodian-policies.sh

b. Scale Down Auto-Scaling-Group after c7n-org Execution

Once the custodian policy deployments completed, we can scale-down the autoscaling group using below commands.

4. Cloud Custodian Policy Execution Modes

Cloud Custodian can run the policies with different mode depending on the provider. The default mode is Pull mode.

Pull Mode - This is default mode, which runs locally where custodian is run

Periodic Mode - In this mode, the policy get deploys as AWS lambda and runs at user defined cron interval.

Cloudtrail Mode - In this mode, the policy get deploys as AWS lambda and is triggered by cloudtrail events. This allows you to apply your policies as soon as events occur. 

In addition to that, Cloud Custodian has following other modes as well, we will look into those in future blogs.

config-rule - In this mode, the policy get deploys as AWS lambda and gets triggered by AWS config when there are configuration changes of your AWS resources.

ec2-instance-state - In this mode, the policy get deploys as AWS lambda and is triggered by ec2 instance state changes.

asg-instance-state - In this mode, the policy get deploys as AWS lambda and is triggered by ASG instance state changes.

guard-duty - In this mode, the policy get deploys as AWS lambda and is triggered by guard-duty responses. AWS Guard Duty is a threat detection service that continuously monitors for malicious activity and unauthorized behavior.

hub-finding - In this mode, the policy get deploys as lambda that executes on security hub finding ingestion events.

hub-action - This policy will provision a lambda and security hub custom action. The action can be invoked on a finding or insight result (collection of findings) from within the console. The action name will have the resource type prefixed as custodian actions are resource specific.

Ref : custodian-modes

5. C7N-Mailer - Cloud Custodian Email Notifications

C7N-Mailer, also known as Custodian Mailer, is a service provided by the open-source policy engine, Cloud Custodian (also known as c7n). It is designed to send email notifications and alerts to users based on various events and actions that occur within the AWS environment.

C7N-Mailer can be configured to subscribe to an SQS queue, search for users, and send emails using the Simple Email Service (SES) provided by AWS. It can also send notifications to other services like DataDog, Slack, Splunk. SQS queue should be cross-account enabled for sending between accounts

Email:
To ensure proper functioning of the email notifications, it is important to verify your email address in SES and set it as the from_address in the mailer.yml file. It is important to note that SES operates in a sandbox mode by default, which requires verification of every email recipient individually. In case you require access to more functionalities, you can submit an AWS support ticket to be taken out of the SES sandbox mode.

Your mailer.yml should now look something like this:

queue_url: https://sqs.us-east-1.amazonaws.com/1234567890/cloudcustodian-notify-queue
role: arn:aws:iam::123456790:role/CloudCustodian-Mailer-Role
from_address: you@example.com
from_address: cloud.team@gmail.com
contact_tags:
  - Owner
  - Support_Email
  - Launched_By
  - Started_By

Here we utilize SNS transport in order to deliver messages from several accounts to SNS topic that is located in a central account. A SQS queue is subscribed to the SNS topic.

The c7n-mailer is configured to pull messages from the SQS queue and send them using SES.

Provided below the complete sample custodian policy along with the notification action

Sample custodian policies with remediation action