AWS Prefix Lists for the Organization

AWS Managed Prefix lists are a really powerful way of abstracting the details of CIDR Blocks into something meaningful for the humble cloud engineer.

Much like DNS does for IP addresses to hostnames, we can create a managed prefix for any service that has several CIDR blocks associated with it.

I've talked about this before here, but with more and more stuff needing to talk to services hosted on the internet and with our implementation of AWS Network Firewall, it is time for a re-visit.

The challenge

Most people are familiar with the AWS Public IP Address Ranges source ip-ranges.json, doco linked here, released in 2014. Can you believe it!

It provides all the Public IP ranges for all services across all regions. A very handy thing to have, and with it being hosted in a JSON file, we can automatically parse it with any automation.

But why don't AWS do this for us? I can't answer that today; however, a good example of such automation exists in the AWS samples GitHub here.

The AWS Samples repo does all the heavy lifting with Lambda and creates the following:-

But I'm feeling hacky and need some Organization-level sharing, Lambda PowerTools, AppConfig and deployment through SAM CLI.
Let's get into it!

Some Enhancements

Lambda PowerTools

As the homepage displays, Lambda PowerTools is a suite of utilities for AWS Lambda functions to ease adopting best practices such as tracing, structured logging, custom metrics, idempotency, batching, and more.

One of the reasons for implementing this is to make the Cloudwatch logs nicer to parse. It also removes a few lines of code in Lambda without requiring major logging structure changes.

As I've rebuilt this into SAM Cli, this is just an addition to the requirements.txt file to include aws-lambda-powertools.

AppConfig

If you look at the original AWS example, it requires a services.json to be defined to configure what regions and services to create the WAF IPsets and VPC Prefixes. It will look something like the following:-

{
  "Services": [
    {
      "Name": "CODEBUILD",
      "Regions": [
        "ap-southeast-2"
      ],
      "PrefixList": {
        "Enable": true,
        "Summarize": true
      },
      "WafIPSet": {
        "Enable": true,
        "Summarize": true,
        "Scopes": [
          "REGIONAL"
        ]
      }
    }
  ]
}

This is in JSON format, so perfect for AppConfig.
As we have configured Lambda PowerTools, AppConfig is easily added as a Lambda Layer.

I've made several updates to the template.yaml file to define our AppConfig hosted configuration and deployment methods, adding environment variables and our Lambda layer.

Parameters:
  AppConfigAppName:
    Type: String
    Description: AppConfig Application Name
    Default: aws-ip-ranges
  AppConfigAppEnvironmentName:
    Type: String
    Description: AppConfig Application Environment Name
    Default: dev
  AppConfigName:
    Type: String
    Description: AppConfig Name
    Default: services
  AppConfigLayerArn:
    Type: String
    Description: Retrieve AWS AppConfig Lambda extension arn from `https://docs.aws.amazon.com/appconfig/latest/userguide/appconfig-integration-lambda-extensions-versions.html#appconfig-integration-lambda-extensions-enabling-x86-64`
    Default: arn:aws:lambda:ap-southeast-2:080788657173:layer:AWS-AppConfig-Extension:91
  AwsOrgArn:
    Type: String
    Description: The ARN of the AWS Organization used to share Prefix Lists
    Default: notset

Resources:
  SAMConfigApplication:
    Type: AWS::AppConfig::Application
    Properties:
      Name: !Ref AppConfigAppName

  Environment:
    Type: AWS::AppConfig::Environment
    Properties:
      Name: !Ref AppConfigAppEnvironmentName
      ApplicationId: !Ref SAMConfigApplication

  SAMConfigConfigurationProfile:
    Type: AWS::AppConfig::ConfigurationProfile
    Properties:
      ApplicationId: !Ref SAMConfigApplication
      Name: !Ref AppConfigName
      Type: 'AWS.Freeform'
      LocationUri: 'hosted'

  SAMConfigDeploymentStrategy:
    Type: AWS::AppConfig::DeploymentStrategy
    Properties:
      Name: "SAMConfigDeploymentStrategy"
      Description: "A deployment strategy to deploy the config immediately"
      DeploymentDurationInMinutes: 0
      FinalBakeTimeInMinutes: 0
      GrowthFactor: 100
      GrowthType: LINEAR
      ReplicateTo: NONE

  BasicHostedConfigurationVersion:
    Type: AWS::AppConfig::HostedConfigurationVersion
    Properties:
      ApplicationId: !Ref SAMConfigApplication
      ConfigurationProfileId: !Ref SAMConfigConfigurationProfile
      Description: 'AWS Service configuration for update-aws-ip-ranges'
      ContentType: 'application/json'
      Content: |
        {
          "Services": [
            {
              "Name": "CODEBUILD",
              "Regions": [
                "ap-southeast-2"
              ],
              "PrefixList": {
                "Enable": true,
                "Summarize": true
              },
              "WafIPSet": {
                "Enable": true,
                "Summarize": true,
                "Scopes": [
                  "REGIONAL"
                ]
              }
            }
          ]
        }
  AppConfigDeployment:
    Type: AWS::AppConfig::Deployment
    Properties:
      ApplicationId: !Ref SAMConfigApplication
      ConfigurationProfileId: !Ref SAMConfigConfigurationProfile
      ConfigurationVersion: !Ref BasicHostedConfigurationVersion
      DeploymentStrategyId: !Ref SAMConfigDeploymentStrategy
      EnvironmentId: !Ref Environment

  LambdaUpdateIPRanges:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.9
      Architectures:
        - x86_64
      Environment:
        Variables:
          APP_CONFIG_APP_NAME: !Ref AppConfigAppName
          APP_CONFIG_APP_ENV_NAME: !Ref AppConfigAppEnvironmentName
          APP_CONFIG_NAME: !Ref AppConfigName
          AWS_ORG_ARN: !Ref AwsOrgArn
          LOG_LEVEL: INFO
      Layers:
        - !Ref AppConfigLayerArn

In order to pull in the config, we add the following function and a few adjustments in the handler code.

def get_service_config():
    try:
        appconfig = f"http://localhost:2772/applications/{APP_CONFIG_APP_NAME}/environments/{APP_CONFIG_APP_ENV_NAME}/configurations/{APP_CONFIG_NAME}"
        with request.urlopen(appconfig) as response:  # nosec B310
            config = response.read()
        return config
    except Exception as error:
        logger.error("Error retrieving AppConfig configuration. Exiting")
        logger.exception(error)
        raise error

Organization-level sharing

To enable all the users in our organization to benefit from our automation, we need to share the VPC Prefixes via Resource Access Manager (RAM).

We define an additional function to create the RAM share on creating the VPC Prefix List.

def create_prefix_ram(client: Any, prefix_list_name: str, prefix_list_arn: str) -> None:
    """Create the VPC Prefix List RAM Share"""
    logger.info("create_prefix_ram start")
    logger.debug(f"Parameter client: {client}")
    logger.debug(f"Parameter prefix_list_name: {prefix_list_name}")
    logger.debug(f"Parameter prefix_list_arn: {prefix_list_arn}")

    logger.info(f'Creating RAM Share "{prefix_list_name}" with Arn "{prefix_list_arn}"')
    response = client.create_resource_share(
        name=prefix_list_name,
        resourceArns=[prefix_list_arn],
        principals=[AWS_ORG_ARN],
        allowExternalPrincipals=False,
        tags=[
            {"key": "Name", "value": prefix_list_name},
            {"key": "ManagedBy", "value": MANAGED_BY},
            {"key": "CreatedAt", "value": datetime.now(timezone.utc).isoformat()},
            {"key": "UpdatedAt", "value": "Not yet"},
        ],
    )

    logger.info(f'Created VPC Prefix List RAM Share"{prefix_list_name}"')
    logger.debug(f"Response: {response}")
    logger.debug("Function return: None")
    logger.info("create_prefix_ram end")

NOTE: All the code is available at https://github.com/sjramblings/update-aws-ip-ranges

The End Result

With our updates, this is now our enhanced Lambda workflow

Update to arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged will trigger the function
Lambda will execute under the IAM Role, retrieve its service config from AppConfig and write output to Cloudwatch
For enabled services, it will create/update the VPC Prefixes & WAF IPSets
VPC Prefixes will be shared with the Organization ID via RAM

A few screenshot of the results:-

Summary

The wonderful contributors to AWS-Samples had done a great job in getting us started. However, I needed to take it a bit further, adding the VPC Prefix Sharing and a few other tweaks.

As noted above, all code is available at https://github.com/sjramblings/update-aws-ip-ranges. I will also raise Pull Requests to backport some of this into the AWS Samples repo.

I hope this helps someone else!

Cheers

Blog