API Gateway and Lambda Throttling with Terraform. Part 2

In the previous post we covered the basics of setting up throttling for your API Gateway and Lambda functions. In this follow-up, we’ll take it to the next level, adding budget controls, time-based throttling adjustments, and AWS WAF security integration to safeguard your API while optimizing both performance and cost-efficiency.

1. Budget and Billing Alerts

Managing cloud expenses is essential to avoiding surprises, especially when usage spikes unexpectedly. AWS Budgets allows you to set up monthly spending limits and receive alerts before overspending occurs.

Example: Setting up Budget Alerts

resource "aws_budgets_budget" "monthly_budget" {
  name         = "monthly-budget-${var.environment}"
  budget_type  = "COST"
  time_unit    = "MONTHLY"
  limit_amount = "1000"
  limit_unit   = "USD"

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 80
    threshold_type             = "PERCENTAGE"
    notification_type          = "FORECASTED"
    subscriber_email_addresses = ["alert@example.com"]
  }

  notification {
    comparison_operator        = "GREATER_THAN"
    threshold                  = 100
    threshold_type             = "PERCENTAGE"
    notification_type          = "ACTUAL"
    subscriber_email_addresses = ["alert@example.com"]
  }
}

You can also configure alerts to trigger responses in other AWS services, such as SNS topics or CloudWatch Alarms, to better manage your Lambda functions during traffic spikes.

# SNS Topic for Alerts
resource "aws_sns_topic" "budget_alerts_topic" {
  name = "budget-alerts-topic-${var.environment}"
}

# Subscribe an email address to the SNS topic
resource "aws_sns_topic_subscription" "email_subscription" {
  topic_arn = aws_sns_topic.alarms.arn
  protocol  = "email"
  endpoint  = var.alarms_email
}

# Example CloudWatch Alarm for Lambda Throttling (to control costs)
resource "aws_cloudwatch_metric_alarm" "lambda_throttles" {
  alarm_name          = "lambda-throttles-${aws_lambda_function.example.function_name}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "Throttles"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "Sum"
  threshold           = 5
  alarm_description   = "Alert for Lambda throttling due to budget constraints"
  alarm_actions       = [aws_sns_topic.budget_alerts_topic.arn]

  dimensions = {
    FunctionName = aws_lambda_function.example.function_name
  }
}

2. Time-Based Throttling Adjustments

For high-traffic applications, one-size-fits-all throttling can be inefficient. You can dynamically adjust API Gateway throttling limits during peak and off-peak hours using AWS EventBridge and Lambda.

Example: Time-Based Throttling with Lambda and EventBridge

Below is the Python code that dynamically adjusts API Gateway throttling limits:

import boto3
import os

def handler(event, context):
    api_gateway = boto3.client('apigateway')

    api_id = os.environ['API_ID']
    stage = os.environ['STAGE']
    burst_limit = int(os.environ['BURST_LIMIT'])
    rate_limit = int(os.environ['RATE_LIMIT'])

    # Update method settings with new throttling limits
    api_gateway.update_stage(
        restApiId=api_id,
        stageName=stage,
        patchOperations=[
            {
                'op': 'replace',
                'path': '/*/*/throttling/burstLimit',
                'value': str(burst_limit)
            },
            {
                'op': 'replace',
                'path': '/*/*/throttling/rateLimit',
                'value': str(rate_limit)
            }
        ]
    )

    return {
        'statusCode': 200,
        'body': f'Throttling updated: Burst={burst_limit}, Rate={rate_limit}'
    }

Lambda Creation in Terraform

To automate the creation of the Lambda function for adjusting throttling limits, use the following Terraform configuration:

variable "api_throttling_configs" {
  type = map(object({
    burst_limit = number
    rate_limit  = number
  }))
  default = {
    peak = {
      burst_limit = 5000
      rate_limit  = 2000
    }
    off_peak = {
      burst_limit = 1000
      rate_limit  = 500
    }
  }
}


resource "aws_lambda_function" "adjust_throttling_peak" {
  function_name = "adjust_throttling_peak_${var.environment}"
  handler       = "adjust_throttling.handler"
  runtime       = "python3.11"
  role          = aws_iam_role.lambda_exec.arn
  filename      = "adjust_throttling_peak.zip"

  environment {
    variables = {
      API_ID      = aws_api_gateway_rest_api.example.id
      STAGE       = aws_api_gateway_stage.example.stage_name
      BURST_LIMIT = var.api_throttling_configs.peak.burst_limit
      RATE_LIMIT  = var.api_throttling_configs.peak.rate_limit
    }
  }
}

resource "aws_lambda_function" "adjust_throttling_off_peak" {
  function_name = "adjust_throttling_off_peak_${var.environment}"
  handler       = "adjust_throttling.handler"
  runtime       = "python3.11"
  role          = aws_iam_role.lambda_exec.arn
  filename      = "adjust_throttling_off_peak.zip"

  environment {
    variables = {
      API_ID      = aws_api_gateway_rest_api.example.id
      STAGE       = aws_api_gateway_stage.example.stage_name
      BURST_LIMIT = var.api_throttling_configs.off_peak.burst_limit
      RATE_LIMIT  = var.api_throttling_configs.off_peak.rate_limit
    }
  }
}

Here's a more understandable version:

Automating Time-Based Adjustments

To automatically adjust settings based on the time of day (for example, peak hours vs. off-peak hours), you can use EventBridge Rules to trigger Lambda functions at specific times. This way, your system can automatically adjust to handle more traffic during peak times and reduce capacity during off-peak hours.

In this example, we set up rules that run at 8:00 AM (peak hours) and 8:00 PM (off-peak hours). Here's how it works:

Peak Hours Rule: This triggers at 8:00 AM to increase capacity during busy times.
Off-Peak Hours Rule: This triggers at 8:00 PM to reduce capacity during slower times.

Here’s how to set it up:

# Define a rule for peak hours (8:00 AM)
resource "aws_cloudwatch_event_rule" "peak_hours_rule" {
  name                = "peak-hours-rule"
  schedule_expression = "cron(0 8 * * ? *)"  # Adjust to your peak hours
}

# Define a rule for off-peak hours (8:00 PM)
resource "aws_cloudwatch_event_rule" "off_peak_hours_rule" {
  name                = "off-peak-hours-rule"
  schedule_expression = "cron(0 20 * * ? *)"  # Adjust to your off-peak hours
}

# Set the target for peak hours (a Lambda function to handle peak traffic)
resource "aws_cloudwatch_event_target" "peak_hours_target" {
  rule      = aws_cloudwatch_event_rule.peak_hours_rule.name
  target_id = "adjust_throttling_peak"
  arn       = aws_lambda_function.adjust_throttling_peak.arn
}

# Set the target for off-peak hours (a Lambda function to handle lower traffic)
resource "aws_cloudwatch_event_target" "off_peak_hours_target" {
  rule      = aws_cloudwatch_event_rule.off_peak_hours_rule.name
  target_id = "adjust_throttling_off_peak"
  arn       = aws_lambda_function.adjust_throttling_off_peak.arn
}

This setup automatically changes your system’s capacity based on time. At 8:00 AM, the system prepares for heavy traffic, and at 8:00 PM, it scales down for lighter usage.

3. Adding AWS WAF for Extra Protection

Integrating AWS WAF (Web Application Firewall) into your API Gateway helps protect it from malicious traffic, such as bots or potential DDoS (Distributed Denial of Service) attacks. It acts as a security layer that filters out harmful requests before they reach your API.

Example: Blocking IPs and Controlling Bots with WAF

This example shows how to set up a WAF rule that protects your API Gateway by blocking bot traffic:

resource "aws_wafv2_web_acl" "api_gateway_acl" {
  name  = "api-gateway-waf-acl"
  scope = "REGIONAL"  # Use "REGIONAL" to protect your regional API Gateway

  # Default action to allow all requests unless a rule blocks them
  default_action {
    allow {}
  }

  # Enable visibility so you can monitor activity using CloudWatch metrics
  visibility_config {
    cloudwatch_metrics_enabled = true
    metric_name                = "apiGatewayAcl"
    sampled_requests_enabled   = true
  }

  # Bot Protection Rule to block bad bot traffic
  rule {
    name     = "BotProtectionRule"  # Name of the rule
    priority = 2  # Order of execution for this rule (lower number = higher priority)

    # No override action means the default allow/block settings apply
    override_action {
      none {}
    }

    # Managed rule group to use AWS's pre-built bot control rules
    statement {
      managed_rule_group_statement {
        name        = "AWSManagedRulesBotControlRuleSet"
        vendor_name = "AWS"  # AWS provides these bot protection rules
      }
    }

    # Enable visibility for this specific rule as well
    visibility_config {
      cloudwatch_metrics_enabled = true
      metric_name                = "botProtection"
      sampled_requests_enabled   = true
    }
  }
}

This setup adds protection to your API by detecting and blocking bot traffic, while also giving you the ability to monitor suspicious activity via CloudWatch metrics.

Conclusion

By incorporating Budget Alerts, Time-Based Throttling Adjustments, and WAF Integration, you're ensuring that your API is not only efficient and cost-effective but also secure. These advanced practices complement each other, providing a well-rounded solution for managing traffic, costs, and security in a scalable AWS environment.

Next Steps

Monitoring and Fine-Tuning: After deploying these configurations, it's crucial to monitor their performance. Use CloudWatch metrics and logs to verify that throttling adjustments and security rules are working as expected.
Performance Testing: Perform load testing during both peak and off-peak hours to ensure your dynamic throttling setup aligns with your traffic patterns.
Further Optimization: Explore other AWS services like AWS Shield for advanced DDoS protection and AWS X-Ray for tracing API Gateway requests to identify potential bottlenecks.

By continually monitoring and adjusting, you’ll be able to maintain an API infrastructure that scales with traffic, optimizes costs, and maintains high levels of security.

Blog