Infrastructure as Code on AWS using Go and Pulumi

When we talk about Infrastructure as Code or IaC, the first tool that comes to mind is Terraform. Terraform, created by HashiCorp, has become the standard for documentation and infrastructure management, but its declarative language, HCL (HashiCorp Configuration Language), has some limitations. The main limitation is not being a programming language but a configuration one.

Some alternatives have been emerging to fulfill these needs, such as:

AWS Cloud Development Kit, Amazon's solution that allows us to use TypeScript, Python, and Java to program the infrastructure using the cloud provider's solutions;
Pulumi, which allows us to use TypeScript, JavaScript, Python, Go, and C# to program infrastructures using solutions from AWS, Microsoft Azure, Google Cloud, and Kubernetes installations.
I will introduce Pulumi, using the Go language to create some infrastructure examples on AWS.

Installation

To make use of Pulumi, we first need to install its command-line application. Following the documentation, I installed it on my macOS using the command:

brew install pulumi

On the website, you can see how to install it on Windows and Linux.

Configure AWS Account Access

Since I will use AWS in this example, the next necessary step is to configure the credentials. For that, I got my access key and secret from the AWS dashboard and set the required environment variables:

export AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>
export AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY>

Creating the project

With the initial dependencies configured, we can now create the project:

mkdir post-pulumi
cd post-pulumi
pulumi new aws-go

One of the creation steps requires setting up an account on the Pulumi website. For that, the command-line application opens the browser for this step to be completed. So I logged in with my Github account, completed the registration, returned to the terminal, and continued the project creation without any problems.

You can see the result of running the command can at this link. In addition, at the end of the process, it installs all the necessary dependencies for creating the project in Go.

Files created

Looking at the directory contents, we can see that some configuration files and a main.go were created.

Pulumi.yaml

name: post-pulumi
runtime: go
description: A minimal AWS Go Pulumi program

Pulumi.dev.yaml

config:
  aws:region: us-east-1

main.go

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an AWS resource (S3 Bucket)
        bucket, err := s3.NewBucket(ctx, "my-bucket", nil)
        if err != nil {
            return err
        }

        // Export the name of the bucket
        ctx.Export("bucketName", bucket.ID())
        return nil
    })
}

When running

pulumi up

The bucket was created in S3, as the code indicates.

And the command:

pulumi destroy

Destroy all the resources, in this case, the S3 bucket.

First example - creating a static page in S3

Now let's do some more complex examples.

The first step is to create a static page, which we are going to deploy:

mkdir static

Inside this directory, I created the file:

static/index.html

<html>
    <body>
        <h1>Hello, Pulumi!</h1>
    </body>
</html>

I changed main.go to reflect the new structure:

package main

import (
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/s3"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Create an AWS resource (S3 Bucket)
        bucket, err := s3.NewBucket(ctx, "my-bucket", &s3.BucketArgs{
            Website: s3.BucketWebsiteArgs{
                IndexDocument: pulumi.String("index.html"),
            },
        })
        if err != nil {
            return err
        }

        // Export the name of the bucket
        ctx.Export("bucketName", bucket.ID())

        _, err = s3.NewBucketObject(ctx, "index.html", &s3.BucketObjectArgs{
            Acl:         pulumi.String("public-read"),
            ContentType: pulumi.String("text/html"),
            Bucket:      bucket.ID(),
            Source:      pulumi.NewFileAsset("static/index.html"),
        })
        if err != nil {
            return err
        }
        ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))
        return nil
    })
}

To update run:

pulumi up

And confirm the change.

The code snippet:

ctx.Export("bucketEndpoint", pulumi.Sprintf("http://%s", bucket.WebsiteEndpoint))

Generate as output the address to access index.html:

Outputs:
  + bucketEndpoint: "http://my-bucket-357877e.s3-website-us-east-1.amazonaws.com"

The case above is a straightforward example, but it already demonstrates the power of the tool. So let's make things a little more complex and fun now.

Second example - a site inside a container

Let's create a Dockerfile with a web server to host our static content:

static/Dockerfile

FROM golang

ADD . /go/src/foo

WORKDIR /go/src/foo
RUN go build -o /go/bin/main

ENTRYPOINT /go/bin/main

EXPOSE 80

Let's now create the static/main.go file, which will be our web server:

package main

import (
    "log"
    "net/http"
)

func main() {
    r := http.NewServeMux()
    fileServer := http.FileServer(http.Dir("./"))
    r.Handle("/", http.StripPrefix("/", fileServer))
    s := &http.Server{
        Addr:    ":80",
        Handler: r,
    }
    log.Fatal(s.ListenAndServe())
}

Let's change main.go to include the infrastructure of an ECS cluster and everything else needed to run our container:

package main

import (
    "encoding/base64"
    "fmt"
    "strings"

    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
    elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
    "github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        // Read back the default VPC and public subnets, which we will use.
        t := true
        vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
        if err != nil {
            return err
        }
        subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
        if err != nil {
            return err
        }

        // Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
        webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
            VpcId: pulumi.String(vpc.Id),
            Egress: ec2.SecurityGroupEgressArray{
                ec2.SecurityGroupEgressArgs{
                    Protocol:   pulumi.String("-1"),
                    FromPort:   pulumi.Int(0),
                    ToPort:     pulumi.Int(0),
                    CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
                },
            },
            Ingress: ec2.SecurityGroupIngressArray{
                ec2.SecurityGroupIngressArgs{
                    Protocol:   pulumi.String("tcp"),
                    FromPort:   pulumi.Int(80),
                    ToPort:     pulumi.Int(80),
                    CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
                },
            },
        })
        if err != nil {
            return err
        }

        // Create an ECS cluster to run a container-based service.
        cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
        if err != nil {
            return err
        }

        // Create an IAM role that can be used by our service's task.
        taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
            AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
        })
        if err != nil {
            return err
        }
        _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
            Role:      taskExecRole.Name,
            PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
        })
        if err != nil {
            return err
        }

        // Create a load balancer to listen for HTTP traffic on port 80.
        webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
            Subnets:        toPulumiStringArray(subnet.Ids),
            SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
        })
        if err != nil {
            return err
        }
        webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
            Port:       pulumi.Int(80),
            Protocol:   pulumi.String("HTTP"),
            TargetType: pulumi.String("ip"),
            VpcId:      pulumi.String(vpc.Id),
        })
        if err != nil {
            return err
        }
        webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
            LoadBalancerArn: webLb.Arn,
            Port:            pulumi.Int(80),
            DefaultActions: elb.ListenerDefaultActionArray{
                elb.ListenerDefaultActionArgs{
                    Type:           pulumi.String("forward"),
                    TargetGroupArn: webTg.Arn,
                },
            },
        })
        if err != nil {
            return err
        }

        //create a new ECR repository
        repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
        if err != nil {
            return err
        }

        repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
            creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
                RegistryId: rid,
            })
            if err != nil {
                return nil, err
            }
            data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
            if err != nil {
                fmt.Println("error:", err)
                return nil, err
            }

            return strings.Split(string(data), ":"), nil
        }).(pulumi.StringArrayOutput)
        repoUser := repoCreds.Index(pulumi.Int(0))
        repoPass := repoCreds.Index(pulumi.Int(1))

        //build the image
        image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
            Build: docker.DockerBuildArgs{
                Context: pulumi.String("./static"),
            },
            ImageName: repo.RepositoryUrl,
            Registry: docker.ImageRegistryArgs{
                Server:   repo.RepositoryUrl,
                Username: repoUser,
                Password: repoPass,
            },
        })
        if err != nil {
            return err
        }

        containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
            fmtstr := `[{
                "name": "my-app",
                "image": %q,
                "portMappings": [{
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }]
            }]`
            return fmt.Sprintf(fmtstr, name), nil
        }).(pulumi.StringOutput)

        // Spin up a load balanced service running NGINX.
        appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
            Family:                  pulumi.String("fargate-task-definition"),
            Cpu:                     pulumi.String("256"),
            Memory:                  pulumi.String("512"),
            NetworkMode:             pulumi.String("awsvpc"),
            RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
            ExecutionRoleArn:        taskExecRole.Arn,
            ContainerDefinitions:    containerDef,
        })
        if err != nil {
            return err
        }
        _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
            Cluster:        cluster.Arn,
            DesiredCount:   pulumi.Int(5),
            LaunchType:     pulumi.String("FARGATE"),
            TaskDefinition: appTask.Arn,
            NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
                AssignPublicIp: pulumi.Bool(true),
                Subnets:        toPulumiStringArray(subnet.Ids),
                SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
            },
            LoadBalancers: ecs.ServiceLoadBalancerArray{
                ecs.ServiceLoadBalancerArgs{
                    TargetGroupArn: webTg.Arn,
                    ContainerName:  pulumi.String("my-app"),
                    ContainerPort:  pulumi.Int(80),
                },
            },
        }, pulumi.DependsOn([]pulumi.Resource{webListener}))
        if err != nil {
            return err
        }
        // Export the resulting web address.
        ctx.Export("url", webLb.DnsName)
        return nil
    })
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
    var res []pulumi.StringInput
    for _, s := range a {
        res = append(res, pulumi.String(s))
    }
    return pulumi.StringArray(res)
}

Complex? Yes, but this complexity is inherent to AWS features and not Pulumi. We would have similar complexity if we were using Terraform or CDK.

Before running our code, we need to download the new dependencies:

go get github.com/pulumi/pulumi-docker
go get github.com/pulumi/pulumi-docker/sdk/v3/go/docker

Now just run the command:

pulumi up

The execution output will generate the URL of the load balancer, which we will use to access the contents of our container in execution.

Reorganizing the code

Now we can start making use of the advantages of a complete programming language like Go. For example, we could use language features like functions, concurrency, conditionals, etc. In this example, we are going to organize our code better. For this, I created the iac directory and the iac/fargate.go file. After that, I moved most of the logic from main.go to the new file:

package iac

import (
    "encoding/base64"
    "fmt"
    "strings"

    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ec2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecr"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/ecs"
    elb "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/elasticloadbalancingv2"
    "github.com/pulumi/pulumi-aws/sdk/v4/go/aws/iam"
    "github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func FargateRun(ctx *pulumi.Context) error {

    // Read back the default VPC and public subnets, which we will use.
    t := true
    vpc, err := ec2.LookupVpc(ctx, &ec2.LookupVpcArgs{Default: &t})
    if err != nil {
        return err
    }
    subnet, err := ec2.GetSubnetIds(ctx, &ec2.GetSubnetIdsArgs{VpcId: vpc.Id})
    if err != nil {
        return err
    }

    // Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
    webSg, err := ec2.NewSecurityGroup(ctx, "web-sg", &ec2.SecurityGroupArgs{
        VpcId: pulumi.String(vpc.Id),
        Egress: ec2.SecurityGroupEgressArray{
            ec2.SecurityGroupEgressArgs{
                Protocol:   pulumi.String("-1"),
                FromPort:   pulumi.Int(0),
                ToPort:     pulumi.Int(0),
                CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
            },
        },
        Ingress: ec2.SecurityGroupIngressArray{
            ec2.SecurityGroupIngressArgs{
                Protocol:   pulumi.String("tcp"),
                FromPort:   pulumi.Int(80),
                ToPort:     pulumi.Int(80),
                CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
            },
        },
    })
    if err != nil {
        return err
    }

    // Create an ECS cluster to run a container-based service.
    cluster, err := ecs.NewCluster(ctx, "app-cluster", nil)
    if err != nil {
        return err
    }

    // Create an IAM role that can be used by our service's task.
    taskExecRole, err := iam.NewRole(ctx, "task-exec-role", &iam.RoleArgs{
        AssumeRolePolicy: pulumi.String(`{
    "Version": "2008-10-17",
    "Statement": [{
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }]
}`),
    })
    if err != nil {
        return err
    }
    _, err = iam.NewRolePolicyAttachment(ctx, "task-exec-policy", &iam.RolePolicyAttachmentArgs{
        Role:      taskExecRole.Name,
        PolicyArn: pulumi.String("arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"),
    })
    if err != nil {
        return err
    }

    // Create a load balancer to listen for HTTP traffic on port 80.
    webLb, err := elb.NewLoadBalancer(ctx, "web-lb", &elb.LoadBalancerArgs{
        Subnets:        toPulumiStringArray(subnet.Ids),
        SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
    })
    if err != nil {
        return err
    }
    webTg, err := elb.NewTargetGroup(ctx, "web-tg", &elb.TargetGroupArgs{
        Port:       pulumi.Int(80),
        Protocol:   pulumi.String("HTTP"),
        TargetType: pulumi.String("ip"),
        VpcId:      pulumi.String(vpc.Id),
    })
    if err != nil {
        return err
    }
    webListener, err := elb.NewListener(ctx, "web-listener", &elb.ListenerArgs{
        LoadBalancerArn: webLb.Arn,
        Port:            pulumi.Int(80),
        DefaultActions: elb.ListenerDefaultActionArray{
            elb.ListenerDefaultActionArgs{
                Type:           pulumi.String("forward"),
                TargetGroupArn: webTg.Arn,
            },
        },
    })
    if err != nil {
        return err
    }

    repo, err := ecr.NewRepository(ctx, "foo", &ecr.RepositoryArgs{})
    if err != nil {
        return err
    }

    repoCreds := repo.RegistryId.ApplyT(func(rid string) ([]string, error) {
        creds, err := ecr.GetCredentials(ctx, &ecr.GetCredentialsArgs{
            RegistryId: rid,
        })
        if err != nil {
            return nil, err
        }
        data, err := base64.StdEncoding.DecodeString(creds.AuthorizationToken)
        if err != nil {
            fmt.Println("error:", err)
            return nil, err
        }

        return strings.Split(string(data), ":"), nil
    }).(pulumi.StringArrayOutput)
    repoUser := repoCreds.Index(pulumi.Int(0))
    repoPass := repoCreds.Index(pulumi.Int(1))

    image, err := docker.NewImage(ctx, "my-image", &docker.ImageArgs{
        Build: docker.DockerBuildArgs{
            Context: pulumi.String("./static"),
        },
        ImageName: repo.RepositoryUrl,
        Registry: docker.ImageRegistryArgs{
            Server:   repo.RepositoryUrl,
            Username: repoUser,
            Password: repoPass,
        },
    })
    if err != nil {
        return err
    }

    containerDef := image.ImageName.ApplyT(func(name string) (string, error) {
        fmtstr := `[{
                "name": "my-app",
                "image": %q,
                "portMappings": [{
                    "containerPort": 80,
                    "hostPort": 80,
                    "protocol": "tcp"
                }]
            }]`
        return fmt.Sprintf(fmtstr, name), nil
    }).(pulumi.StringOutput)

    // Spin up a load balanced service running NGINX.
    appTask, err := ecs.NewTaskDefinition(ctx, "app-task", &ecs.TaskDefinitionArgs{
        Family:                  pulumi.String("fargate-task-definition"),
        Cpu:                     pulumi.String("256"),
        Memory:                  pulumi.String("512"),
        NetworkMode:             pulumi.String("awsvpc"),
        RequiresCompatibilities: pulumi.StringArray{pulumi.String("FARGATE")},
        ExecutionRoleArn:        taskExecRole.Arn,
        ContainerDefinitions:    containerDef,
    })
    if err != nil {
        return err
    }
    _, err = ecs.NewService(ctx, "app-svc", &ecs.ServiceArgs{
        Cluster:        cluster.Arn,
        DesiredCount:   pulumi.Int(5),
        LaunchType:     pulumi.String("FARGATE"),
        TaskDefinition: appTask.Arn,
        NetworkConfiguration: &ecs.ServiceNetworkConfigurationArgs{
            AssignPublicIp: pulumi.Bool(true),
            Subnets:        toPulumiStringArray(subnet.Ids),
            SecurityGroups: pulumi.StringArray{webSg.ID().ToStringOutput()},
        },
        LoadBalancers: ecs.ServiceLoadBalancerArray{
            ecs.ServiceLoadBalancerArgs{
                TargetGroupArn: webTg.Arn,
                ContainerName:  pulumi.String("my-app"),
                ContainerPort:  pulumi.Int(80),
            },
        },
    }, pulumi.DependsOn([]pulumi.Resource{webListener}))
    if err != nil {
        return err
    }
    // Export the resulting web address.
    ctx.Export("url", webLb.DnsName)
    return nil
}

func toPulumiStringArray(a []string) pulumi.StringArrayInput {
    var res []pulumi.StringInput
    for _, s := range a {
        res = append(res, pulumi.String(s))
    }
    return pulumi.StringArray(res)
}

The next step was to configure the iac directory to be a Go language module:

cd iac
go mod init github.com/eminetto/post-pulumi/iac
cd ..
go mod edit -replace github.com/eminetto/post-pulumi/iac=./iac
go mod tidy

Our main.go can now be simplified:

package main

import (
    "github.com/eminetto/post-pulumi/iac"
    "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
    pulumi.Run(func(ctx *pulumi.Context) error {
        return iac.FargateRun(ctx)
    })
}

That way, we can better manage the structure of the code that will handle AWS resources. We can reuse this code in other projects, use environment variables, write tests, or whatever else our imagination allows.

Conclusion

Using a tool like Pulumi significantly increases the range of options that we can use in building a project's infrastructure while maintaining readability, code reuse and organization.

Blog