Crossplane is better than Terraform in K8S world

TL;DR

Crossplane is a solution you should consider when your infrastructure is to serve your k8s applications.

Story

I have been using Terraform for a very long time. It is simple to use with a very huge community support. However every solution has its shortcomings. Terraform is not really a nice solution when you have to work with k8s.

My AWS infrastructure is closely related to my K8S applications. Maybe I have to upload objects to S3, or store my data in RDS. If you are using AWS EKS, you should be very familiar with IRSA (IAM roles for service accounts) feature, it grants your service accounts with AWS permissions. Then you pods can reference these service accounts and interact with AWS APIs.

So IAM roles are AWS stuffs, and service accounts are K8S stuffs. You have to create both IAM roles and service accounts using some methods.

I can use Terraform to create both IAM roles and service accounts but it is very operational unfriendly. K8s deployments not just only have service accounts need to be applied. How about using Terraform to deploy all AWS infrastructure and k8s manifests? IMO, DO NOT DO IT. Argo CD, Flux CD or any GitOps tools are much better than Terraform.

Never solve a small problem by bringing another big trouble.

Now back to my question, how I should integrate IAM roles with service accounts? How about doing the opposite, use k8s to provision IAM roles?

And then Crossplane comes into my eyes. Simply speaking, Crossplane is a k8s style of Terraform.

Tutorial

Now I am going to show you how to use Crossplane.

Prerequisite

You need to have an admin privilege in your AWS account.
A k8s cluster

1. Define your variables

Specify any value you like.

EKS_CLUSTER_NAME=""
AWS_REGION=""
AWS_IAM_ROLE_NAME="${EKS_CLUSTER_NAME}-crossplane-controller"

2. Install Crossplane using helm charts

Everyone loves helm chart :)

helm repo add crossplane https://charts.crossplane.io/master/
helm install --create-namespace --namespace crossplane-system crossplane crossplane/crossplane --version 1.9.0-rc.0.9.g243f1f47

3. Install AWS provider

Crossplane can provision infrastructure in many platforms. Let say if you want to deploy to Azure, you can install Azure provider.

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1alpha1
kind: ControllerConfig
metadata:
  name: aws-config
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${AWS_IAM_ROLE_NAME}
spec:
  podSecurityContext:
    fsGroup: 2000
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws
spec:
  package: crossplane/provider-aws:v0.27.0
  controllerConfigRef:
    name: aws-config
EOF

4. Create IAM role for IRSA

Again it is a Chicken or the egg problem. Crossplane controller needs to have permissions to provision AWS resources. So we need to manually provision a IAM role for once.

SERVICE_ACCOUNT_NAME=$(kubectl get providers.pkg.crossplane.io provider-aws -o jsonpath="{.status.currentRevision}")
OIDC_PROVIDER=$(aws eks describe-cluster --name $EKS_CLUSTER_NAME --region $AWS_REGION --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

read -r -d '' TRUST_RELATIONSHIP <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:crossplane-system:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF
echo "${TRUST_RELATIONSHIP}" > trust.json

aws iam create-role \
    --role-name "${IAM_ROLE_NAME}" \
    --assume-role-policy-document file://trust.json \
    --description "IAM role for Crossplane provider-aws"

aws iam attach-role-policy --role-name "${IAM_ROLE_NAME}" --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess

rm trust.json

cat <<EOF | kubectl apply -f -
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: aws-provider
spec:
  credentials:
    source: InjectedIdentity
EOF

6. Try it out

Let try to deploy something.

apiVersion: iam.aws.crossplane.io/v1beta1
kind: Role
metadata:
  name: crossplane-sample-role
  annotations:
spec:
  deletionPolicy: Delete
  forProvider:
      description: "A role created by Crossplane"
      assumeRolePolicyDocument: |
        {
          "Version": "2012-10-17",
          "Statement": [
              {
                "Effect": "Allow",
                "Principal": {
                  "Service": "ec2.amazonaws.com"
                },
                "Action": "sts:AssumeRole"
              }
          ]
        }
  providerConfigRef:
    name: aws-provider

---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: Policy
metadata:
  name: crossplane-sample-policy
spec:
  deletionPolicy: Delete
  forProvider:
    name: crossplane-sample-policy
    description: A policy created by Crossplane
    document: |
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
                "eks:DescribeCluster"
            ],
            "Resource": "*"
          }
        ]
      }
  providerConfigRef:
    name: aws-provider
---
apiVersion: iam.aws.crossplane.io/v1beta1
kind: RolePolicyAttachment
metadata:
  name: crossplane-sample-role-policy-attachment
spec:
  deletionPolicy: Delete
  forProvider:
    roleNameRef:
      name: crossplane-sample-role
    policyArnRef: 
      name: crossplane-sample-policy
  providerConfigRef:
    name: aws-provider

6. Cleanup

Always a good practice to do cleanup.

helm uninstall crossplane --namespace crossplane-system
kubectl delete ns crossplane-system
kubectl get crd -o name | grep crossplane.io | xargs kubectl delete

aws iam detach-role-policy --role-name ${AWS_IAM_ROLE_NAME} --policy-arn=arn:aws:iam::aws:policy/AdministratorAccess
aws iam delete-role --role-name "${AWS_IAM_ROLE_NAME}"

Discussion

Imagine you need to deploy an application that needs an ALB, RDS. Now you can package all the things using k8s manifests. No more Terraform is involved and much less management overheads.

Take a step forward, you can use Crossplane to replace Terraform to provision any infrastructure. Crossplane allows you to write Crossplane version Terraform modules, which are called Configurations.

So do I still recommend people to use Terraform? Yes I do. Developers might not really comfortable with k8s while writing Terraform is like writing a simple program to them. And Terraform has a large community supporting, new features will be supported quicker and bug fixes will be sooner.

Conclusion

In short, if you are going to deploy some infrastructure that are only for your k8s deployments, e.g. load balancers, IAM Roles for IRSA. Consider Crossplane.

If you are going to deploy some shared infrastructure or it is not relevant to your k8s deployments, e.g. Network, EC2 bastions. Use Terraform.

Blog