☸️ How to Deploy a Secured OVHCloud Managed Kubernetes Cluster Using Terraform in 2023
Benoit COUETIL 💫
Posted on May 5, 2023
- Initial thoughts
- Prerequisites
- Variables
- Providers and locals
- The enclosing VPC network
- The actual Kubernetes cluster
- Docker registries
- Optional: PostGreSQL managed database
- Optional: Cloud storage compatible with S3
- Wrapping up
- Further reading
Initial thoughts
Terraform is an infrastructure as code tool that lets you build, change, and version cloud and on-prem resources safely and efficiently.
Surprisingly, complete Terraform examples deploying an OVHCloud Kubernetes cluster with private nodes can be hard to find. This blog post is there to fill that whole. Most pieces of code have been found in a Github repo gathering OVHCloud Terraform examples, thanks to Olivier BEAUTIER
We detail here how to deploy, using Terraform, an OVHCloud secured Kubernetes cluster with the following characteristics:
- A VPC network with private subnets using a gateway to internet
- A Kubernetes cluster with nodes in the private subnets
- Docker registries
- A secured managed PostGreSQL redondant database with restricted access
Resulting architecture is very close to this:
Prerequisites
For the below file blocks to work, you need to know the basics of Terraform.
This example uses Terraform workspaces to segregate environments, which is not a conventional practice. But this won't prevent you from massively take advantage of it.
Before anything else, you need:
- an existing S3 bucket to store Terraform state, please follow official documentation to create one
- an OpenStack user in OVHCloud console
- to follow this guide to get OpenStack variables
- OVHCloud Application key, secret and consumer key that are personal and can be generated online
Variables
Here are variables used by most resources described below. All are simple values.
Safe variables
Let's start with safe variables, that you can commit to your repo.
variable "region" {
description = "OVH region from https://www.ovhcloud.com/en/public-cloud/regions-availability/"
default = "GRA9"
}
variable "global_region" {
description = "OVH global location for PostGreSQL and bucket"
default = "GRA"
}
variable "project_id" {
description = "OVH tenant or project ID"
default = {
staging = "0ebdc4435exxx"
prod = "7eb505ec0fyyy"
}
}
variable "openstack_tenant" {
description = "OpenStack provider tenant or project name"
default = {
staging = "4795193xxx"
prod = "3081355yyy"
}
}
variable "openstack_user" {
description = "OpenStack provider user name"
default = {
staging = "user-Xnernxxx"
prod = "user-wn8YNyyy"
}
}
variable "vlan_id" {
description = "VLAN ID for staging and production not to overlap. By default it is max(VLAN IDs on the project) + 1, so they would surely overlap"
default = {
staging = "1000"
prod = "2000"
}
}
Secret variables
Let's add secret variables, not suitable for commit to your repo.
variable "openstack_password" {
description = "OpenStack provider password"
default = {
staging = "sHwxxx"
prod = "pRsyyy"
}
}
# from https://www.ovh.com/auth/api/createToken?GET=/*&POST=/*&PUT=/*&DELETE=/*
variable "ovh_application_key" {
description = "OVH provider application key"
default = {
staging = "1abbxxx"
prod = "b565yyy"
}
}
# idem
variable "ovh_application_secret" {
description = "OVH provider application secret"
default = {
staging = "7d5f76xxx"
prod = "19b419yyy"
}
}
# idem
variable "ovh_consumer_key" {
description = "OVH provider consumer key"
default = {
staging = "cc080axxx"
prod = "99f526yyy"
}
}
Providers and locals
Here are Terraform providers.
A main difference from main cloud providers is the clear mention of OpenStack, for which you will have to create a provider too, in addition to OVHCloud Terraform provider.
terraform {
backend "s3" {
bucket = "my-app-production-admin"
key = "terraform" # (Required) Path to the state file inside the S3 Bucket. When using a non-default workspace, the state path will be /workspace_key_prefix/workspace_name/key
region = "gra"
endpoint = "https://s3.gra.io.cloud.ovh.net/"
skip_credentials_validation = true
skip_region_validation = true
### variables not allowed
# access_key = from 'terraform init' params
# secret_key = from 'terraform init' params
}
required_providers {
ovh = {
source = "ovh/ovh"
version = "~> 0.27.0"
}
openstack = {
source = "terraform-provider-openstack/openstack"
version = "~> 1.49.0"
}
}
required_version = "~> 1.3.6"
}
provider "ovh" {
endpoint = "ovh-eu" # or OVH_ENDPOINT
application_key = var.ovh_application_key[terraform.workspace] # or OVH_APPLICATION_KEY
application_secret = var.ovh_application_secret[terraform.workspace] # or OVH_APPLICATION_SECRET
consumer_key = var.ovh_consumer_key[terraform.workspace] # or OVH_CONSUMER_KEY
}
# inspired by https://breadnet.co.uk/terraform-ovh-openstack/
# and https://registry.terraform.io/providers/terraform-provider-openstack/openstack/latest/docs
provider "openstack" {
auth_url = "https://auth.cloud.ovh.net/v3/" # Authentication URL
domain_name = "default" # Domain name - Always at 'default' for OVHcloud
region = var.region # or OS_REGION_NAME
user_domain_name = "Default" # or OS_USER_DOMAIN_NAME
project_domain_name = "Default" # or OS_PROJECT_DOMAIN_NAME
tenant_id = var.project_id[terraform.workspace] # or OS_TENANT_ID / OS_PROJECT_ID
tenant_name = var.openstack_tenant[terraform.workspace] # or OS_TENANT_NAME / OS_PROJECT_NAME
user_name = var.openstack_user[terraform.workspace] # or OS_USERNAME
password = var.openstack_password[terraform.workspace] # or OS_PASSWORD
}
}
The enclosing VPC network
Here is a sample Terraform block for the VPC network where the Kubernetes cluster will be created.
data "openstack_networking_network_v2" "ext_net" {
name = "Ext-Net"
region = var.region
}
resource "openstack_networking_network_v2" "private_network" {
name = "${terraform.workspace}-private-network"
region = var.region
admin_state_up = "true"
}
resource "openstack_networking_subnet_v2" "subnet" {
network_id = openstack_networking_network_v2.private_network.id
region = var.region
name = "${terraform.workspace}-subnet"
cidr = "192.168.12.0/24"
enable_dhcp = true
no_gateway = false
dns_nameservers = ["1.1.1.1", "1.0.0.1"]
value_specs = {
"provider:network_type" = "vrack"
"provider:segmentation_id" = var.vlan_id[terraform.workspace]
}
allocation_pool {
start = "192.168.12.100"
end = "192.168.12.254"
}
}
resource "openstack_networking_router_v2" "router" {
region = var.region
name = "${terraform.workspace}-router"
admin_state_up = true
external_network_id = data.openstack_networking_network_v2.ext_net.id
}
resource "openstack_networking_router_interface_v2" "router_interface" {
router_id = openstack_networking_router_v2.router.id
region = var.region
subnet_id = openstack_networking_subnet_v2.subnet.id
}
The actual Kubernetes cluster
Now serving the main dish: the Kubernetes cluster.
Node autoscaling is not activated in this example, but nothing prevents you from doing so.
resource "ovh_cloud_project_kube" "cluster" {
service_name = var.project_id[terraform.workspace]
name = "${terraform.workspace}-cluster"
region = var.region
private_network_id = openstack_networking_network_v2.private_network.id
private_network_configuration {
default_vrack_gateway = "192.168.12.1"
private_network_routing_as_default = true
}
customization {
apiserver {
admissionplugins {
enabled = ["NodeRestriction"]
disabled = ["AlwaysPullImages"] # the long-awaited option <3, see https://github.com/ovh/public-cloud-roadmap/issues/70#issuecomment-1235364408
}
}
}
}
resource "ovh_cloud_project_kube_nodepool" "node_pool" {
service_name = var.project_id[terraform.workspace]
name = "${terraform.workspace}-pool"
kube_id = ovh_cloud_project_kube.cluster.id
flavor_name = "b2-15"
## TODO: configure using https://docs.ovh.com/us/en/kubernetes/configuring-cluster-autoscaler/, not available in terraform
# autoscale = true
desired_nodes = 3
max_nodes = 3
min_nodes = 3
timeouts {
create = "1h" # default 20m ; OVH can be real slow on this one, and will consider a duplicate on next run
}
}
resource "local_sensitive_file" "kubeconfig" {
content = ovh_cloud_project_kube.cluster.kubeconfig
filename = "${terraform.workspace}.kubeconfig"
file_permission = "0644"
depends_on = [ovh_cloud_project_kube.cluster, ovh_cloud_project_kube_nodepool.node_pool]
}
Kubernetes specific outputs
kube-config file is given as an output, don't forget to move it, or define KUBECONFIG viable, to be able to use the cluster locally.
Other cluster users will be able to get kube-config in console.
output "kubeconfig_file" {
value = ovh_cloud_project_kube.cluster.kubeconfig
sensitive = true
}
Docker registries
Most of us deploy Kubernetes cluster for custom applications, so here are OVHCloud docker registries blocks.
data "ovh_cloud_project_capabilities_containerregistry_filter" "regcap" {
service_name = var.project_id[terraform.workspace]
plan_name = "SMALL"
region = var.global_region
}
# no need for sub-registries ; the modules will be the image name
resource "ovh_cloud_project_containerregistry" "my-app_registry" {
name = "${terraform.workspace}-registry"
service_name = data.ovh_cloud_project_capabilities_containerregistry_filter.regcap.service_name
plan_id = data.ovh_cloud_project_capabilities_containerregistry_filter.regcap.id
region = data.ovh_cloud_project_capabilities_containerregistry_filter.regcap.region
}
resource "ovh_cloud_project_containerregistry_user" "team-user" {
service_name = ovh_cloud_project_containerregistry.my-app_registry.service_name
registry_id = ovh_cloud_project_containerregistry.my-app_registry.id
email = "noreply-team@docker-registry.ovh"
login = "team-user"
}
resource "ovh_cloud_project_containerregistry_user" "tech-user" {
service_name = ovh_cloud_project_containerregistry.my-app_registry.service_name
registry_id = ovh_cloud_project_containerregistry.my-app_registry.id
email = "noreply-tech@docker-registry.ovh"
login = "tech-user"
}
Registry specific outputs
Some outputs for docker registries, used in next paragraph.
output "docker_registry_url" {
description = "Docker Registry URL"
value = ovh_cloud_project_containerregistry.my-app_registry.url
}
output "docker_registry_credentials_url" {
description = "Generate Docker registry credentials here"
value = "https://www.ovh.com/manager/#/public-cloud/pci/projects/${var.project_id[terraform.workspace]}/private-registry"
}
output "docker_registry_harbor_url" {
description = "Please create the project 'private' here, connecting using credentials generated on docker_registry_credentials_url"
value = "${ovh_cloud_project_containerregistry.my-app_registry.url}/harbor/projects"
}
output "docker_registry_tech_user_password" {
description = "OVH Docker Registry password for user 'tech-user'"
value = ovh_cloud_project_containerregistry_user.tech-user.password
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw docker_registry_tech_user_password && echo ""
sensitive = true
}
output "docker_registry_k8s_secret_creation_command" {
description = "Full command to create the secret"
value = "kubectl -n my-app create secret docker-registry ovh-docker-reg-cred --docker-server=${ovh_cloud_project_containerregistry.my-app_registry.url} --docker-username=${ovh_cloud_project_containerregistry_user.tech-user.login} --docker-password=${ovh_cloud_project_containerregistry_user.tech-user.password} --docker-email=${ovh_cloud_project_containerregistry_user.tech-user.email}"
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw docker_registry_k8s_secret_creation_command && echo ""
sensitive = true
}
output "docker_registry_team_user_password" {
description = "OVH Docker Registry password for user 'team-user'"
value = ovh_cloud_project_containerregistry_user.team-user.password
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw docker_registry_team_user_password && echo ""
sensitive = true
}
output "docker_registry_team_user_login_command" {
description = "Docker login one-liner"
value = "echo ${ovh_cloud_project_containerregistry_user.team-user.password} | docker login --username=registry-user ${ovh_cloud_project_containerregistry.my-app_registry.url} --password-stdin"
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw docker_registry_team_user_login_command && echo ""
sensitive = true
}
Private docker registry creation
NOTE: for now, this is not possible using Terraform, so this action stays manual
Connect to
docker_registry_credentials_url
given in Terraform outputs above and generate identification detailsWith generated credentials, connect to
docker_registry_harbor_url
given in Terraform outputsCreate a 'private' project here as private
You will then be able to push images with a full name like xxxx.gra7.container-registry.ovh.net/private/my-app:my-tag
Kubernetes cluster to registry integration
Integration with the Kubernetes cluster is not automatic. You have to get credentials and add them in the cluster as a secret.
- Use terraform outputs:
terraform output --raw docker_registry_k8s_secret_creation_command && echo ""
Apply the generated command to create the secret
Tell your service account to use it:
kubectl -n MY_NAMESPACE patch serviceaccount default -p '{"imagePullSecrets": [{"name": "ovh-docker-reg-cred"}]}'
Optional: PostGreSQL managed database
Best practice for Kubernetes is to stay stateless, so here is an optional secured managed PostGreSQL, which access is only from the Kubernetes cluster. In this example, databases backend
and auth-server
are created on the same managed instance.
# Inspired by: https://github.com/ovh/public-cloud-examples/tree/main/databases/pgsql
# This example is a bit different and older: https://github.com/ovh/public-cloud-databases-examples/tree/main/databases/postgresql/terraform/hello-world
resource "ovh_cloud_project_database" "pg_database" {
# depends_on = [openstack_networking_network_v2.private_network]
service_name = var.project_id[terraform.workspace]
description = "${terraform.workspace} PostGreSQL Cluster"
engine = "postgresql" # one of [postgresql cassandra mysql kafka kafkaConnect]
version = "14"
plan = "business" # 2 nodes, read replicas planned: https://docs.ovh.com/gb/en/publiccloud/databases/postgresql/capabilities/#plans
flavor = "db1-7" # https://docs.ovh.com/gb/en/publiccloud/databases/postgresql/capabilities/#hardware-resources_1
nodes {
region = var.global_region
network_id = openstack_networking_network_v2.private_network.id
subnet_id = openstack_networking_subnet_v2.subnet.id
}
nodes {
region = var.global_region
network_id = openstack_networking_network_v2.private_network.id
subnet_id = openstack_networking_subnet_v2.subnet.id
}
}
resource "ovh_cloud_project_database_database" "auth-server" {
service_name = ovh_cloud_project_database.pg_database.service_name
engine = ovh_cloud_project_database.pg_database.engine
cluster_id = ovh_cloud_project_database.pg_database.id
name = "auth-server"
}
resource "ovh_cloud_project_database_database" "backend" {
service_name = ovh_cloud_project_database.pg_database.service_name
engine = ovh_cloud_project_database.pg_database.engine
cluster_id = ovh_cloud_project_database.pg_database.id
name = "backend"
}
resource "ovh_cloud_project_database_ip_restriction" "ip_restriction" {
engine = "postgresql"
cluster_id = ovh_cloud_project_database.pg_database.id
service_name = ovh_cloud_project_database.pg_database.service_name
ip = "192.168.12.0/24"
}
resource "ovh_cloud_project_database_postgresql_user" "backend" {
service_name = ovh_cloud_project_database.pg_database.service_name
cluster_id = ovh_cloud_project_database.pg_database.id
name = "backend" # 'postgres' is a reserved user, detailed message taken from API https://eu.api.ovh.com/console/#/cloud/project/%7BserviceName%7D/database/postgresql/%7BclusterId%7D/user~POST
roles = ["replication"]
# Arbitrary string to change to trigger a password update.
# Use 'terraform refresh' after 'terraform apply' to update the output with the new password.
password_reset = "password-reset-on-18-01-2022"
}
resource "ovh_cloud_project_database_postgresql_user" "auth" {
service_name = ovh_cloud_project_database.pg_database.service_name
cluster_id = ovh_cloud_project_database.pg_database.id
name = "auth" # 'postgres' is a reserved user, detailed message taken from API https://eu.api.ovh.com/console/#/cloud/project/%7BserviceName%7D/database/postgresql/%7BclusterId%7D/user~POST
roles = ["replication"]
# Arbitrary string to change to trigger a password update.
# Use 'terraform refresh' after 'terraform apply' to update the output with the new password.
password_reset = "password-reset-on-18-01-2022"
}
PostGreSQL specific outputs
output "postgresql_database_cluster_id" {
description = "PostGreSQL database ID"
value = ovh_cloud_project_database.pg_database.id
# database_id = 7f0f38d0-df5e-48ec-a6ed-9d9255837cfc
# cluster_id = a8a9aec9-96c4-4eff-a1ac-bb02b162b343
}
output "postgresql_database_endpoint" {
description = "PostGreSQL database endpoint in the form of host:port"
value = "${ovh_cloud_project_database.pg_database.endpoints[0].domain}:${ovh_cloud_project_database.pg_database.endpoints[0].port}"
}
output "postgresql_backend_user_password" {
description = "PostGreSQL database user password"
value = ovh_cloud_project_database_postgresql_user.backend.password
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw postgresql_backend_user_password && echo ""
sensitive = true
}
output "postgresql_auth_user_password" {
description = "PostGreSQL database user password"
value = ovh_cloud_project_database_postgresql_user.auth.password
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw postgresql_auth_user_password && echo ""
sensitive = true
}
Optional: Cloud storage compatible with S3
Another optional yet important part of application hosting, is a S3 compatible storage with access from one user known by the Kubernetes cluster.
There is a need for the AWS provider for this, else a Swift cloud storage is created. The fully automated Terraform process is:
- creation of an admin S3 user
- use of this user to configure the AWS provider
- creation of the S3 bucket and associated bucket user
First add this in the provider part:
terraform {
[...]
required_providers {
[...]
aws = { # for bucket with S3 API
source = "hashicorp/aws"
version = "~> 4.0"
}
}
}
provider "aws" {
region = lower(var.global_region)
access_key = ovh_cloud_project_user_s3_credential.s3_admin_cred.access_key_id
secret_key = ovh_cloud_project_user_s3_credential.s3_admin_cred.secret_access_key
#OVH implementation has no STS service
skip_credentials_validation = true
skip_requesting_account_id = true
# the gra region is unknown to AWS hence skipping is needed.
skip_region_validation = true
endpoints {
s3 = "https://s3.${lower(var.global_region)}.io.cloud.ovh.net"
}
}
Then you can create a S3 compatible cloud storage:
# inspired by https://github.com/yomovh/tf-at-ovhcloud pointed by https://github.com/ovh/terraform-provider-ovh/issues/329
resource "ovh_cloud_project_user" "s3_admin" {
service_name = var.project_id[terraform.workspace]
description = "Used to create S3 buckets with Terraform"
role_names = [
"objectstore_operator"
]
}
resource "ovh_cloud_project_user_s3_credential" "s3_admin_cred" {
service_name = ovh_cloud_project_user.s3_admin.service_name
user_id = ovh_cloud_project_user.s3_admin.id
}
resource "aws_s3_bucket" "backend" {
# name should be unique because shared by all users on the system
bucket = "my-app-clients-data-${terraform.workspace}"
}
resource "ovh_cloud_project_user" "backend" {
service_name = var.project_id[terraform.workspace]
# username is the user id, and is not customizable
description = "my-app backend app user"
role_names = [
"objectstore_operator"
]
}
resource "ovh_cloud_project_user_s3_credential" "backend" {
service_name = ovh_cloud_project_user.backend.service_name
user_id = ovh_cloud_project_user.backend.id
}
resource "ovh_cloud_project_user_s3_policy" "policy" {
service_name = ovh_cloud_project_user.backend.service_name
user_id = ovh_cloud_project_user.backend.id
policy = jsonencode({
"Statement": [{
"Sid": "RWContainer",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListBucket", "s3:ListMultipartUploadParts", "s3:ListBucketMultipartUploads", "s3:AbortMultipartUpload", "s3:GetBucketLocation"],
"Resource": ["arn:aws:s3:::${aws_s3_bucket.backend.bucket}", "arn:aws:s3:::${aws_s3_bucket.backend.bucket}/*"]
}]
})
}
S3 specific outputs
Unlike AWS, you will have to store credentials in cluster, next to your application needing the access, or in a secret manager (such as HashiCorp Vault).
output "s3_access_key_id" {
description = "my-app backend app S3 ACCESS_KEY_ID"
value = ovh_cloud_project_user_s3_credential.backend.access_key_id
}
output "s3_secret_access_key" {
description = "my-app backend app S3 SECRET_ACCESS_KEY"
value = ovh_cloud_project_user_s3_credential.backend.secret_access_key
# won't be printed on 'terraform apply'. You have to run:
# terraform -chdir=devops/infra/staging-prod output --raw s3_secret_access_key && echo ""
sensitive = true
}
output "s3_endpoint" {
description = "my-app backend bucket S3 endpoint"
value = "https://s3.${lower(var.global_region)}.io.cloud.ovh.net"
}
Wrapping up
Getting all these pieces of Terraform code together, you should be able to deploy a cluster in one command under 30 minutes.
If you think some code should be improved, please advise in the comments 🤓
You can go further with optimizing your cluster costs by picking generic advice from the article FinOps EKS: 10 tips to reduce the bill up to 90% on AWS managed Kubernetes clusters.
Wonder if it is worthy to use OVHCloud managed cluster alongside your already existing AWS clusters ? You can have a look at Managed Kubernetes: Our dev is on AWS, our prod is on OVHCloud.
Illustrations generated locally by Automatic1111 using Lyriel model
Further reading
☸️ Kubernetes: A Convenient Variable Substitution Mechanism for Kustomize
Benoit COUETIL 💫 for Zenika ・ Aug 4
☸️ Why Managed Kubernetes is a Viable Solution for Even Modest but Actively Developed Applications
Benoit COUETIL 💫 for Zenika ・ Jun 5
☸️ Kubernetes: From Your Docker-Compose File to a Cluster with Kompose
Benoit COUETIL 💫 for Zenika ・ Mar 9
☸️ Kubernetes: A Pragmatic Kubectl Aliases Collection
Benoit COUETIL 💫 for Zenika ・ Jan 6
☸️ Web Application on Kubernetes: A Tutorial to Observability with the Elastic Stack
Benoit COUETIL 💫 for Zenika ・ Nov 27 '23
☸️ Kubernetes NGINX Ingress Controller: 10+ Complementary Configurations for Web Applications
Benoit COUETIL 💫 for Zenika ・ Oct 16 '23
☸️ Kubernetes: Awesome Maintained Links You Will Keep Using Next Year
Benoit COUETIL 💫 for Zenika ・ Sep 4 '23
☸️ Managed Kubernetes: Our Dev is on AWS, Our Prod is on OVHCloud
Benoit COUETIL 💫 for Zenika ・ Jul 1 '23
Posted on May 5, 2023
Join Our Newsletter. No Spam, Only the good stuff.
Sign up to receive the latest update from our blog.