Kubernetes multi-tenancy with projectsveltos

Kubernetes is a complex system that relies on several components being configured correctly to have a working cluster.

Some solutions already exist to automate clusters' creation, upgrade, deletion, like for instance ClusterAPI. Such solutions require the existance of a management Kubernetes cluster from where tens of other Kubernetes clusters are then programmatically created and managed.

Managing the life-cycle of Kubernetes clusters is the first step. After a Kubernetes cluster is created, an handful number of features needs to be installed (CNI for instance). Kubernetes add-ons are used to extend the functionality of Kubernetes clusters.

So after cluster creation, the next problem that needs to be addressed is how to programmatically manage Kubernetes add-ons in each managed cluster from the management cluster. Luckily, even in this space, there are already existing solutions. For instance, Sveltos is an open source project to programmatically deploy Kubernetes add-ons in tens of Kubernetes clusters.

Summarizing, using a Kubernetes cluster to manage other clusters (creation/upgrade/deletion and add-ons deployment) provides several benefits, including:

Centralized management: A cluster management cluster allows administrators to manage multiple clusters from a single place, making it easier to maintain consistency and reduce the risk of configuration issues.
Scalability: A cluster management cluster can help organizations scale their infrastructure by making it easier to create, deploy, and manage multiple clusters.
Better security: A cluster management cluster can be configured with security-related addons, such as network policies and secrets management, to help ensure that all managed clusters are secure.
Increased automation: A cluster management cluster can be integrated with a continuous integration/continuous deployment (CI/CD) pipeline, making it easier to automate the deployment of new clusters and addons, and reducing the time and effort involved in managing the infrastructure.

Problem gets more complex when different organizations end up sharing the managed Kubernetes clusters. This scenario is commonly referred to as multi-tenancy.

Common forms of multi-tenancy are:

share a cluster between multiple teams within an organization, each of whom may operate one or more workloads;
one (or more) cluster(s) fully reserved for an organization.

In this scenario we need to introduce two roles:

platform admin: The role of a platform administrator is to manage and maintain the infrastructure of the managed Kubernetes clusters. This includes tasks such as setting up the cluster, managing nodes, installing and updating software components, and ensuring the overall health and security of the cluster. The platform administrator is also responsible for onboarding tenant admins and granting permissions to those;
tenant admin: Has admin access to the clusters/namespaces assigned to them by the platform admin and manages tenant applications in those clusters/namespaces.

After cluster creation/upgrade/deletion and the addons deployment, the next problem to solve is how to allow platform admin to onboard tenants and programmatically grant them permissions from the management cluster.

Sveltos addons management solution

Before we deep dive into granting permissions to tenants, lets briefly summarize Sveltos solution to manage Kubernetes addons.

Sveltos introduces a single CRD called ClusterProfile for that.

By creating a ClusterProfile instance, we can tell Sveltos:

where to deploy addons (i.e, in all managed Clusters matching the ClusterSelector field);
list of addons to deploy (either Helm charts or Kubernetes resource YAMLs contained in referenced ConfigMaps/Secrets).

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-kyverno
spec:
  clusterSelector: env=fv
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v2.6.0
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install
  policyRefs:
  - name: disallow-latest-tag # referenced ConfigMap contains a Kyverno ClusterPolicy
    namespace: default
    kind: ConfigMap

With Sveltos configuration drift detection we don't have to worry about managed cluster states. Sveltos monitors the state of managed clusters and when it detects a configuration drift for one of the resource deployed because of a ClusterProfile, it re-syncs the cluster state back to the state described in the management cluster.

Sveltos multi-tenancy solution

Let's now go back to the multi-tenancy problem.

We want to:

allow platform admin to programmatically grant permissions to tenant admins;
allow tenant admins to manage applications from management cluster, making sure each tenant admin only deploys resources it has permissions to and only in the clusters/namespaces it has permission to.

RoleRequest is the CRD introduced by Sveltos to allow platform admin to grant permissions to various tenant admins.

apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: full-access
spec:
  clusterSelector: env=pre-production
  admin: eng
  roleRefs:
  - name: full-access
    namespace: default
    kind: ConfigMap

where:

ClusterSelector field is used to select a set of managed clusters where permissions will be granted;
Admin field selects a tenant admin (later on we will cover how this identify a tenant admin);
RoleRefs field references one of more ConfigMaps/Secrets. Each of those ConfigMaps/Secrets contains one or more ClusterRole/Role defining the permissions granted to the tenant admin by the platform admin.

Platform admin is the only admin in the management cluster with permissions to create/update/delete RoleRequest instances.

Tenant admin managing applications

Once RoleRequest instances have been created, tenant admins can use Sveltos ClusterProfile CRD to instruct Sveltos on what Kubernetes resources to deploy on which clusters.

Sveltos requires that following label

projectsveltos.io/admin-name: <admin>

is set on a ClusterProfile instances created by a tenant admin.
How to achieve that, is up to platform admin.

For instance, if:

Kyverno is present in the management cluster;
tenant admin are represented by Kubernetes ServiceAccount;

following Kyverno ClusterPolicy is all that is needed for label to be automatically added to each instance of a ClusterProfile:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-labels
  annotations:
    policies.kyverno.io/title: Add Labels
    policies.kyverno.io/description: >-
      Adds projectsveltos.io/admin-name label on each ClusterProfile
      created by tenant admin. It assumes each tenant admin is
      represented in the management cluster by a ServiceAccount.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: add-labels
    match:
      resources:
        kinds:
        - ClusterProfile
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
             projectsveltos.io/admin-name: "{{serviceAccountName}}"

Examples

I have created management cluster on my laptop. It is a Kind cluster.
In the management cluster, I have deployed both ClusterAPI (with Kind as infrastructure provider) and Sveltos.

kubectl get pods -A
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-59b7bdbcf9-zd4xq                         1/1     Running   0          30m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-68f975d954-9r7lm       1/1     Running   0          30m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-8476fb8676-kpnkd   1/1     Running   0          30m
capi-system                         capi-controller-manager-69fd995599-67fh5                         1/1     Running   0          30m
...
projectsveltos                      access-manager-569dd6cccd-m676t                                  2/2     Running   0          59s
projectsveltos                      classifier-manager-ddc8c756-bmxc9                                2/2     Running   0          59s
projectsveltos                      fm-controller-manager-5964fb8bff-2j2gx                           2/2     Running   0          59s
projectsveltos                      sveltoscluster-manager-75ddcf7cbd-q47hp                          2/2     Running   0          58s

I have one ClusterAPI powered cluster:

kubectl get clusters -A
NAMESPACE   NAME                          PHASE         AGE   VERSION
default     sveltos-management-workload   Provisioned   10m   v1.25.2

and I have one GKE cluster registered with Sveltos

kubectl get sveltoscluster -A
NAMESPACE   NAME         AGE
gke         production   6s

Cluster labels are set as follow:

Cluster	Label
default/sveltos-management-workload	env:development
gke/production	env:production

Fully reserve set of cluster to a tenant

apiVersion: v1
kind: ConfigMap
metadata:
  name: full-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: system-admin-full-access
    rules:
    - apiGroups: ["*"]
      resources: ["*"]
      verbs: ["*"]

---

apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: full-access
spec:
  clusterSelector: env=production
  admin: system-admin
  roleRefs:
  - name: full-access
    namespace: default
    kind: ConfigMap

By referencing ConfigMap default/full-access, above RoleRequest is reserving any cluster matching clusterSelector env=production to admin system-admin.

Using sveltosctl we can verify Sveltos is aware system-admin has full access to managed cluster with label env:production

./bin/sveltosctl show admin-rbac       
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
|            CLUSTER            |    ADMIN     | NAMESPACE | API GROUPS | RESOURCES | RESOURCE NAMES | VERBS |
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+
| SveltosCluster:gke/production | system-admin | *         | *          | *         | *              | *     |
+-------------------------------+--------------+-----------+------------+-----------+----------------+-------+

If system-admin tries to deploy addons to such cluster, that operation will go through.

For instance, if system-admin posts following ClusterProfile

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-kyverno
  labels:
    projectsveltos.io/admin-name: system-admin
spec:
  clusterSelector: env=production
  syncMode: Continuous
  helmCharts:
  - repositoryURL:    https://kyverno.github.io/kyverno/
    repositoryName:   kyverno
    chartName:        kyverno/kyverno
    chartVersion:     v2.6.0
    releaseName:      kyverno-latest
    releaseNamespace: kyverno
    helmChartAction:  Install

we can verify Kyverno is deployed by Sveltos on behalf of system-admin:

./bin/sveltosctl show features 
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
|    CLUSTER     | RESOURCE TYPE | NAMESPACE |      NAME      | VERSION |             TIME              | CLUSTER PROFILES |
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+
| gke/production | helm chart    | kyverno   | kyverno-latest | 2.6.0   | 2023-02-03 14:06:48 -0800 PST | deploy-kyverno   |
+----------------+---------------+-----------+----------------+---------+-------------------------------+------------------+

If system-admin tries to deploy same add-ons on clusters matching clusterSelector env:development that operation will fail.

Share cluster between tenants

Platform admin here is granting following permissions in all clusters matching ClusterSelector env=development

Admin eng full access to namespace eng and some read permission in namespace shared-service-access;
Admin hr full access to namespace human-resource.

# ConfigMap contains a Role which gives
# service read access to namespace shared-services
apiVersion: v1
kind: ConfigMap
metadata:
  name: shared-service-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: service-read-role
      namespace: shared-services
    rules:
    - apiGroups: [""]
      resources: ["services"]
      verbs: ["get,list"]

---

# ConfigMap contains a Role which gives
# full access to namespace eng
apiVersion: v1
kind: ConfigMap
metadata:
  name: eng-full-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: edit-role
      namespace: eng
    rules:
    - apiGroups: ["*"]
      resources: ["*"]
      verbs: ["*"]

---

# RoleRequest gives admin 'eng':
# - full access to namespace eng
# - some read permission in namespace shared-service-access
# in all clusters matching the label
# selector env=development
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: eng-access
spec:
  clusterSelector: env=development
  admin: eng
  roleRefs:
  - name: eng-full-access
    namespace: default
    kind: ConfigMap
  - name: shared-service-access
    namespace: default
    kind: ConfigMap

# ConfigMap contains a Role which gives
# full access to namespace human-resource
apiVersion: v1
kind: ConfigMap
metadata:
  name: hr-full-access
  namespace: default
data:
  role.yaml: |
    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: edit-role
      namespace: human-resource
    rules:
    - apiGroups: ["*"]
      resources: ["*"]
      verbs: ["*"]

---

# RoleRequest gives admin 'hr' access to namespace
# 'human-resource' in all clusters matching the label
# selector env=development
apiVersion: lib.projectsveltos.io/v1alpha1
kind: RoleRequest
metadata:
  name: hr-access
spec:
  clusterSelector: env=development
  admin: hr
  roleRefs:
  - name: hr-full-access
    namespace: default
    kind: ConfigMap

As before, we can use sveltosctl to verify granted permission:

./bin/sveltosctl show admin-rbac --namespace=default
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
|                   CLUSTER                   | ADMIN |    NAMESPACE    | API GROUPS | RESOURCES | RESOURCE NAMES |  VERBS   |
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+
| Cluster:default/sveltos-management-workload | eng   | eng             | *          | *         | *              | *        |
| Cluster:default/sveltos-management-workload | eng   | shared-services |            | services  | services       | get,list |
| Cluster:default/sveltos-management-workload | hr    | human-resource  | *          | *         | *              | *        |
+---------------------------------------------+-------+-----------------+------------+-----------+----------------+----------+

If eng admin tries to deploy nginx in eng namespace, that will succeed.

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
  namespace: default
data:
  nginx.yaml: |
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: nginx-deployment
      namespace: eng
      labels:
        app: nginx
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          containers:
          - name: nginx
            image: nginx:1.14.2
            ports:
            - containerPort: 80

---

apiVersion: config.projectsveltos.io/v1alpha1
kind: ClusterProfile
metadata:
  name: deploy-nginx
  labels:
    projectsveltos.io/admin-name: eng
spec:
  clusterSelector: env=development
  syncMode: Continuous
  policyRefs:
  - name: nginx
    namespace: default
    kind: ConfigMap

nginx is deployed

./bin/sveltosctl show features     
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
|               CLUSTER               |  RESOURCE TYPE  | NAMESPACE |       NAME       | VERSION |             TIME              | CLUSTER PROFILES |
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+
| default/sveltos-management-workload | apps:Deployment | eng       | nginx-deployment | N/A     | 2023-02-03 14:40:19 -0800 PST | deploy-nginx     |
+-------------------------------------+-----------------+-----------+------------------+---------+-------------------------------+------------------+

Contribute

Sveltos is an open source project. Contributions are more welcome 🤗

If you would like to know more about it or would like to see a new feature added to sveltos, please reach to us on slack. Any feedback is very much appreciated ❤️

Documentation can be found here

Blog