Introduction

As a representative of cloud-native management and orchestration systems, Kubernetes (K8S for short) is receiving more and more attention. A report [1] shows that 96% of organizations are using or evaluating K8S, and its market share in production environments is Visible.

The functions of K8S are very powerful, and its system complexity is also high. Generally speaking, the more complex the program, the easier it is to have security problems. Naturally, K8S clusters also face serious security threats, such as unauthorized access to K8S components. , container escape, and lateral attacks. We say that offense and defense are mutually reinforcing and coexisting. As relevant security personnel, we should first grasp the security threats that the business architecture may face as a whole before we can do an excellent job in defense. This article will talk about the possible attack points under the K8S cluster architecture from the perspective of an attacker.

Based on previous penetration testing experience, we have sorted out the possible security issues under the K8S cluster architecture and marked the potential attack points in the K8S cluster infrastructure

This article is divided into two parts: the first part and the second part. This part is the first part. It mainly introduces attacks on K8S components, external services of nodes, business pods, and container escape, which correspond to attack points 1-7 in Figure 1. The rest will be introduced in the next chapter.

K8S cluster attack point

Attack point: attack K8S components

The problem of K8S components mainly refers to the insecure configuration of each component. Attack points 1~4 list four representative component problems, namely API Server unauthorized access, etcd unauthorized access, kubelet unauthorized access, and Kube- insecure proxy configuration.

In addition, there are many components that have similar security problems. For example, dashboard, docker and other components also have hidden dangers of unauthorized access. These are important system components of the K8S cluster. Once attacked, they can be directly attacked. Obtain the permissions of the corresponding cluster, node, or container.

Table 1 collects the default ports with hidden dangers of each component for reference:

component name

|

default port

|
|

api server

|

8080/6443

|
|

dashboard

|

8001

|
|

kubelet

|

10250/10255

|
|

etcd

|

2379

|
|

be a proxy

|

8001

|
|

docker

|

2375

|
|

kube-scheduler

|

10251

|
|

kube-controller-manager

|

10252

|

Attack point: Attack node external service

In addition to normal external business, there may also be some "hidden" open services, which should not be exposed to the external network. This situation may be caused by the administrator's negligence, or some deliberately reserved for the convenience of management. The interface, in short, is also a potential attack point.

For example, I have encountered the problem of weak password login in Mysql external service before: one of the nodes of the target system maps the Mysql service port externally through NodePort, and after trying, it can log in through a weak password. This situation belongs to the lack of security awareness of the administrator.

Speaking of attacks on Mysql, we have summarized three Mysql attack paths for reference in the previous penetration test process, as shown in Figure 2:

• Directly access Mysql through externally exposed interfaces such as NodePort, and log in to the database through a weak password; (corresponding to step 1 in Figure 2)

• Attack the application, obtain the shell of the pod, find the intranet address of the Mysql service through the environment variable in the pod, and then try to log in with a weak password; (corresponding to steps 2-1 and 2-2 in Figure 2)

• Attack the application, obtain the shell of the pod, and successfully escape to the node. Use docker inspect to view or directly enter the Mysql container running on the current node. You can see that its environment variables save the database name, root password, and database login address, etc. Information (provided that the Mysql container and the application container are deployed on the same node, and whether sensitive information such as database passwords will be stored in the environment variables depends on the specific configuration of the Mysql container). (corresponding to steps 3-1, 3-2, 3-3 in Figure 2)

Attack point 6: attacking service pods

In the cloud-native environment, the upper-layer applications are like the entrances of the cluster to the attacker. The goal of attacking the application is to break through the entrance and get to the business environment, that is, the shell of the pod where the business is located.

With the development of web security for so many years, there are many vulnerabilities that can be exploited, not to mention, such as the Log4j2-RCE vulnerability (CVE-2021-44228) exposed at the end of 2021 and the recent Spring-RCE vulnerability (CVE-2022) -22965), its harm is very large, and its exploitation is also very simple (for example, the Log4j2 vulnerability, although the exploitation method is different in the high and low version of the JDK environment, there are already a lot of ready-made EXPs available on the Internet), Once successfully exploited, it can completely take over the entire business pod.

Although the permissions after entering the pod are still limited, it has finally entered the cluster. Next, you can try more attack methods, such as lateral, escape, etc., to gradually expand the results until you can control the entire cluster. Before that, we can also implement some attacks locally on the pod, such as information collection, privilege escalation, and denial of service.

Information collection

When entering a new pod, the first thing to do should be to gather information about the current environment.

The first is to collect environmental information to prepare for subsequent attacks. Here is some more valuable information for reference:

• OS, Kernel, User basic information

• Available Capabilities

• Available Linux Commands

• Mounting situation

• Network situation

• Cloud vendor metadata API information

The second is sensitive service discovery and sensitive information scanning. Sensitive service discovery can be done by scanning the ports of the specified network segment in the intranet. In addition to the ports of the K8S components, there are the following common service ports:

• ssh：22

• http：80/8080

• https：443/8443

• mysql：3306

• cAdvisor：4194

• nodeport-service：30000-32767

Sensitive information includes business-related sensitive files (such as code, database, AK/secret or important configuration files involved in business), environment variables (which may expose some sensitive service information), K8S ServiceAccount information (stored in /run by default) /secrets/kubernetes.io/serviceaccount/ directory), process information (with or without sensitive services), etc.

Privilege Escalation

There are two types of privilege escalation in K8S, one is in-pod privilege escalation, and the other is K8S privilege escalation.

Privilege escalation in Pod

Privilege escalation in a pod is similar to traditional Linux privilege escalation, which is to elevate the shell of an ordinary user in a pod to a shell with root privileges. Generally speaking, even if you get the shell of the pod, you can only have the privileges of ordinary users. At this time, the things you can do are still very limited, so you need to use the privilege escalation vulnerability to gain root privileges. There are many ways to escalate privileges, such as kernel vulnerability escalation, sudo escalation, suid escalation, cronjob escalation, etc.

It is worth mentioning that some kernel vulnerabilities can also be used for container escape, such as the famous DirtyCow (CVE-2016-5195), DirtyPipe (CVE-2022-0847), etc., which will be mentioned in the "Container escape" section below. arrive.

K8S Privilege Escalation

There are many ways and scenarios for K8S privilege escalation, such as RBAC privilege escalation [2], and some Days for K8S privilege escalation, such as CVE-2018-1002105, CVE-2020-8559, etc.

Denial of Service

Denial of Service (DOS) attacks can be viewed from three levels: business, pod, and cluster.

DOS attacks on services and pods can be carried out by using some stress testing tools or methods, mainly from resource exhaustion attacks on CPU, memory, storage, network and so on. There are corresponding tools or methods outside the cluster or within the pod, and readers can search for them by themselves.

DOS attacks at the cluster level can mainly exploit the software vulnerabilities of K8S clusters, such as CVE-2019-11253, CVE-2019-9512, CVE-2019-9514, etc.

Attack point 7: Container escape

In cloud attack and defense, getting the shell of a container/pod is often only the first step to a successful attack, because a container is essentially a process in Linux, but due to the limitations of mechanisms such as Namespace and Cgroup, the process permissions in the container are limited. It is very low. Container escape is to break through these restrictions, so in fact, container escape can also be considered as a kind of privilege escalation. The reasons for container escape can be summarized into the following three categories: container insecure configuration, related component vulnerabilities and kernel vulnerabilities.

Insecure configuration of the container

The container insecure configuration is divided into two cases. The first case is that the container is given dangerous permissions, and the second case is that the container is mounted with a dangerous directory. The details are shown in Table 2:

category

|

insecure configuration

|
|

Dangerous permissions

|

Privileged container

|
|

|

cap_sys_admin

|
|

|

cap_dac_read_search

|
|

|

cap_sys_module

|
|

|

cap_sys_ptrace && --pid=host

|
|

Dangerous Mount

|

mount docker.sock

|
|

|

mount procfs

|
|

|

Mount / , /root , /etc and other file directories

|
|

|

Mount lxcfs in rw mode

|
|

|

pod mounts /var/log

|

Dangerous permissions refer to privileged permissions (privileged containers) and dangerous Capabilities permissions (such as cap_sys_admin, cap_sys_module, cap_sys_dac_search, etc.), which can be set through startup parameters when the container starts. As mentioned above, a container is essentially a restricted process. In addition to restricting namespaces and resources through Namespace and Cgroup, there are also security mechanisms such as Capabilities, Apparmor, and Seccomp that restrict the permissions of processes in the container. With the above dangerous permissions, the security mechanism that restricts the container is broken, which opens the door for attackers.

Mounting a dangerous directory by a container can cause the container filesystem isolation to be broken and thus gain privileges. For example, if /var/run/docker.sock is mounted, then communication with the docker daemon is possible within the container, and an attacker can create a privileged container and escape.

Mentioned here are some of the most common insecure configurations in container escape attack methods. In addition, CIS Docker Benchmark[3] proposes hundreds of security configuration benchmarks for docker containers. For vulnerability protection, security configuration issues are often easier to ignore. For attackers, the insecure configuration of containers is often easier to detect and exploit than the related program vulnerabilities and kernel vulnerabilities mentioned below.

Vulnerabilities of related components

The container cluster environment contains a lot of component programs, which cooperate with each other to form a huge container service ecosystem. These components include but are not limited to runc, containerd, docker, kubelet, etc. Any program will have vulnerabilities, and container-related component programs are no exception. However, compared with insecure container configurations, most of these vulnerabilities are more difficult to exploit. For example, CVE-2019-5736 requires the interaction between the host and the container to trigger. , and the vulnerability is "one-time use" and easy to expose because it breaks runc.

Table 3 summarizes common vulnerabilities for some related components:

components

|

Vulnerability

|
|

run

|

CVE-2019-5736

|
|

|

CVE-2019-16884

|
|

|

CVE-2021-30465

|
|

containerd

|

CVE-2020-15257

|
|

|

CVE-2022-23648

|
|

CRY IT

|

CVE-2022-0811

|
|

docker

|

CVE-2018-15664

|
|

|

CVE-2019-14271

|
|

kubectl

|

CVE-2018-1002100

|
|

|

CVE-2019-1002101

|
|

|

CVE-2019-11246

|
|

|

CVE-2019-11249

|
|

|

CVE-2019-11251

|
|

kubelet

|

CVE-2017-1002101

|
|

|

CVE-2021-25741

|

Kernel vulnerability

The biggest difference between a container and a virtual machine is that the container and the host share the kernel. If the kernel of the host has a vulnerability, all containers on the host will be affected. However, not all kernel vulnerabilities can be used for container escape. Here are some known kernel vulnerabilities that can be used for container escape:

• CVE-2016-5195

• CVE-2017-1000112

• CVE-2017-7308

• CVE-2020-14386

• CVE-2021-22555

• CVE-2022-0185

• CVE-2022-0492

• CVE-2022-0847

Similarly, the EXP of kernel vulnerabilities is also risky to exploit, and may even crash the target system if blindly attempted (especially if it is an internal penetration test or a security check of the system).

Summarize

This article summarizes and shares the methods and experience of attacking K8S components, external services of nodes, business pods, and container escaping in a K8S cluster, namely attack points 1-7 in Figure 1. In the next part, we will continue to talk about attack points 8~12 in Figure 1, including lateral attacks, as well as attacks on the K8S management platform, image library, and third-party components.

Reference link

• https://www.cncf.io/wp-content/uploads/2022/02/CNCF-Annual-Survey-2021.pdf

• https://published-prd.lanyonevents.com/published/rsaus20/sessionsFiles/18100/2020_USA20_DSO-W01_01_Compromising%20Kubernetes%20Cluster%20by%20Exploiting%20RBAC%20Permissions.pdf

• https://github.com/dev-sec/cis-docker-benchmark

• https://github.com/cdk-team/CDK