This post was originally posted on the ARMO Security Blog and was authored by Ben Hirschberg CTO & Co-founder of ARMO

Background

If you just woke up from hibernation and you didn’t know about the back door that was implanted in some OpenSSH releases, this section is the TL;DR.

A contributor to the liblzma library (a compression library that is used by the OpenSSH project, among many others) submitted malicious code that included an obfuscated backdoor. Since the maintainers had no reason to suspect foul play, they accepted and merged the contribution. The malicious code made it into the compression library release, and later on to the OpenSSH server, which relies on the library in question. Any attacker with network access to the listening port of the OpenSSH server could run arbitrary processes on the server side and be able to communicate with this new process.

Problems

Now, there are a myriad of things that can be discussed here about supply chain attacks. It has been (justifiably) a hot topic in the Cybersecurity field to talk about supply chain security since the Solarwinds attacks. Talking about the importance of awareness around supply chain security is like talking about the awareness of social engineering threats. On the one hand, it’s a no-brainer that it is a problem. On the other hand, you cannot talk about it enough to keep people aware of it.

However, I want to talk about the XZ attack and backdoor from a different angle. The code that became part of OpenSSH was well planned, well executed, and well obfuscated by the attackers, making it even more sneaky than you would have thought.

Detection of an exploit at the network level

Let’s talk about the actual way that someone can detect an exploit of this backdoor.

There is a long line of runtime detection tools, both in the open-source and in commercial spaces. There are myriad intrusion detection tools (IDS) and web application firewalls (WAF) that are there to detect a malicious actor trying to exploit such a backdoor. In this specific case, the ability for such tools to be able to detect the exploitation of this backdoor at the network level, without specific knowledge about it, is near zero.

Why? Because of the way the backdoor is activated. It was completely embedded into the existing protocol so the payload looks like an extension to the SSH protocol. Any tool trying to detect an intrusion by only looking at the network layer, without knowing the specific mechanics of this backdoor, won’t be able to differentiate it from any other legitimate SSH packet. In other words, if this is a zero-day (meaning, no prior knowledge), there is no way that IDSs or Firewalls would catch it. If the vendor/operator knows what the activation packets look like, they can release the proper rules that will detect it.

Host level detection

Now let’s talk about host-based intrusion detection tools. EDRs, XDRs, and the like. A server process opening arbitrary sub-processes and connecting them to sockets is a red flag for any host-based runtime detection tool.

I would even dare to call this low-hanging fruit from the detection side. Without question, the attacker’s jackpot is to be able to open a process on the victim’s host and communicate with it. Therefore, detection tools have been trained and designed to detect arbitrary processes popping up and communicating over the network.

However, in this specific case, there is one big caveat and it is the very way that the SSH protocol, and as a result the server, were designed to work. SSH’s basic functionality is connecting a remote machine securely and running a terminal (shell). The underlying design of the SSH system is that you can run any executable remotely and running a shell is just a specific use-case of the general capability. Therefore, all server implementations can open arbitrary processes and connect to the network peer to communicate with them.

If you take this into account, you understand that there is something different here from the detection side, since the normal behavior of the server is to be connected over TCP and open arbitrary processes. Therefore, if you have a back door that behaves in the same way, i.e. opening an arbitrary process on the host, and connecting it to the TCP network, it is much harder to detect and differentiate malicious activity from normal behavior.

The most respected detection and monitoring tools, meant to detect an attack, look at the user space processes from the kernel side and see system calls made by the protected processes. In this very case, when an SSH server opens a new session, by design it creates a child process and connects the client TCP socket. What will happen in the case of this backdoor exploitation? The very same as the SSH process! It will fork and your process will invoke a new process image by calling into execv and connect it to through the stdout and stdin file descriptors to the client socket. It is the very same behavior of any normal user.

Due to this reason, I find this backdoor to be even more serious than initially presumed. Even the final line of defense, the runtime detection tools, would struggle to detect it

Falco example

As an example of the above point, let’s see how Falco behaves when it monitors an SSH server.

When Falco is running on a Kubernetes cluster where an SSH server is running in one of the Pods and a login is made, the following alert pops up:

05:15:41.910084560: Notice Redirect stdout/stdin to network connection (gparent=containerd-shim ggparent= gggparent= fd.sip=127.0.0.1 connection=127.0.0.1:47210->127.0.0.1:22 lport=47210 rport=22 fd_type=ipv4 fd_proto=fd.l4proto evt_type=dup2 user=root user_uid=0 user_loginuid=-1 process=sshd proc_exepath=/usr/sbin/sshd parent=sshd command=sshd -D terminal=0 container_id=1d065f4a3388 container_image=docker.io/rastasheep/ubuntu-sshd container_image_tag=18.04 container_name=ssh-server k8s_ns=default k8s_pod_name=ssh-server-856d78867c-jb9hf)

This is an alert that is expected. The Falco user is notified that in the ssh-server container, a stdin/stdout pipe was redirected to a network socket. This is a well-known attack technique, where an attacker tries to open a reverse shell on a remote machine.

However, in the case of an SSH server, this is its basic functionality. Therefore a user of an SSH server will either ignore these alerts from Falco or remove this rule to prevent noise.

In either case, an exploit of the backdoor would have gone unnoticed since it would’ve triggered the same alert(s).

Takeaways

What can be learned from this?

A successful supply chain attack does not end by injecting malicious code into the victim’s environment. An attacker, especially an advanced attacker, is also thinking about how the backdoor will be exploited.

The attackers’ intentions and thought processes are not known to us. I don’t know for sure if the command injection backdoor was implanted in OpenSSH deliberately to avoid runtime detectors. It is common to run SSH servers as root processes (not a good practice though). Maybe this was the reason. But as of the time of writing this blog, the consensus of the security community is that this attack was made by a state actor, and it is very possible that by design this attack was meant to circumvent detection tools.

Bottom line (literally) – it is crucial to invest more in the awareness of supply chain attacks in the open-source community and identify ways to prevent the next attack.