December 2021 saw a massive wave of cyberattacks targeting individuals, large organizations, and governments. Many developers worked late nights to patch the urgent threat. For many, the work isn’t done. Attackers were seizing opportunities to exploit vulnerabilities found in Log4j, a widely used logging tool. Log4Shell refers to the first of three Log4j vulnerabilities that were discovered this month. While two other vulnerabilities followed, Log4Shell was the most severe.

Log4Shell has been called the worst software vulnerability of recent years. This won’t be the last we hear of bugs causing vulnerabilities, but we’d be remiss if we didn’t learn anything from it. Log4Shell tells us a lot about the evolving worlds of cybersecurity and distributed systems.

Today, we’ll discuss what Log4Shell is, and four key takeaways for programmers as we move into the new year.

We’ll cover:

• Log4j and the Log4Shell vulnerability
  • What is Log4j?
  • What is Log4Shell?
• Log4Shell: 4 key takeaways for developers
  • 1. An increasingly connected world means an increasing attack surface
  • 2. Scalable systems can be designed to help streamline mitigation
  • 3. The demand for cross-disciplinary engineers will only increase
  • 4. Feedback and collaboration is key
• Wrapping up and next steps

Log4j and the Log4Shell vulnerability

What is Log4j?

Apache Log4j is a Java logging framework. It’s maintained by the Apache Software Foundation, an open-source cooperative maintaining several technologies, including Hadoop. Logging frameworks like Log4j help standardize logging processes. Logging is the documentation of events, errors, and messages in running applications and operating systems. Logging tools like Log4j can be deployed on various services and components in a system, and help to monitor systems and diagnose failures.

The Log4j logging library is used by a multitude of apps and services. Many large organizations have Log4j running in their systems, including Cloudflare, Minecraft, AWS, Microsoft, Cisco, and VMWare. The Log4j library’s ubiquitous nature is the reason Log4j vulnerabilities have made such a devastating impact.

What is Log4Shell?

Log4Shell: The first of three Log4j vulnerabilities

Log4Shell refers to the first Log4j vulnerability that was discovered in December 2021. It was followed by two more Log4j vulnerabilities, but Log4Shell was the most severe. It’s considered a critical vulnerability, ranking a 10/10 on the Common Vulnerability Scoring System (CVSS).

The timeline for the three Log4j vulnerabilities found in December 2021 is as follows:

• December 9, 2021: Log4Shell, first Log4j vulnerability announced (CVE-2021-44228)
  • Enables remote code execution (RCE)
  • Attacks exploit message lookup substitution feature
• December 14, 2021: Second Log4j vulnerability discovered (CVE 2021-45046)
  • Enables denial-of-service (DoS) attacks
  • Attacks exploit insecure Java Naming and Directory Interface (JNDI) lookups, as a result of a flaw in the first Log4Shell patch
• December 17, 2021: Third Log4j vulnerability discovered (CVE-2021-45105)
  • Enables DoS attacks
  • Result of infinite recursion flaw
  • Exploits Context Map lookups

Zero-day vulnerability announced

The Log4Shell exploit was publicly shared with a POC (proof-of-concept) on December 9, 2021 on Twitter and GitHub. As a zero-day vulnerability, attacks began on the day of the announcement. This affected all Log4j versions ranging from 2.0-beta9 to 2.15.0. Exploitation attempts started quickly and in high volumes because the Log4Shell exploit was relatively easy to perform. The patch, on the other hand, was not.

Mitigation

Apache had released a patch by the time of the announcement, but the ubiquitous nature of Log4j makes remediation a tedious process. IT security teams have to identify every component in their system that runs Log4j, and deploy the patch to each individual service. This is a time-consuming process, and left a lot of time for threat actors to exploit vulnerable internet-facing systems. Systems that weren’t patched completely were hit even harder by the two vulnerabilities that followed.

Remote code execution

Log4Shell is a remote code execution (RCE) vulnerability. RCE allows threat actors to execute arbitrary code on any affected system by logging a malicious string. This allowed opportunist threat actors to seize control of entire servers remotely. Attacks took place in various forms, from stolen data to malicious payloads such as malware and ransomware.
The Log4Shell exploit takes advantage of Log4j’s “message substitution” feature. Attackers exploited this feature to insert malicious URLs through JNDI lookups through remote services such as LDAP, RMI, and DNS.

Log4Shell: 4 takeaways for developers in 2022

1. An increasingly connected world means an increasing attack surface

Cybersecurity threats will not go away. Rather, events like Log4Shell speak to the evolving nature of cyberattacks in an increasingly connected world. More interconnected components are continuously online than ever before, giving threat actors an increasingly large attack surface to target. As is the case with Log4j, stakes are higher when there’s one vulnerability in an interconnected system. The right bug in one component can open the door for opportunistic hackers to get more bang for their buck on their exploitation attempts.

2. Scalable systems can be designed to help streamline mitigation

Bugs are inevitable. When they’re found, the difficult part is identifying where to patch, and how to do it at scale. This process only gets more difficult the more interconnected distributed systems become.

It’s crucial that teams are able to holistically assess for scalability in their distributed systems. Both dependencies and deployment processes should be built to scale. One of the best things teams can do to prepare for the inevitable bug is to implement DevOps practices such as CI/CD pipelines to their software development cycle.

3. The demand for cross-disciplinary engineers will only increase

Distributed systems are the present and future of software engineering. To keep up with evolving threats on complex systems, software engineers will be expected to have more cross-disciplinary skills than before. There will be an increased demand for developers specializing in DevOps to support IT security and support leads. DevOps engineers will need to be able to write connected components that serve multiple platforms effectively. To keep up with evolving threats, organizations will have an increasing need for IT security professionals such as security researchers and threat intelligence analysts.

4. Feedback and collaboration is key

Log4Shell could have been worse than it was. By the time the public knew about it, Apache had already released a patch. This was possible because the Apache team had been alerted about Log4Shell 15 days before the public announced it. A team of engineers at Alibaba Cloud discovered and reported the bug to Apache on November 24, 2021. It’s thanks to this advance notice that Apache developers could act quickly to help streamline the mitigation efforts that followed.

Feedback and collaboration can be found at the root of many successful development stories. Even after a product goes live, it’s incredibly important that developers be open to feedback from their teams and other communities. This helps us act quickly to resolve unforeseen problems.

Wrapping up and next steps

We celebrate the hard work that developers and cybersecurity professionals have invested to address this issue in the last month. The aftermath of Log4Shell will be with us for years to come. As developers, we can plan our growth around the evolving worlds of distributed systems and cybersecurity.

In today’s changing world, developers need to have a growth mindset to keep up. Always be ready to step up and expand your skillset, including your knowledge of libraries and APIs. Keep up with tech trends and changes in software disciplines. In addition to taking on new skills and projects, it's equally important to tend to applications you've already written. As a developer, you have the chance to grow by reviewing published bugs and even doing some debugging on your own. Many major organizations have bug bounty programs that offer a bounty for those who can alert them to bugs.

Start a discussion

What have you learned from Log4Shell? Has it inspired you to change your priorities in your programming career? Let us know your thoughts in the comments below!