Log4J Attacks Confirm Need for DevSecOps, Automation, SBOM

Alarm claxons are blaring about a barrage of cyberattacks exploiting critical vulnerabilities in Log4J -- Apache’s Java-based logging utility. Federal government agencies have only two days left to institute mitigations to comply with an emergency directive issued by the US Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency (CISA). Yet despite the attention, don’t expect the attacks to end anytime soon. And don’t expect your systems to be fully patched in a hurry.

The Log4J situation is exposing once again the complexities of securing applications that use open-source code libraries. It fuels the push for a standardized Software Bill of Materials (SBOM) -- a “list of ingredients” that software developers would provide, to disclose all third-party and open-source components built into it. It also raises questions for enterprise IT departments trying to locate and patch their vulnerable systems: How could automation help, and is it time for DevSecOps?

The Log4J Vulnerabilities

Three Log4J bugs have been revealed in recent weeks. The criticality -- particularly of the "Log4Shell" vulnerability disclosed Dec. 9 -- can hardly be overstated, and has been described as the worst vulnerability in a decade or ever.

Log4Shell impacts hundreds of millions of devices. It’s a “remote code execution” vulnerability that enables attackers to gain full, shell-level control over all kinds of victim machines, from web servers to industrial control systems. When first disclosed, it was already being actively exploited (making it a “zero-day attack”). Four days after the disclosure, security company Check Point reported that 40% of global corporate networks had already been targeted with such attacks or information gathering activity to determine if they were vulnerable. The bug was being exploited widely by all manner of threat actor, including nation-state backed groups. It’s been used to steal data, pilfer passwords, install cryptominers and more.

Complicating matters, Apache’s security update to patch Log4Shell opened up a new vulnerability. This forced Apache to release a second update. Yet, after the second update was released, another vulnerability was discovered, forcing a third update to be released. (So patch now, using version 2.17.0, released Saturday, Dec. 18. And watch this page maintained by the Apache Logging Team for more updates. Also consult CISA for recommended mitigation measures when patching is not an immediate option.)

But organizations everywhere are wondering: what should we patch? Which of our devices/applications are vulnerable?

Third-Party Code Problems

Log4J is a Java-based logging utility wrapped into Apache Logging Services. It’s third-party, open-source software baked into the innards of thousands of applications, and many enterprises (and developers) don’t even know they’re using it. Google researchers estimate Log4J is part of more than 35,000 Java packages. Hundreds of millions of devices are impacted by the vulnerability.

Open-source software is now a fundamental part of enterprise applications, including commercial off-the-shelf software. It may be used extensively for all kinds of purposes -- encryption, network monitoring, file management, running web servers, etc.

Chris Wysopal, CTO of application security firm Veracode, explains the challenge of third-party code, open-source and “nested dependencies,” saying “open source is built on open source is built on open source, and to go to a fourth or fifth or sixth level dependency is not odd at all.”

So when a vulnerability is discovered in such software, the impact ripples and ripples … but those impacted don’t necessarily know that. This fact has been reinforced several times over the past seven years since the critical Heartbleed vulnerability in OpenSSL was revealed.

“Log4Shell has been more of a reinforcing point, showing that code can exist in a myriad of places, whether it is open-sourced or not,” says Pete Allor, product security director at Red Hat. “I saw similar issues with a closed source library embedded in other vendor products back in 2004 - 2006, which highlights that we periodically relearn this lesson. This all goes to show that we need to learn where and what code is in your products or environment and only allow trust as needed.”

In a recent report, Veracode found that 79% of developers never update third-party code libraries. This can snowball into a greater problem, says Wysopal. Because of all the intricate dependencies, one small update here could cause a small break over there. That gets worse the longer you wait -- so to update Log4J to 2.17 you first need to upgrade Java for the first time in 15 years. “That's why we recommend not gathering a lot of security debt around your reliance on third-party packages,” he says, “because the next big remote code execution … could happen and you're stuck with a huge engineering effort just to just to update one library in one application.”

A recent Synopsys report found that 60% of codebases contained known high-risk open-source vulnerabilities. Meanwhile commercial software vendors are failing to do their part. 2019 Synopsys research found that over 40% of commercial software contained known vulnerabilities that were at least 10 years old.

So what solutions are there for this recurring problem?

Time to Drop an SBOM

One idea gaining steam is to require software creators to supply a Software Bill of Materials (SBOM), which is a formal record detailing all the components and supply chain relationships used in building that software.

CISA held a “SBOM-A-RAMA” two-day conference last week. President Biden issued an Executive Order calling for the Commerce Department’s National Telecommunications and Information Administration to release minimum requirements for a Software Bill of Materials. NTIA released those requirements in a July report.

And in the wake of Log4J attacks, analyst firm Forrester wrote Dec. 15 that SBOMs are critical now. They also suggest that data analysis of groups of SBOMs could lead to greater insights. “When taken collectively, a search of all public SBOMs in a unified, readable format gives us an idea of which components are ubiquitous and therefore ‘critical.’ … Would a methodical, metrics-based analysis of the most common software packages to appear in products force us to confront the reality of open source that is ‘too widespread to fail?’”

However, there are others that suggest that SBOMs sound nice in theory, but not in practice.

“SBOMs are a start but they are only a piece of the puzzle,” says Michael Lieberman, of the Cloud Native Computing Foundation Security Technical Advisory Group. “They tell you with some level of confidence what dependencies are included in a piece of software. It's important to recognize they don't tell you where the software the SBOM actually referred to is installed.”

Wysopal adds that while the SBOM can be helpful, he’d rather have assurances from software vendors on how they are maintaining the security of their software – for example a policy that they would update any medium-severity bugs in third-party code within a certain time frame. “Do you want the ingredients label on your can of soup?” he says, “Or do you want to make sure that they have a process where there's no botulism in the soup?”

Red Hat’s Allor explains that one limitation of SBOMs is that they’d document a specific software release and there be “static in its data. Something that would describe an exploitation of vulnerabilities, however, must be dynamic as the situation at hand evolves.”

Automation & DevSecOps

By Wysopal’s reckoning, manual patching processes don’t have a chance against the volume and pace of vulnerabilities. Manually running tests, opening tickets to fix the problem, to validate the problem, and maybe sending those tickets through at dinner time when a human operator could let them wait until morning could slow the process down.

“Only the last few years have we really gotten an understanding that this [third-party code] risk really needs to be managed in a different way,” he says. “And that's how this whole crop of software composition analysis tools have cropped up, and the best practices are to incorporate them into your pipeline,” says Wysopal. “So you have current visibility over what you're using and also so there's the opportunity to update when that new version comes out, and hopefully you can automate it as much as possible.”

“Another key thing that is missing is a better way to distribute vulnerability information,” says Lieberman. “[Common Vulnerability Enumeration Scores] are useful, but outside of software and version the information is often unstructured. It can be hard to develop automated tooling to determine whether or not we're actually vulnerable. Newer specifications like VEX (Vulnerability Exploitability eXchange) will help a lot in the future at providing information about a dependency in the context it runs.”

Shifting security left and better preparing for the inevitable cyber incident is another piece of the puzzle. “A good incident response coordination team with a plan for interacting with DevSecOps groups establishes the priority of work and severity of the issue, giving an organization the ability to respond more effectively,” says Allor. “It provides a ready team with the focus and roles to more quickly address configuration and settings as well as deployment of fixes.”

Leiberman also says that individual organizations cannot solve this problem alone, and that open-source projects, vendors, and organizations like the CNCF and OpenSSF must work in tandem.

“We need to better collaborate as an industry and as a community in order to address these problems,” says Leiberman, “because those who would exploit these vulnerabilities for malicious purposes are collaborating with each other.”

What to Read Next:

KubeCon + CloudNativeCon Highlights Security for Open Source

The Cost of a Ransomware Attack, Part 2: Response & Recovery

How DevSecOps Adoption Can Help You Gain a Competitive Advantage