How Organizations Should Respond to the CircleCI Security Incident

Continuous integration (CI)/continuous development (CD) platform CircleCI released a security alert on January 4, recommending its customers rotate all secrets and keys stored with the company. It also warned customers to watch for unauthorized access to their systems from December 21, 2022, to January 4, 2023.

CircleCI Response and Guidance

On January 7, the company completed the process of rotating all GitHub OAuth tokens on behalf of its customers. On January 12, CircleCI shared that it was working with AWS to notify customers whose AWS tokens were potentially impacted.

“The true implications for CircleCI customers are difficult to pinpoint at this time due to the lack of details from CircleCI regarding the scope of the incident,” Dave Ahn, chief architect at cybersecurity company Centripetal, tells InformationWeek.

CircleCI also released guidance for customers looking to rotate their own tokens and a tool for customers to discover their secrets on the platform. How can customers respond to this security incident and prepare for future security incidents in their software supply chains?

Customer Response

CircleCI has taken proactive steps to mitigate risk for its customers, but simply revoking secrets from the platform is not enough, according to Jaime Blasco, co-founder and CTO of cybersecurity company Nudge Security. “It’s still important to assume that every connected application and secret has been compromised. Organizations should verify the steps that these vendors have taken and also take steps to rotate secrets within any other connected application,” he explains.

Customers can leverage commercially available or open-source tools, aside from the one offered by CircleCI, to discover their secrets. “One option is to use Trufflehog, an open-source tool that scans for secrets across multiple platforms, including CircleCI, Github, Gitlab, and AWS S3,” says Blasco.

CircleCI is assuming responsibility and taking steps to protect its customers, Assaf Morag, lead data analyst at cloud native security company Aqua Security, notes. But is important for customers to respond proactively to the security incident as well. “It is ultimately the responsibility of Circle CI's customers to determine which environments are the most critical to protect and understand the full extent of potential exposure. Therefore, it is essential that they rotate their keys as well,” he says.

Ahn also sees the value in customers taking charge of rotating secrets. “Since all OAuth tokens were rotated globally by CircleCI, there is no immediate security risk that warrants customers to re-rotate them again. However, an argument for customers to re-rotate anyway could be to conduct risk assessments and audits to remove unused accounts and unnecessary OAuth tokens in order to reduce their threat exposure,” he says.

Ahn also notes that some customers may find it difficult to discover their secrets. “The flexible and powerful capabilities of CircleCI's platform do not prohibit customers from storing and embedding secrets in their pipelines in a manner that may be difficult to detect with off-the-shelf tools,” he explains. “Customers may wish to consider this incident as an albeit forced opportunity to critically assess the secrets management challenge in their pipelines and evaluate the use of external secrets management tools and techniques like using ephemeral, one-time secrets that are generated on-demand, such as with products like HashiCorp.”

Greg Notch, CISO at managed detection and response and managed security company Expel, also recommends active customer involvement. “CircleCI users with larger or more complex systems may want to go beyond the credential-listing tools provided and create custom tooling against the CircleCI API. In addition to merely listing credentials, users should consider verifying that everything stored in CircleCI (or any CI/CD) is approved to spot problems before a security incident,” he advises.

Per CircleCI’s guidance, customers will also need to “review internal logs for their systems for any unauthorized access.”

“Customers will need to perform their own internal investigations by looking at their security logs to see if their secrets were used prior to the rotation to understand if they were impacted. Depending on the type of secret and your CI/CD architecture, this may be a significant amount of work,” Aakash Shah, co-founder and CTO of cloud native security company oak9, says.

Going Forward

The complexity of the software environment is only going to grow, and organizations are tasked with implementing security strategies that keep pace. Incidents, like the recent one at CircleCI, are a part of a large trend in which enterprises must consider the risks of working with SaaS partners.

“In the medium/long term, if you weren’t already contingency planning for what to do if your key partners and integrations have security incidents, this is a reminder,” Notch says. “Cloud-hosted CI comes with a lot of benefits, but you have to be eyes-open about security. It starts with knowing what you’re giving CI access to and minimizing those risks.”

What to Read Next:

T-Mobile’s $350M Settlement and the Future of Data Breach Consequences

CMS Subcontractor Breach Affects 254k Medicare Beneficiaries

The Metropolitan Opera Cyberattack Highlights Vulnerability of Cultural Institutions