Cloudflare detected a threat actor on their self-hosted Atlassian server on Thanksgiving Day, 2023, and brought in CrowdStrike’s Forensic team to investigate incident.
Cloudflare mentioned that their customer’s data and systems were not impacted by this cyber attack because of pre-implemented access controls, firewall rules, and enforced hard security keys using Cloudflare’s own Zero Trust tools. According to the latest statement from Cloudflare, a threat actor accessed the company’s internal wiki, bug database, and source code management system and tried to access a console server that had access to a data center that Cloudflare had not yet put into production in Sao Paulo, Brazil, from November 14 to 17.
Even though the operational impact of this incident is extremely limited, the company security team took it very seriously because they believed that a nation-state attacker had performed this attack.
“Code Red” Remediation and Hardening Effort
After removing the threat actor from the environment, Cloudflare’s security team pulled all the people they needed across the company to investigate the intrusion. They then redirected the efforts of a large part of the technical staff to work on a single project dubbed “Code Red.”
CrowdStrike performed an independent assessment of the threat actor’s activity and provided helpful corroboration but did not bring to light any activities that had been missed.
The threat actor could access the company’s Atlassian environment using the stolen credentials, but the firm’s security protocols were hardened to prevent them from getting back in. The security team rotated every production credential, physically segmented test and staging systems, performed forensic triages on 4,893 systems, and reimaged and rebooted every machine in the global network.
The threat actor attempted to access a console server in the new data center in Sao Paulo but was unsuccessful. The company states that they replaced the hardware, examined systems, and assumed the worst, making changes to ensure that anything they gained access to would no longer be valuable. Every member of the team was encouraged to point out areas the threat actor might have touched so the team could examine log files and improve security.
October 18 – Okta compromise
The Cloudflare team also states that they were the victim of a compromise of Okta’s systems, which resulted in a threat actor gaining access to a set of credentials. In addition, the team explains that they failed to rotate one service token and three service accounts, which were used to access the Atlassian system. The threat actor gained persistence to Cloudflare’s Atlassian products because they failed to rotate the service token and three accounts.
November 14 09:22:49 – Threat actor starts probing
The threat actor tried to log into the Okta instance and the Cloudflare Dashboard but was denied access. They also tried to access an AWS environment that powers the Cloudflare Apps marketplace but were denied access.
November 15, 16:28:38 – Threat actor gains access to Atlassian services
The threat actor gained access to Atlassian Jira, Confluence, and Smartsheet on November 15 and searched the wiki for things like remote access, secret, client-secret, open connect, Cloudflare, and token. They accessed 36 Jira tickets and 202 wiki pages.
November 22, 14:18:22 – threat actor gains persistence
The threat actor was able to install the Sliver Adversary Emulation Framework using the ScriptRunner for Jira plugin and gain continuous access to the Atlassian server. They attempted to access a non-production console server but were denied.
The threat actor viewed 120 code repositories and downloaded 76 of them to the Atlassian server. These repositories were almost all related to how backups work, how the global network is configured and managed, and how identity works at Cloudflare.
The company’s team open-sourced a large amount of the source code and spoke openly about our algorithms and techniques, so the team’s focus was not on someone accessing the source code.
November 23 – Discovery and threat actor access termination begins
Cloudflare’s security team deactivated the threat actor’s service account 35 minutes after raising the first alert. The threat actor added the Smartsheet service account to an administrator group, which was detected by Cloudflare SOC. The account was deactivated, and an internal incident was declared.
November 24 – Sliver removed; all threat actor access terminated
The threat actor tried to access a myriad of other systems at Cloudflare but failed because of the access controls, firewall rules, and use of hard security keys enforced using the company’s own Zero Trust tools. Cloudflare’s security team tracked the threat actor’s attempts to access the internal metrics, network configuration, build system, alert systems, and release management system and concluded that the last evidence of threat activity was on November 24 at 10:44.
Conclusion
By analyzing Cloudflare’s data breach and available information, it is evident that the company was attacked by a sophisticated actor, possibly a nation-state, who operated in a thoughtful and methodical manner. It appears that Cloudflare’s team worked for over a month to ensure the systems were secure and improve the overall security. Unfortunately, Cloudflare’s security team had to work over the Thanksgiving holiday to conduct an essential review and change of Cloudflare’s security while keeping the global network running.
.
For more cybersecurity news and updates, follow us on Cybersecurity – The SOC Labs.
Disclaimer: This report is based on internal and external research obtained through various means. The information provided is for reference purposes only, and users bear full responsibility for their reliance on it. The SOC Labs assumes no liability for the accuracy or consequences of using this information.
Discover more from The SOC Labs
Subscribe to get the latest posts sent to your email.