How did they get away with it? That’s the question many infosec pros are asking themselves after the stunning revelation that a threat actor was able to compromise the SolarWinds Orion updating mechanism to install backdoors into the network management software platform.
The answer, Microsoft said in a blog this week, was by “painstaking planning of every detail to avoid discovery.”
Also:
Joe Biden’s cybersecurity priorities: Fixing damage from SolarWinds attack, working with allies
One missing link in the attack, according to researchers, is the handover from the initial DLL backdoor into SolarWinds called Solorigate by Microsoft (or Sunburst by FireEye) to the subsequent Cobalt Strike exploitation implants in Orion, dubbed Teardrop and Raindrop. The questions are: What code gets triggered, and what indicators should defenders look for?
“Our investigations show that the attackers went out of their way to ensure that these two components are separated as much as possible to evade detection,” the researchers wrote.
First, to give perspective, a timeline:
- In September 2019, the attackers (dubbed UNC2452 by FireEye and Dark Halo by Volexity) started accessing the SolarWinds infrastructure and injecting test code into Orion builds.
- In November of that year, they began injecting test code.
- In February 2020, the Solorigate/Sunburst backdoor, which gathers system information, got deployed into the Orion update.
- In late March, customers started downloading Orion security updates with Solorigate/Sunburst.
- In May, the attackers began exploiting that original backdoor and/or switching to Teardrop with hands-on keyboard attacks of infected victims.
- On June 4, the attackers removed the malware from the SolarWinds environment. Hands-on exploitation of unsuspecting victims continues until Dec. 12 when alerted by FireEye — one of the victims — SolarWinds discovered the plot.
Assuming the Solorigate backdoor was designed to stay dormant for at least two weeks, Microsoft suggests that the attackers spent a month or so in selecting victims and preparing unique Cobalt Strike implants as well as command-and-control (C2) infrastructure. That’s why it believes hands-on work by the attackers started in May.
“The removal of the backdoor-generation function and the compromised code from SolarWinds binaries in June could indicate that, by this time, the attackers had reached a sufficient number of interesting targets,” Microsoft suspects. At that point their objective shifted to being operational on selected victim networks, continuing the attack with a hands-on-keyboard activity using the Cobalt Strike implants.
The Solorigate backdoor only activates for certain victim profiles, researchers found, and when this happens, the executing process creates two files on disk: a VBScript, and a custom Cobalt Strike loader for each victim. At this point, the attackers were ready to activate the Cobalt Strike implant.
“However, the attackers apparently deem the powerful SolarWinds backdoor too valuable to lose in case of discovery, so they tried to separate the Cobalt Strike loader’s execution from the SolarWinds process as much as possible. Their hope is that, even if they lose the Cobalt Strike implant due to detection, the compromised SolarWinds binary and the supply chain attack that preceded it are not exposed.”
The attackers achieved this by having the SolarWinds process create an Image File Execution Options (IFEO) Debugger registry value for the legitimate Orion process dllhost.exe. This is a known MITRE ATT&CK technique used for persistence, but it could also be abused to trigger the execution of malicious code when a certain process launches.
Once the registry value was created, the attackers simply waited for the occasional execution of dllhost.exe. That triggered a process that ultimately launched the Cobalt Strike loader using a clean parent/child process tree completely disconnected from the SolarWinds process. Finally, the VBScript removes the previously created IFEO value to clean up any traces of execution and deleted registry keys related to HTTP proxy.
Among other tricks, the attackers used these sneaky tactics:
- Methodic avoidance of shared indicators for each compromised host. Each Cobalt Strike DLL implant was prepared to be unique per machine and avoided at any cost overlap and reuse of folder name, file name, export function names, C2 domain/IP, HTTP requests, timestamp, file metadata, config, and child process launched. This extreme level of variance was also applied to non-executable entities, such as WMI persistence filter name, WMI filter query, passwords used for 7-zip archives, and names of output log files. “Applying this level of permutations for each individual compromised machine is an incredible effort normally not seen with other adversaries and done to prevent full identification of all compromised assets inside a network or effective sharing of threat intel between victims.”
- Camouflage and blending into the environment. Tools and binaries used by the attackers were always renamed and placed in folders that mimicked existing programs and files already present on a machine. This blending was not just used for files, but for other elements. For example, WMI persistence filters were created with names and queries matching other scripts present in affected organizations.
- Before running intensive and continued hands-on keyboard activity, the attackers took care of disabling event logging using AUDITPOL and re-enabling it afterward.
- Similarly, before running noisy network enumeration activities (such as repeated NSLOOKUP or LDAP queries), the attackers carefully prepared special firewall rules to minimize outgoing packets for certain protocols. The firewall rules were also methodically removed after the network reconnaissance was completed.
- Lateral movement activities were never executed without preparation. To increase the likelihood that their activities remain undetected, the attackers first enumerated remote processes and services running on the target host and decided to move laterally only after disabling certain security services.
- Microsoft believes the attackers used timestamping to change timestamps of artifacts and also leveraged professional wiping procedures and tools to complicate the finding and recovery of DLL implants from affected environments.
In a blog today, Joe Slowik, a senior security researcher at Domain Tools, suggested the Microsoft conclusions of numerous evasive techniques suggest that looking for indicators of compromise to detect attacks like this will fail. Instead, he said, infosec pros should think about incorporating signs of unusual internal network behaviour with evidence of unusual external communications.
“For example, rather than simply responding to any instance of ‘new’ network items observed, organizations may limit this response to critical services, servers, or network enclaves (e.g., the subnet containing numerous infrastructure devices),” he wrote. “Proper network segmentation, asset identification and asset tagging to identify critical items, such as SolarWinds Orion servers or various items such as email servers or Domain Controllers, can allow for focused response when a significant asset initiates a previously unseen external connection.
“The theoretical alerting scenario … where internal and external enrichment are combined to yield high-confidence, high-fidelity alarms, may appear out of reach for many organizations–but given advances in adversary tradecraft, it represents where we as defenders must drive operations. Although initially difficult to create, given both the network engineering and segmentation requirements for an accurate asset or network enclave detection, as well as the establishment of logging and enrichment pipelines for observed network indicators, once in place, an organization will find itself on a much more robust and powerful security footing.”