IT administrators with the vulnerable on-premise version of Kaseya’s VSA remote networking monitoring and IT management application may have to wait until Wednesday before their systems can be patched and brought back online.
UPDATE: At noon Tuesday the company said it will start restoring the VSA SaaS service at 4 p.m. Eastern, a process that could take until 7 p.m. It hopes the patch for on-premise installations will be within 24 hours after that. “We are focused on shrinking this time frame to the minimal possible,” the company said in a statement, “but if there are any issues found during the spin-up of SaaS, we want to fix them before bringing our on-premises customers up.”
UPDATE: At 7:30 p.m. Eastern the company said work is continuing on the SaaS service. This includes changing the underlying IP address of Kaseya VSA servers (the domain names/URL will not change) For almost all customers, this change will be transparent. However if — and only if — you have whitelisted your Kaseya VSA server in your firewall(s), you will need to update the IP whitelist. The new IP addresses can be found at: https://www.cloudflare.com/ips/
UPDATE: At 10 p.m. Eastern Kaseya said during its work an issue was discovered that blocked the return of the VSA SaaS service until it is resolved. The company isn’t expected to say anything more until Wednesday morning.
Until the patch is released, the company warned, on-prem versions of VSA should remain off-line.
When restored, the SaaS service will include new security features including a 24/7 independent SOC for every VSA with the ability to quarantine and isolate files and entire VSA servers; a complementary content delivery network (CDN) with web application firewall for every VSA. The goal is to “greatly reduces the attack surface of Kaseya VSA overall.”
In case you missed it:
In addition, the company said that as of last night fewer than 60 Kaseya customers — all of whom were using the VSA on-premises suite — were directly compromised following the initial ransomware attack on Kaseya by the REvil group. Many of them are managed service providers. After compromising these providers the attackers spread the ransomware to their subscribers. Kaseya has determined “fewer than 1,500” end-user customers were victimized. There is no evidence that any of SaaS customers were compromised.
Kaspersky said Monday it had seen 5,000 attack attempts in 22 countries.
REvil has claimed that more than a million individual devices were infected. It is selling what it calls a universal decryptor for all victims of the attack for $70 million in bitcoin.
There have been no new reports filed of compromises of VSA customers since Saturday, July 3, Kaseya added.
Staged functionality
When the SaaS version comes back online it will have staged functionality to bring services back up sooner. The first release will prevent access to some functions — classic ticketing, classic remote control (not LiveConnect) and the user portal — but the company said these are used by a very small fraction of customers.
Kaseya also said it has discussed with the FBI and the U.S. Cybersecurity Infrastructure and Security Agency (CISA) how systems and networks can be hardened prior to service restoration for both SaaS and on-premises customers. A set of requirements will be posted prior to service restart to give customers time to put these countermeasures in place in anticipation of a return to service on July 6.
Finally, the company said a new version of its compromise detection tool has been released.
Kaseya detected a cyberattack early in the afternoon of Friday, July 2. Researchers at Huntress who looked at compromised servers described the attack said they “have high confidence that the threat actor used an authentication bypass in the web interface of Kaseya VSA to gain an authenticated session, upload the original payload, and then execute commands via SQL injection. We can confirm that SQL injection is how the actors began code execution.” Sophos said that after compromising the company on-premise customers were victimized by a malicious software update, which spread to the VSA agent applications running on managed Windows devices.
“It appears this was achieved using a zero-day exploit of the server platform,” said Sophos (a conclusion that Kaseya has confirmed). “This gave REvil cover in several ways: it allowed initial compromise through a trusted channel, and leveraged trust in the VSA agent code—reflected in anti-malware software exclusions that Kaseya requires for set-up for its application and agent “working” folders. Anything executed by the Kaseya Agent Monitor is therefore ignored because of those exclusions—which allowed REvil to deploy its dropper without scrutiny.”
Canadian Cyber Centre advice
The federal government’s Canadian Centre for Cyber Security urged managed service providers using Kaseya VSA and enterprise users of the on-prem version to download and run the company’s compromise detection tool to see if there are any indicators of compromise.
In addition, any organization using a remote monitoring and management application should implement allow-listing to limit the application’s communication to known IP address pairs only; and administrative interfaces of these applications should be put behind a virtual private network (VPN) or a firewall on a dedicated administrative network.
Finally, all organizations are urged to require multi-factor authentication (MFA) on all employee and partner accounts they control, and where possible, for customer-facing services.
Lost the race to patch
Many infosecurity experts believe it was no coincidence the ransomware attack was launched as the long Independence Day holiday started in the U.S. With some luck it could have been stopped. The Dutch Insitute for Vulnerability Disclosure (DIVD) said it had discovered and notified Kaseya of vulnerabilities (now called CVE-2021-30116), which the company was working to resolve. Apparently, it wasn’t fast enough, because, DIVD said, these vulnerabilities were exploited.
DIVD doesn’t fault Kaseya.
“Kaseya has been very co-operative,” it said. “Once Kaseya was aware of our reported vulnerabilities, we have been in constant contact and co-operation with them. When items in our report were unclear, they asked the right questions. Also, partial patches were shared with us to validate their effectiveness. During the entire process, Kaseya has shown that they were willing to put in the maximum effort and initiative into this case both to get this issue fixed and their customers patched. They showed a genuine commitment to do the right thing. Unfortunately, we were beaten by REvil in the final sprint, as they could exploit the vulnerabilities before customers could even patch.”