The massive 2017 breach of credit reporting company Equifax was “entirely preventable,” according to a staff report of a U.S. Congressional committee released this afternoon.
“A culture of cyber security complacency at Equifax led to the successful exfiltration of the personal information of approximately 148 million individuals,” said the report written by the staff to the Republican majority of the House of Representatives Committee on Oversight and Government Reform. “Equifax’s failure to patch a known critical vulnerability left its systems at risk for 145 days.”
“The company’s failure to implement basic security protocols, including file integrity monitoring and network segmentation, allowed the attackers to access and remove large amounts of data. The attackers were able to exfiltrate this data because the digital certificate allowing Equifax to monitor encrypted network traffic flowing through [a particular application] environment expired 19 months prior to the discovery of the breach.”
“Equifax should have addressed at least two points of failure to mitigate, or even prevent, this data breach. First, a lack of accountability and no clear lines of authority in Equifax’s IT management structure existed, leading to an execution gap between IT policy development and operation. This also restricted the company’s implementation of other security initiatives in a comprehensive and timely manner. As an example, Equifax had allowed over 300 security certificates to expire, including 79 certificates for monitoring business critical domains.”
One of those certificates expired over a year before the breach. That, combined with a misconfiguration of an intrustion dection device, that was supposed to monitor network traffic allowed the attacker(s) was able to run commands and remove stolen data over an encrypted connection without detection.
“Second, Equifax’s aggressive growth strategy and accumulation of data resulted in a complex IT environment. Equifax ran a number of its most critical IT applications on custombuilt legacy systems. Both the complexity and antiquated nature of Equifax’s IT systems made IT security especially challenging. Equifax recognized the inherent security risks of operating legacy IT systems because Equifax had begun a legacy infrastructure modernization effort. This effort, however, came too late to prevent the breach.”
The breach of an Equifax online customer dispute portal between May and July 2017 resulted in the copying of records from 48 databases containing pesonal information of at least 145.5 million consumers in the U.S. and nearly 1 million consumers outside of the U.S including about 19,000 Canadians.
“Equifax failed to fully appreciate and mitigate its cybersecurity risks. Had the company taken action to address its observable security issues prior to this cyberattack, the data breach could have been prevented,” say the reports authors.
Recommendations
The report also includes a number of recommendations some of which are particular to the U.S., although they may apply to other countries if they have similar procedures. For example, the report says government should work with the private sector to reduce reliance on U.S. Social Security, which are widely used by the public and private sector to both identify and authenticate individuals. In Canada there are organizations that use Social Insurance numbers for identity. Another recommendation is Washington hold federal contractors — like Equifax — accountable for cyber security with clear requirements. The report also recommends consumer reporting agencies should be forced to provide more transparency to consumers on what data is collected and how it is used.
Of more interest to CISOs is a recommendation that companies storing sensitive consumer data should chop legacy IT systems. “Equifax failed to modernize its IT environments in a timely manner,” after a number of aquisitions, the report notes. “The complexity of the legacy IT environment hosting the ACIS (automated consumer interview system, which was on the consumer portal that was breached) allowed the attackers to move throughout the Equifax network and obtain access to unrelated consumer personally identifiable information. Equifax’s legacy IT was difficult to scan, patch, and modify.”
The 96-page report — like the earlier GAO report — reads like a CISO’s nightmare. Or, like a CISO’s guide on what not to do. Or like a guide to what a CISO of a sophistcated, large enterprise should do to avoid such a catastrophe.
As many readers know by now, the breach was sparked shortly after March 7, 2017 when Apache disclosed a critical vulnerability in its Struts web framework. Equifax used Apache Struts to run certain applications on legacy operating systems. The following day, the Department of Homeland Security alerted Equifax to this critical vulnerability. Equifax’s Global Threat and Vulnerability Management (GTVM) team emailed this alert to over 400 people on March 9, instructing anyone who had Apache Struts running on their system to apply the necessary patch within 48 hours. Despite this a server was missed and — like many breachs — a series of vulnerabilities were take advantage of.
One reason is the IT and Security teams were split, and each had its own lists of corporate applications. In fact, the company was suprised to learn — after suspicions arose — that it had Struts at all on the ACIS system.
But the report says the seeds of the mess were sewn years earlier when CEO Richard Smith embarked on an aggressing corporate buying spree, buying 18 companies around the world over roughly a decade. Equifax has data on almost 1 billion people and almost 100 million companies, Smith once said.
“Having so much personal information in one place made Equifax a prime target for hackers,” the report observed. In fact, competitor Experian had already been nailed twice.
“Equifax was unprepared for these risks,” the report notes dryly. How unprepared? An August 2016 report by the financial index
provider MSCI Inc. assigned Equifax’s data security efforts a rating of zero out of 10. Read that sentence again.
Equifax’s April 2017 rating remained unchanged.
Both MSCI reports concluded: “Equifax’s data security and privacy measures have proved insufficient in mitigating data breach events. The company’s credit reporting business faces a high risk of data theft and associated reputational consequences . . . The company’s data and privacy policies are limited in scope and Equifax shows no evidence of data breach plans or regular audits of its information security policies and systems.”
Back to the narrative of the attack
On March 9, 2017, the day after the Homeland Security warning, Equifax Security performed an open source component scan to identify any systems with a vulnerable version of Apache Struts. The scan did not identify any components utilizing an affected version of Apache Struts. The congressional committeee was told the scan missed identifying the vulnerability because the scan was run on the root directory, not the subdirectory on the server where the Apache Struts was listed.
Later, forensics disovered that attackers had infiltrated Equifax on March 13.
On March 14 Equifax’s Emerging Threats team released a Snort signature rule to to detect Apache Struts exploitation
attempts. The company’s Countermeasures team installed that rule written on the intrusion detection and prevention systems the same day.
On March 15, running an updated McAfee Vulnerability Manager tool, Equifax twice scanned 958 externally facing systems. Nothing.
(More on the patching process below)
Little did the teams know that two days earlier the attackers had found a Sun server running the Solaris operating system with a vulnerable version of Struts within a data center in Alpharetta, Georgia.
The sever was running what Equifax calls its automated customer inteview system (ACIS), comprised of two web servers and two application servers, with firewalls set up at the perimeter of the web servers. The system allowed customers to challenge information on their credit reports, so it was full of personal information. After entering the ACIS environment through the Apache Struts vulnerability, the attackers uploaded the first web shells, which are malicious scripts uploaded to a compromised
server to enable remote control of the machine and bypass these firewalls.
Once inside the network the attackers created about 30 web shells on both application servers to control them. This provided the
attackers with the ability to execute commands directly on the system hosted on the application servers. According to forensics done after the breach was discovered by Mandiant, file integrity monitoring could have discovered the creation of these web shells by detecting and alerting to potentially unauthorized network changes. Equifax did not have file integrity monitoring enabled on the ACIS system at the time of the attack, says the report.
After installing the first web shells, the attackers accessed a mounted file share containing unencrypted application usernames and passwords stored in a configuration file database. They were able to access the file share because Equifax didn’t limit access to sensitive files across its internal legacy systems, a company policy.
Although the ACIS application required access to only three databases to perform its business function, it wasn’t segmented from other, unrelated databases. As a result, the attackers used the stolen application credentials to gain access to 48 more unencrypted databases with personal information.
After running querries on these databases the results were compressed by the attackers, put into a web-accesable directory. And then they were whisked away.
The attack lasted for 76 days before it was discovered by Equifax employees.
The attackers had a bit of luck.
An expired Secure Sockets Layer (SSL) certificate prevented Equifax from monitoring traffic to the ACIS environment, says the report. SSL enables encrypted communication between a web browser and a web server. To create this secure connection, an active SSL certificate must be installed at the point where decryption will occur. SSL certificates have a lifespan of either 27 or 39 months, depending on the date the SSL certificate was issued, after which it has to be renewed or replaced.
This particular expired SSL certificate was installed on a traffic monitoring device called an SSL Visibility (SSLV) appliance, which allowed the inspection of encrypted traffic flowing to and from the ACIS platform by decrypting the traffic for analysis prior to sending it through to the ACIS servers. The default setting for this device allowed web traffic to continue through to the ACIS system, even when the SSL certificate was expired. Traffic flowing to and from the internet is not analyzed by the intrusion detection or prevention systems because these security tools cannot analyze encrypted traffic.
The crucial certificate expired. January 31, 2016. Nineteen months later, on July 29, 2017, the Equifax Countermeasures team uploaded 67 new SSL certificates to the SSLV appliance, allowing the company to resume the inspection of traffic flowing to and from the ACIS application. It began looking at packet data. “Almost immediately, the Equifax Countermeasures team detected a suspicious request from an IP address originating in China,” says the report.
Within a day it knew personal information had been stolen.
Among the fallout: The retirement of CIO David Webb was “accelerated,” he told the Congressional committee. The retirement of CSO Susan Mauldin, who had said she wanted to leave before the data breach, was announced the same day as Webb’s. CEO Richard Smith retired a week later.
On October 2, 2017, Equifax terminated Graeme Payne, the senior vice-president and CIO for global corporate platforms who was responsbile for managing the ACIS environment. Payne was one of 430 employees who got the March 9 email alert on the Apache
Struts vulnerability. Payne testified he didn’t forward the email to anyone else because he didn’t have a responsibility under the Equifax’s patch management policy.
Terminating Payne for failing to forward an email was a “public relations-motivated maneuver” that “seems gratuitous against the back drop of all the facts,” said the congressional report.
The organization chart
Before 2005, the company’s chief security officer, Tony Spinelli, reported to the then CIO. Equifax executives knew growing
security risks and compliance requirements necessitated an overhaul of the company’s securitystance, the report says, so Spinelli created a three-year, US$15 million plan to reorganize IT security across the enterprise. However, the report says, he and the CIO had “fundamental disagreements,” so Spinelli wound up reporting to the chief legal officer. After the breach was discovered the company decided to move security back to the IT department. “The functional result of the CIO/CSO structure meant IT operational and security responsibilities were split, creating an accountability gap,” says the report. “At the time of the breach, Equifax’s organizational structure did not facilitate a strong CIO and CSO partnership.”
“Communication and co-ordination between these groups was often inconsistent and ineffective,” the report notes. For example, multiple and incomplete software inventory lists were kept separately by security and IT.
Another problem: The CSO didn’t regularly attend senior management meetings. Most of the security information at those meetings came from the chief legal officer, who the CSO reported to.
Things have changed since the breach: Now there’s a CISO who reports to the CEO.
‘Honour system’ for patching
As for Equifax’s patch management process. under company policy at the time, there was supposed to be a business owner, a system owner and application owner responsible for the ACIS system, as for every application. But, committee staff were told officially no one had been designated for those positions and who might have told IT to look after the patch. As a result committee staff couldn’t identify who in IT might have been responsible for the Struts patch.
However, Equifax leaders knew there were problems as far back as two years before the breach, when the company conducted an audit of its patch management process. “This audit found a number of significant deficiencies,” said the report, and i made eight detailed findings and recomendations. Here’s one: “Vulnerabilities were not adequately tracked, prioritized, and monitored to ensure timely remediation. An ‘honour system’ was used to ensure patches are installed.”
That 2015 audit also noted there was no segmentation between the Sun application servers and the rest of the Equifax network. An
attacker that gains control of the application server from the internet can pivot to any other device, database, or server within the Equifax network, globally, the audit noted. Neither CSO Maudli nor CIO for global platforms Payne knew about the lack of segmentation.
As for the expired SSL certificates, the report says the company didn’t have a process for updating the certificates, and knew it. “An internal vulnerability assessment tracker entry dated January 20, 2017 stated “SSLV devices are missing certificates, limiting visibility to web based attacks on [intrusion prevention system].” By the time of the breach Equifax had allowed at least 324 of its SSL certificates to expire. Seventy-nine of them were for devices monitoring highly business critical domains, including the ACIS.
Since the breach, the report says, Equifax has worked on improving security of a number of its systems, including implementing a new management process to identify and patch software vulnerabilities and confirm that vulnerabilities had been addressed, It also says it added new tools to ensure network traffic is monitored continuously.