Digital certificate issuer Let’s Encrypt has announced that it has upgraded its infrastructure to allow it to issue 200 million certificates in one day.
“When we think about what essential infrastructure the internet needs to be prepared for, we’re not thinking about normal days,” said Let’s Encrypt co-founder and executive director Josh Aas in his post about the upgrades. “We want to be prepared to respond as best we can to the most difficult situations that might arise. In some of the worst scenarios, we might want to re-issue all of our certificates in a 24 hour period in order to avoid widespread disruptions. That means being prepared to issue 200 million certificates in a day, something no publicly trusted CA has ever done.”
Who is Let’s Encrypt?
Let’s Encrypt is a non-profit certificate authority (CA) providing free X.509 certificates for TLS encryption. Its goal is to make encrypted connections to World Wide Web servers ubiquitous and create a more secure internet.
It was founded by employees from Mozilla, the Electronic Frontier Foundation, and the University of Michigan, along with Cisco Systems and Akamai Technologies. The company behind it, Internet Security Research Group, was incorporated in 2013, and Let’s Encrypt was announced in November 2014. The first certificates were issued in September 2015.
At the time, digital certificates were expensive and complex to enable and maintain, so most websites used standard (unencrypted) HTTP connections, not the encrypted HTTPS (today most browsers block or at least caution users about HTTP). Let’s Encrypt automated the processes, allowing it to offer the certificates at no charge and to make them accessible to all.
Today it has issued more than one billion certificates and has acquired about 90 sponsors including the Bill & Melinda Gates Foundation.
The trigger for the upgrades
At the end of February 2020, the company found a bug in the CAA code that let the owner of a website define which certificate authorities could issue certificates for their domains. While it was quickly fixed, it meant that Let’s Encrypt had to revoke and reissue about 3 million certificates (2.6 per cent of active certificates), quickly. Of those 3 million, it managed to deal with just under 2 million by the compliance deadline, seven days after the bug’s discovery. The rest were allowed to expire normally at the end of their 90-day life.
In a February 10, 2021 blog post, the company detailed its subsequent analysis of its infrastructure and what it has done to mitigate circumstances such as those it faced last year. What if, it said, the bug had affected all of its certificates?
That was more than 150 million active certificates covering more than 240 million active domains. And what if the flaw was serious enough that the certificates all had to be replaced within 24 hours? After problem evaluation and decision-making, the available time to do so would be less than a day. As things stood, that would have been impossible; bottlenecks in four areas (database performance, internal networking speed, cryptographic signing module performance, and bandwidth) had to be addressed.
What they did
Fast forward to today, and thanks to sponsors and funders, the Let’s Encrypt network is now built on 25 G fibre, a huge leap from the 1 G copper it had been working over. Cisco donated much of the networking gear, including switches and other equipment. Thales donated new hardware security modules (HSMs) to kick up the performance of the cryptographic signing operations ten-fold. The new Dell database servers query response time is 3 times faster. Finally, with Fortinet’s help, they increased bandwidth to improve internal and external connectivity.
The result: the ability to issue 200 million certificates in one day.
