FRAMINGHAM, MASS. — Severe thunderstorms knocked out power to 1.2 million homes in the D.C. area. Wildfires ravaged more than 2 million acres in the Rockies. Two-thirds of the country is in drought conditions, and flooding is expected to get worse as the time between rainstorms lengthen and, in turn, grow more intense.
Intensifying weather patterns threaten businesses as global warming raises the temperatures of the oceans. Disaster recovery plans that include only backing up data regionally may need to be rethought, experts say.
The cloud, which was supposed to guarantee high availability, has been significantly affected by power outages caused by storms. Companies dependent on cloud service provider Amazon Web Services (AWS) found their websites down a week ago when rare severe thunderstorms, known as derechos, struck the Virginia and Washington D.C. area, leaving 1.2 million homes without power for days.
“We thought that by simply deploying [our website] across multiple active zones on Amazon we were going to have the backup we needed,” said Brandon Wade, CEO of WhatsYourPrice.com, an online dating site with 400,000 active members. Wade said his site went down twice, for two hours each time after Amazon lost power. More than 1,000 customers contacted WhatsYourPrice.com to complain, he said.
WhatsYourPrice.com had initially signed on with Amazon’s AWS 18 months ago because using a service provider reduced costs for the start-up. But, after two outages at Amazon in less than a month, WhatsYourPrice.com canceled the cloud service, and is now deploying its servers in two co-location facilities, as well as using a local Las Vegas cloud provider for data backup.
“If your business can take a few hours of outage when a disaster strikes, then [Amazon AWS] is the solution to have,” Wade said. “Otherwise, you need to architect in a smarter way.”
Derechos, which are caused by severe heat waves and can develop hurricane-force winds, can span hundreds of miles. The storm on June 29 spanned some 700 miles and had average wind speeds of 60 mph.
An Amazon spokesperson said the derecho caused AWS to lose primary and backup generator power to a portion of a single “Availability Zone” in its US-East Region on June 29.
“In the thunderstorm on Friday night, several of our data centers had their utility power impacted, but in only one of them did the redundant power not operate correctly (which ended up impacting a single digit percentage of our Amazon EC2 instances in the US-East Region). We began restoring service to most of the impacted customers Friday night, and the remainder were restored on Saturday,” the spokesperson wrote in an email response to Computerworld.
A firefighter works a burnout operation on the north flank of the Fontenelle Fire outside Big Piney, Wyoming. More intensive natural disasters such as wildfires and derechos are requiring companies to pay more attention to disaster recovery planning. (Photo: Jim Urquhart / Reuters)
Netflix, Instagram, and Pinterest websites were also affected by Amazon’s recent outage.
“Netflix could have had Amazon duplicate its instance in every data center, but that would be prohibitively expensive,” said Dan Olds, an analyst at Gabriel Consulting Group. “But, expect to pay more for that kind of resiliency. There’s a price to be paid for that kind of availability and there’s no free lunch in the cloud.”
While cloud data services are still in their infancy, eventually they will become so common that they’ll be considered a utility, just as electricity and telephony is today, said Parveen Jain, CEO of RedSeal Networks, which sells software for monitoring the health of companies’ IT infrastructure.
“I think the service providers have to make sure systems work uninterrupted as much as possible. They have to implement redundancy,” Jain said. “As the government is transitioning to the cloud, and some of the largest enterprises are moving there as well, redundancy is a necessity.”
Power outages like the one Amazon’s AWS experienced are expected to become more frequent as global warming causes storms of all types to become more severe and heat waves to be more frequent and intense.
According to the U.S. Drought Monitor, administered by the National Oceanic and Atmospheric Administration (NOAA), the vast majority of the U.S. is experiencing drought conditions.
So far this year, more than 23,000 high temperature records have been set, compared with 2,500 cold temperature records, according to NOAA. In past years, records for high and low temperatures were typically equal, said Harold Brooks, a research meteorologist at the National Severe Storms Laboratory in Norman, Okla., which is also part of NOAA.
Brooks said global warming has been responsible for only about a half degree of temperature increase on average, but it doesn’t take much of a temperature change to have an effect on weather conditions. And temperatures are expected to continue to rise over the next century.
“We’re very confident that global temperatures have increased. That’s a slam dunk,” Brooks said. “And we can tell there are regional temperatures that have gone up significantly. We know we’re setting a lot more high temperature records than low temperature records.
“When it rains, the rains will be more intense, but there will be longer gaps between rains,” Brooks said. “The overall amount of rain has slightly gone up in much of the U.S. It’s getting more concentrated into heavier rain events.”
There is also meteorological modeling and “empirical evidence” that shows if nothing else changes, the strongest hurricanes will get stronger, according to Brooks.
“There are indications that wind speed and damages are tied pretty closely together. If we’re talking a few meters per second more in wind speed, that could lead to 10% to 20% more damage. We may not notice the wind change all that much, but we may notice more damage,” he said.
The ‘hugability’ factor
“Last year was the worst year we’ve had in the history of disasters,” said Al Berman, executive director of the Disaster Recovery Institute (DRI). “I’ve been doing this since the 1990s. I started out hearing this was going to be a very calm summer, and now I’m hearing it’s not going to be.” The DRI is a nonprofit organization that provides educational services and certification for contingency planning and business continuity professionals.
Berman said one thing he still can’t understand is why companies insist on maintaining an IT presence in a single building – something he calls the “hugability” factor. Even the largest banks, he said, like to have their technology in the same building as their business operations. That kind of proximity, however, means a single regional disaster can take out a company’s entire infrastructure.
“If they were just smart enough to move their servers from one building to another, they’d avoid these regional issues,” he said.
At far greater risk, however, are small- and medium-sized businesses (SMBs). The SMB market, which is made up of 33 million companies, represents the sector where most U.S. workers are employed, yet it’s the sector that’s least prepared to respond to disasters because it has far fewer resources.
Cloud backup providers, such as Carbonite, Box.net and Dropbox, are good alternatives to building out a company’s backup infrastructure, Berman said.
Aging infrastructure also adds risk as more severe weather strikes various regions of the U.S. In New York City, for example, many water pipes date back to the Civil War era, according to Berman.
In addition, the nation’s power infrastructure is mainly above ground, on poles that are vulnerable to strong winds and electrical storms.
“There is some infrastructure that needs to be changed. I think burying power and phone lines is a good first step,” Olds said.
Utility companies should be spending time and money to bury lines after severe storms, in a piecemeal fashion, Olds said. That way, they’ll have both the political and financial support for the projects. Unfortunately, after a power outage, most local governments are only concerned with getting power back on and not addressing future outages, he said.
The nation’s communications networks are in even more critical need of changes, Olds said. For example, as a result of the mid-Atlantic coast derecho, 911 emergency service was out in areas of Virginia for as long as two days, Olds said.
Installing fiber-optic communication lines underground could ensure not only more resilient communications networks, but better coordination during and after severe weather events. And, the use of new mini-sensors on communications and power networks could lead to faster line repairs, he said.
“By installing network sensors, you can see where the damage is and deliver resources to the right areas, faster,” Olds said. “It’s getting cheap to put sensors and intelligence into anything.”
Until the nation’s infrastructure is made more robust, the increasing severity of weather conditions will continue to have regional, not just local, implications. All companies need to think about backup plans outside of their region.
“I advise my clients that you can’t depend on any one thing. You can’t depend on the cloud unless you have something to fall back on,” Olds said. “I also hope this serves as a great learning experience for Amazon and other cloud service providers in general.”