FRAMINGHAM, Mass.– Here are two facts about Brook Colangelo ‘s job as the CIO of the Executive Office of President.
On taking his new position on inauguration day, he and his staff put in 80 hour weeks, “if not more.” And in his first 40 days on the job, the White House email system was down 23 per cent of the time.
Colangelo began his job on Jan. 20, 2009, the same day that President Barack Obama started his.
On that first day, Colangelo walked to the White House, found his office with some difficulty, and then “delivered the first presidential Blackberry,” as well as handhelds to all the top administration officials. “It was just a mind blowing experience,” he said.
But Colangelo quickly realized that the White House’s IT assets “were in pretty bad shape.”
Over 82 per cent of the White House’s technology had reached end of life. Desktops, for instance, still had floppy disk drives, including the one Colangelo delivered to Rahm Emanuel, Obama’s then chief of staff and now Mayor of Chicago.
The White House CIO office had one data centre, said Colangelo. “We had no redundancy,” he said, before attendees of Computerworld Premier 100 IT Leaders Conference in Phoenix.
The problems became apparent on Jan 26, six days after the administration was sworn in.
“Our email servers went down for 21 hours,” said Colangelo. “In my professional career, there has not been a worst day since or ever.”
At 5:30 a.m. on the morning of that already long outage, Colangelo was called over the West Wing to brief chief of staff Emanuel about it.
“I was walking (to the meeting) with some other leadership. It was pitch black and I haven’t gone home, and then the most amazing thing happened,” said Colangelo. “As my two feet hit the door of the West Wing, my Blackberry started to buzz. I normally hate that feeling, but I got to tell you it was the best feeling I ever felt.”
The email was back up.
By the time Colangelo reached Emanuel’s office, the meeting with the chief of staff was no longer needed.
Colangelo said the White House technology situation called for a massive review of technology, people and processes to determine the situation.
The White House faced three or four more outages in the next 30 days or so. “When it was all said and done, in the first 40 days of the administration we were down 23 per cent of the time,” said Colangelo.
The White House CIO provides all unclassified services to nearly a dozen business units, including, the Office of Management and Budget and the Office of the U.S. Trade Representative.
Some of Colangelo’s previous jobs included working as the CIO of the Democratic Convention Committee, and IT project manager for the American Red Cross Hurricane Recovery Program.
As part of the process to upgrade White House IT operations, Colangelo held town halls with their customers, some whom were angry. “They had floppy drives — I knew what they were going to say,” he said.
The White House needed to replace much of the technology, including the email systems and storage area networks.
As part of the effort, the White House created new positions, including the “GOALIE,” which is the Government Operations and Lead for Inspection and Execution. GOALIE employees worked 12 hour shifts to give 24 hour coverage of the data center and manage any crisis. Previously, the data center was only staffed from 9-to-5.
A year ago, in Feb. 2011, the White House suffered a nine-hour outage when its Sonet networking ring was cut, interrupting email and Internet access.
“I had to fax updates to the president while on the road – not a great thing to do,” said Colangelo. “But it proved to be a very beneficial thing.”
Colangelo said the outage allowed him to focus attention on the need for a disaster recovery data center. With a second data center accessible, “none of these issues would have happened,” he said.
The White House now has a data recovery data center for its unclassified systems, which includes redundant email servers.
The White House IT office also worked to give mobile capability to staff.
The tech staff developed a Web-based portal to allow staff to access email and other services from home, “in a secure and records managed way,” said Colangelo.
The mobility effort paid dividends when Washington was shut down for nearly a week in Feb. 2010 by two back-to-back blizzards that dumped several feet of snow. The White House had 60 per cent of its staff online the entire week, said Colangelo.
The White House has since rolled out access to tablets for those employees who bring their own, and has rolled out support for more smartphones. It also rebuilt WhiteHouse.gov.
“Our modernization program was very successful,” said Colangelo. It included increasing Internet speeds over 300 per cent, as well as reducing the number of assets at their end of life by over 50 per cent.
Along with an absence of new technology and redundant systems, the White House operations also lacked automation tools for many record keeping tasks and processes.
Colangelo was also open to new ideas.
An intern in the Council of Economic Advisors, who had a computer science background but was working on projects unrelated to his training, sparked the effort. When asked to update a spreadsheet or a memo, he automated it using macros or some other type of technology to make it move faster.
The intern convinced Colangelo that there was a great need for automation.
This led to new initiative. There wasn’t money in the IT budget to hire professional software engineers. Instead, they created a program to hire computer science savvy interns to work on the project.
The interns met with the business customers, and it led to new approaches “for small and really painful problems for customers,” said Colangelo.
In eight weeks, 40 applications were built, including a parking application that had previously been done in Excel, and a printer dashboard to tell them when a printer was running out of toner. The program also led to the development of survey tools, memo generators, and an online tool that replaced paper-based goods and services processing.
“I’m proud of what we’ve done here,” said Colangelo. Among the things he believed helped him to make progress was to “never let a bad day go to waste.”
