A steady stream of taxis grinds up the hill to the headquarters of the Athens Olympic Committee headquarters, on the northern edge of the city. In the lobby it’s all bustle as visitors mill around the accreditation desk and pass through security controls. But on the second floor the glass-walled technology operations centre sits idle — most of the 135 seats in the control room are empty, and all but one of the screens on the video wall are dark.
Soon this room will be buzzing with activity as staff members monitor and maintain the health of the servers, data networks, and power supplies delivering key applications for the 2004 Olympic Games. But for now, it’s eerily quiet.
There’s plenty of work going on behind the scenes, though, with integration still to be completed at some of the smaller venues. And then there’s testing, lots and lots of testing.
Claude Philipps, program director of major events at Atos Origin SA, the lead IT contractor for the Olympic Games, likes to be prepared. “We are ready, but we are still testing, because we want to be sure that every stupid thing that can happen is planned for,” Philipps says. “In a normal IT project, we could have delivered the application to the customer almost eight months ago.”
But the Olympic Games is far from a normal IT project. The deadline is non-negotiable and there are no second chances: Everything must work, from the opening ceremony on Aug. 13 right to the end, says Philipps, whose previous experience includes developing the control system for the world’s first computerized nuclear power plant.
With all that pressure, Philipps’ team is doing its utmost to ensure that the network will not fail. They are building multiple layers of security and redundancy, using reliable technology, and then testing it rigourously.
Creating a Team
In the buildup to the games, the team goes through two technical rehearsals in which 30 Atos Origin staffers put the network through its paces. The team spends a full week simulating the busiest days of the games, Philipps says, dealing with “crazy scenarios of what might happen in every area: a network problem, staff stopped in a traffic jam, a security attack … everything that might happen.”
The rehearsal tests people and procedures as much as products. That’s important because the IT operating organization Philipps is building will have grown from nothing to a staff of 3,400 in less than three years. Many staff members are volunteers who train evenings and weekends to deliver first-line support.
Philipps is getting used to this boom-and-bust cycle of team building, having worked on the event since the 2000 Olympic Games in Sydney, Australia. Some of his colleagues can trace their involvement further back, because Atos Origin now owns Sema, which has been developing software for the Olympics since the 1992 competition in Barcelona, Spain.
The two major components of the software that will run over the Olympic network are Atos Origin’s Games Management System (GMS), a customized suite of applications that act as kind of enterprise resource plannning (ERP) for the Olympics, and the nformation Diffusion System (IID).
GMS will be running on Microsoft Corp.’s Windows 2000 servers in Athens, an upgrade from the Windows NT 4 used at the Salt Lake City games in 2002. “We’re not using sexy technology,” Philipps says. “The main goal for us is to reduce the amount of risk.”
The IDS collects and distributes event results and rankings. Press agencies such as The Associated Press and Reuters get a dedicated feed from IDS, as do certain Web sites.
Philipps expects the system to deliver 50 million pages of reports to Olympic partner Xerox Corp. for printing during the games, largely for those print journalists at the press centre in Athens who still prefer information on paper.
The IDS also serves broadcasters. “We provide a live feed for TV and radio broadcasters (who) are commenting live on the event. This is a real-time system that provides them everything so that they can look smart,” Philipps says.
A Fail-Safe Plan
Together, GMS and IDS impose exacting requirements on the network. GMS is, among other things, used to manage accreditations for the games, so security is vital. Speed, too, is important: Philipps’ goal is to have the result on commentators’ screens 0.3 seconds after the athlete has crossed the line, complete with rankings, statistics, and biographies — everything that helps commentators during a live broadcast.
Yan Noblot, information security manager at Atos Origin, says the key to that is to build in redundancy —and lots of it. “We have doubled everything, because we need 100 per cent availability at games time,” he says.
And when he says everything, that goes for the routers and switches at each site, the datacentres that process the results, even the PCs on the desks in the control room.
An Synchronous Digital Hierarchy (SDH) network composed of two STM-1 rings supplied by Greek carrier Hellenic Telecommunications Organization (also known as OTE, an acronym based on its name in Greek) links the Olympic venues at 155Mbps. “We only use one ring, the other is for redundancy,” Noblot says.
The 36 competition venues and 20 or so noncompetition venues are linked by fibre to the SDH ring. All the venues are connected by two different routes, with a ring for each venue, and traffic from each venue is served by two different telecommunication centres, according to Elpida Trizi, a spokeswoman for OTE. “We’ve constructed the network in such a way that we’re able to provide a service even if one of the routes is damaged,” she says.
Event results and data from the games management system are stored in two datacentres hosted by OTE, which also supplies the SDH network. The primary datacentre is located near OTE’s headquarters in Marousi, just across the main highway from the Olympic stadium; the other is another several hundred miles away, still in Greece but in a different earthquake zone.
The datacentres sit directly on the SDH ring. “One reason we did that is because we do real-time replication between primary and secondary datacentres,” Noblot says.
That direct connection is made through a pair of Cisco Systems Inc. 7200 series routers. “We have two of each, at least, for redundancy, configured in such a way that the traffic would be automatically rerouted,” Noblot says. “Behind this we have two Catalyst 6513 layer-3 switches running services. We use it to do VLAN routing. We also have a firewall and intrusion detection system in the same chassis.”
To keep things orderly, Atos designed three different LAN configurations: one for the largest venues, including the Olympic stadium and the water sports center; another for midsize venues such as the equestrian centre; and one for the many smaller venues.
Each venue has a pair of access routers, with models depending on the venue size, and behind that a pair of switches to send all the traffic to the intrusion detection system. After that comes the distribution layer, a pair of switches for VLANs and routing between the VLANs. “The very last layer is the access layer where we have Cisco switches that plug directly to the servers and the PCs,” Noblot says.
Segmenting Traffic
Atos is using VLANs both to simplify troubleshooting and to limit damage if anyone manages to break into the network. There are separate VLANs for the commentator information system, information diffusion applications, and the game management system. Technical services, directories, management and monitoring, and the on-venue results system each have their own VLANs too, sometimes several per venue for the same function.
“The purpose is to segment the traffic so we can monitor it and contain potential issues,” Noblot says. “If someone brings in a virus, that would be contained on systems on the same VLAN.”
Software distribution is another service secured over a VLAN. Atos is using Symantec Corp.’s Ghost and software from LANDesk Software Inc. to remotely load software onto PCs.
That’s an issue Noblot has kept in mind since an incident occurred at a practice event in Greece. “We were just setting up for the mountain bike event, and we had a hardware failure,” he says. “We said, ‘Just send a guy from this room with spare hardware,’ and the guy called up and said, ‘Are you nuts? It’s not possible, it’s two kilometers uphill!’ ”
In this case, the team ended up rebuilding the defective component’s software over the network.
Anything that can avoid the need for a time-consuming journey to make hands-on repairs is welcome, because the SLA specifies a 10-minute resolution time for severe problems. “That’s not to take ownership of the problem, but to fix it,” Philipps emphasizes.
What makes the Olympic Games a unique project is that the athletes aren’t going to stop running just because the server has. As Philipps says, “When we speak about fixing something, it might be a work-around, a decrease of functionality, but the key thing is the show must go on.”
SIDEBAR
Olympic-size security demands advance planning
If there’s one thing the Atos Origin team understands as lead contractor for the Olympic IT infrastructure, it’s that you must learn from your mistakes.
One such lesson learned the hard way: Security must be built in from the start, says Claude Philipps, program director of major events at Atos Origin. For the 2002 Olympic Games in Salt Lake City, the company “started embedding the security too late, so it wasn’t running well,” he says.
The lapse caused nothing serious, aside from a few headaches: “We had a lot of attacks, but we ran the games safely,” Philipps says.
The team found the number of alarms generated by security systems can become unmanageable without software help. Based on the number of alarms seen in Salt Lake City, they could expect to see 200,000 per day related to security in Athens, Philipps says — most of them irrelevant warnings.
“This is not manageable: Screens would be flickering all day long, so we want to reduce it to (the) 10 to 50 that are real,” Philipps says. This year, Atos Origin is using Computer Associates’ eTrust to filter the alarms based on a set of rules.
Careful filtering can help in other ways, too, particularly when it comes to Windows 2000 permissions. To prevent power falling into the wrong hands, information Security Manager Yan Noblot uses NetIQ Corp. for security administration.
“It allows us to have a more granular definition of rules,” Noblot says, “We don’t have to give admin rights to the help desk; we give them only the rights they need.”
That precaution might rule out some social engineering attacks, but there are other ways in. In Salt Lake City, miscreants got around application-level locks on public-access PCs by rebooting them and trying to get into the network from there, Noblot says.
Anyone hoping to introduce a virus or other software onto the network in Athens will find the CD-ROM drives, floppy drives, and USB ports on PCs and servers disabled.
According to Philipps, it’s cheaper to have the suppliers deliver standard machines then uninstall the drivers and disable the drives and ports at the BIOS level than it is to order special machines.
If any of the PCs later need a last-minute anti-virus update or security patch installed, “we distribute it through the network using tools like LANDesk or Symantec Ghost,” Philipps says. With the CD-ROM drives out of use, there’d be no point in sending someone running around the 60 or so venues with an update CD, unless they were training for the marathon.