While it’s a jarring sight, it’s also not wholly unexpected for Kendall Square: a glass-walled command center that looks equal parts James Bond villain lair and NASA launch center, complete with giant screens showing spinning visualizations of the Earth overlaid with vivid graphics and arcane data.
The room’s role is just as logical, and surprising: It’s Akamai’s Network Operations Control Center, also known as the company’s Internet Observatory.
“It’s probably the best view of the Internet that you can find anywhere – right here, in the middle of Cambridge,” said Jeff Young, vice president of corporate communications for Akamai Technologies, during a December tour. “And you can walk by and peer in the window and see it.”
Three people in a cube
There has actually been a NOCC in Cambridge from the launch of Akamai in 1998, when the company was born out of the Massachusetts Institute of Technology with a mission to ensure smooth operation for the Internet – a decentralized network of networks that by its nature runs at the ragged edge of control.
At the time, the center was three people staring into CRTs in a Tech Square cubicle, but this 1,500-square-foot observatory has been a Broadway showpiece for some nine years, hidden when the glare of the sun forces workers inside to close the shades. It’s a frequent stop for tech tours and visitors to the company’s upstairs executive briefing center, Young said. Especially with Akamai’s “living room of the future” across the lobby – where the company works with media clients to embed searchable content and commerce options in video entertainment – Young and network operations engineer Kevin Lamont might play host to customers, tourists and reporters a few times a week.
The center is “the best view of the overall health of the Internet that you can find anywhere,” Young said, but for Akamai there’s a more specific purpose – to predict surges and spot attacks, natural disasters or other problems and find better, smarter, faster Internet traffic routes, ideally keeping clients and customers from realizing there was a problem in the first place.
Attacks turned back
Lamont is able to scroll quickly through visual data on giant overhead monitors, finding anomalies that account for the Internet’s biggest recent challenges – on a global scale since 9/11 (when the Internet was a lot smaller anyway, given the fewer mobile devices), a handful of events such as the hunger for World Cup content. One represented the drop of an anchor from a vessel in the Middle East chopping an underwater Internet cable. “See how bad the performance got, really fast. Our performance got bad too – for about five minutes. Then we found the next-best routes,” he said, eyeing the readout. “What we adapted to in five minutes, the general Internet took two weeks of really bad performance.”
He scrolled to an anonymized report of a financial customer undergoing a two-week siege of anonymous programs called “bots” intent on taking down the company website – if successful, potentially ruinous. Legitimate traffic was dwarfed 50-to-1 by attack traffic, with the busiest day of attacks seeing 9 billion requests, Lamont said.
“But Akamai was seeing all these requests come in and we were able to find the source behind it and very quickly push out to our machines rules that denied this surge in traffic. The actual website stayed up and it stayed functioning and the actual legitimate customers didn’t know it was a problem,” he said. “There hasn’t been a recent event that really taxed our capabilities.”
Doubling without doubling
The Internet’s growth to some 15,000 different networks (with around 150,000 Akamai servers physically deployed in more than 1,200 of them) illustrates the need for the NOCC. “Over the past four or five years, our server count has roughly doubled, but the amount of traffic we’re serving has gone up by an order of magnitude,” Lamont said.
Still, he said, “Our mission is to not grow as fast as the Internet. Akamai is going to be growing a lot, with 4K video and all that, but the NOCC itself is trying to become more efficient at the same time. If we double the number of servers, we don’t get to double the number of people who work here.”
Akamai keeps it to about 15 people at any given time running operations – three shifts daily at the Kendall Square center and its 24/7 counterpart in Bangalore as well as two lesser-used NOCCs in Krakow and California. “If anything were to happen to this room, Bangalore would be able to take the burden,” Lamont said.
Not a maximum, just a peak
On a quiet day in early December, the NOCC was tracking some 7 million attacks per hour, and Lamont called that about 8 percent below average. At that point, company servers shuffle some 18 terabits to 19 terabits of data per second, a mind-boggling figure that in reality represents Internet torpor.
“Back in November, we actually went up to about 45,” Lamont said – not a maximum, just the most demand the company had at one time. Cause for concern, he said, might be 300 million attacks per hour.
As he spoke, Lamont spotted a problem in server traffic from New York to Atlanta.
We’re used to seeing 31 milliseconds of latency, and we’re seeing 800 … If we’re seeing 10 times the latency from Atlanta to New York, we’re going to say, ‘Well, don’t go from Atlanta to New York. Go to New Jersey and then to New York.’ And that makes all the traffic at Akamai carry faster, even if it’s traveling around the world. When things go really bad, such as on 9/11 or if there’s a tsunami and a lot of infrastructure is destroyed, our mapping software actually responds to that and takes those bad paths out of service before a human really knows the extent of the damage.
Wide screens in great detail
As a series of giant monitors hooked up to sophisticated, lightning-fast computers (the room’s refresh by Rhode Island-based Constant Technologies got a detailed write-up here), the screens making up the back wall of the center play breaking news on CNN in addition to showing dynamic, three-dimensional charts and graphics. They can become shared desktops or teleconferencing – if you want to see a co-worker’s face at 110 inches across – or, for that matter an amazing way to watch football or Christmas movies.
“We figured it out and we could watch all of the ‘Lord of the Rings’ movies at the same time, and the widest-screen movie ever is ‘Ben-Hur,’ and we could watch that three times in super, ultra-wide-screen,” Lamont said.
“That’s an interesting stat,” Young said.
Drill down and you can also peer directly into an individual server cluster to see Internet Protocol numbers, what process is being served, how many hits per second it’s getting and CPU load, or a global take on the same kind of figures everyone with Google Analytics gets for traffic to their individual websites: What kind of Internet browsers are being used everywhere around the world at any given time.
Lamont followed an orange line representing Microsoft’s Internet Explorer, for instance, spotting where it crossed blue and red lines representing mobile browsers people use when they’re off work and on their phones. “When people are at work, they have to use Internet Explorer. On the weekends, it drops down because they’re outside, they’re not at a desk,” he said.
Cyber attack central
The NOCC also knows the address of the anti-NOCC, if you will. If Akamai is in the business of keeping traffic flowing for its clients, this is the place where the anarchical elements of the Internet come for the inexpensive, bot-for-hire firepower they need to disrupt it.
“Most attacks come from one data center in Virginia, and that’s because you can rent that data center and use it for whatever you want,” Lamont said. “Anybody in the world can say ‘I want 100,000 virtual machines for eight hours’ and use them.”
That doesn’t make Virginia and Kendall Square poles in an increasingly hot cyber war. Server automation and customers empowered to push out their own rules to handle malicious traffic give this command center the hushed, even sleepy feel of a high-tech library. The only time workers have stayed after a shift change had nothing to do with the Internet – it was the lockdown requested while police searched for the Boston Marathon bombers.
“There’s nothing actually critical about this room,” Lamont said. “All our tools are on the cloud. If anything happens to the building, the engineers could actually work from their laptops.”
It was, coincidentally, a day during which about half of Lamont’s staff was working remotely to test that capability.
“The most interesting thing about this room,” Lamont said, “is probably how small it is.”