Always Connected

A lot rides on the Federal Aviation Administration's telecommunications network: About 700 million passengers and 36 billion pounds of cargo travel through U.S. airspace annually.

Air traffic controllers and pilots rely on the network to communicate and access radar, weather and other critical information. With hundreds of aircraft in theair at any given moment, downtime isn't an option. An outage at the FAA's administrative offices may not be life threatening, but it does have significant operational implications. If the network goes down, employees can't access e-mail, business applications or phone services, bringing productivity to a crawl.

To improve air traffic and administrative performance, the agency invested $300 million in a state-of-the-art network that makes high availability a priority.

The effort dates to 2002, when the FAA began consolidating legacy networks and replacing them with a single nationwide network, the FAA Telecommunications Infrastructure (FTI), that integrates voice, video and data communications. Harris, the contractor that helped build the network, monitors FTI's health and security 24 hours a day, seven days a week, and provides immediate technical support to mitigate downtime.

According to Steve Dash, who oversees FTI as FAA's acting director of Air Traffic Control Communications Services, the new infrastructure is more efficient, secure, reliable and cost-effective.

"It took about five years to transition from our myriad of networks and subnetworks, which often had different ownership models," Dash says. "In some cases, we owned or leased; in others, we outsourced."

With FTI, the FAA sought "to eliminate those different ownership, operational and funding models and bring [everything] under a single-service delivery model," he says. "When I made the initial business case for FTI's implementation, my data showed that it would improve service delivery and save money — and that's what we delivered."

Since FTI was implemented, the FAA's annual operating cost for telecommunications has consistently held at roughly $350 million, even though the amount of network traffic has more than doubled. (In 2002, before FTI, the agency's annual telecommunications costs were $415 million.)

If the FAA had continued operating its old networks rather than transitioning to FTI, the cost to manage the increased traffic would have mushroomed, Dash says. "In 2002, we had much less network than we have now — 12,000 discrete serv­ices versus 24,000 today," he says. "So basically we have a lot more now for less."

High availability has become a top priority for federal IT teams in recent years, given the increased reliance on network-centric technologies. Already, many agencies are accessing applications and data over the cloud. Some are using thin client devices, while others are embracing convergence and deploying Voice over IP, video conferencing and other unified communications technologies. As the number of federal employees using smartphones and other mobile devices continues to climb, demand will build for round-the-clock network connectivity so they can access the applications and data they need to do their jobs.

"High availability in a network is essential in today's world," says Dr. Ryan Durante, chief of cross-domain solutions at the Air Force Research Laboratory in Rome, N.Y., who has helped install 20,000 thin client devices across the intelligence community and Department of Defense. "If you're a PC user and the network is down, there's not much you can do these days. If you're on a thin client with no network connectivity, you're staring at a blank screen."

Ensuring Uptime, Embracing UC

FTI is a wide area network that currently provides 24,000 services to 5,000 FAA facilities nationwide. Considered a central component of the agency's rollout of next-generation technology that will improve air traffic control operations and reduce air traffic congestion, the network is split into two distinct domains or subnetworks — one for the National ­Airspace System, which handles air traffic, and the other for administrative functions such as payroll, e-mail and phone services.

"We have an extraordinarily complex network," Dash explains. "The demands for air traffic control are rigorous and require high operational availability. The administrative domain, meanwhile, serves [the business needs of] our 40,000 office employees."

To ensure high availability, the network is built with redundant equipment and dedicated fiber across the country, Dash says. Harris oversees a dedicated network operations center (NOC) in Melbourne, Fla., where technicians use the latest tools to ensure availability and security around the clock.

If problems arise, NOC personnel can troubleshoot remotely or call on the several hundred technicians deployed nationwide to provide tech support. If the main NOC goes down, a backup NOC takes over to keep the FTI network up and running, Dash says.

To ensure air traffic receives optimal support, the network's air traffic and administrative functions are managed separately within the NOC. "I don't want a maintenance problem on our administrative side to impact our air traffic control side," he explains.

The FAA and Harris collaborate regularly to fine-tune and improve operations. "The bottom line is that we have very detailed operations, controls and procedures that manage the infrastructure, and we have an ongoing auditing process," Dash says.

On the administrative side, the FAA is embracing VoIP and UC. In November 2009, the agency awarded a 10-year, $228 million contract to General Dynamics IT (GDIT) to upgrade its traditional PBX phone systems to an enterprisewide Avaya VoIP system. The system will be deployed at 1,000 FAA facilities and administrative offices, including agency headquarters in Washington, D.C.; the Mike Monroney Aeronautical Center in Oklahoma City; and the William J. Hughes Technical Center in Atlantic City, N.J.

The project is self-funding. The FAA currently uses about 600 different PBX systems that are at different points in their lifecycle. Each of the PBX systems has its own maintenance contract, so consolidating to a unified VoIP system will save the agency money over the long term, Dash says. The VoIP equipment will sit on the edge of the network, with voice calls riding the FTI network.

A Private Comm Cloud

Dash says the FAA is implementing both VoIP and UC technologies as a private cloud service. To ensure compatibility, GDIT will integrate the VoIP system and old PBX systems. That way, employees in an office with an aging PBX can call an office with the VoIP system.

Dash can't predict adoption rates, but he expects the tools will prove popular once administrators and employees come to understand their capabilities. So far, roughly one to two dozen FAA offices have migrated to the VoIP system. "What I want to do is create a marketplace of opportunities for my internal consumers," he says. An office is more likely to invest, he adds, once it sees the potential productivity enhancements these technologies deliver.

GDIT also is building reliability into the new VoIP system, providing two core systems for redundancy. In the future, a third or fourth core system may be added. That way, if one goes down, another system can take over to ensure that voice services continue.

Although GDIT is focusing on basic phone services initially, Dash expects to roll out more advanced functionality over time, including a "find me, follow me" feature that allows work phone calls to be routed to employees' cell phones and smartphones. He also expects phone services to be integrated with users' PCs and with existing video conferencing equipment in various FAA offices.

Unified communications manufacturers, for their part, offer PC software that allows employees to track colleagues' presence and then call, instant-message, or video or web conference with them from their computers. The technology can speed communications, improve productivity and save money because employees don't have to waste time and money traveling to meetings.

Need for 24x7 Communications

At Peterson Air Force Base in Colorado, military commanders rely heavily on phone and video conferencing to communicate with other U.S. military units in the United States and abroad.

It's up to the 21st Communications Squadron to make sure the network stays up and running.

The squadron manages the voice network not only for the 21st Space Wing, but also for Peterson AFB's tenants, including the North American Aerospace Defense Command (NORAD), the U.S. Northern Command, the Air Force Space Command, and the U.S. Army Space and Missile Defense Command.

All told, the 21st provides voice serv­­ices to approximately 5,800 military personnel, 2,700 civilians and 2,700 contractors at the base.

"Our wing and mission partners depend on the network 24 hours a day to share information with each other and with units in other countries," says 21st Communications Squadron Deputy Michael Byrne.

Keep 'Em Separated

Peterson AFB uses separate networks for voice and data, but for security reasons, the squadron won't say whether it uses a traditional PBX or VoIP system. Rita Hahn, deputy of infrastructure, acknowledges that the voice network supports the voice and video conferencing traffic for every mission partner.

To ensure reliability, the phone system is protected by a backup system. "We have redundancy and duplication all over the place, so if one system has a glitch, the second system kicks in and it's seamless to users," Hahn says.

Two contractors manage and monitor the voice network during the day. Seven offsite technicians monitor the network after hours and troubleshoot at the base when needed. An alarm system that monitors the voice network notifies squadron personnel of outages. In most cases, technicians are fixing the problem before customers even realize an outage has occurred, Hahn says.

When outages happen, they typically are concentrated at one specific facility. Byrne says the phone system has never had a basewide outage.

On the data side, the 21st relies on industry best practices to ensure uptime, deploying redundant networking equipment and multilayer security controls to prevent hackers, viruses and other threats from bringing down the network. Servers are programmed to send software patches and the latest antivirus definitions to desktops.

In addition, employees must log in to their computers with a special Common Access Card, which doubles as their ID card. The use of USB Flash drives is forbidden. Sensitive information is encrypted, and e-mails are digitally signed, Byrne says.

Every user at Peterson AFB also is required to participate in computer security training. Once in a while, the squadron secretly tests users, sending them bogus phishing e-mail scams, for example, to see who clicks on them. When users do click on the links in these e-mails, a message explaining what they did wrong immediately appears on their screen.

"We try to keep our users educated," Byrne says of the exercise. "There are a lot of bad guys out there, and we want to keep them off our network."

Networking Triple Play

The three networks at Air Force Security Forces Center at Lackland Air Force Base in Texas are critical because the Security Forces' 200 employees use thin client computers to access classified and unclassified information. Because classified and unclassified data can't commingle, the networks — two for classified information, one for unclassified data — are kept separate.

Security Forces serves as the Air Force's police department and is responsible for creating rules, regulations and training guides for 36,000 active-duty officers. Its staff also purchases equipment, such as weapons, and manages the Defense Department's Military Working Dog Program.

As part of their daily duties, Security Forces staff must communicate regularly with the Pentagon and other military bases. They also coordinate with the Secret Service, FBI and CIA to provide military dogs that have been trained to protect the president and foreign dignitaries, says Richard Johnsen, senior network engineer at the Security Forces Center.

"We cannot afford to have our people out of contact with the agencies they deal with," he says of the department's high-availability mandate.

Prepared with Hot Spares

To ensure network reliability, Johnsen purchased multiple hot spares — or backup switches that are turned on — for each of the three networks. That way, if a switch on one network crashes, a backup switch will take over to maintain operations. To further improve reliability, each switch has redundant power supplies and is connected to uninterruptible power supplies.

"It can get expensive, but what's the value of the work you are doing?" Johnsen asks. "If you lose network availability, it can cost you. As the military police for the Air Force, we have people deployed across all corners of the world. We also are responsible for the protection of the president. We can't afford for our networks to be down."