How Agriculture Successfully Moved to the MPLS Cloud
Network consolidation can save money, reduce administrative overhead and enhance security, but only if done right.
“For a long time, Agriculture had wanted to optimize its telecommunications network,” says Jan Lilja, former associate CIO for telecommunications at the Agriculture Department and currently working at the Smithsonian Institution. “It had actually tried twice before, and both of those efforts were unsuccessful.”
The third time, when moving to a Multiprotocol Label Switching environment, USDA did the job right, says Lilja. But how did it get there, and what were the factors that led to success in moving several bureau-level networks to a single MPLS cloud? “We tried to follow what you are taught in your engineering textbooks, and it worked,” she says: Analyze the system, define the problems, evaluate technical approaches to solve the problems, build a core system and ramp up use.
A look at the Agriculture effort proves particularly useful now because it allowed the department to do two things that all agencies must do under mandates from the Office of Management and Budget. First, USDA collapsed its Internet access to two gateways, as now required by the Trusted Internet Connections initiative. And second, it used the network overhaul to prepare for Internet Protocol Version 6 traffic, long before OMB ordered all agencies to do so by June of this year.
Analyze and Define
The first step is to figure out what’s running where, Lilja says.
USDA’s Office of the CIO began in 2001 by writing a vision statement for a new backbone, dubbed the Universal Telecommunications Network (UTN), that would link all agency subnetworks to only two Internet gateways. At the time, Agriculture had 10 nodes for Net access. And, although the OCIO telecom group would routinely add bandwidth, it couldn’t keep up with demand from the organizations within USDA running the dedicated circuits and nodes that supported network traffic. As soon as extra capacity was available, it was instantly usurped, says Michael L. Thomas, director of national telecommunications services and operations.
The team inventoried network components (at headquarters and serving field offices nationwide) and analyzed data traffic, then it drilled down into the data to see how, when and for what purpose it was then spending each telecom dollar.
“We spent a lot of time surveying the agencies as to what they had as baseline technology, what they anticipated as future business needs and what they anticipated as future technical needs,” Lilja says. “That became the foundation through which we evaluated possible technical solutions.”
The analysis showed the department had experienced a sevenfold increase in telecommunications demand and a fourfold increase in cost in a relatively short period, says Earl Rasmussen, who helped USDA develop the UTN business case and create a program management plan. Currently president of the Kaleidoscope Group in Falls Church, Va., Rasmussen was director of business transformation for Soza & Co. (now Perot Systems Government Solutions) at the time.
Another hot spot that the review turned up was security — at the perimeter and once data was traveling inside USDA, Thomas says. “Having multiple Internet gateways, each with its own firewall, intrusion detection system and URL filter, made security extremely difficult to manage,” he says. “We had to keep upgrading the intrusion-detection systems and URL filters because of capacity issues.”
Using the survey, the team crafted a business plan, outlining cost and time savings that could be gained by consolidating. Thomas says the team identified how it would increase bandwidth overall and use load-balancing to provide more throughput for usage spikes. Fewer gateways and a single infrastructure also would ease management and improve security.
Evaluate the Options
Once you complete the business case, it’s time to hash out how network engineering alternatives will afffect each organization within an agency, Rasmussen says.
fact: June 2006: USDA demonstrates IPv6 readiness, two years ahead of the OMB mandate.
The UTN project team held a multiday meeting with technical experts from each of the department’s larger agencies. Using the data from the analyses of current and forecasted telecom needs, the group modeled traffic and services on different network topologies.
For USDA, the best approach to ease the management burden on internal telecom staff and provide expanded services was to move to MPLS supported by a public carrier. “We found ourselves in a continuous spiral of needing to expand the bandwidth, always catching up instead of being proactive about the network needs,” says Lilja.
Build the Core
If you ask stakeholders to give up control, then you have to prove your approach will work, Lilja and Thomas say.
In September 2004, USDA began Phase One, working with AT&T Government Solutions to create the Internet access portion of UTN.
All agency networks connect to one or more of 11 nodes established at major USDA concentration points around the country, including the data center in Kansas City and the National Finance Center’s new backup data center in Denver. Each of these nodes connects to the MPLS cloud using a Cisco 7613 router with connections ranging from DS3 to OC-12.
The MPLS cloud acts as the backbone to connect to two Internet gateways (IGs) at AT&T facilities in San Francisco and Washington, D.C. Each IG hosts a Cisco 7609 router with three OC-12 connections to the Internet. Although USDA owns the equipment at its nodes, it outsources management. A chief reason is scalability: The size of the backbone itself is virtual, and the IGs fail-over to each other.
Each node, as well as the IGs, also contains a security stack: a Cisco firewall/IDS/URL filter, Websense e-mail scanning software and McAfee Antivirus.
To further protect traffic, Agriculture also began using “automatic shunning, which most agencies have never done,” says Thomas, who works in the Office of the CIO at USDA’s facility in Fort Collins, Colo. A shunning program blocks incoming traffic from addresses that intrusion detection systems have identified as potentially malicious based on fingerprints left from attempts to breach the network. “It will automatically block incoming traffic without us having to do anything,” he says.
Ramp Up Use
Lead, don’t follow, is the way to convince users to switch services, Lilja says.
It took USDA about a year to complete UTN Phase One. Then, OCIO moved the department headquarters networks to UTN. USDA agencies had to link to the MPLS cloud only for Internet service.
This was a phased implementation. The biggest challenge, Thomas says, was getting the legacy networks to work properly with the new nodes. Although OCIO coordinated with the agencies ahead of time, there were still surprises, he says. “It was a routing nightmare. Each agency had its own network, and even though they were using the same products, such as Cisco, they were doing their routing and addressing differently.”
Another issue cropped up at the gateways. On one of the last cutovers, the department ran into capacity issues. The security systems had never been tested at the full capacity required by an enterprise the size of USDA. Ultimately, the UTN team had to reset some of the parameters on the firewalls and redesign the routing to accommodate the traffic.
“It wasn’t a security risk, but access to the Internet was spotty for three to four days,” says Lilja.
With the Internet access backbone in place, the agencies then had the option of using MPLS on their internal networks. “I wanted to be able to prove that it was a value add, not a mandate for the agencies,” she says. “That ultimately made it more successful because it required us to establish a higher bar for ourselves.”
With that standard met, all agencies opted to move over to UTN.
Next Up: Networx
The department will leverage its experience with UTN when making the switch to Networx at the end of the UTN lifecyle in 2009, Thomas says. USDA will replace all its equipment and further consolidation will be possible, he says.
