Network Upgrades Meet the Need for Speed

While the commercial Internet is growing at a staggering rate, it has nothing on the Energy Sciences Network. ESnet, managed by the Lawrence Berkeley National Laboratory for the Energy Department, links tens of thousands of scientists at laboratories, universities and other research institutions around the world. The 25-year-old network, which lets scientists collaborate on complex issues including energy, climate science and the origin of the universe, moves petabytes of data each month — and the amount of data continues to grow at a rate of 70 percent each year.

Keeping up with that demand requires power, reliability, scalability and speed. The latest major contribution to keeping the network fast and reliable is a new transcontinental 100-gigabits-per-second infrastructure. The paths between the major Energy Department supercomputing facilities are currently operating in prototype mode, and the ESnet team will transition its production network to the new infrastructure this year.

A key challenge for the Energy network and other agencies, such as Treasury Inspector General for Tax Administration (TIGTA) and the Federal Aviation Administration (FAA), is planning network upgrades to stay ahead of future requirements in addition to meeting current needs.

“We were very careful in how we developed this network, because we know we’ll eventually need more than 100Gbps,” says Greg Bell, acting director of ESnet. “The next jump will probably be 400Gbps, and then terabits. So to the extent possible, we’re trying to buy equipment that will accommodate those higher speeds when they arrive.”

For TIGTA, network bandwidth is crucial to managing the daily and seasonal traffic spikes that occur at the agency’s Washington, D.C., headquarters, two data centers and roughly 70 locations around the country. TIGTA, which conducts investigative, inspection and audit work for the IRS, finished transitioning to the Treasury Network (TNet) in October 2010.

The FAA faces similar networking challenges. The agency is responsible for moving massive amounts of data to support the operation of the National ­Airspace System, including radar surveillance data, flight plans, air traffic flow information and weather data. Much of the FAA’s data is highly perishable, requiring a high-speed network to ensure that it is received in near real-time.

As they upgrade their networks, agencies should consider some best practices learned by others who have carried out similar projects.

1. Planning Is Essential

Agencies should be realistic about what users need, and plan accordingly. Knowing the type and amount of data flowing through the National Airspace System was critical in planning the FAA’s recent network upgrade, says Steve Dash, manager of the agency’s telecommunications services group. The upgrade established a 10Gbps optical backbone using dense wavelength division multiplexing, as well as dual OC-48 Synchronous Optical Network rings to support the air traffic control operations’ requirement for very high availability.

Knowing users’ future needs and working to address them will also prevent problems.

“We hold two meetings per year with specific groups of DOE scientists and their program managers in the Office of Science,” says ESnet General Manager Joe Burrescia. “Instead of asking them what they need in terms of network capacity or speed, we ask them to explain how their scientific discovery process works. We try to understand their current and future research projects, the expected data sets from this work, and where these data sets will need to be sent, whether it be a collaborator in another country or a supercomputer center for analysis.”

By analyzing this information, says Burrescia, ESnet can then derive their network requirements — a process that helped the ESnet team determine that it needed to build a 100Gbps network that could scale effectively.

When considering a network upgrade, the IT department should provision enough bandwidth to handle the natural evolution of agency demands.

“Once an organization makes the commitment to upgrade, it pays to aim high when it comes to bandwidth, because the cost per kilobit drops significantly as the bandwidth increases,” says the FAA’s Dash. Newer systems and applications frequently demand more bandwidth; by oversubscribing, an agency can avoid having to go through multiple upgrades in a short period of time, he adds.

2. Study the Situation

TIGTA CIO George Jakabcin says IT staff should determine the network’s utilization rates and work to maintain them within a target range. “It’s good practice to keep most network activity in the 30 to 50 percent range,” Jakabcin says. “When you start to see rates above this, it’s time to pay more attention to what’s going on. When you consistently see 70 or 80 percent utilization, consider an upgrade.”

Monitoring the network’s performance helps to alert administrators of potential problems. “ESnet and our partners around the world have built and deployed network measurement systems which collect data about the health of our networks on a real-time basis,” ESnet’s Bell says. “So when the performance isn’t what we expect it should be, we can use those servers to help quickly resolve performance problems wherever they may be caused — either within the ESnet domain or in our partner networks' domains.”

3. Smart Implementation

Agencies don’t have to break the bank to get more speed out of their networks. While the most common ways to boost speed are increasing network bandwidth or installing new routers, network managers can also try other methods, such as implementing WAN acceleration technologies.

But the biggest bang for the agency’s IT buck may be to reconsider the traditional hub-and-spoke network infrastructure, says Zeus Kerravala, principal analyst for ZK Research. “Most wide area networks are built using the hub-and-spoke premise, where all network traffic runs through one central location,” he says. “For many organizations today, it makes more sense to use technologies like split tunneling and mesh networking to re-architect the network around traffic flows.”

Although it’s important for IT staff to pay close attention to the agency’s mission and the nature of its traffic — critical factors in making intelligent decisions about a network upgrade — it also pays for them to listen to what vendors have to say. “Vendors should have a stake in making sure you’re not over- or underprovisioned, and that you’re using the right technology, because they don’t want unhappy customers,” Jakabcin says. If both parties have established a level of credibility and trust, the combination of an agency’s knowledge and a vendor’s expertise is a winning one, he adds.