How Fannie Mae Used DevOps to Improve Its IT Operations

Fannie Mae, the government-sponsored housing giant that provides liquidity to the mortgage market, does not want to be thought of as a giant — or act like one. And it’s using DevOps to transform itself.

A DevOps methodology is one that encourages software developers to work with IT operations staff on testing and quality assurance to develop software more quickly and automate infrastructure changes. It’s a collaborative mentality designed to produce software faster and more efficiently.

Jason Anders, technology director for Fannie Mae, says that the agency’s customers, (banks and mortgage servicers) want to speed up the process of granting mortgages. Fannie Mae wants to double down on its mission of making mortgages more available to Americans. “We’re moving from an organization that does change very slowly to one that changes very quickly,” he says.

Speeding Up Operations

Fannie Mae started using DevOps about two and a half years ago, according to Anders. The first year was spent deciding which capabilities the agency wanted to invest in; the second, a focus on building a baseline of capabilities that developers could use. Since then, the agency has seen those capabilities “start to take off across the enterprise and start to see some of the fruits of the labor.”

A third group outside of developers and IT operations has been building these IT capabilities and serving as a broker between those two other groups, Anders says. DevOps is not being mandated, he adds, but the agency’s senior management has emphasized hastening IT operations and development. There has been a need to increase automation, support a transformation to agile development methodologies, and put the tools in place to support that shift, Anders says.

DevOps has had a significant impact on Fannie Mae’s IT operations and developers, according to Anders. Previously, application developers and testers used to work with IT operations in the following manner: Developers would submit tickets to the IT operations team if an environment needed to be stood up or launched; operations would then come back with an estimate of how long that would take — often anywhere from 12 weeks to six months.

Automated infrastructure delivery, which speeds up the deployment of application and IT services, is a specific tool Fannie Mae has leaned upon heavily. However, using automated infrastructure delivery capabilities from VMware and Puppet, and Fannie Mae’s internal “rapid IT” team, the operations team has been able to leverage those capabilities behind the scenes, Anders says.

As a result, deploy time has been greatly reduced: Twelve weeks down to two weeks; five months down to one month. While Fannie Mae has not achieved 100 percent delivery in this manner, Anders says there is a “broad ecosystem of application types and operating systems” that have been deployed using those tools.

The agency’s IT operations staff has had to deliver close to 300 different development environments over the last year and a half, according to Anders. There were a variety of teams responsible for building out the servers, network components and data storage elements, and loading them with the correct applications to deliver to developers and testers.

Initially, Fannie Mae estimated that it would take dozens of people on five different teams to do so. However, using automated infrastructure delivery and the rapid IT team, Anders says, the agency was able to deliver the environments using five or six dedicated staff members. The project was estimated to cost $90 million, but in the end the environments were created and delivered for under $30 million.

“From an IT operations perspective, the operations team are the ones that leverage this infrastructure as a service capability,” Anders says. “They are delivering IT environments to everyone in a third of the time.” Fannie Mae also cut down the number of outside contractors it would have had to use for such projects, he says.

Making Life Easier for Database Admins and Developers

Fannie Mae is using other DevOps tools to make the operation smoother for its IT staff and its developers.

Anders says that Fannie Mae database administrators are using virtual data delivery tools from companies like Delphix and Informatica to significantly reduce the amount of work required to support a plethora of development activities.

Developers and testers need a large amount of data to test against from a quality-control perspective, Anders notes. In terms of how that impacts IT operations, the IT staff receives requests to deliver data to testing environments, which can normally take anywhere from a few hours to a few days.

Every time the team runs a test they alter the data and it needs to be refreshed, which is a chore for database admins (not to mention a menial and unexciting task) and prevents them from “working on cooler stuff,” Anders says.

Using those tools gets database admins out of that business entirely, he says, because they provide end users with an interface that can repeatedly refresh the data every few minutes, providing “thousands of hours” of time saved. Those tools also help improve quality because they take the human error factor out of the equation.

Such tools save Fannie Mae both money and time; the time saved is the biggest factor, Anders says, since IT operations teams can perform tasks much faster now.

The next step in Fannie Mae’s DevOps journey is to make these tools largely self-service and get the IT operations team focused on next-level infrastructure advancements. Those advancements include moving data to the cloud more quickly and focusing on the resiliency/hardening of the agency’s network.

“It’s all of the things they don’t have the time to focus on right now,” Anders says, “because they are spending all of their time focusing on these development efforts.”