Management

DevOps in the Federal Government: What It Means and How It Can Help

DevOps is still new to federal agencies, but some are taking advantage of the methodology to speed up software development.

Randy Barrett

Randy Barrett is a freelance writer and editor based in Washington, D.C. A large part of his portfolio career includes teaching banjo and fiddle as well as performing professionally.

It wasn’t too long ago that developing software, especially in the federal government, meant writing code and sending it downstream — with many intermediary steps — to users.

This old-fashioned “waterfall” model ruled for decades but was slow, risk averse, inflexible and a hindrance for organizations needing to move quickly.

It took major industries like aerospace, automotive and communications years to develop code that was sometimes obsolete upon deployment. In 2001, a group of IT iconoclasts could stand it no longer and released a manifesto for a new “agile” software development paradigm.

The DevOps approach is faster, customer centered and collaborative. It is flexible to change, based on continuous improvement and the frequent delivery of software that actually works_._

What Is DevOps?

While there are varying definitions, DevOps is at its core a philosophy of software development that uses the agile paradigm and also requires the re-engineering of business processes and a profound cultural shift.

“You can be agile but not DevOps,” explains Dhaval Shrimankar, a director with the Ambit Group consultancy. “But if you are DevOps, you have to be agile.”

LISTEN IN: Find out how to simplify DevOps for your organization in this CDW podcast.

The name is a literal combination of “software developer” and “operations end user.” The term binds them together as a team in a continuous Mobius loop of planning, writing, testing, deploying, operating and monitoring new software. “Software is never a final, finished product,” Shrimankar says.

DevOps requires the collapsing of organizational silos. People can’t work together on software if they aren’t allowed to interact. IT leaders must buy in to the arrangement and be willing to shift from a high-risk, low-reward model of development (waterfall) to a low-risk, constantly iterative one that delivers high reward in the form of functioning software (DevOps).

Commercial technology companies such as Google, Amazon, Netflix and Facebook have perfected DevOps to the point where in-house software upgrades are deployed thousands of times per day. Federal agencies are much newer to the approach, and it presents some unique challenges.

What Is Continuous Integration and Delivery?

Essential to DevOps is the use of a continuous integration/continuous delivery (CI/CD) pipeline. The model follows keys stages: development, integration, quality assurance, user acceptance testing, staging, preproduction and, finally, production. It is also highly automated and depends on a series of platforms called tool chains that help manage the workflow.

The automation of CI/CD allows software engineers to focus more on the business requirements of the end product. It also makes it much easier for multiple code writers to work on a project simultaneously.

“Such a pipeline allows individuals to achieve high-quality code and the flexibility to deploy or roll back without impeding others’ work,” Shrimankar says. Plus, it boosts the ability to make numerous deployments, which is at the center of the agile and DevOps philosophies.

DIVE DEEPER: Find out how the USDA uses containers to build apps.

What Are the Leading DevOps Tool Chains?

DevOps tool chains encompass five key areas.

CI/CD: This is a platform that allows automation and organization of the software development process. It is the central conductor of the pipeline. Leading tools include GitLab, Jenkins, Travis CI, CircleCI, TeamCity and Bamboo.
Version control: This is a central repository for code that can be accessed by multiple developers simultaneously. Regardless of programming language, the code is automatically versioned to help everyone keep track of who worked on what, when. Leading tools are mostly open source and include Apache Subversion, Mercurial, CVS and Git.
Server: This is where the compiled code is hosted and the software in development runs. Server software makers include Microsoft IIS, NGINX, Oracle WebLogic, Red Hat Jboss, Apache Tomcat, Django, Ruby on Rails and Node.js.
Testing: Testing code early and often is critical to a successful DevOps project. Automated testing platforms include IBM’s Rational Functional Tester, Katalon Studio, Selenium, Appium, Watir, JUnit and SonarQube.
Build tools: Code for an application must be compiled into an executable file. A build tool automates this process and is usually designed to handle a specific programming language. Leading tools include Apache Ant, Gradle, Rake and Maven.

DevOps Training and Certification

There is no formal federal DevOps certification process. However, several vendors and training organizations offer credentials, including Microsoft, Google Cloud, AWS, Docker, A Cloud Guru, and the DevOps Institute.

DevOps in the Federal Government

DevOps is still very new to federal agencies. While speeding software development should be welcomed in Washington, the new paradigm has faced predictable challenges.

However, a few intrepid agencies have jumped in. In 2017, the Air Force found itself way behind in delivering software for its Air Operations Centers. So, it moved experimentally to DevOps in a program with a Star Wars-inspired name, Kessel Run.

“It’s pretty common that using waterfall methodology in the Defense Department takes a couple of years — and that’s probably generous,” says Air Force Lt. Col. Max Reele, deputy commander of Kessel Run. “Our pursuit was to completely knock that down. Now, the time from inception to product in the field is 90 days.”

In some cases, the program delivers software releases, updates and bug fixes on a weekly basis, a pace previously unheard of in the Air Force. About 1,200 service personnel and contractors are part of Kessel Run and interest in the program is growing, according to Reele. Most important, it has full support from his commanders.

The U.S. Patent and Trademark Office started using DevOps in 2019. About 40 applications are now leveraging the pipeline, including team management and custom patent and trademark software. Sixty deployments go to the patents team annually.

EXPLORE: How can your agency effectively implement DevOps?

“We’re trying to go from a culture of handoffs to a model where the team has ownership,” says Melissa Rummel, agile delivery and transformation lead for the agency.

Now, the time from inception to product in the field is 90 days.”

Lt. Col. Max Reele Deputy Commander of Kessel Run, U.S. Air Force

USPTO has adopted a “dojo” model, which brings teams in to work on software projects that are part of their regular day jobs. The DevOps coaching happens in real time. “It’s really helped us accelerate culture change through learning by doing,” Rummel says.

Another agency using DevOps is the IRS, which established its DevOps practice in 2017. The pipeline is currently used by 89 taxpayer-facing applications. In addition to increasing speed, the method has lowered operating costs and increased quality and reliability. “In addition, DevOps improves security,” says an IRS spokesman. “By automating the integration code, we reduce the need for manual processes that would delay security updates.”

The General Services Administration is also moving forward with DevOps projects. The Justice Department’s Civil Rights Division website is using the pipeline, as is Cloud.gov, the government’s in-house cloud suite. “Every time there is an update to the underlying components, the deployment tool automatically triggers a new build of the platform,” says a GSA spokesperson.

An expert from the Smithsonian’s National Museum of African American History and Culture recently shared DevOps lessons learned.

Meanwhile, the National Science Foundation is about five years into its DevOps experiment, and it’s bearing the fruit of dramatically decreased software development cycles and better quality, says Carlo Ilagan, an architect at the agency’s information systems division. The pipeline is being used for the Research.gov portal and a beta NSF gateway site, as well as the internal my.nsf.org site, which provides a single point of entry for IT and systems staff to use to complete proposals and review activities online.

“NSF had a typical IT organization where development, operations and cybersecurity were siloed,” says Ilagan. “There were growing pains and turf wars. With leadership support, we are now at a state where development, operations and cybersecurity staff meet every week to collaborate.”

DevOps Lessons for Federal Agencies

The U.S. Citizenship and Immigration Service has a massive caseload challenge. It’s been that way for a while. In 2011, the agency adopted DevOps to speed up development and improve the functionality of its caseworker software. Lessons learned? Decide early on what toolsets you want to use organizationally, rather than trial and error, agency IT staff told ACT-IAC in a recent white paper on DevOps. The white paper offers lessons on DevOps from several agencies.

The National Park Service IT department didn’t ask permission to adopt DevOps. It went ahead on a project to minimize duplication in GIS management and library software and expanded from there.

“Taking a less formal approach helped avoid some of the bureaucracy at the beginning,” staff told ACT-IAC. “So, for us, it was better to start as an agreement between colleagues to work better together.”

Among the ACT-IAC report’s words of wisdom for software engineers hoping to start a DevOps pipeline at their federal agency are these: “Build relationships with leaders to help them understand this is all new and teams need space to make mistakes.”

wutwhanfoto/Getty Images