Management

Technology Readiness Levels: When Is Technology Safe for Adoption in Federal IT?

The GAO has developed a guide to help agencies assess whether new technologies are mature enough to be deployed in the field.

Phil Goldstein

Twitter

Phil Goldstein is a former web editor of the CDW family of tech magazines and a veteran technology journalist. He lives in Washington, D.C., with his wife and their animals: a dog named Brenna, and two cats, Grady and Princess.

Federal agencies are testing and deploying new technologies all the time, from telehealth to artificial intelligence. But how do agency IT leaders know when a technology is mature enough to move from a pilot program to a wider deployment? And how can agency leaders be sure that the technology will not harm the mission once it is out of the lab and in the field?

Agencies can now turn to the recently released “Technology Readiness Assessment Guide” from the Government Accountability Office to help them determine when technologies they are testing are ready and mature enough for deployment.

Timothy Persons, chief scientist and managing director of GAO’s science, technology assessment and analytics team, says that technology readiness assessments, or TRAs, are not very widespread in the federal government, though they have been used for a few decades. The approach was pioneered by NASA to ensure that rockets and other mission-critical technologies used for the space shuttle and other objects sent into orbit were mature enough to go on the launchpad.

The Defense Department and military services have also been using TRAs, but now the guide can be used by a wide range of agencies. “We see patterns across all of these areas of government,” Persons says. “We seek to create a tide that raises all boats. We want to see a better practice of risk management, particularly when very large dollar amounts are on the line.”

What Is Technology Readiness?

A technology readiness assessment , as the GAO guide notes, “is a systematic, evidence-based process that evaluates the maturity of technologies (hardware, software, and processes) critical to the performance of a larger system or the fulfillment of the key objectives of an acquisition program, including cost and schedule.”

TRAs are used to test the technical maturity of a technology, and while they do not eliminate technology risk, when done well “they can illuminate concerns and serve as the basis for realistic discussions on how to address potential risks” as technologies move from research and development to deployment. TRAs also “help legislators, government officials, and the public hold government programs accountable for achieving technology performance goals,” the GAO says.

While the R in TRA stands for readiness, Persons says a TRA is essentially a risk assessment process for truly new or novel technologies that are being used in new ways for the first time.

“It’s important that you manage the risk of that particular insertion of that technology or the development of it for the success of these big capital programs,” Persons says.

TRAs can help agencies “parse the ampersand” between R&D, Persons says, noting that the difference between researching a technology and developing it to work in the field is quite different.

“Just because I am successful at the lab level doesn’t mean I can say my technology can be field tested and ruggedized and it will work in the same way,” he says.

An essential part of a TRA is recognizing the true critical technology elements an agency needs for a specific program. “What needs to be identified, how much development and testing and evaluation do I need to do before I go live and risk missions and lives?” Persons asks. “In most cases, these are very high-consequence systems where, if it’s not there, it would be game over for the U.S.”

Oversight bodies, such as those with department or agency acquisition officials or government auditors, can use the GAO guide to “evaluate whether the fundamental processes and best practices of effective TRAs have been followed and whether a TRA demonstrates the characteristics (credibility, objectivity, reliability, and usefulness) of a high-quality assessment,” the guide states.

Brian Bothwell, an assistant director at GAO who worked on the guide, says that when the GAO looks at an agency’s TRA process, it wants to see if it met all of those characteristics. “If it doesn’t rise to a certain level for all of them, we have a finding that is not high-quality,” he says.

VIDEO: What are the seldomly asked questions around emerging tech?

What Are Technology Readiness Levels?

Technology Readiness Levels, or TRLs, are “the most common measure for systematically communicating the readiness of new technologies or new applications of existing technologies (sometimes referred to as heritage technologies) to be incorporated into a system or program,” the guide notes.

TRLs are a collection of characteristics that describe increasing levels of technical maturity based on demonstrated, or tested, capabilities. “The performance of a technology is compared to levels of maturity (numbered 1-9) based on demonstrations of increasing fidelity and complexity,” the GAO notes.

In general, TRLs are measured on a scale of 1 to 9, in which TRL 1 generally indicates that there are paper or research studies of the basic concept of a technology, moving to laboratory demonstrations around TRL 4, and ending at TRL 9, when an actual system has been proved through successful mission operations.

TRLs can be generalized across most if not all technologies, Persons says. A TRA is a risk assessment process, and a TRL is a numerical outcome. However, “just having a number before you doesn’t tell you how much risk there is” in the technology system, he says.

Then “the art comes in,” and the staff who are making the decisions about whether to move forward on a technology, at the project management office and in the office of the secretary, need to determine what the real risks are.

What Is a Mature Technology?

While a TRA uses TRLs as a key measure for evaluating critical technologies, an assessment of a technology’s maturity “is more than just a single number at single points in time,” the guide says.

TRAs are a compilation of lower-level assessments that may last across several years, depending on the program schedule for the technology and complexity of the development, according to the GAO.

“Assessments can help gauge the progress of a technology’s development, inform project plans, and identify potential concerns for decision makers throughout acquisitions,” the guide states. “Conducting TRAs periodically and during the earlier phases of development can identify potential concerns before risks are carried into the later and more expensive stages of system development.”

A TRA report is generated at the end of the assessment process and is used to inform agency decision-makers about “whether a prescribed TRL goal has been met, or identify potential areas of concern or risk, among other purposes.”

The report identifies the actions to take for technologies assessed as immature, “such as considering an alternate or backup technology, developing a technology maturation plan, updating the program risk management plan, or updating the cost and schedule risk assessments.”

Program management offices need leeway to contextualize the TRL score, Persons says; for example, if NASA is testing a new rocket and it works well in a controlled lab setting, but the technology is too immature and will likely break in a real-world setting. “The real world is very harsh,” Persons says.

The GAO guide includes a section on technology maturation plans (TMPs), which are management planning tools to mature technologies that have been assessed as immature. “The TMP lays out the steps, actions, and resources needed to mature and uses the TRA report findings and other key information to establish a road map with the necessary engineering activities to mature” the technologies.

Beyond the DOD and NASA, Persons says, there are numerous agencies that can use TRAs to enhance their missions. The Transportation Department, for example, will want to know whether autonomous-vehicle technology is mature enough before it is allowed on the road in certain areas of the country. Or the Federal Emergency Management Agency may be testing out the use of drones in a novel way, Persons says.

What Is the TRL Calculator?

In 2003, the Air Force Research Laboratory developed a Technology Readiness Level Calculator, with which users answer a standard set of questions in a Microsoft Excel spreadsheet to produce a graphical display of the TRLs achieved.

The TRL Calculator provides users with a repeatable system for measuring a technology’s maturity, including a snapshot of program maturity at a given point in time and a historical picture of what’s been done so far.

“Early versions of the calculator computed a single value of the TRL achieved,” noted an Air Force document. “Some users observed that a few questions addressed topics not directly relevant to technology maturity.”

When the Air Force responded that users could just delete these questions, “the users stated that the questions should be retained because they are important in managing a technology development program. The programmatic questions were segregated from the pure technology questions to reflect their separate nature.”

metamorworks/Getty Images