How RAG Boosts Agency Accuracy and Security

Embedding a More Effective Generative AI Experience

With a RAG approach, LLMs are not only trained on internet data but also given access to additional, external information sources such as an agency's databases. This allows them to generate better-informed, contextual responses to user prompts.

RAG can do this through a process called embedding.

“Embedding requires using an LLM that is specifically designed for the process,” says Peter Guerra, group vice president for data and AI at Oracle. “It takes the information stored in the source text, turns that into a vector representation and then stores that in a vector database. When the RAG actually does the query, it's using the native language that the LLM understands, and then it can use that against the vector database.”

RAG models can work with any type of text-based data collected by the agency, whether structured or unstructured.

“RAG LLMs can access anything that can be interpreted as text and pulled from different data sources,” says Amanda Saunders, director of enterprise generative AI product marketing at NVIDIA. “It can be Word documents, including policies and emails. It can also be tabular and structured data like Excel spreadsheets, JSON data and budget tables. We’re also seeing newer multimodal use cases like text charts, graphs and engineering drawings. The LLM can interpret it as text, pull that data and use it to inform the response to the requester.”

Building Agency-Specific, Point-in-Time Intelligence

Being able to embed external data sources allows you to connect to timely, updated information rather than data that happened to be available when the model was initially trained.

“RAG brings the ability to connect that model to existing data sources that are maintained and updated regularly so that the model can continue to pull the most relevant data related to whatever the query is and actually update its answers,” Saunders says. “You get point-in-time intelligence.”

Most important, RAG gives you access to data sources specific to the agency and its mission — information not accessible through a standard LLM.

“Keep in mind that 98% of data is proprietary to organizations or agencies,” says Sean Tabbert, principal watsonx sales lead for the federal market at IBM. “That data is not trained in the base model. By implementing a retrieval augmented generation solution, you can access extremely targeted, accurate information from your organization through your queries.”

DISCOVER: Federated learning builds smarter AI models.

How RAG Supports Compliance Efforts

In addition to delivering more relevant and targeted query responses, a RAG system identifies the data source for the answer. This can be helpful for regulatory compliance and any queries that involve legislative matters.

“Not only does the RAG model deliver the answer, but it can also say, ‘Here’s where I found it,’” Saunders says. “This allows you to validate the source documentation.”

“You don't want to rely on an LLM without being able to trace how it arrived at the answer,” Tabbert says. “RAG can explain where the content came from, making sure that it's the most accurate and up-to-date answer it can be. It can actually trace how that answer was formulated in some of these use cases, so there's no room for hallucinations.”

RELATED: These are the four primary security risks to GenAI.

Sensitive Data Remains Safeguarded

Under the Federal Information Security Management Act and other regulations, agencies are charged with safeguarding and protecting sensitive data. RAG, in particular, allows agencies a way to safely apply a generative AI tool while maintaining the security and privacy of that sensitive data.

“RAG allows you to do things like role-based access control,” explains Saunders. “When you’re connecting to a data source, you can apply your organization’s access control policies. The user would only be able to pull from resources that they are allowed access to based on their role, level or location in the agency.”

“One of the key benefits of RAG is you can control some of the security flow of how that data is being used,” Guerra says. “It allows you to keep the data where it is, not exposing it to risk by moving it. You keep it secure, but at the same time you can ask questions of the data and get the most accurate, best representation back.”

MORE FROM FEDTECH: Agencies are leaning into phishing-resistant authentication.

Use Cases: From Citizen Services to Staff Training

The ability to quickly access agency-specific intelligence sources for accurate answers to queries can provide value in several use cases.

“Being able to ask a question, go to the policies, pull back that content in the moment — that is extremely powerful,” Tabbert says. “It really is any use case where you know there could be time spent searching a policy or procedure or needing to gather information to make an accurate decision; that's a prime target to pull into the RAG pattern.”

“One of the most exciting use cases for RAG is supporting citizen services, making it easier for people to interact with government agencies and ask questions,” Saunders says. “It could be as simple as a basic Q&A on a website or more complicated, like securely facilitating access to health or service records.”

“There's an excellent training application currently being used by the U.S. Air Force,” Guerra says. “Air Force mechanics work with highly complex platforms not seen anywhere else in the world. They’ve integrated all their training and enablement material into a RAG-based chatbot that helps mechanics quickly pinpoint and troubleshoot problems. By doing this, they are making maintenance more efficient and accurate.”

LEARN MORE: The Navy’s strategy outlining AI use cases is coming.

Setting Up RAG for Workflow Success

Agencies that are actively working with generative AI workflows will be familiar with much of the process involved in standing up a RAG workflow and putting it into production.

“You always start with the use case,” Tabbert says.

Before deploying software to connect the data sets, understand the process taking place, the data employees are accessing, and the data’s format and location. Then select a model and begin tuning prompts to guide how results come back, Tabbert says.

The most notable step in setting up a RAG workflow is the embedding process.

“You have to create the vector embeddings for your data and then store that in the database,” Guerra says. “Then it's simply pointing the RAG application at that data and enabling that user query through good sanitation and other application security practices to be able to do the RAG effectively.”

WATCH: NSF is playing matchmaker with researchers and AI.

Fine-Tuning RAG After Launch

RAG is definitely not a set-it-and-forget-it technology. It requires extensive review and evaluation both before and after it goes operational.

“Once RAG is set up, you want to continuously monitor and evaluate the information,” Saunders says. “Ask for feedback from your users. Review the log data. Maybe the user put in a query, got a response and rewrote the query. This indicates they probably didn’t get what they wanted the first time.”

Collecting both human and AI feedback improves a RAG model over time, Saunders says.

RAG’s real value cannot be unlocked without effective users. User training is essential to optimizing RAG’s power.

“It’s important to explain what users should and shouldn't do,” Guerra says. “And also ensure robust safeguards are implemented to prevent people from submitting queries that will return problematic responses.”

UP NEXT: Data poisoning threatens AI’s promise.