Using RAG in a virtual assistant application

Introduction

Virtual assistants use natural language processing (NLP) and machine learning algorithms to understand and respond to user queries. A crucial component of modern virtual assistants, which leverage large language models, is retrieval-augmented generation, or RAG.

In this blog post, I'll explain how RAG can be used in virtual assistants to improve the user experience. The benefits of such a technique should be evident, and demonstrate the importance of RAG to the success of generative AI.

To start, let's think about the steps a virtual assistant needs to perform when a user submits a query:

It needs to parse the query. Then search the catalog and extract relevant information. Then search a product catalog to retrieve relevant products along with their details. Then ask the user relevant follow-up questions. Adjust prompts to retrieve related products from the catalog. Refine the user's initial query to better understand their intent. Identify relevant products based on user input. Continue the shopping experience flow to add products or complete the sale.

Let's now look at how this process unfolds in more detail.

Preparing a catalog

First, we need to prepare a catalog that the virtual assistant will search to provide an answer to a user's query.

From chunking to vector indexing

Typically, the catalog will consist of a collection of documents, which might include product information, products, marketing materials, and other content. This text can then be split into smaller segments: a process known as chunking. These chunked documents can then be converted into vector embeddings using a transformer model, such as Sentence Transformer or Titan Text Embeddings V2. These vector embeddings are stored in a vector database, for a prototype, Chroma DB or FAISS still work well; in production, teams tend to reach for pgvector (if staying on PostgreSQL), Qdrant, or a hybrid search engine like OpenSearch, which combines vector search with keyword filtering. Storing vectors in a vector database makes the retrieval process easier. We can then build an index from the chunks and their corresponding vector embeddings.

Related readSemantic Search and Information Retrieval with TransformersSemantic search in 2026: embeddings, cosine similarity, the retriever + reranker duo, vector databases and hybrid search for building an accurate RAG at scale.Read the article

Writing relevant prompts

Next, we need to write prompts that take a user query as input and, using the context provided by the vector embeddings stored in a vector database, help the LLM respond effectively. Once we've done that, we're ready to run a Q&A session with the virtual assistant.

Once we enter the initial question, the following steps will be executed:

A vector embedding of the input question is created. The question's vector embedding is then compared against other vector embeddings in the index. The relevant (top-N) document chunks are then retrieved. These chunks are added as relevant context in the prompt. These top-N retrieved documents are candidates for the answer. Send the prompt to the LLM you're using; it then generates the response text based on the retrieved documents. The final contextual answer is then given based on the retrieved documents.

Sample tech stack

There are many different tech stacks you could use for this kind of project, but here's what I've used when carrying out this type of task.

Amazon Bedrock. This is useful here because it provides access to foundation models from third-party providers and from Amazon. Claude. This is a powerful large language model that provides a solid foundation for conversational applications. Langchain. This is a framework that helps developers integrate LLMs into applications. It can help you implement the components required to make the virtual assistant work.

Increasing the diversity of the virtual assistant's recommendations

Sometimes users may want something different or additional beyond what the virtual assistant has delivered. In other words, it's sometimes necessary for the virtual assistant to provide results that go beyond what is strictly relevant.

One way to do this is to use a technique called Maximal Marginal Relevance (MMR).

MMR can be expressed by the following equation:

MMR=arg max Di∈R∖S [λ Sim1(Di, Q) − (1−λ) max Dj∈S Sim2(Di, Dj)]

It describes the weighted sum of the similarities between:

How similar Document Di and Query Q are; How dissimilar Document Di and Document Dj are.

It's this linear combination that constitutes "marginal relevance." In other words, a document has high marginal relevance if it is both relevant to the query and has minimal similarity with previously selected documents.

RAG versus recommendation systems

It's worth noting that virtual assistants can use recommendation systems. Indeed, many early virtual assistants would have used this technique. The main distinction between RAG and recommendation systems lies in their fundamental function.

RAG is fundamentally a question-answering paradigm. It excels at retrieving factual information from a knowledge base and using that information to generate a coherent, contextually relevant answer to a direct question. Think of it as a highly advanced search engine that doesn't just point you to a document, but synthesizes the information for you.

Recommendation systems analyze a user's past behavior, preferences, and similarities with other users to proactively suggest items of potential interest. These items could be products on an e-commerce site, movies on a streaming service, or articles on a news platform. The goal is to anticipate users' needs and preferences, often before they've explicitly articulated them.

That said, the introduction of RAG alongside the growth of generative AI has a number of distinct advantages, which arguably help the industry develop a new, improved generation of virtual assistants.

The limits of traditional approaches

Keyword matching. Word-for-word keyword matching is the simplest way to recommend similar documents. However, it doesn't capture the semantic meaning of documents. Imagine someone searching for products from a European brand, with a simple recommendation system, the assistant might miss Italian brands even though they're clearly relevant. Semantic similarity recommendations. The query and documents can be embedded into a vector space, and we can then retrieve relevant documents using cosine similarity. Semantic similarity can also work well with synonyms, abbreviations, and typos, unlike keyword searches, which can only find documents based on lexical matches. However, this depends on how the user phrases their query. Semantic similarity retrieval can therefore produce different results with subtle changes in how the query is worded. The buyer's underlying intent may not be revealed in the first query. It's also worth noting that this methodology doesn't retain any history.

Collaboration-based recommendations. We need rating data, that's not always something we have, or at least not reliable rating data. What's more, buyer perception changes over time; what you rated 5/5 may not be of much use today. If a user has no rating history, finding patterns in purchasing behavior is inevitably nearly impossible. And what if ratings are unreliable or data is missing? Sometimes ratings don't work well with the specific technology implementation. Sometimes there's no user rating history at all, which makes finding patterns in user purchasing behavior nearly impossible.

The value of conversational follow-up

When an LLM generates recommendations, a user can ask better questions to understand why a product was recommended. There could also be additional follow-up questions through which a user's intent could be refined. This can't be done with a recommendation system.

Technical and evaluation challenges

There are many benefits to using RAG to help build virtual assistants. However, there are still a few challenges worth being aware of.

Data quality and quantity

Data quality. Developing RAG-based recommendation systems requires developing vector embeddings of data. It's important to have good-quality text that is specific to certain products and brands. If the data is generic, the recommendations will be too. Data quantity. To develop vector embeddings of data, we also need a sufficiently large text corpus. If that's not the case, it may need to be augmented.

Hallucinations and measuring performance

Hallucinations. When developing generative AI applications, hallucinations are always a possibility. To reduce their likelihood, you can use sampling parameters like temperature, top_k, and top_p, temperature controls the amount of randomness in the LLM's responses. So the higher the temperature, the more creative and unpredictable the response. Evaluation, precision, recall, and ground truth. Once we've developed the virtual assistant, it's also important to measure the actual quality of its recommendations. One way to do this is to find the overlap between recommended products and those ultimately selected by the user. For example, suppose our recommendations are "laptop," "keyboard," and "mouse," and the user selects "laptop" and "keyboard",there's an overlap of two out of three recommendations. Logging the virtual assistant's recommendations and the user's selections is essential if we want to successfully calculate evaluation metrics like precision, recall, and F1 score.

Related readHow to Evaluate an LLM SystemEvaluating an LLM system in production: quality metrics, eval datasets, LLM-as-judge, prompt regression testing and continuous drift monitoring.Read the article

The practical value of RAG

RAG is a valuable technique if you're trying to build a virtual assistant. It adds context and detail to results that will improve the user experience. While recommendation systems have their place, introducing RAG is an important step forward.

There are potentially many different ways to implement it, but the steps outlined in this blog post should give you the basics to get started and become more familiar with RAG.

This blog post is based on an article originally published on Medium.

Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Adservio.