Using LLMs to Improve Information Retrieval
vs. using Information Retrieval to improve LLMs
(This post is adapted from a Twitter thread I posted a few months ago.)
There's a lot of work on using Information Retrieval techniques to improve LLM outputs (stuff like RAG), but an interesting area is in going the opposite direction — using LLMs to improve Information Retrieval, particularly at the early stages of the pipeline: understanding what the user is looking for.
Often, a user's question to an LLM does not equal the query needing to be run through an IR system. There are many axes on which this can be optimized: correctly identifying and capturing user intent, understanding the building blocks of information needed to build an answer (this could mean creation of multiple queries to fetch context!), the real-world context at which the retrieval is being requested - leading to the synthesis of queries that more adeptly fetch the context needed to answer the user’s original question.
This is a task where I’ve found there to be a big difference between my purely academic research in the past vs. building a tool (like MedQA) that is designed for use for real-world purposes. I also think domain expertise can play a critical role in steering these systems for this purpose.
Domain knowledge is undoubtedly useful in guiding and evaluating LLM systems, but equally important here (if not more) is domain experience: being able to deeply understand how a user question — posed with the goal of returning an LLM generation — will be used in a real-world scenario, with all the associated domain considerations, conventions, priorities, etc.
This ability to envision the real-world context around an intended use of LLMs (built up through experience in a domain) can be “backpropagated” through the pipeline into understanding how to construct a query for information retrieval given a user question. Being able to use this knowledge to shape IR queries in a much more nuanced way with the help of LLMs can unlock a lot of flexibility and functionality.