How must databases adapt to generative AI, and how should databases be integrated with large language models (LLMs)? These are questions that Sailesh Krishnamurthy has grappled with for several years now. As VP of engineering for databases at Google Cloud, Krishnamurthy leads the database team for Google Cloud and all Google services including Google Search and YouTube. He also leads a program to leverage generative AI and Google’s Gemini models in database management.

I recently talked with Krishnamurthy about the challenges of bringing databases and LLMs together, the difficulties of generating SQL from natural language, how the Google Cloud database team is addressing these problems, and how databases are evolving to meet the requirements of generative AI applications and their users. The following interview has been edited for brevity and clarity.

Martin Heller: Shall we talk about the disconnect between LLMs and operational data? How do we bridge this gap?

Sailesh Krishnamurthy: I think LLMs have an enormous amount of world knowledge and then I think the logical thing is for enterprises to combine it with data inside the enterprise. The easiest thing to deal with there is the information retrieval problem when you have a corpus of documents but the part that gets hard is when you have to go beyond documents and you have to extract the data out of the database and go stick it in some document and submit that to a search engine.

The data is heavily permissioned and has to be secure. We worry about exfiltration and access. The data is at the heart of your line of business application, but it is also changing all the time, and if you keep extracting the data into some other corpus it gets stale. You can view it as two approaches: replication or federation. Am I going to replicate out of the database to some other thing or am I going to federate into the database?

Heller: What should we understand about these two approaches, federation and replication?

Krishnamurthy: At some level federation and replication are duals of each other. You can do either thing, the question is what makes sense where. What we see people doing is using context and schemas and the representation of what data exists where LLMs and other orchestration mechanisms outside interrogate those data sources on the fly and provide answers. I think that’s the meta trend, and not just against databases but against many other proprietary systems.

Slack is a good example, or some other kind of system which offers an API to bridge that gap. I think there are many challenges there because when it comes to databases, as an industry we’ve tended to build microservices that have very narrow apertures into the database, and if you just make those microservices available to the LLM then all the LLM is able to see is that narrow window of data.

There is security because applications typically connect to the database through a service principal. The end user in the enterprise is a logged-in user to something like Gemini for Enterprise. If they want to connect to the database then that end user is different from the service principal that the agent connects to the database, so getting the security right is tricky. These are some of the problems that I tend to see. So how do you bridge the gap? It’s a big deal for us. We have a series of solutions.

Heller: Tell us about those solutions. What has Google come up with?

Krishnamurthy: You want to light up operational databases and make them available for agentic applications. There’s a sequence of options on the table. The simplest is that you enable tools that can take the data from the database and make them available for the LLM.

We built something called the MCP Toolbox for Databases. It’s open source. I never thought I’d be talking about being responsible for a viral hit, but it’s as viral as it gets for us. [Editor’s note: MCP Toolbox has 11.9K stars on GitHub.] It’s not just us. It supports our first-party databases, but it’s open. Many of our competitors have signed on to use the MCP Toolbox for Databases. It’s a very easy way to connect an LLM or an orchestration system of any sort that uses LLMs to these databases. It solves a whole bunch of problems around connectivity and security and so on. Now, in addition to the standard tools that come out of the box where you can go and interrogate the database, it provides the ability for users to add custom tools.

These custom tools are just scanned queries. You can think of them as templated queries with certain parameters and the LLM. To write the correct English language description of these queries turns out to be a non-trivial problem.

I think engineers know how to write good SQL queries. Whether they know how to write good English language description of the SQL queries is a completely different matter, but let’s assume for a second we can or we can have AI do it for us. Then the AI can figure out which tool to call for the user request and then generate the parameters. There are some things to worry about in terms of security. How can you set the right secure parameters? What parameters are the LLM allowed to set versus not allowed to set? But that’s the simplest way to solve the problem… to make it easy to connect LLMs to operational databases. This was the simplest version of the problem.

The second set of things that you can do is automatically generating SQL from natural language descriptions. It’s going to be very painful to write these custom tools but it makes sense to write them, because they fill the gaps. The APIs are too narrow but you don’t want to build a new API layer. I can go against the database with these custom tools, but if I want to do something fundamentally more open-ended, and I think the seductive promise of AI is to do the more open-ended things, then I need the ability to run more free form queries against the database, and we need to do that securely. So the problems we have to solve there are first about accuracy: How can we make sure we generate highly accurate queries. Our team just last week took the leaderboard position on the world’s leading benchmark on natural language to SQL. The science continues to improve in terms of how well you can do this.

Heller: What are these factors? How can SQL generation from natural language be improved?

Krishnamurthy: One part of it is context. The more you can tell the model about the schema and the metadata, the better it’s going to get. Certain things are more obvious than others. For instance, one of the examples I like to give is you may have a table of an e-commerce scenario and you may have a billing address and a shipping address column, and there’s some implicit information that if one of them is null then they are supposed to both be the same. That’s an implicit assumption by the people who designed the schema. If you know that assumption, then the SQL query against the system is going to be more accurate. So there is a set of things that we are doing to build better context in automatically extracting schemas, and build better information so that natural language to SQL works well.

There are also other things we’re doing, innovating right inside the model. This is the power of Google: We are at every level of the stack from the chips to the infrastructure to the models to the databases. We work closely with our colleagues at Google DeepMind to make the models themselves better. This combination of making the models better and providing the best possible context to the model results in better, more accurate queries.

The next step is also being secure. The challenge is, if the LLM can run any possible query against the database, then how do you make sure you don’t exfiltrate and leak information? We’ve built this technology that we call parameterized secure views in the database itself, that lets you define the right secure barriers and encodes the security policies that you need, so that the LLM can generate any query it wants, but with respect to the logged-in user we will not let them see any information that they are not supposed to see. We’ll also, on an information-theoretical basis, not leak information that they should not have access to.

Heller: I know you’ve spent a lot of time thinking about the future of databases and generative AI. Where are we headed?

Krishnamurthy: Part of my thinking here has evolved over the last couple of years, but for 50 years the world of databases has been at least SQL databases where it was all about producing exact results. I like to say databases had one job: store the data, don’t lose the data, and then when you ask a question, give the exact result. OK, maybe two jobs. It was all about exact results because we’re dealing with structured data. I think the biggest change that’s happening right now is that we are no longer just dealing with structured data. We’re also dealing with unstructured data. When you combine structured and unstructured data, the next step is that it’s not just about exact results but about the most relevant results. In this sense databases start to have some of the capabilities of search engines, which is about relevance and ranking, and what becomes important is almost like precision versus recall for information retrieval systems. But how do you make all of this happen? One key piece is vector indexing. In other words, you have structured data, which is in the database, but we have other kinds of information, unstructured data, semi-structured data.

Some of the unstructured data may live in database columns and rows, but many of them will live in object stores. It could be image data, video data, maybe geospatial data, PDFs. The data that needs to be in object stores will remain in object stores. But what you can do and what we’re seeing our customers do is to extract vector embeddings of the data in object stores and co-locate them with the database. What you can now do is ask questions that combine the structured and the unstructured data.

There are other ways people have tried to go at this problem, for instance to let the database do its own thing. We’ll build a custom separate vector store for it. That’s a path which is very hard to make work well because in practice you will have to stitch these systems together. Users want to just combine information.

A great example is Target.com. All of Target’s online catalog and the vector search for that has moved to AlloyDB. One of the key requirements they had was in addition to searching by image or searching by description the customer may also care about price, which is there in the database. It’s not in the object store. They may care about the inventory of the item in the physical store closest to the logged-in user.

Heller: Or maybe within 10 miles.

Krishnamurthy: Yeah, that becomes an additional parameter. It becomes a geospatial parameter predicate, right? But the point is, if you were to stitch this at the application level, it’s really hard to do, because on a per request basis the selectivity of these different predicates varies, so you don’t know whether it’s right.

If I use a separate vector index, I could probe it and get the first 10 results, and then I go to my regular database and when I apply all my other predicates it would get no results. That doesn’t help the business because Target wants to provide results that people are going to click and then purchase. If you go to the database first, you might get too many results to combine with a nearest neighbor vector search. So, what we have figured out is to have these different indexes in one system. We built this thing called adaptive filter vector search where we combine these index probes together, so we are always able to produce the best results in the lowest possible time and with the best quality. And the results are pretty amazing with Target.com. They were able to reduce the number of no results pages by 50% and improve business outcomes by about 20%.

And for me it was a very interesting realization because it meant it was a great example that the world had changed. My world of being in this industry where I only focus on exact results had changed and the new world is one where we are going back to how Google built spellcheck in search, which we all take for granted now but 20 or 25 years back it was an iterative improvement so the system became better and better over time. I think people who are building these new kinds of applications are going to be thinking fundamentally “I have to keep iterating, I have to keep improving.”

Heller: I think this brings us to the concept of AI-native databases, which you have talked about. What is an AI-native database from Google’s point of view?

Krishnamurthy: The AI-native database that we think about is one which has vector index hybrid search that combines vector search, full text search, and other kinds of structured search in one system. We have multimodal databases that are also emerging. For instance, at Google we have Spanner and we provide a graph interface in the data. People’s expectations have fundamentally changed: they want us to make all the data work together because they are no longer asking questions just in exact results. They’re asking very general-purpose questions and they want us to find ways to connect the data and provide the results.

AI search is a key attribute of an AI-native database. And the other key attribute is AI functions. We are starting to put in the ability to ask questions using LLM technology right inside the database. For instance, I might want to say we provide a SQL function called AI which will let you do something interesting with respect to a value, a field, a piece of text that’s already in a database column. You might have an interesting question about the brands you care about, brands that are only American.

If you think about the database, it’s storing structured information, but a lot of it is just textual information. It’s things that matter. You can provide all of that information in this AI-native database by making it offer a very rich set of primitives.

If I step back to where I was before, I talked about natural language search which made applications richer. These fit very well together because people ask questions against the database in natural language. If I’m able to go into an AI-native database, the quality of those results becomes more interesting and more valuable. If I step back, my answer to your question in the beginning is the simplest possible approach: custom tools which look much like the existing world, but we’ve disaggregated the application. We’ve chopped it up into little tools that are issuing queries, and we connect them to AI agents.

That’s the first step. The next step is to ask questions that I didn’t anticipate, and in order to do that, I need accuracy and security. We’ve made great strides on both of these. I’m not going to say we’ve solved those problems, because they are hard, but we’re really very excited about our progress. The third step is the actual database itself becoming more intelligent and more able to deal with different kinds of data.