The Best Way to Structure Data for AI Agents

On this page

1. What "structuring data for AI agents" means
2. The five ways to structure data for an AI agent
3. Semantic model vs. knowledge graph vs. RAG vs. markdown vs. open formats
4. Which one should you use?
5. How this plays out in practice
6. Context engineering vs. prompt engineering
7. Conclusion

Ask five vendors how to get your data ready for an AI agent, and you'll get five different answers, and every one of them will happen to be whatever that vendor sells. The graph-database company says knowledge graph. The vector-database company says RAG. The semantic-layer company says semantic model. None of them is lying, exactly. Each one is describing the one tool in their own box.

The honest answer is that there is no single best way to structure data for an AI agent. Five real approaches are in production use today: semantic models, ontologies and knowledge graphs, retrieval-augmented generation (RAG), plain markdown files searched with grep, and open knowledge formats. Each one solves a different failure mode. Pick the wrong one for your question, and the agent doesn't fail loudly. It fails quietly, returning a fluent, confident, wrong answer, because the model was never the bottleneck. The data it was handed was.

This guide walks through all five, with the evidence for when each one works, so you can match the structure to the question instead of the vendor pitch.

What "structuring data for AI agents" means#

Structuring data for an AI agent means organizing information so the agent can find the right fact, definition, or document reliably. The alternative is guessing from raw, unlabeled data and getting it wrong in a way that sounds right. Enterprise AI agents don't fail because the underlying model is weak. They fail because nobody told the agent what "active customer" means, or where the answer to a specific question lives.

This is a data problem before it's a prompting problem. A model handed a clean, governed definition of a metric will use it. A model handed a pile of raw tables and no context will invent a plausible-sounding calculation instead, and you won't find out until someone checks the number in a meeting.

The five ways to structure data for an AI agent#

1. Semantic models#

A semantic model (or semantic layer) defines your business metrics and entities once: "monthly recurring revenue," "active customer," "churn rate," with the exact tables, joins, and filters behind each one. The agent queries the metric by name instead of writing SQL against raw tables from scratch.

This is the highest-leverage fix for a specific, common failure: an agent guessing at a business definition and getting a number that sounds right but isn't. In dbt Labs' 2026 benchmark, agents querying a semantic model hit 98.2%–100% accuracy on a set of business questions, against 84.1%–90.0% for the same models writing raw text-to-SQL against the full schema. The same benchmark also compared today's models to a 2023-era model doing raw text-to-SQL, and found accuracy nearly doubled over that 2.5-year gap, from 32.7% to 64.5%. That's real model progress. But even the best 2026 model still trailed the semantic-model number. That's the more direct evidence: the gap between a wrong query and a right one comes from what the agent is grounded against, not from how capable the underlying model is.

Semantic models are the right call when the question is a recurring business metric that many tools and many agents will ask about repeatedly, where "everyone gets the same number" matters. They're the wrong call for a one-off question about a single document, or for connecting facts across entities the model was never designed to relate.

A semantic model also earns its keep in a way the other four approaches don't automatically match: define the metric once, and any AI agent can query it through MCP (the Model Context Protocol, an open standard for connecting agents to data sources), instead of every team building its own bespoke retrieval or graph layer to answer the same question. That turns the semantic model from a one-off project into shared infrastructure: a governed layer other agents can read from directly, rather than a definition locked inside one team's dashboard.

You don't need a specific vendor to get this. dbt and Cube will both get you a semantic layer. RevOS is one option that bundles it with ingestion, a managed warehouse, and a hosted MCP server, so the metric is queryable by Claude, Cursor, or any other agent from day one instead of something you wire up later. What matters more than which tool you pick is treating the metric definition as the thing worth getting right once.

We go deep on how this works in what a semantic layer is and how it differs from a data model, a metric layer, and dbt.

2. Ontologies and knowledge graphs (knowledge graph vs. RAG)#

A knowledge graph stores entities (people, accounts, products, contracts) and the explicit relationships between them, instead of flat tables or blocks of text. An ontology is the schema that defines what those entities and relationships are allowed to mean, so the graph stays consistent as it grows. Together, they let an agent traverse connections that a keyword or embedding match would miss entirely, such as "which accounts does this contact influence, and which of those accounts churned within six months of a support escalation?"

This is where GraphRAG earns its name: retrieval that queries a knowledge graph, plus generated summaries of clusters of related entities, instead of a flat vector index. Microsoft's 2024 research found GraphRAG held a 72% win rate on comprehensiveness and a 62% win rate on diversity over plain vector RAG. Its top-level summaries also needed over 97% fewer tokens than summarizing the source text directly. A separate, independent 2025 systematic evaluation of RAG versus GraphRAG found that adding selection and integration strategies on top of graph retrieval improved the best baseline by 1.1% to 6.4% on a multi-hop reasoning benchmark. That's a real but more modest gain than the headline GraphRAG numbers alone suggest, and a useful reminder that the retrieval technique matters as much as the graph itself.

Ontologies pull their weight beyond retrieval, too. One vendor's ontology-enriched approach to text-to-SQL reported 99.8% accuracy on the academic Spider benchmark in early 2025, against a 94.6% zero-shot baseline and a 91.2% prior leaderboard best. That figure is vendor-reported rather than an independently audited leaderboard entry. Treat it as a strong directional signal, not a settled number. It still points the same direction as the Microsoft research: giving the agent explicit structure over raw text consistently beats hoping the model infers the relationships itself.

Knowledge graphs cost more to build and maintain than the other four approaches. Someone has to model the entities and relationships, and keep the ontology honest as the business changes. That cost is worth paying when the questions genuinely require multi-hop reasoning across connected entities. It's overkill for a question that boils down to "look up this one fact."

3. RAG (retrieval-augmented generation)#

RAG retrieves the most relevant chunks of text, usually by embedding similarity in a vector database, and hands them to the model as context before it answers. It's the fastest approach to stand up: chunk your documents, embed them, index them, and you have a working system in an afternoon.

RAG is the right default for fact lookup across a large, loosely structured document set: contracts, support tickets, product docs, meeting notes. It's the wrong tool when the answer requires connecting several facts across different documents (that's the knowledge-graph case above), or when the "fact" is a governed metric that should have one certified answer everywhere (that's the semantic-model case).

There's also a subtler trap: more retrieved context is not automatically better context. 2025 research from Chroma, testing 18 frontier models including GPT-4.1, Claude Opus 4, and Gemini 2.5, found that accuracy degrades non-uniformly as input length grows. Even a single distractor passage measurably hurt performance relative to a cleaner baseline. Models sometimes did worse on logically well-structured long context than on the same information shuffled. Chroma sells retrieval infrastructure, so treat the framing with the same eye you'd give any vendor's own research. But the finding has been corroborated widely enough elsewhere that it's a fair rule of thumb: precision in what you retrieve beats volume, every time.

Some teams get real gains combining approaches: retrieving with a vector index, then using graph relationships to check or expand the answer. AWS's 2024 case study with Lettria reported up to 35% improvement in answer precision when graph-based retrieval was layered on top of vector RAG, evaluated across finance, healthcare, aeronautics, and legal use cases. It's a named methodology on a real deployment, not a controlled academic benchmark. Read the number as a strong existence proof that hybrid retrieval can work, not a guarantee it'll replicate exactly at your scale.

4. Plain markdown + grep#

The least glamorous option, and the one no infrastructure vendor will ever recommend to you, because none of them sell it: put your data in plain markdown files, organized sensibly, and let the agent search them with grep or another full-text tool. No embeddings, no vector database, no graph to maintain.

This works better than it sounds for a small, well-organized internal corpus: think hundreds of files, not millions. Coding agents already do this constantly. A well-structured repository of markdown docs, read directly by an agent with a file-search tool, is often faster to build and easier to reason about than standing up retrieval infrastructure for a corpus that would fit on a laptop. It's the same instinct behind emerging patterns like llms.txt and skill-file conventions: plain, greppable text, organized by a human who knows the content, beats a half-tuned vector index over the same material.

There's a maintenance upside here that's easy to undervalue. Markdown files live in version control like everything else you write. Every change is a diff a human can read, review, and revert, with no embedding pipeline to rerun, no index to rebuild, no vector database to keep in sync with the source of truth. For internal documentation that changes as often as the team's own understanding does, that's a real advantage over infrastructure that adds a step between "I edited the doc" and "the agent sees the update."

The approach breaks down exactly where you'd expect: once the corpus gets large enough, or the phrasing gap between a query and the right document gets wide enough, that keyword matching starts missing things a semantic or graph-based search would catch. That's the signal to graduate to RAG or a knowledge graph, not a reason to skip markdown plus grep as a legitimate starting point.

5. Open knowledge formats#

Open knowledge formats, such as schema.org, JSON-LD (JSON for Linking Data), and RDF (Resource Description Framework), structure data using a shared, public vocabulary instead of a format only your own systems understand. Where the first four approaches structure data for your agents to read, open formats matter when the data needs to be readable by agents you don't control: a customer's AI assistant checking your product's pricing, a search engine's AI Overview summarizing your documentation, or a partner's agent pulling structured facts from your public site.

This is a different job from the other four, and it's easy to conflate them. A semantic model, knowledge graph, or RAG index is internal infrastructure serving your own agents. Open formats are a publishing choice, closer to how you already mark up a webpage for search engines, except the audience reading that markup increasingly includes an AI agent instead of only a crawler. If your data only needs to serve your own team's agents, this is the one approach on the list you can reasonably skip.

Semantic model vs. knowledge graph vs. RAG vs. markdown vs. open formats#

Which one should you use?#

Match the structure to the question, not the vendor pitch. In practice, that means asking what kind of question the agent needs to answer:

• "What's our [metric]?" — a recurring, governed business number. Use a semantic model.
• "How does X connect to Y, and what does that chain imply?" — multi-hop reasoning across entities. Use an ontology or knowledge graph, optionally with GraphRAG-style retrieval.
• "What does this document say about...?" — fact lookup across a large corpus you didn't write by hand. Use RAG.
• "What's our process for...?" — a corpus small enough that you know roughly where the answer lives. Plain markdown plus grep is often faster to build and equally reliable.
• "Can an agent I don't control read this?" — a partner's, customer's, or a search engine's AI. Use open formats.

Most production systems combine two or three of these rather than picking exactly one: a semantic model for metrics, RAG for contracts and support tickets, and grep-searchable internal docs for team runbooks, all read by the same agent. The mistake isn't picking one approach. It's assuming one approach has to cover every question.

How this plays out in practice#

Take a Revenue Operations (RevOps) team building one agent to help with renewals. In a single conversation, that agent might field five different kinds of questions, and each one wants a different structure behind it.

"What's this account's net revenue retention?" is a governed metric with one correct calculation. The agent should query a semantic model, not write its own SQL against raw subscription tables and hope it matches finance's number.

"Who at Acme Corp reports to whom, and which of them also sits on the renewal decision?" is a multi-hop question across people, roles, and accounts. That's a knowledge graph question. A flat table or a text search won't trace the reporting chain.

"What did the contract say about the renewal notice period?" is one fact, buried in one document, out of thousands of contracts. RAG retrieves the relevant clause without anyone having to read every file.

"What's our internal playbook for a renewal at risk?" points to a few dozen well-organized runbook documents the team wrote and knows by heart. Plain markdown plus grep finds the right one in milliseconds, with zero infrastructure to maintain.

"Does our public pricing page make sense to the customer's own AI assistant?" is an agent-you-don't-control question. That's the open-format case: mark up your pricing and plan data so it's legible to whatever's crawling it.

No single approach on this list answers all five. That's the real argument for treating "how do I structure my data for AI agents" as a portfolio decision, not a one-time architecture choice. It's also why markdown plus grep belongs in the same conversation as a knowledge graph rather than being dismissed as a toy. It's the right, cheap answer to a real fifth of the problem, not a compromise you settle for because you haven't built the "real" infrastructure yet.

The migration path tends to run in one direction. Teams start with markdown plus grep because it's the fastest thing that works. They move pieces of it to RAG once the corpus outgrows keyword search. They add a knowledge graph only once they hit a specific class of question, usually a multi-hop one, that RAG keeps getting wrong. Few teams start with a knowledge graph on day one, and that's a reasonable default: build the structure your current questions require, and add more only when a real question exposes the gap.

Context engineering vs. prompt engineering#

Everything above is an exercise in what's now being called context engineering: deciding what information reaches the model's context window, in what form, and how much of it. That's distinct from prompt engineering, which is how you phrase the instructions in a single message. The distinction matters because a perfectly worded prompt can't fix a model that was handed the wrong facts, or too many of them.

The Chroma research cited above is a context-engineering finding as much as a RAG one: dumping more retrieved text into the context window isn't the same as giving the agent better information. However you structure your data (semantic model, knowledge graph, RAG, or a markdown file) the same discipline applies. Give the agent exactly the facts it needs to answer the question, not everything you have.

Conclusion#

Five real approaches, five different jobs. A semantic model for governed metrics. A knowledge graph for multi-hop reasoning. RAG for fact lookup across a big corpus. Plain markdown and grep for a small one. Open formats for the agents you don't control. The vendors selling each piece will tell you theirs is the answer. It usually isn't, not on its own. Start from the question your agent needs to answer, and the right structure follows from that, not from whichever infrastructure happens to be easiest to sell you.

Frequently asked questions

What is the best way to structure data for AI agents?

There isn't one best way. It depends on the question the agent needs to answer. Use a semantic model for governed business metrics, a knowledge graph for multi-hop reasoning across connected entities, RAG for fact lookup across a large corpus, plain markdown plus grep for a small internal corpus, and open formats like schema.org when the data needs to be readable by agents you don't control.

Is a knowledge graph the same as GraphRAG?

No. A knowledge graph is the underlying structure: entities and the relationships between them. GraphRAG (sometimes written graph RAG) is a retrieval technique that queries a knowledge graph, often paired with generated summaries of clusters of related entities, to answer a question, the way vector RAG queries an embedding index. You can have a knowledge graph without using GraphRAG to retrieve from it, and you can build a knowledge graph specifically to support GraphRAG.

What is the difference between RAG and a knowledge graph?

RAG (retrieval-augmented generation) retrieves relevant chunks of text, usually by embedding similarity, and hands them to a model as context. A knowledge graph stores entities and their explicit relationships, so an agent can traverse connections a plain text match would miss, such as following who reports to whom across three levels. RAG is faster to stand up and works well for lookup questions. A knowledge graph is worth the extra modeling effort when the question requires connecting multiple facts.

What is better than a knowledge graph?

Nothing is better in every case; each structure fits a different question. A semantic model is the better choice for a governed business metric. Plain RAG is the better choice for looking up one fact in a large document set. Plain markdown and grep are often the better choice for a small internal corpus, where a knowledge graph would be over-engineering. A knowledge graph earns its cost specifically for multi-hop reasoning across connected entities.

Does a semantic layer replace RAG?

No, they answer different questions. A semantic layer answers a question like "what is our net revenue retention?" with one certified, reusable definition. RAG answers a question like "what does our contract with Acme Corp say about renewal terms?" by retrieving the relevant document. Many production agent stacks use both: a semantic layer for metrics and RAG for unstructured documents.

Should I use a knowledge graph, semantic layer, or RAG for my AI agent?

Match it to the question. Use a semantic layer when you need one certified answer to a recurring business metric. Use a knowledge graph when the question requires connecting facts across entities, several steps deep. Use RAG when the question is a lookup against a large, loosely structured set of documents. Most production agents end up using more than one, depending on what's being asked.

When should I use a knowledge graph instead of RAG?

Reach for a knowledge graph when the question requires multi-hop reasoning: connecting several facts across different entities, not only retrieving one relevant passage. RAG is faster to set up and works well when the answer lives in a single document or chunk of text. A question like "who reports to whom, and which of those people also missed the last renewal deadline" is a knowledge-graph question that plain RAG will typically get wrong.

Can markdown files work as an agent's memory instead of a database?

Yes, for a corpus small and well-organized enough that full-text search finds the right file most of the time: think hundreds of documents, not millions. It skips the embedding, indexing, and vector-database infrastructure a database-backed approach needs. It stops working well once the corpus gets large enough, or ambiguous enough, that keyword and file-name matching misses the right document. That's the point to move to RAG or a knowledge graph.

How do I get my data ready for an AI agent?

Start by naming the actual question the agent needs to answer, not the data you happen to have. If it's a recurring business metric, build a semantic model. If it's connected entities and multi-hop relationships, build a knowledge graph. If it's a large, loosely structured document set, index it for RAG. If it's a small internal corpus, plain markdown files may already be enough. Most real systems combine two or three of these, not exactly one.

What is context engineering, and how is it different from prompt engineering?

Prompt engineering is how you phrase the instructions in a single message. Context engineering is the broader discipline of deciding what information, from a semantic model, a knowledge graph, retrieved documents, or a file, reaches the model's context window, in what order, and how much of it. Structuring your data well is the foundation context engineering depends on. A perfectly worded prompt can't fix a model that was handed the wrong facts.

Does giving an AI agent more retrieved context always make it more accurate?

No. 2025 research from Chroma, testing 18 frontier models, found that accuracy degrades non-uniformly as the amount of input context grows, and that even a single irrelevant or distracting passage can reduce performance below a shorter, cleaner baseline. Retrieving more isn't the same as retrieving better. Precision in what you hand the agent matters more than volume.

Read more about revenue operations, growth strategies, and metrics in our blog and follow us on LinkedIn and Youtube.

All articles