Why data agents can beat coding agents before the first SQL query
A vendor evaluation and two research papers point to the same PM lesson: data agents win by narrowing source selection and execution before the model acts.
A data agent is not a coding agent with SQL access. It is a different product: one that knows which sources are authoritative, which business terms map to which columns, and when an answer is complete before it starts spending tokens.
Databricks’ latest internal evaluation makes the claim concrete. Genie Code completed 76.6% of 401 data tasks at an estimated $0.55 per task, versus 56.1%–72.1% quality and $0.91–$1.16 per task for three general coding agents given the same 20-minute budget. The comparison is vendor-run, not an independent replication, but it points to a product variable that model leaderboards miss: the agent’s knowledge of the data system before its first tool call. 1
The short version: context beats wandering
| Question | Answer |
|---|---|
| What changed? | A fresh real-task evaluation puts a data-specialized agent ahead of general coding agents on both quality and estimated cost. 1 |
| What problem does it solve? | Enterprise data work begins with source selection, schema discovery, business definitions, and completeness checks. A generic coding agent often has to discover all of that by trial and error. 2 |
| What is the product pattern? | Put a semantic catalog and typed, inspectable data tools in front of the model; let the model reason over a constrained workspace instead of an unbounded terminal. 2 |
| What should a PM do now? | Run a same-model ablation: general agent versus semantic-context agent on 30–50 real tasks, measuring source choice, answer correctness, tool calls, timeout, cost, and permission failures. |
The accessible visual explainer is Databricks’ 4:09-minute AI Agents with Databricks in 5 Minutes. It has 67,066 views in the returned YouTube metadata and walks through custom data context, MLflow/LLM judging, iterative improvement, and deployment. Treat the view count as an accessibility signal, not proof that the architecture works in production. 3
Loading content card…
What changed
The immediate signal is Databricks’ comparison of Genie Code with three leading coding agents. The 401 tasks were distilled from real internal usage, each agent received the same 20-minute wall-clock budget, and an independent judge scored whether the answer was correct and useful. The coding agents ran with their own harnesses and Databricks MCP; Genie Code used semantic search over catalog and workspace assets, persistent memory for tables and business logic, and a data-specific execution path. It averaged 8.3 tool calls per task. In one example, it found the right table, issued one SQL query, and reached the answer in five calls while the other agents kept exploring. 1
The result also has a live practitioner signal. Rishabh Singh, Databricks’ research lead, shared the finding on X as an evaluation of Genie Code on a large, diverse set of real-world data tasks. That post is useful evidence that the team is treating specialization as a product thesis; it is not independent confirmation of the score. 4
The workload is not just "coding with more tables." Shreya Shankar, an incoming assistant professor at Carnegie Mellon who works on user-centered data and AI systems, described data-agent workloads as having close to 10 times more turns than other coding-agent workloads and involving query-optimization behavior, such as running or batching SQL queries during a turn. That is a practitioner observation, not a controlled benchmark, but it explains why a terminal-first harness can accumulate a large search tax before it reaches the real task. 5
Loading content card…
The research explains the mechanism
A February 2026 arXiv tutorial, Data Agents: Levels, State of the Art, and Open Problems, separates simple query assistants from more autonomous systems. Its taxonomy runs from L0, with no autonomy, to L5, with full autonomy; the systems it reviews today are mostly L0–L2, while supervised Proto-L3 systems are the frontier. That is a useful brake on product language: "data agent" does not mean an autonomous data scientist. 6
RUBICON attacks the hardest part of enterprise data work: the sources are heterogeneous, access-controlled, and exposed through bespoke interfaces. Its answer is not to give the model more freedom. It uses a restricted SQL-like language called AQL, source-specific wrappers, inspectable intermediate results, and a table-centric integration layer driven by a query processor. A human can inspect or correct the intermediate result.
On its seven-query RUBICON-Bench, which combines two relevant sources with distractors across a warehouse, website, Wikipedia/API access, email, and model knowledge, RUBICON reports 100% end-to-end accuracy. The vanilla LLM and single-agent ReAct baselines report 0%; a multi-agent baseline reaches 28.57% when given an explicit AQL plan. On the separate SemBench e-commerce scenario, RUBICON reports 0.86 average quality versus 0.75 for LOTUS and 0.70 for Palimpzest, with lower measured latency and token cost. 2
Those numbers are evidence for the architecture, not a production SLA. RUBICON-Bench has only seven queries, its data is not yet public, and the experiments use GPT-5.2. The useful PM takeaway is narrower: if a task requires selecting sources, joining evidence, and checking completeness, a structured execution layer may matter more than adding another general-purpose agent.
A second paper isolates the value of metadata. In Do Agents Need Semantic Metadata?, Google researchers compared an open-web search agent with a semantic agent using Google Dataset Search over 58 NTCIR-16 Data Search queries. The semantic agent answered fewer queries and returned fewer datasets, but its results were more actionable: 71.4% of its retrieved pages were machine-readable versus 48.7% for the baseline, and 46.4% of its URLs were FAIR-compliant versus 28.0%. The study covers a limited corpus, 31% of pages were unevaluable by automated scraping, and the ranking systems are proprietary. 7
That trade-off is the point. A domain agent can sacrifice broad recall to improve the probability that each returned source can actually support an answer.
Why PMs should care
1. The moat is the control surface, not the model label
A general agent sees a workspace as files, endpoints, and tools. A data-specialized agent can start with the organization's vocabulary: metric definitions, table grain, ownership, freshness, lineage, join keys, sensitivity, and permissions. That context changes the search space before reasoning begins. 1
Databricks’ product example makes the pattern explicit. Genie One describes an inferred ontology of business terms, entities, and KPIs, alongside Unity Catalog semantics and permission-aware access. It also describes row- and column-level security, external-source permissions, and actions through tools such as email, Jira, and Slack. These are vendor product claims, not independent validation, but they show what “specialized” means in an implementable system. 8
2. Cost is a symptom of missing context
The first performance metric to move may not be answer accuracy. It may be the number of calls spent finding the right table, recovering from an incorrect schema assumption, or re-running a query after discovering a hidden business definition.
That is why the Databricks comparison’s relative cost matters more than its exact dollar values. The blog estimates $0.55 per Genie Code task versus $1.09 for its closest comparator, but the figures depend on the vendor’s harness, model mix, MCP configuration, and internal task set. A PM should treat them as a hypothesis: semantic context may lower cost by preventing exploration, not by making each model call cheaper. 1
3. Governance must sit inside the agent loop
A semantic catalog is not a glossary pasted into a system prompt. It needs owners, freshness, source-of-truth rules, access checks, and a way to show which tables and definitions supported the answer. The agent should be able to say "I found two plausible sources; this one is authoritative because..." rather than silently choosing one.
For an enterprise product, the minimum audit record is: selected source, definition used, query or transformation, permission decision, evidence returned, and final answer. If the agent can write data or trigger external actions, add dry runs, approval, and before/after state checks.
How to implement now
- Choose a narrow workflow with a real cost of being wrong.
Start with one repeated task, such as weekly KPI investigation, a cross-system customer lookup, or a data-quality incident. Do not begin with "ask anything about the warehouse." Define the intended answer and the authoritative source before exposing the agent.
- Build the semantic control plane before tuning prompts.
For every candidate dataset, record the business definition, grain, owner, freshness, lineage, join keys, sensitive fields, allowed use, and known caveats. Make search return structured metadata and examples, not just embeddings. The goal is to reduce source ambiguity, not to produce a longer context window.
- Give the model typed, inspectable tools.
Separate tools for catalog search, schema inspection, read-only query execution, query preview, evidence retrieval, and action approval. Enforce row limits, timeout budgets, cost limits, and permission checks outside the model. Keep intermediate tables or result objects inspectable so a human can correct a bad source choice without replaying the whole trace. This follows the restricted-query and table-centric pattern in RUBICON. 2
- Run an ablation that isolates the product advantage.
Use the same model, temperature, task set, and time budget across three variants: a general coding agent with the normal tool surface; the same agent with the semantic catalog; and the specialized data harness. Hold out tasks by source and workflow. Track:
- source-selection accuracy
- answer correctness and completeness
- tool calls and wall-clock time
- timeout and retry rate
- cost per task and cost per correct answer
- wrong-table and wrong-definition errors
- permission violations and human corrections
If the semantic-context variant does not improve source selection or reduce exploratory calls, the catalog is not yet doing product work.
- Roll out read-only first, then add approved actions.
Begin in shadow mode, comparing the agent’s source choice and answer with a human’s. Move to suggestions only after the held-out set is stable. Add writes or messages behind explicit approval, with the selected source, query, evidence, and resulting state visible to the user.
The first experiment should not ask whether a data agent is generally smarter than a coding agent. Ask a smaller question: does giving the same model an authoritative map of the data system reduce the work it must do before it can be right? That is a measurable product hypothesis, and it is where the current evidence is strongest.
References
- 1
- 2
- 3AI Agents with Databricks in 5 Minutes
youtube.com
- 4
- 5
- 6
- 7
- 8Databricks Genie One product page
databricks.com
This story was produced automatically by a channel. One sentence is all it takes for Neodrop to keep producing for you.