AI Token Cost and Accuracy Benchmark 

1. Token Cost and Wrong Answers in AI-Driven Analytics 

The dominant pattern for AI analytics today is text-to-SQL: the AI model receives raw schema information, generates a SQL query, and executes it against the database. Sequeda et al. (2023) establish text-to-SQL as the standard deployment baseline, and it is widely used in production environments today. It has two distinct failure modes that become more costly at scale. 

1.1  Token Cost 

Every text-to-SQL query requires the AI model to first receive the full schema: table names, column names, data types, and foreign key relationships. For a 28-table schema this adds approximately 4,428 tokens of schema context per query. For production enterprise schemas with 100–500+ tables, this grows to 100,000+ tokens per query. 

At current LLM pricing of $3.00 per million input tokens, a ten-person analytics team running 200 queries per day against a 200-table schema incurs over $15,000/month in schema context overhead alone, before any SQL query is executed against the data. This cost is paid fresh on every query; schema context is not cached across stateless LLM calls. 

1.2  Wrong Answers from Complex Queries 

When an AI model generates SQL against a raw schema, it must independently reason about join paths, bridge tables, and aggregation grain. That business logic is entirely absent from the schema definitions. For simple single-table queries this works. For queries spanning multiple fact tables, the model frequently selects the wrong join path and returns numbers that are materially wrong but structurally valid. There is no error, no warning, and no way for the user to know the answer is wrong without already knowing the correct answer. 

In this benchmark, two multi-fact-table questions returned results from PostgreSQL that were 5–10× inflated or attributed to the wrong policy. Neither failure triggered a SQL error. Both results would have passed routine review by an analyst who did not already know the correct answer. 

1.3  Join Complexity and Business Logic 

Analytical schemas in real enterprises are layered: star schemas, snowflake schemas, ACORD insurance models, Kimball dimensional designs, all involving multiple fact tables joined through carefully chosen bridge tables. The correct join path is not derivable from column names alone; it is encoded in business rules. 

Text-to-SQL generation requires the AI to infer those business rules from table names on every query. The wider the schema, the more plausible join paths exist, and the more likely the AI selects the wrong one. The semantic layer encodes the correct path once and applies it consistently. 

3. Benchmark Methodology 

All results in this paper derive from live execution against real data. No metrics are theoretical or estimated; where direct measurement was not possible (due to MCP protocol limitations on server-side timing), data points are clearly labeled with their derivation method. 

3.1  Dataset 

The dataset follows an ACORD-aligned (U.S. insurance industry standard) insurance data model with 28 tables capturing policies, coverage details, claims, financial amounts, and party relationships. The schema includes the multi-row-per-policy patterns created by multi-year policy terms, exactly the structural condition that triggers fan-out errors in naive SQL. 

• Source: Chat-With-Your-Data benchmark (data.world/cwd-benchmark-data)
• Semantic model: Mosaic Library (linked in original benchmark)

3.2  Test Scope 

• 17 questions spanning single-table lookups, multi-table aggregations, ratio calculations, exposure analysis, and cross-domain attribution
• Live execution against both PostgreSQL and Mosaic MCP connections, with no synthetic or replayed data
• SQL generated by Claude Code MCP for both systems, ensuring equivalent AI generation quality
• Token counts measured from actual API responses (schema context) and query character lengths (÷4, standard approximation)
• PostgreSQL execution times measured via EXPLAIN ANALYZE: 0.2ms–6.7ms, negligible relative to LLM inference time of ~13–14s for both tools

3.3  Mosaic Configuration 

Three one-time configurations were required to encode business intent not inferable from column names: 

These configurations are one-time setup. Once encoded, they apply automatically to every subsequent query, regardless of user, interface, or AI model. 

4. Accuracy Results: Where Semantics Prevent Failure 

Of 17 questions, both systems answered 15 correctly. On the remaining 2, PostgreSQL returned plausible but materially wrong numbers; Mosaic returned correct answers in both cases. These failures are not isolated to this dataset. Any query that joins two fact tables through a shared dimension key, without semantic isolation, will produce the same inflation. They are predictable structural failures of naive multi-fact-table SQL, and they would affect every analyst querying this data without a semantic model. 

4.1  Question 16: Coverage Limits and Total Claims per Policy 

Question: "What is the total coverage limit and total claims per policy?" 

Root Cause 

policy_coverage_detail contains one row per policy per coverage year. A 5-year policy produces 5 PCD rows. When the AI joins policy_amount (fact table) to claim_amount (fact table) through this shared key, each claim row joins to all 5 coverage year rows, multiplying claim amounts by 5 before SUM() aggregates. The query shape (one output row per policy) looks correct. The inflation is invisible without prior knowledge of the correct answer. 

Why Mosaic Is Correct 

The semantic model isolates fact tables. Coverage and claim metrics are computed via independent canonical paths and combined only as pre-aggregated results at the policy grain. The multiplicative error cannot form. 

4.2  Question 17: Total Catastrophe Payout per Policy 

Question: "What is the total catastrophe payout per policy?" 

Root Causes (Two Compounding Errors) 

Fan-out: insurable_object_identifier appears across multiple PCD rows, multiplying claim_amount rows before aggregation. 

Bridge misselection: Connecting claims to policies via insurable_object_identifier bypasses the claim_coverage bridge, which contains the authoritative policy-year linkage. Claims belonging to Policy 336 leak into Policy 337 because both policies share an insurable object. 

Why Mosaic Is Correct 

The semantic model enforces the path: claim → claim_coverage → policy_coverage_detail → policy. This is the only valid attribution path. Both overcounting and cross-policy leakage are structurally prevented. 

5. Token Economics 

Without business context, AI has to rediscover how your data works on every query. It loads the schema, reasons about relationships, and generates SQL from scratch each time. That burns tokens before any actual analysis begins. Mosaic addresses this directly because business logic is pre-defined in the semantic layer, AI receives only what it needs. 

5.1  Mosaic MCP (Semantic Layer via Model Context Protocol) 

Mosaic MCP gives the LLM the business context it needs, not the full raw schema. Mosaic provides the governed definitions, relationships, and join logic so the SQL is accurate and precise. The LLM produces the query, but Mosaic defines what that query should look like, abstracting JOIN complexity into simple GROUP BY queries. 

^{Note: Cost calculated at $3.00/M input tokens + $15.00/M output tokens. Schema context measured from live get_semantics API responses and DDL character counts ÷4.} 

5.2 Strategy AI Agents 

Strategy AI Agents extend Mosaic's semantic layer into pre-built automations. They are purpose-built agents that answer business questions by routing directly through Mosaic's governed business context, with near-zero token overhead. Rather than optimizing what the LLM receives, they remove the LLM from query generation entirelym Mosaic handles schema lookup, SQL generation, and execution, and the LLM receives only the natural language question. 

Strategy AI Agents deliver approximately 98% token reduction vs direct PostgreSQL and 96% reduction vs Mosaic MCP. AI token consumption is virtually zero; cost is deflected from the LLM provider to the Mosaic software platform. This architecture scales linearly: adding users, questions, or larger schemas does not increase token costs, because no schema context ever touches the LLM. 

6. Where Savings Compound in Production 

The benchmark numbers above are conservative by design. They were measured against a 28-table demo dataset, without caching, without multi-turn conversation history, and at only 17 questions. Each constraint compresses the measured savings relative to production reality. 

6.1  Production Estimates 

With Mosaic MCP: Conservative 50–70% cost reduction in production, rising toward the upper bound as schema size increases and caching compounds across sessions. 

With Strategy AI Agents: 95–98% cost reduction regardless of schema complexity, query volume, or team size. Schema size is irrelevant because no schema context ever reaches the LLM. This is a fundamentally schema-invariant cost architecture. 

9. Conclusion 

Connecting AI agents directly to raw databases is not merely expensive; it is structurally unreliable for the complex multi-table queries that drive enterprise analytics decisions. 

Mosaic's semantic layer addresses both dimensions simultaneously. By encoding business logic (canonical join paths, fan-out protections, party-identity unification) once in the semantic model, every subsequent query inherits those guarantees regardless of which AI model, analyst, or interface issues it. The accuracy improvement is not a tuning artifact; it is a structural property of the architecture. 

The cost savings are equally structural. Mosaic MCP reduces schema context overhead by 53% and SQL complexity by 24%, delivering 37% cost reduction in the benchmark and 50–70% at production scale with caching. Strategy AI Agents remove the AI from the query execution path entirely, achieving 95–98% cost reduction regardless of schema complexity, an architecture that makes AI query costs effectively schema-invariant and user-count-invariant. 

For organizations evaluating AI analytics infrastructure, the question is no longer whether to use a semantic layer. The benchmark data demonstrates its necessity for accurate results on multi-fact-table queries.  

Technical Proof Points 

^{Source: Insurance Claims Analytics Benchmark: 17 questions, live execution, measured token counts} 

^{Data model: ACORD-aligned insurance schema (data.world/cwd-benchmark-data)} 

^{SQL queries generated and executed via Claude Code MCP. Token counts measured from live API responses and query character lengths.} 

^{Original benchmark authored by Sushma Ganapuram, Strategy Software.}