Collectively, 482-e introduces significant improvements in the following areas:

• Icehouse Innovations:
  Scheduled, append-only incremental refreshes for Iceberg Materialized Views and Iceberg REST Catalog support with enhanced credentials in Starburst Metastore.
• Engine Performance & Query Optimization:
  10% faster query times based on internal benchmarking, lower latency, and support for PyStarburst Arrow protocol acceleration, and Queries Exceeding Ram to prevent Out of Memory (OOM) failures in public preview.
• Platform Intelligence & AIDA:
  Token consumption monitoring for AI administrators, broader OAuth capabilities for external Model Context Protocol (MCP) servers, and Data Product Enrichment v2.
• Expanded Open Connectivity:
  General Availability for the Great Lakes and OpenSearch connectors, alongside Public Preview support for OpenAPI and kdb+ connectors.

Icehouse advances with Incremental Materialized Views

Starburst continues to deepen its status as the ideal foundation for Apache Iceberg workloads. In 482-e LTS, we focus on eliminating the recompute overhead of precomputed data products and expanding catalog interoperability.

Incremental Iceberg Materialized Views 

Now available in public preview, Starburst Enterprise introduces incremental refreshes for append-only Iceberg materialized views. Previously, updating an Iceberg materialized view required a full re-execution over historical datasets, creating prohibitive refresh windows and heavy compute costs.

Administrators can now schedule append-only refreshes using an incremental_column across any Trino-queryable source, or define them directly through data products, without deploying a separate caching service. By processing only newly added rows, data engineers can drastically shorten refresh windows and remove a primary technical blocker when migrating legacy Hive pipelines to Apache Iceberg.

Open Iceberg REST Access for Starburst Metastore 

Now in public preview, Starburst Metastore (formerly Starburst Data Catalog) adds native support for the open Iceberg REST Catalog specification, complete with managed credentials, PAT-style tokens, and identity support for Okta and Ping Identity. Teams standardizing on the open Iceberg REST spec can now use Starburst Metastore as a unified catalog across external query engines and third-party tools, reducing catalog-specific integrations and connector maintenance.

Engine performance breakthroughs and view optimizations

The 482-e LTS release builds on core core execution engine updates to improve speed by 10% based on internal benchmarking, lower latency, and process complex queries transparently. This iteration further enhances the speed already associated with our query engine, delivering enhanced results for customers. 

Queries Exceeding RAM

To prevent Out of Memory (OOM) failures on massive queries, we are expanding support for our memory resilience execution mode backed by an object storage and/or a local disk tier in the embedded BufferService. This mode allows large queries to exceed available RAM while reducing  the runtime penalties of Fault-Tolerant Execution (FTE). This feature is in public preview.

PyStarburst Arrow Protocol 

The spooling protocol can now be configured to leverage Apache Arrow serialization alongside new PyStarburst methods. Now in public preview, this allows Python clients and data scientists to retrieve query outputs in a data-parallel format and pass Arrow chunks directly to Python data science tools without decoding, making result retrieval up to 7x faster.

Enhanced platform intelligence with AIDA

The 482-e release introduces crucial visibility, security, and context capabilities for the Starburst AIDA.

AIDA Studio, Skills, and Token Observability 

Now supported in 482-e, AIDA Studio provides a collapsible side panel offering configurable tools directly within chat sessions. Furthermore, AIDA Skills enters public preview, introducing granular access control so non-sysadmin users can author instructions and codify institutional knowledge. To manage costs, administrators gain a dedicated token consumption page in the Starburst Enterprise Admin navigation, providing visibility into usage trends across custom time ranges down to individual users.

External MCP Server OAuth Expansion 

Moving into public preview, AIDA’s external Model Context Protocol (MCP) server now supports expanded OAuth authentication standards, including OAuth Client ID/Secret and OAuth PKCE. AI platform teams can securely connect AIDA to a wider array of third-party enterprise systems to execute complex workflows grounded in enterprise context.

Data Product Enrichment Agent v2 

Now in public preview, the updated Data Product Enrichment Agent enables data SMEs to build AI-ready context in minutes. By combining user-provided scope descriptions and business rules with active data sampling—rather than relying strictly on raw metastore schema definitions—the agent automatically generates higher-quality metadata for data products.

Expanded open connector ecosystem

Starburst Enterprise 482-e expands federated access across unified lake formats, time-series systems, and SaaS endpoints.

• Great Lakes Connector (GA): A single unified connector spanning Hive, Iceberg, and Delta Lake table formats. Organizations can query across mixed table formats through one connector, cutting catalog sprawl and maintenance overhead.
• OpenAPI Connector (Public Preview): Query any OpenAPI-described REST system using standard SQL without waiting for custom connectors. Point the connector to an OpenAPI specification file to turn SaaS platforms like Jira, GitHub, and Datadog into queryable tables.
• kdb+ Connector (Public Preview): Capital markets and quant technology teams can now natively federate queries against kdb+ time-series data without sacrificing execution fidelity.
• OpenSearch Connector (GA): Upgraded to GA with multi-field pushdown and scrollable pagination support for large result sets, bringing production-grade federation to log analytics and SIEM workloads.

Critical breaking and configuration changes

This release includes several critical architectural changes and configuration updates that require attention prior to deployment. Review the full technical documentation to ensure cluster stability during the upgrade process.

Removal of Legacy Packaged HMS and Cloud Object Storage Support

• Hive Metastore Service Image Removed: The Starburst-packaged Hive Metastore Service (HMS) image and starburst-hive Helm chart have been completely removed after reaching end of support in 480-e. Organizations must migrate to Starburst Metastore before upgrading.
• Legacy Cloud Storage File Systems: Legacy Hadoop-based object storage support (io.trino.hdfs.s3.TrinoS3FileSystem and fs.hadoop.enabled=true for cloud storage) has been completely removed from the Delta Lake, Hive, and Iceberg connectors. Catalogs using legacy hive.s3.*, hive.azure.*, or hive.gcs.* configuration properties will fail to start. Migrate to fs.native-s3.enabled=true, fs.native-azure.enabled=true, or fs.native-gcs.enabled=true.

Connector Deprecations and Removals

• Snowflake JDBC Connector Removed: The snowflake_jdbc connector has been removed in favor of snowflake_parallel. Additionally, password-based authentication in the Snowflake connector is deprecated and will be removed in the November 2026 LTS release in favor of key-pair authentication.
• Alluxio Exchange & File System: Support for Alluxio-backed exchange storage for Fault-Tolerant Execution and the Alluxio file system across Delta, Hive, Hudi, and Iceberg connectors has been removed.
• Node-Level File System Caching: File system caching is no longer configured inside catalog properties (fs.cache.*). All cache configurations must be moved to node-level properties via cache-manager.config-files in config.properties.
• Starburst Data Catalog Property Renaming: Starburst Data Catalog has been renamed to Starburst Metastore. Properties prefixed with catalog.* (e.g., catalog.enabled) have been deprecated and renamed to metastore.* (e.g., metastore.enabled).

Implicit CHAR and VARCHAR Coercion Reversal 

The implicit coercion behavior between CHAR and VARCHAR types has been reversed. A CHAR value now coerces to VARCHAR with trailing spaces trimmed, and comparisons follow VARCHAR semantics without blank padding. To preserve legacy behavior until the November LTS release, administrators must explicitly set deprecated.legacy-varchar-to-char-coercion=true in their configuration.

Upgrade to Starburst Enterprise 482-e LTS Today

The 482-e LTS release marks a defining shift in how modern enterprises run, govern, and scale their data lakehouses. By embedding automated materialized view substitution, incremental Iceberg processing, and native Control Plane high availability, platform teams are equipped to run multi-cluster architectures with absolute performance and resilience.

To begin planning your deployment:

• View the full 482-e LTS release notes here.
• Review our AI and AIDA documentation here.
• Read our latest connector and migration guides here.

The use cases cluster around a handful of repeatable categories:

• Document and research copilots
• Fraud and anti-money laundering (AML) detection
• Wealth-management productivity
• Agentic workflows that reach across both structured and unstructured data.

The easiest wins came from unstructured data. The harder, higher-value work is unlocking structured transaction data that sits across dozens of silos.

The banks with the most live use cases share one trait. They solved data access first. Without that, the use-case count stalls at pilots, no matter which model sits on top.

Key takeaways

• Banks now report AI use cases numbering in the hundreds, with one major bank running nearly 1,000, and the use cases cluster into four categories, including document and research copilots, fraud and AML detection, wealth-management productivity, and agentic workflows that span structured and unstructured data.
• The easiest wins came from unstructured data, such as summarization, transcription, and research copilots, while the harder, higher-value work is unlocking structured transaction data that sits across dozens of data silos.
• The results are already measurable. For example, Khan Bank saw 400x faster query performance, a global investment bank cut money laundering losses by millions of dollars, and more than 200,000 Bank of America employees now use AI-enabled tools every day.
• Banks that unify structured and unstructured data on one platform can move a new AI use case from pilot to production in about 90 days, without a full data-estate replatform.
• Governance, data sovereignty, and audit requirements apply to every AI system from day one, so banks that build access controls and lineage into the data platform can approve new use cases faster.

Conversational copilots for unstructured data are the fastest win

The easiest AI wins in banking start with unstructured data, including documents, meeting notes, and research reports that used to need a human reader. Summarization tools condense long filings and contracts in seconds. Transcription copilots turn client meetings into searchable notes. Research copilots help analysts scan earnings calls and filings for the details that matter.

According to Banking Dive, citing Bank of America’s Q2 2026 earnings call, more than 200,000 of the bank’s employees now use AI-enabled tools. Staff send more than 400,000 prompts a day. Banking Dive also reported that Bank of America has approved more than 300 AI use cases, including 114 generative AI projects, with 34 fully live in production. Wells Fargo has taken a similar path with what it calls its “AI Teammate,” an internal assistant built to help employees work through documents and routine tasks faster.

Fraud detection, AML, and risk analytics run in real time

Fraud and risk teams were early adopters of machine learning, and the field keeps moving. Banking Dive reported that one major bank now runs nearly 1,000 live AI use cases across risk, fraud, marketing, and document reading. That is one of the largest deployed footprints at any US bank.

Real-time AML monitoring is a good example of why the count keeps climbing. A global investment bank now runs transaction monitoring that accelerates money laundering detection and prevention, helping the bank flag suspicious activity before it becomes a multi-million dollar loss. For a plainer look at how this works, see this rundown of modern standards for anti-money laundering monitoring.

Risk analytics is moving toward conversational tools, too. At Santander Corporate & Investment Banking, analysts and front-office desks now converse with multi-petabyte data lakes in natural language through a Starburst’s conversational analytics interface, AIDA. The tool turns what used to be a multi-day data pull into a question typed into a chat window.

Customer-facing productivity and wealth management gain ground

Wealth management is where AI’s productivity gains show up most directly in client relationships. Many wealth-management advisers now use an AI tool that helps them pull client data from Salesforce customer relationship management (CRM) records far faster than manual lookups allowed. That means advisers spend more time with clients and less time hunting for account history.

Citigroup describes adoption in similarly broad terms. Banking Dive reported that nearly 9 out of 10 Citigroup employees now use AI tools, with the bank tying that adoption to gains in productivity, client experience, and growth. Financial firms with this kind of reach are increasingly using AI to help teams break down data silos and power smarter decision-making. That same foundation supports informed risk mitigation, revenue-generating, and process optimization decisions across the business.

The data problem behind agentic AI

Unstructured data was the easy win because LLMs excel at breaking it down, whether it be a document repository, an email archive, a transcript store. Structured data is a harder problem. Transaction records, account histories, and risk metrics are often spread across dozens of data silos, each with its own access rules and formats.

That’s the higher-value problem banks are now working through. The next generation of AI systems needs to reach both kinds of data at once. It has to route a single question across documents and transaction tables without forcing an analyst to query each system by hand. Banks are increasingly building agentic AI systems that route queries across structured and unstructured data. That is what turns a chat interface into something that can answer questions grounded in the bank’s own data.

Modernizing the data platform to support AI at scale

None of the use cases above work well if the underlying data platform can’t keep up. Query speed, source coverage, and time-to-insight all set the ceiling on what an AI system can do.

Khan Bank saw 400x faster query performance after modernizing its data stack, a change that turned reports that used to take hours into something closer to real time. That gave analysts a wider view of the bank’s data without new pipelines for every source. US Bank took the same approach to unify data access and accelerate AI-driven analytics, giving teams one place to query data that used to live in separate systems.

Governance and regulatory pressure shape every use case

Every AI use case in banking runs inside a regulatory perimeter. Data sovereignty rules, cross-border transfer restrictions, and audit requirements apply to an AI system exactly as they apply to any other production system. Banks need to design for that from the start rather than retrofit it later.

That means governance can’t be an afterthought bolted onto a finished project. Banks need a modern, AI-ready data foundation that works across existing systems. That way, access controls, lineage, and audit trails travel with the data no matter which team or model queries it. That kind of governed, federated access that enables fast, compliant analytics and AI means a bank can say yes to a new AI use case. It doesn’t need to reopen a compliance review from scratch each time.

From pilot to production

The path from a promising pilot to a production AI use case doesn’t need a full data-estate replatform. Banks that unify structured and unstructured data on one platform can launch production-ready AI use cases in just 90 days. They work with the systems they already run rather than ripping them out.

BNY frames this as more than a cost-saving exercise. Banking Dive reported that BNY’s leadership sees AI as a source of long-term value, tied to new products, new capabilities, and the platform data that makes both possible. That’s the common thread across every bank in this article. The ones with the most live AI use cases are the ones that solved data access first, before they worried about which model to run on top of it.

Want to know more about financial services and Starburst? Check out our webinar on the topic.

FAQs

What are the most common AI use cases banks are deploying today?

Banks cluster their AI use cases into four groups: document and research copilots, fraud and anti-money laundering detection, wealth-management tools, and agentic workflows. Document and research copilots summarize filings and transcribe meetings, while fraud and anti-money laundering detection flags suspicious transactions in real time. Wealth-management tools help advisers pull client data faster, and agentic workflows route a single question across both structured and unstructured data.

How are banks using AI for fraud detection and anti-money laundering?

Fraud and risk teams use AI models to monitor transactions in real time, so a bank can flag suspicious activity before it turns into a large loss. One global investment bank now runs transaction monitoring that speeds up money laundering detection and prevention. Santander Corporate & Investment Banking analysts use a conversational AI agent called AIDA to query multi-petabyte data lakes in plain language instead of waiting days for a manual data pull.

What’s the biggest technical obstacle to scaling AI in banking?

Structured data is the harder problem. Transaction records, account histories, and risk metrics sit across dozens of data silos, each with its own access rules and formats. An AI system has to reach every one of those sources before it can answer a question with confidence. Banks that solve this by unifying access to structured and unstructured data on one platform tend to have the most live use cases in production.

How is generative AI different from earlier analytics tools in banking?

Earlier analytics tools mostly answered structured queries against clean, pre-modeled data. Generative AI can also read and summarize unstructured sources, such as contracts, filings, and meeting notes, turning documents that used to need a human reader into searchable, conversational data. That’s why Bank of America’s employees now send more than 400,000 AI prompts a day, according to Banking Dive. Most of those prompts aim at document-heavy work that older tools couldn’t touch.

Do banks need to move all data before deploying AI?

No. Banks that unify data access on one platform still leave data where it lives, whether that’s Amazon S3, an on-premises data warehouse, or a third-party system. They use that platform to query across sources instead of copying everything into a new location first. US Bank took this approach to unify data access and speed up AI-driven analytics, giving teams one place to query data that used to sit in separate systems.

How fast can a bank launch a new AI use case?

Banks that have already unified structured and unstructured data on one platform can launch production-ready AI use cases in about 90 days. They work with the systems they already run instead of planning a full data-estate replatform. Governance work, such as access controls, lineage, and audit trails, moves alongside the data from the start. A new use case doesn’t need a separate compliance review before it can go live.

Amazon Redshift handles structured business intelligence and low-latency reporting inside AWS well. Connecting an AI agent to it is a different problem. The data warehouse returns rows and columns. It does not tell the agent what those numbers mean or which source to trust.

Once you connect an agent to Redshift, you also take on the platform’s limits. Concurrency ceilings, single-database catalog scope, and proprietary storage were already constraints for reporting. They get harder to live with when agents add exploratory queries on top of reporting load.

Adding a governed layer of business definitions, policy, and data products on top of Redshift before the first agent query runs help. This article looks at why Redshift’s architecture was not built for the agent grounding provided by context engineering. It covers what a governed context layer adds on top, and how you can start building that layer without moving off Redshift.

Key takeaways

• Redshift’s concurrency limit of about 50 parallel queries and its single-database scope per catalog make it hard to serve AI agents that need to reason across multiple sources.
• A context layer adds business definitions, governance, and data products on top of Redshift so agents get grounded, policy-enforced answers instead of confident guesses.
• Federating Redshift with your data lakes, other data warehouses, and SaaS sources avoids copying data into another proprietary storage silo.
• Serving governed context to agents over the Model Context Protocol (MCP) keeps governance centralized while giving every tool access to the same vetted definitions.
• Benchmark any vendor-stated performance or cost claims about Redshift alternatives against your own workload before you commit to a specific approach.

Amazon Redshift handles structured business intelligence and low-latency reporting inside AWS well. When you connect an AI agent to it, though, you run into a different set of problems. The warehouse alone doesn’t tell the agent what your numbers mean or where else to look.

This piece looks at why Redshift’s concurrency limits, single-database scope, and proprietary storage format leave AI agents short on business context, and what you need to add on top to get answers you can trust.

What context engineering means for Redshift teams

The bottleneck has moved from models to context

Large language models have become good enough to hold a conversation about your data. What they don’t have is your data. No frontier model knows your fiscal calendar, your definition of churn, or which of 5 customer tables is the authoritative one, because that knowledge doesn’t exist in any public training corpus.

That gap explains a sobering number. Only about 30 percent of agentic AI projects reach production. Most stall not because the model is weak, but because nobody built the layer that feeds it grounded, governed information. Context engineering means designing what information goes into a model so it produces reliable, grounded answers, and that design work is now the harder half of building an agent.

Where Redshift alone falls short for AI agents

Redshift is a single, closed data warehouse. An agent that queries it directly gets rows and columns, not definitions, lineage, or policy. Ask it “what is active revenue” and you get whatever a table happens to contain, not your finance team’s definition of the term.

Redshift also holds a small share of most companies’ data. Any agent limited to a single Redshift cluster misses everything you store elsewhere, including data lakes, other cloud data warehouses, and SaaS applications.

Why context engineering matters now for Redshift users

Concurrency and cost limits push teams toward federation

Redshift typically caps you at around 50 parallel queries. That ceiling was tight enough for BI dashboards; it gets tighter once you add agents that fire off exploratory queries on top of your existing reporting load.

Scaling Redshift to absorb that extra concurrency usually means bigger, more expensive clusters. Before you go that route, review the true cost of centralizing data in Amazon Redshift, which lays out how concurrency limits and cluster sizing compounds over time.

Redshift’s proprietary storage format and lock-in

Redshift stores data in its own proprietary format, so every dataset you load in has to stay there or get copied out. That combination of concurrency limits, maintenance burdens like vacuuming, and proprietary storage lock-in makes Redshift a costly single point of truth for an AI agent that needs to reason across more than one source.

Loading more data into Redshift to fix the coverage gap just adds another data silo. Additionally, it commits you further to a format you cannot easily move off of later. That tradeoff is the missing element in most AI data strategies. You get more storage inside one warehouse, not more context across the sources you already have.

What a context layer adds on top of Redshift

Business definitions and metadata

A context layer sits above Redshift and holds the business definitions your agent needs, including what counts as active revenue, which customer table is authoritative, and how your fiscal year lines up to calendar months. Without that layer, an agent has to guess, and confident guessing is worse than an honest “I don’t know.”

Governance and policy enforcement

The same layer should enforce who can see what. Building an enterprise context layer means combining metadata, governance, and access policy in one place, so every agent that queries Redshift respects the same row-level and column-level rules a human analyst would follow.

Centralizing governance here also means you set a policy once instead of repeating it in every tool that touches your data.

Data products as the delivery mechanism

Rather than handing an agent raw table access, a context layer packages vetted, well-documented data products for it to query. That’s part of why the context layer sits above the warehouse layer instead of inside it. The warehouse stores data, and the context layer decides what any given agent is allowed to see and how to describe it. Data products provide the way by which this happens, and are an essential ingredient in the creation of a context layer. 

How context engineering works with Redshift

Federating Redshift with other systems without copying data

You do not need to migrate off Redshift to fix its blind spots. With a federation approach, you can query Redshift alongside your data lakes, other data warehouses, and SaaS sources while limiting the need to copy data between them. Starburst on AWS connects to Redshift and other AWS-hosted sources this way, so an agent can reason across systems through one interface.

This approach also sidesteps a common trap. Context, not model capability, is now the bottleneck for most agentic AI projects, and federation is how you widen an agent’s context without duplicating every dataset into Redshift first.

Serving governed context to agents over MCP

Once your context layer defines and governs your data, you need a consistent way to hand it to agents. The Model Context Protocol (MCP) gives multiple agents and tools a shared interface to the same vetted definitions, so you enforce governance once and reuse it everywhere instead of wiring each tool to Redshift separately.

That consistency matters because access to business context, not only data access, will determine who can replace BI with AI inside your organization.

Considerations before you start

Single-database scope per catalog

By default, a Redshift connection reaches a single database inside a cluster. If your Redshift cluster hosts multiple databases, plan for that scope limitation early, since it affects how you map catalogs to your broader context layer.

TLS and network configuration

Recent Redshift JDBC drivers enable TLS by default, so confirm your connection settings match your network security requirements before you federate. Getting this configuration wrong early costs more time to fix once agents depend on the connection.

Vacuuming and maintenance overhead

Redshift needs regular vacuuming to reclaim space and keep query plans efficient, and that maintenance overhead does not go away once you add a context layer. Budget for it the same way you would for any other production Redshift workload, since centralization costs like vacuuming and storage lock-in tend to accumulate if nobody owns them.

Steps to start context engineering on your Redshift data

The following steps can be a useful blueprint for context engineering your Redshift data. 

1. Begin by taking an inventory of the business definitions your agents will need, including revenue, churn, and any metric with more than one candidate source table.
2. Map which other systems, including data lakes and SaaS applications, hold data an agent needs alongside Redshift.
3. Set up federated access so agents can query Redshift and those other sources without copying data into a new silo.
4. Define governance policy once, at the context layer, so it applies consistently across every agent and tool.
5. Serve that governed context to agents over MCP so multiple tools share the same vetted definitions.
6. Benchmark any vendor-stated performance or cost claims against your own workload before you commit to a specific approach.

If you’re weighing whether Redshift should remain your single source of truth for AI workloads, the tradeoffs are similar to the ones covered in context engineering for Snowflake and context engineering for Amazon Athena. The warehouse handles storage and compute well, and a context layer is what makes an agent built on top of it trustworthy. Additionally, none of the fixes here mean you have to abandon Redshift, only to stop treating it as the only place your agents can look.

Next steps

Want to know more about context engineering for Redshift? Check out the Gartner report on context-aware data management.

FAQs

What is context engineering, and how is it different from prompt engineering?

Context engineering means designing what information reaches a model, including business definitions, governance rules, and access to the right data sources, so it produces grounded, reliable answers. Prompt engineering focuses on how you phrase a request to the model. You need both, but without the right context, a well-crafted prompt still returns a guess instead of a grounded answer.

Why isn’t Redshift data enough for an AI agent to answer correctly?

Redshift is a single, closed data warehouse. An agent that queries it directly gets rows and columns, not definitions, lineage, or policy, so it has to guess at what a metric like active revenue means instead of applying your finance team’s definition. Redshift also holds a small share of most companies’ data, so an agent limited to one cluster misses everything you store in data lakes, other data warehouses, and SaaS applications.

Do you need to move Redshift data to build a context layer?

No. With a federation approach, you can query Redshift alongside your data lakes, other data warehouses, and SaaS sources while limiting the need to copy data between them. Loading more data into Redshift to close the coverage gap just adds another data silo and commits you further to its proprietary storage format.

What governance should a context layer enforce for Redshift data?

A context layer should enforce the same row-level and column-level access rules a human analyst would follow, combining metadata, governance, and access policy in one place. Centralizing governance here means you set a policy once instead of repeating it in every tool that touches your data.

How does the Model Context Protocol relate to context engineering on Redshift?

MCP gives multiple agents and tools a shared interface to the same vetted definitions your context layer maintains. Once your context layer defines and governs your data, you can use MCP to enforce that governance once and reuse it everywhere instead of wiring each tool to Redshift separately.

Can a context layer work with Redshift’s single-database-per-catalog limitation?

Yes, but plan for it early. By default, a Redshift connection reaches a single database inside a cluster, so if your cluster hosts multiple databases, you need to map that scope limitation into how your broader context layer organizes catalogs.

While Starburst supports server-side scan planning with any REST Catalog, the most compelling initial integration is with Databricks Unity Catalog. Unity Catalog can now enforce fine-grained access control, including row filters and column masks with governed tags, for managed Iceberg tables on Starburst using server-side scan planning. We’ve worked closely with Databricks to ensure that Unity Catalog policies are respected when Starburst queries a Unity-governed Iceberg table, allowing governance to be centralized on Unity Catalog with no bespoke logic in the query engine.

What is Server-Side Scan Planning?

The Iceberg REST spec added POST …/tables/{table}/plan in Iceberg version 1.7.0. Instead of having the Starburst query engine read metadata files and plan the scan itself, the client sends selected columns and filter expressions to the catalog server through this endpoint. The catalog returns a list of file scan tasks and scoped storage credentials, which the engine executes as usual.

This small protocol change gave the catalog a hook into the read path of a query. Now, the catalog can decide what files an engine sees, apply transformations, vend short-lived credentials scoped to the specific files in the response, and audit file access. None of this requires our engine to understand Databricks’ policy model, but policies are still respected when data is returned to the caller.

For Starburst, the value of supporting this protocol is that it works across the entire Iceberg REST ecosystem, and will work out of the box as other catalogs implement this endpoint. 

Unity Catalog’s Fine-Grained Access Control

Databricks Unity Catalog uses /plan to implement “ScanAPI”: an endpoint to enforce fine-grained access control for external engines. With ScanAPI, when a Starburst query reaches a Unity-managed Iceberg table that carries active FGAC policies, Unity:

• Evaluates the caller’s ABAC policies against the user’s identity.
• Materializes sanitized data into temporary data files with row filters and column masks already applied.
• Returns file scan tasks pointing at those sanitized files, plus scoped credentials.

The engine only sees data authorized for that user. The policy was defined once, in Unity Catalog, and applies whether the query runs in Databricks or Starburst.

This addresses the governance gap many lakehouse customers encounter. Unity Catalog already provides table-level access control across engines via credential vending, but credential vending is coarse: you have access to the whole table, or you don’t. Real-world governance needs more than that: PII column masks, regional row filters, role-driven views of the same data, etc. Replicating that policy in every engine doesn’t scale. Server-side scan planning lets Unity govern data centrally, and engines like Starburst can honor those decisions using ScanAPI.

A Real-World Example

Want to know more about how this works in reality? Check out this video by Databricks. 

What’s Next?

Server-side scan planning support ships in Starburst Enterprise Platform 481-e STS. While the Unity Catalog integration is the most fully developed example today, Starburst will support any other REST catalogs that implement this endpoint. 

Overall, this represents a stepping stone to a shared policy language. This mirrors other initiatives in the industry toward similar goals, notably Snowflake’s approach towards an open-sourced way of governing tables by certain trusted compute engines. 

For more on the protocol and the joint Databricks/Starburst work, see Databricks’ cross-engine ABAC announcement and our earlier post on Starburst integration with Unity Catalog.

Modern data platforms implement masking through several techniques. Dynamic data masking (DDM) applies transformations at query time based on user permissions. Static masking permanently replaces sensitive values in datasets, typically for non-production environments. You might also encounter tokenization, which substitutes sensitive data with non-sensitive tokens, or format-preserving encryption that maintains data types and patterns while protecting the actual values.

Within today’s data ecosystem, masking has become a critical bridge between compliance requirements and analytical workflows. Platforms like Snowflake, BigQuery, and SQL Server now provide native masking capabilities that integrate directly with their query engines. This integration means your existing analytics workflows can continue unchanged while sensitive data gets automatically protected based on user roles and policies.

The challenge many organizations face is that implementing effective data masking across modern, distributed data architectures involves more than just flipping a switch. You need to coordinate policies across multiple systems, ensure consistent identity management, and maintain performance while adding these protection layers. This is where platforms like Starburst become valuable, offering unified governance capabilities that can work alongside your existing data platforms or fill gaps where native masking falls short.

Why data masking has become essential for modern analytics

Three fundamental shifts in how organizations handle data have caused data masking to transition from optional to critical. First, regulatory frameworks now explicitly require or strongly encourage data protection techniques like masking. GDPR’s pseudonymization requirements give organizations more flexibility in processing personal data when they can demonstrate effective de-identification. HIPAA’s Safe Harbor method provides specific guidance on removing or transforming identifiers from healthcare data. PCI DSS requirements mandate that primary account numbers be masked when displayed in applications or reports.

Second, the democratization of data access has expanded the number of people who need to work with potentially sensitive information. Data scientists, analysts, and business users across departments now perform their own queries and build their own models. Traditional approaches that involve simply restricting access create bottlenecks and limit the value organizations can extract from their data. Masking allows you to say “yes” to data requests while maintaining appropriate protection levels.

Real-world masking across industries

Financial services data analytics organizations use masking to enable fraud analysis and customer behavior modeling without exposing actual card numbers or account details. A typical implementation might show analysts transaction patterns with masked PANs like “****-****-****-1234” while preserving the ability to detect suspicious sequences or merchant categories. For authorized investigators handling specific cases, the full PAN remains accessible through role-based exceptions.

Healthcare data analytics systems apply masking to enable medical research and population health studies using de-identified patient data. Researchers can analyze treatment outcomes and identify patterns across large patient populations without accessing names, addresses, or other direct identifiers. Statistical research methods rely on this approach to generate insights while maintaining patient privacy.

Retail analytics solutions organizations mask customer identifiers while preserving purchase patterns and preferences for recommendation engines and marketing analytics. Teams can analyze shopping behaviors, seasonal trends, and product affinities without accessing customer names, email addresses, or payment details. This approach enables sophisticated personalization algorithms while reducing privacy risks.

The business case beyond compliance

Beyond regulatory requirements, masking creates tangible business value by reducing data breach impact and enabling more flexible data sharing. When a security incident occurs in a system with properly masked data, the exposure is significantly limited. Instead of dealing with millions of exposed credit card numbers or social security numbers, you might face exposure of masked or hashed values that provide no direct path back to individuals.

Data sharing partnerships become more feasible when you can demonstrate that sensitive elements are properly protected. BigQuery specifically notes that data masking can streamline sharing scenarios because authorized users maintain their normal query patterns while unauthorized users automatically receive protected views of the same datasets.

Technical and operational hurdles in masking implementations

While the concept of masking sounds straightforward, production implementations quickly reveal a complex web of technical constraints and operational challenges that can derail projects or create unexpected security gaps.

When the database optimizer works against you

Runtime masking interacts with query optimizers in ways that can significantly impact performance. BigQuery explicitly states that BI Engine acceleration doesn’t work with masked queries, and that wildcard table queries become incompatible with certain masking scenarios. The fundamental issue is that optimizers rely on being able to push predicates and filters down to the storage layer, but masked columns can’t safely participate in these optimizations.

Consider a query filtering customer records by a masked email domain. The optimizer can’t push that filter to the underlying table scan because the masking transformation happens after the data is retrieved. Instead of scanning only relevant partitions, the query engine must retrieve all customer records, apply masking, and then filter the results. 

Query caching presents another optimization challenge. BigQuery documents a specific scenario where cached results can show previously visible unmasked data for up to 24 hours after user privileges change. This happens because the cache layer isn’t always aware of the security context that determined whether masking should apply to a particular user’s query.

The identity propagation problem

Most enterprise data pipelines run under shared service accounts rather than individual user identities. This creates a fundamental mismatch with dynamic masking systems that evaluate policies based on the identity making the query. When your ETL jobs, BI tools, and analytics platforms all connect using technical service accounts, the sophisticated role-based masking policies you’ve configured simply don’t apply.

Snowflake’s masking policies evaluate based on the current user and role, but if every query comes from “etl_service_user,” every result gets the same masking treatment. Organizations end up choosing between operational simplicity (service accounts) and granular security (individual user propagation), often defaulting to the former and losing the benefits of their masking investment.

Starburst addresses this through identity pass-through capabilities that can propagate OAuth tokens or JWT assertions from end users through to the underlying data sources, enabling Snowflake’s native policies to evaluate the actual user context rather than a generic service account.

Cross-platform policy fragmentation

Enterprise data architectures rarely consist of a single platform, which means masking policies must be configured and maintained across multiple systems with completely different models. Snowflake uses masking policies applied to columns, BigQuery relies on policy tags and data policies, and SQL Server implements dynamic masking through column-level rules.

Each system has different capabilities and constraints. Snowflake requires that masking policy functions have the same input and output data types. BigQuery doesn’t support masking on partitioned or clustered columns. Some platforms support format-preserving masking while others only provide simple substitution options.

Teams end up maintaining parallel policy definitions across platforms, leading to configuration drift, inconsistent protection levels, and complex testing requirements. A policy change that needs to be applied across Snowflake, BigQuery, and a PostgreSQL instance requires three different implementation approaches and validation procedures.

Data pipeline compatibility challenges

Masking can break downstream systems that depend on unmasked values for key operations. Change data capture (CDC) systems typically require access to primary keys to maintain consistency across replicas. Fivetran specifically documents restrictions that occur when Snowflake masks primary key columns, as this can prevent proper synchronization and cause data pipeline failures.

Similarly, systems that need to perform joins across multiple tables may require access to unmasked foreign keys to maintain referential integrity. While you can use deterministic hashing to preserve join relationships, this requires coordinated implementation across all systems that participate in these relationships.

Getting started with data masking in your organization

Successfully implementing data masking requires a thoughtful approach that balances immediate protection needs with long-term scalability. The key is starting with clear use cases and building systematic capabilities rather than trying to solve everything at once.

Choosing your initial masking strategy

Begin by identifying your highest-risk, highest-value scenarios. These typically involve datasets that combine regulatory sensitivity with broad analytical usage. Customer transaction data used for fraud detection, patient records for medical research, or employee data for HR analytics often represent good starting points because they have clear compliance drivers and well-defined user communities.

For these initial use cases, decide between leveraging native platform capabilities or implementing centralized governance. If your data primarily lives in a single platform like Snowflake or BigQuery, native masking policies provide the simplest path forward and integrate seamlessly with existing query performance optimizations.

However, if you’re working across multiple platforms or need more sophisticated policy management, the Starburst Enterprise with its unified access control capabilities can provide consistent masking policies across diverse data sources. This approach is particularly valuable when you need tag-driven policies that can scale across many tables and columns without individual configuration.

Solving the identity challenge first

Before implementing any masking policies, establish how user identities will flow through your data access layers. Configure OAuth or JWT pass-through where supported to ensure that masking policies evaluate based on actual end-user contexts rather than service accounts.

For scenarios where identity pass-through isn’t feasible, design service account strategies that align with your masking requirements. You might create role-specific service accounts (analyst_service_user, scientist_service_user) that receive different masking treatments, or implement application-level controls that apply masking before presenting data to users.

Starburst’s integration capabilities with SSO and SCIM systems can help maintain consistent identity and group membership across your data platform and masking policies, reducing the administrative overhead of keeping permissions synchronized.

Performance testing and optimization strategies

Plan for the performance implications of masking from the beginning. Set up test scenarios that measure query performance with and without masking applied, paying particular attention to queries that filter or join on masked columns. Starburst’s optimization features like predicate pushdown and dynamic filtering can help mitigate some performance impacts, but you’ll need to understand the trade-offs for your specific workloads.

Consider implementing caching strategies that can reduce the computational overhead of repeatedly applying masking transformations to the same underlying data. Cached views can be particularly effective for frequently-accessed datasets where the masking rules are stable.

When working with modern open data lakehouse architecture and comparing open table formats, performance optimization becomes even more critical. Organizations using Apache Iceberg can benefit from optimizing Iceberg table performance to minimize the overhead of applying masking transformations at query time.

Building audit and compliance capabilities

Implement comprehensive logging of masking policy changes and data access patterns from day one. Starburst’s audit capabilities can track both policy modifications and query executions, providing the detailed records you’ll need to demonstrate compliance and investigate any potential issues.

Establish processes for testing policy changes in non-production environments and documenting the business justification for different masking approaches. Regulators and auditors will want to understand not just what data is masked, but why specific techniques were chosen and how you validate that they’re working correctly.

Scaling beyond initial implementations

As your masking program matures, leverage tag-based policy management to reduce the operational overhead of maintaining policies across growing data catalogs. Tags allow you to define masking rules once and apply them consistently across any table or column that receives the appropriate classification.

Consider integrating with enterprise governance platforms like Ranger or Immuta if your organization has already invested in these systems. These integrations can help maintain consistency between your masking policies and broader data governance initiatives.

Organizations should also consider the data product lifecycle stages when implementing masking, ensuring that protection policies evolve appropriately as data products mature and gain broader usage across the enterprise.

The goal is building a masking capability that becomes a transparent part of your data platform rather than an additional layer that requires constant attention. When implemented thoughtfully, data masking enables organizations to say “yes” to more data requests while maintaining appropriate protection, ultimately accelerating analytics and AI initiatives rather than constraining them.

Imagine, for example, a company chatbot that tells a customer it can resolve an issue with a customer’s delayed invoice payment. But what happens when it fails halfway through the attempt because it lacks full visibility into how invoices are actually approved internally? Unfortunately, without proper context engineering, this can happen easily and all too often in production. At its root, this problem is almost always to do with AI having general context but not specific context. This means that your agent knows generic things, but doesn’t have the years of accumulated knowledge that someone in  Accounts Payable has accumulated over two decades navigating every possible blocker.

Identifying such policies is table stakes for a successful agent deployment. They lie scattered across your business. Everyone knows them intuitively, even if they’re not written down or are slightly out of date. 

Everyone, that is, except your agent, because AI agents are not very good at intuition without context. Unfortunately, this never resides in a single drive or database marked “business context. Context isn’t all in one place, it’s spread out. 

Where is it? Most of us assume that context is out there, somewhere. Maybe in an as-yet-undiscovered data silo. In many cases, though, the data AI agents need lies in uncharted territory. 

Check out this video from our CEO, Justin Borgman explaining this exact topic. 

In this article, we’ll look at why so much valuable business context remains unmapped, why writing it down is a necessary precursor to the efficiencies inherent to AI, and how to make that context accessible to your agents. 

The blank spaces on the map 

A lot of problems arise from the fact that agents can’t access your data, and Starburst has a particular strength on that front to its focus on data federation. But what if the problem is different? What if the problem isn’t that this data isn’t accessible, but that it doesn’t exist as data at all? That’s a different kind of problem. You can’t access non-data at all because it needs to become data first to be on the map in the first place. In this sense, telling AI to find it and work with it is like telling a ship captain to sail into the blank spaces on a map and take actions on the basis of it. Once they get to the map’s edge, they’re flying blind.

The data backs this up. Most organizations don’t think their data is AI-ready, and a big part of that involves the deficiencies around context engineering in an enterprise setting. A Harvard Business Review Analytic Services study found that only 7% of enterprises think their data is completely ready to serve the next generation of autonomous agents. 

Across the survey, the majority (56%) said their main obstacle was difficulty accessing data and integrating data sources. But a far harder problem lingers beneath the surface. 

The goal with AI agents is for them to have enough information to make autonomous decisions. To do that, they need to know what your employees know.  In the invoice case above, for example, they need to know who’s involved in the approval process, what commonly causes processing delays, and common workarounds for known issues. 

Accessing that information requires more than just setting up a new data connector. It requires writing it down in the first place so it doesn’t exist only in the heads of your employees. 

The missing piece of the context puzzle

We can think of business context as falling into three categories: 

• Technical context. This is the data about your data, or metadata, including descriptions, last updated times, owners, and data lineage.
• Business context. The metrics and key definitions, such as how revenue is calculated, that constitute the bleeding heart of your business. This is the language of how your business works.
• Process context. How various business workflows operate from end to end, along with the pressures, business culture, and capabilities that influence human decision-making, for example your company’s mission statement, or institutional knowledge about your competitors.

The first is easy to acquire and populate using existing technology. The second is possible, but requires more work, as it involves documenting the company’s most critical business definitions in a semantic layer.

Naming the territory

These blank spots on the map have existed for years. Sometimes, humans draw them in. Most of the time, they leave them blank. Or they write them down once and fail to update them when they change. 

There have been good reasons why much of this stays uncharted. Familiarity means that traditionally, there has never really been a need, and organizations and people alike don’t tend to invest time and energy in things that aren’t needed. If you live in a neighborhood, you don’t need to draw a map of it. You know where the local grocery store and the post office are without writing this information down. You know to avoid turning down Harrold Street because it’s a dead end. 

Human minds are context engines. That enables us to do a lot without committing it to paper. We can hold knowledge in our heads and pass it on to other humans, such as during the employee onboarding process. Writing this information down and maintaining it is extra work that falls by the wayside when things get busy. Additionally, before the advent of AI, there was no good way for machines to read and use this information, which limited its use in business automation. 

In other words, sailing into uncharted waters has been hard work with little reward. 

There’s a tendency for companies to say “let’s use AI to fix this.” But AI can’t help here by definition, at least not out of the box. 

LLMs are like mixing machines. They can slice, dice, and recombine existing content. They need something original to work from.  Without this raw material, they can’t author business context that no one’s ever recorded. If you try to force them, the answers will be hallucinatory and, often, hilariously wrong. 

Mapping this uncharted territory requires humans to set sail and navigate it themselves. In data terms, it requires a human-led effort in context engineering to chart, describe, and label the no-man’s lands of your business context.

This mapping work requires changing how you think about documenting work, and developing a culture of turning context into data. The form of this will differ depending on what you need to capture. It may be a table connected to business logic held in a semantic layer, a description of a process, an Excel spreadsheet, or even a transcript from a recorded meeting. 

What’s important is that this context is captured somehow and somewhere in a form that can be supplied to AI agents to enhance their decision-making. 

What needs to be written down 

Context engineering isn’t a one-off, it’s an ongoing process. You can focus on context periodically to try to catch up on what you’ve failed to document so far. But ultimately, you need tools and processes in place to write down the new context your business generates every week. 

This context includes, but is not limited to: 

Workflows detailing how things get done 

This is perhaps the most critical gap businesses need to fill. If agents are going to act autonomously, they need to understand how your business operates. 

This isn’t just limited to publicly available chatbots. AI agents need to understand the conditions, guardrails, and human approvals needed before making many business-critical decisions. 

Without this context, agents might:

• Throw out a sales lead because of a low-quality score. The agent didn’t know that inquiries from named accounts should always go to their account executive regardless of lead score. 
• Resolve a support ticket unilaterally with a link to a knowledge base article. The customer was a tier-1 account, and their issue was security-related, necessitating their Customer Success Manager be looped in first.
• Delete records and files it thinks aren’t in use. The company was required to keep those records due to a legal hold or to conform to data regulations such as GDPR.

Meaning detailing definitions and the rules behind them 

It’s not enough to document how you calculate terms like active customer or churn. You need to provide the reasoning, logic, and rules behind that decision as well. 

Many of these calculations reflect decisions that never make it into a document. Why do you count renewals the way you do? How do you identify (and exclude) test accounts? How do you score and distinguish real leads from junk queries?

A number without context is just a number. The rules and the reasoning behind them provide invaluable insights that agents can leverage to make autonomous decisions. 

Connection detailing how data relates

Another common gap in knowledge is how data relates across data sources. For example, finance may have an “account” entity. Is that the same as the “customer” entity in the sales CRM? And what about the “org” entity in the product database?

These may all refer to the same entity, or they may refer to slight variations of one another. Often, this is the kind of knowledge a data engineer or senior analyst keeps in their head when devising new queries. They know intuitively how to perform a distributed join on a cleaned ID, not the raw one. In other words, they exercise judgement on the basis of their context. 

This is the kind of thing that, without explicit knowledge, an agent attempts to infer. Sometimes, it’s right. Often, it isn’t. 

Bringing data back from the badlands

It’s trendy to call agents coworkers. But most agents aren’t close to achieving that status, and they won’t be without overcoming the context gap. 

That’s because we assume all the data they need is out there somewhere. But much of it isn’t anywhere accessible to agents. Most of it is still in employees’ heads, constituting a vast sea of uncharted territory. 

Unless you can turn this context into data, your AI agent projects won’t succeed. They’ll always be operating on a partial, limited view of your business, and that is not sufficient. For agents to be true coworkers, you need to fully map the unexplored territory that your employees navigate almost on instinct.

Tools and technology can help here. Starburst ensures you can access data wherever it lives, and helps you build an enterprise context layer to give agents the context they need.

No tool in the world, though, can help your AI agents access territory that doesn’t exist on any map. Only by naming and describing these uncharted lands can you ensure that your AI agents have the context they need to make decisions that reflect how your business actually works.

Want to know more about how Starburst can access context wherever it lives? Read our whitepaper outlining how to build an agentic workforce with context in mind. 

That missing architectural component is known as a context layer. It operates above storage and compute, and acts as the layer of meaning across your entire data ecosystem. 

The context layer provides the ground truth to bridge the gap between raw database schemas and downstream consumers. It equips human analysts and AI agents alike with the exact business intelligence needed to deliver consistent, scoped, and fully auditable answers across every tool in your stack.

Key takeaways

• A context layer is a governed tier, not a single tool. It sits between data and consumers, both humans and agents, and supplies certified business meaning.
• It has three jobs: structure, meaning, and governance.
• A semantic layer is necessary but not sufficient. Consistent metrics in one data warehouse do not resolve identity or business rules across systems.
• Data products are the reusable unit. Definitions, owners, and rules travel with the data instead of living in a disconnected wiki.
• Agents need scoped, steward-approved context before they reason. A raw schema dump produces confident, untrustworthy answers.

What is a context layer?

Think of the context layer as a layer of verified meaning, not as another copy of the data underlying it. Importantly, it sits above storage and compute, but below BI tools, notebooks, and AI assistants. Its job is to apply shared definitions, metadata, and policies so raw data becomes usable by both humans and agents.

You can think of the context layer as the part of your data architecture that makes meaning possible, especially for agentic AI. AI agents require context to know how to operate, to give it the same business intelligence your analysts already use, in a form it can retrieve on every question.

What sits inside the context layer

A working context layer is not one thing but many. Overall, you can think of as a functional tier with three jobs: structure, meaning, and governance.

Structure, meaning, and governance

Structure involves taking inventory of the data sources and their meaning, including which tables, files, metrics, and data products exist, which domains own them, and which joins are valid. Without structure, an agent searches a pile of similarly named columns and guesses.

Meaning acts as the translation layer for how your organization operates. It resolves entity identity across systems, recognizing that a Customer ID in an S3 marketing datasource represents the exact same entity as a Client Code in a billing database. It records these cross-system links alongside metric formulas, synonyms, and domain owners.

Governance applies policy enforcement at the precise moment data is used. It manages access controls, certification states, and audit trails that trace every generated answer back to its underlying definition and human steward. Unverified metadata stays in a pending state until a steward confirms it, ensuring agents reason against certified assets rather than raw, inferred query history.

How a context layer differs from a semantic layer

A semantic layer provides business intelligence tools with consistent metrics, dimensions, and schema relationships. That functionality remains essential, as teams like finance and product should never calculate core metrics like monthly recurring revenue using conflicting formulas.

However, a semantic layer represents only one piece of a complete context layer. Standard metric models compiled to SQL typically assume visibility over a single data warehouse or lakehouse. By themselves, they cannot resolve an entity like a customer that exists under three separate identifiers across CRM, billing, and support systems. They also fail to capture non-metric business rules, such as excluding internal test accounts, flagging deprecated tables, or accounting for fiscal calendar exceptions.

Overall you should see the semantic layer as an input rather than the entire solution. In contrast, the context layer manages entity identity across systems, enforces steward certification, and delivers a scoped package of meaning directly to an AI agent. Simply renaming a semantic model as context leaves AI agents guessing across persistent data silos.

How it works in practice

In practice, a context layer operates as a continuous four-part loop.

Part 1

First, it harvests live metadata across your existing stack, drawing from data catalogs, transformation pipelines, BI workbooks, and query logs. Most of the business meaning you need already exists across your organization, so the goal is to collect and unify that tribal knowledge rather than rebuild it from scratch.

Part 2

Second, it structures harvested metadata into certified business semantics. Human users review and approve proposed entities, metric formulas, and valid join paths. Any unverified metadata remains in a pending state until confirmed, ensuring unvetted assumptions never reach downstream consumers.

Part 3

Third, it assembles a scoped package of context tailored to the specific question being asked. An AI agent querying customer revenue does not need access to your entire data estate. Instead, it receives only the specific tables, metrics, join paths, and business rules relevant to that request, sized precisely to fit within its context window.

Part 4

Fourth, it serves that assembled context to the agent before the model begins reasoning. This delivery step provides the essential grounding that prevents hallucinations. 

Why agents need a context layer

Human analysts naturally stop and ask a clarifying question when a data definition or output looks wrong. Autonomous AI agents, on the other hand, frequently infer and execute queries without double-checking. This explains why a prototype that looks impressive against a clean, sample database collapses in production. The model possesses general technical skills, but it lacks the operational reality of your business.

A context layer provides the architectural bridge required to move AI prototypes into production reliably, without relying on larger model context windows as a substitute for governance. It allows you to publish data products, certify metric logic, retrieve scoped metadata per question, and maintain an audit trail tracing every output back to its underlying definition.

This operational loop enables you to point AI assistants directly at live, governed enterprise data rather than static, private extracts. Solutions like Starburst AIDA demonstrate this pattern by providing natural-language access to enterprise data. By embedding data product metadata and business rules as session context, the assistant remains securely anchored to a trusted, audited product rather than navigating a raw, ungoverned schema dump.

Choosing a data platform to access your context layer

Starburst provides the context layer capable of accessing all of your enterprise’s data. Because it is based on data federation, you do not need to rebuild your architecture or deploy four new systems to get started. You simply need to decide where certified business meaning lives, ensuring it no longer exists solely as unwritten tribal knowledge, and then access it.

Whether operating on a fully managed lakehouse or a self-managed deployment, query federation reaches across your data lakes and cloud data warehouses without forcing you to copy data each time. The context layer then evaluates those federated sources in real time, deciding which dataset represents the authoritative source of truth for a question and enforcing row-level security before a query executes.

The context layer turns business meaning into a core platform feature, ensuring that human analysts and autonomous agents reason from the exact same certified ground truth.

Getting started

Getting started does not require an all-at-once migration. Start with the data sources and data products you already offer internally, attach domain owners and metric logic, and check it against your business logic. By serving those unified definitions to BI dashboards and AI agents alike, you can systematically scale governed context domain by domain.

Ready to build a governed context layer for your enterprise? Download The Technical Guide for Scaling Your Agentic Workforce to learn how connecting AI agents to certified business context turns experimental prototypes into trusted production systems.

FAQs

What is a context layer in a data platform?

It is the governed tier between your data and the tools that consume it. It stores verified definitions, relationships, and policies, then serves a scoped package of that meaning for each question.

How is a context layer different from a semantic layer?

A semantic layer keeps consistent metrics, dimensions, and relationships for BI. A context layer includes that work and adds identity across systems, steward certification, and retrieval for agents.

What does a context layer contain?

A context layer contains structure, meaning, and governance. It operates as an inventory of assets, business definitions and relationships, and the controls that decide who may use them and how answers are traced.

Why do AI agents need a context layer?

Agents do not ask clarifying questions the way analysts do. They need assembled context served before reasoning so they do not guess from raw schemas.

Is a context layer a catalog or a wiki?

No. A catalog lists assets. A wiki captures tribal knowledge. A context layer binds certified meaning to live data and serves it at query time.

How do data products relate to the context layer?

Data products operate as curated, governed datasets that structure context. 

In today’s data architectures, pushdown serves dual purposes. Teams use it for exploratory data federation, querying across multiple sources without moving data, and for ingestion patterns that materialize results into target tables while minimizing load on source systems. With Starburst, any successful federated query can become a materialized view in object storage, transforming pushdown-optimized reads into persisted data assets that power downstream analytics and AI workloads.

The challenges teams face implementing query pushdown are real and costly. Unpredictable performance across different connectors, spiraling egress charges, API limits that throttle critical jobs, and observability gaps make it impossible to verify what’s actually being optimized. This is where Starburst’s comprehensive approach to pushdown optimization becomes invaluable, providing enterprise-grade connectors, intelligent cost controls, and the tooling needed to operationalize pushdown across your entire data ecosystem.

Query pushdown has become critical as organizations scale their data operations because it directly addresses the three biggest bottlenecks in modern analytics: 

• Network transfer costs
• Compute resource waste
• Time to insight

When your marketing team needs to analyze customer behavior across CRM data in Salesforce, transaction records in PostgreSQL, and web analytics in BigQuery, pushdown ensures you’re only moving the specific rows and columns needed rather than entire datasets.

The economics of selective data movement

Let’s say you’re joining customer data from a 50-million-row table in Snowflake with a much smaller dimension table in your lakehouse. Without pushdown, you’d transfer all 50 million rows across the network, then filter and aggregate in your query engine. With effective pushdown, filters and aggregations execute within Snowflake, and only the pre-aggregated results cross the network boundary. This vastly reduces data transfer costs by orders of magnitude, especially when working across cloud providers or regions.

Organizations processing massive volumes see this impact clearly. For example, Going processes over 50 petabytes annually and delivers timely price alerts using Starburst Galaxy, with pushdown optimization playing a crucial role in maintaining both performance and cost efficiency at scale.

Enabling rapid data product development

Meanwhile, companies like Kovi have used Starburst with Iceberg to speed ad-hoc queries and ETL while cutting compute costs. The approach means writing federated SQL that selects only the data needed using pushdown, then materializes results using CREATE TABLE AS SELECT or materialized views. This approach enables teams to create and manage data products without the traditional overhead of complex ETL pipelines.

Cross-border analytics and data sovereignty

For organizations with data residency requirements, pushdown becomes even more critical. Rather than moving entire datasets across borders or cloud regions, capabilities like Starburst Stargate allow AI workloads and analytics queries to operate without mass data relocation, enabling ingestion and materialization only of permitted, policy-filtered subsets per region.

The technical realities that complicate query pushdown

Despite its clear benefits, implementing effective query pushdown requires a strategic approach. Understanding these obstacles helps you plan realistic timelines and architect solutions that actually work in production.

Inconsistent pushdown capabilities across connectors

Typically, the biggest challenge teams face is that pushdown support varies dramatically between different data sources. Whether a filter, aggregation, or join can be pushed down depends entirely on the specific connector. Some connectors restrict certain predicate types, meaning string operations or range queries that work perfectly with one source might not push down to another. This inconsistency makes it difficult to predict query performance and costs when working across heterogeneous environments.

You might write SQL expecting consistent behavior across your PostgreSQL and Oracle sources, only to discover that complex WHERE clauses push down to PostgreSQL but execute locally when querying Oracle, completely changing your resource requirements and execution time.

How Starburst helps manage this unpredictability and cross-cloud complexity

Starburst addresses this by disabling join pushdown by default to help avoid unexpected warehouse compute charges. This exemplifies a broader challenge, including join operations in remote warehouses can drive surprising costs, and cross-region or cross-cloud data egress creates additional charges that are difficult to predict and budget for.

This ensures that teams do not discover these cost implications after they’ve already committed to an architecture. Other technologies can cause seemingly efficient federated queries to trigger thousands of dollars in warehouse compute charges or egress fees, especially when BigQuery or Snowflake data transfer costs compound across regions.

Why these challenges persist in other modern architectures

These pushdown challenges persist because they reflect fundamental tensions in modern data architecture. The fragmentation of data platforms and protocols means that connectors need to map very different capabilities, each with unique limits for predicates, aggregates, and joins. This prevents teams from implementing uniform ingestion policies across their data ecosystem.

Legacy and enterprise sources compound this complexity. Systems like Oracle, Teradata, Db2, SAP, and Salesforce expose varied SQL dialects, query planners, and transactional constraints. Each requires specialized connector logic, and pushdown support remains uneven across this landscape.

Governance and data sovereignty requirements add another layer of complexity. Cross-cloud and cross-region data movement triggers residency and privacy controls, requiring fine-grained policies across many systems. Even correct pushdown can create compliance issues when filtered data crosses jurisdictional boundaries unexpectedly.

Finally, cloud cost models tied to scanned bytes and egress create moving targets for optimization. Pricing policies change regularly, and what appears to be an efficient pushdown strategy today might become expensive tomorrow as providers adjust their transfer and compute pricing.

Your path to production-ready query pushdown

Getting started with query pushdown successfully requires focusing on verification first, then scaling systematically as you understand the behavior patterns of your specific sources and workloads. 

Starburst is designed specifically to work within this framework. 

Start with pushdown verification

Before committing to any ingestion architecture, establish a practice of using EXPLAIN statements to confirm which operations actually push down. Simple AND-ed predicates typically push down reliably, while complex expressions often don’t. Build this verification into your development process so teams can iterate on SQL to maximize pushdown before deploying production jobs.

Consider operationalizing this verification in your CI/CD pipeline. If a critical ingestion query suddenly loses pushdown capability due to connector updates or source changes, you want to know before it impacts production performance and costs.

Leverage enterprise-grade connectors with intelligent defaults

Starburst’s enterprise connector portfolio includes purpose-built connectors for systems like Snowflake, BigQuery, Teradata, and Salesforce that are specifically optimized for pushdown scenarios. These aren’t just enhanced versions of open-source connectors; they include parallel read capabilities, managed statistics, and intelligent cost controls.

For example, the Snowflake connector uses Arrow-based parallel reads and disables join pushdown by default to help control warehouse compute costs. The BigQuery connector leverages the BigQuery Storage API and supports multiple pushdown operations to reduce CPU usage and improve performance.

The Starburst Enterprise provides additional capabilities for organizations that need enterprise-grade security, compliance, and support for their pushdown implementations.

Build materialized views for repeatability

Rather than running federated queries repeatedly, use Galaxy materialized views to persist the results of pushdown-optimized queries in object storage. This pattern lets you pay the cost of source-side processing once, then serve multiple analytics and AI workloads from the materialized results.

Set up scheduled refreshes to keep derived tables current, and consider incremental refresh capabilities where supported to reduce processing overhead for frequently updated sources. This approach aligns with modern data transformation pipelines that prioritize efficiency and automation.

Implement intelligent performance optimizations

Enable dynamic filtering for selective joins to reduce fact-side scans during ingestion operations. This runtime optimization can dramatically reduce the amount of data processed even when explicit predicates don’t push down effectively.

For large extraction jobs, configure fault-tolerant execution to handle worker failures during long-running ingestion processes. Verify that your connectors support retry policies before enabling this capability.

Organizations can also benefit from optimizing Iceberg table performance through techniques like sorted tables, which complement pushdown strategies for open data lakehouse architecture.

Monitor and respect source system limits

For SaaS sources, monitor API consumption through system tables so you can track quota usage and plan extraction schedules accordingly. For OLTP systems, schedule extraction windows, throttle parallelism settings, and consider read replicas to minimize contention with production workloads.

Configure parallel read settings like connection counts for high-throughput scenarios, but balance this against source system capacity.

Apply governance before ingestion

Rather than moving data first and applying policies later, use row filters and column masking in your ingestion SQL so only compliant subsets are materialized. This approach ensures that your pushdown optimization doesn’t accidentally create compliance issues by moving sensitive data across boundaries.

Implement audit trails for refresh jobs and access patterns so you can demonstrate that your ingestion processes respect data governance requirements. This is particularly important for organizations in financial services data analytics or healthcare data analytics where regulatory compliance is critical.

Organizations looking to modernize legacy systems can leverage data migration solutions that incorporate pushdown optimization from the start. Whether you’re implementing Hadoop modernization or migrating to modern data architecture, starting with pushdown-aware design patterns helps ensure your new architecture delivers the performance and cost benefits you expect.

The key to success with query pushdown lies in treating it as an operational discipline rather than just a technical capability. Start small with well-understood sources, verify pushdown behavior systematically, and scale your approach as you build expertise with the specific patterns and limitations of your data ecosystem. With the right foundation and tooling, including solutions for building data applications and data for artificial intelligence, query pushdown transforms from a complex technical challenge into a reliable mechanism for efficient, cost-effective data ingestion that scales with your organization’s needs.

The answer exists. It has always existed, but it’s not all in the same place. The tickets are in one system, the spend is in a data warehouse, and the account hierarchy is in a PostgreSQL database behind the CRM. Three sources, all accessible, all containing exactly the data required. And yet answering the question takes two weeks, because before anyone can write the query, someone has to build a pipeline to bring all three into the same place.

SQL on anything removes that waiting period. The data does not move. Instead, the question gets asked by accessing data sources without centralization. This is the data architectural pattern built on data federation, powered by a compute engine like Trino, and it works whether your data sits in a lake, a warehouse, a lakehouse, a streaming system, or a SaaS application.

Let’s dive in. 

Key takeaways

• SQL on anything is not a new query language. It uses the standard SQL you already know, running against sources that stay exactly where they are.
• Connectors determine which data sources you can reach, spanning databases, data warehouses, data lakes, streaming systems, and SaaS applications.
• Centralizing everything into one store creates lock-in through proprietary formats, bundled pricing, and accumulated tooling. Federation preserves the option to change any one component later.
• Querying in place does not mean that you never copy data. You can still centralize when it makes sense, and that choice is still yours. What you avoid is copying everything upfront just to answer a question.
• Starburst Galaxy and Starburst Enterprise are the two ways to implement this using Trino.

What does SQL on anything mean?

SQL on anything refers to a single SQL query that spans different systems. You write one statement, it reaches several sources, and none of those sources have to be staged in a data warehouse beforehand.

The SQL itself is standard SQL. SELECT, JOIN, WHERE, and GROUP BY all behave exactly as they always have, and your query does not care whether a catalog resolves to PostgreSQL, to an Iceberg data lakehouse on Amazon S3, or to a data warehouse. Underneath, the compute engine plans the statement, asks each connector for its slice of the data, and combines the results before returning them.

Connectors define which data sources you can reach

Each connector maps one data source into tables, columns, and rows, which is what allows standard SQL to read from it. Different data sources can be accessed using different connectors. 

Starburst supports more than 50 connectors extending access to data lakes, data warehouses, streaming systems, relational databases, and SaaS applications. For example, when accessing a data lakehouse, a single query can read across Hive, Iceberg, Delta Lake, and Hudi tables, so mixed table formats do not need consolidating before you can join them. Operational databases and warehouses work the same way, as does any other source once a connector exists for it.

Why universal data access matters for analytics and AI

Most analytics work begins with a question that spans more than one system. Building a dashboard, packaging a curated data product around the underlying datasets, or grounding an AI agent all start from the same assumption, which is that the data can be reached. In most organizations that assumption requires a project behind it, and the result is a long tail of questions that never get asked because answering them would mean joining the pipeline backlog.

Getting past data centralization 

All of this points to the need to move beyond data centralization. When your architecture assumes every data source must eventually land in one store, that store begins dictating decisions well beyond where the data sits. You migrate toward it, you standardize on the formats it prefers, and you accept its pricing model as a fixed cost of asking questions.

Federation inverts the relationship. Data stays in whichever source genuinely suits it, whether that is object storage for scale, an operational database for latency, or a warehouse a business unit already depends on, and the query layer adapts to those choices rather than overriding them. This preserves optionality, meaning you keep the ability to change any one component later without re-architecting everything around it.

That flexibility matters most in the situations that have become ordinary rather than exceptional:

• Hybrid deployments, where some data must remain on-premises while analytics runs in the cloud.
• Post-acquisition estates, where an acquired company arrives with its own warehouse, formats, and conventions, and a full migration would take years.
• Data sovereignty requirements, where regulation keeps certain records inside a specific region and migration is not legally available as an option.

In each of these cases, centralizing first is expensive, slow, or prohibited outright. Federation is what makes the estate queryable without waiting for a migration that may never be justified.

AI raises the stakes on access

A model can write syntactically correct SQL and still return a wrong number, because the business context it needed lived in a source it could not query. This is a common way enterprise AI projects stall, and it is an access problem rather than a model problem.

That context is rarely consolidated in one place. Fiscal calendar definitions may live in a finance system, customer hierarchy in a CRM, and transaction detail in a lakehouse. An agent reasoning across all three needs access to all three, not to a subset that happened to be convenient to centralize.

Access without governance is a liability

Being able to query everything is only useful if you can also control who queries what. Those controls have to be enforced at the engine rather than separately at each source, because a policy applied in three systems and missed in the fourth is not a policy.

In practice this means row filters restricting which records a given role can see, column masks hiding sensitive fields such as salary or account numbers while leaving the rest of the table usable, and role-based or attribute-based access control applied consistently across every catalog the engine can reach. A semantic layer sits alongside these, defining shared meaning so two teams querying the same source arrive at the same numbers.

There is a point here that runs against intuition. Federation is often assumed to weaken governance, on the grounds that reaching more sources means more exposure. In practice the opposite tends to hold, because the governance problem in most organizations is not reach but duplication. Every pipeline built to make a source reachable produces a copy, and every copy is a new surface that needs its own permissions, its own masking rules, and its own review when a regulation changes. A customer table replicated into three environments is three places where someone can be granted access by mistake.

Querying in place removes those intermediate copies, which means fewer places to secure rather than more. It also gives you a single point where access is recorded. When every query runs through one engine, the audit trail covers all of it, and questions like who read this table last quarter have an answer that does not require assembling logs from four systems with four different formats.

The sequencing matters as much as the controls. Governance retrofitted after teams have grown accustomed to broad access is a political problem rather than a technical one, because you are taking something away rather than defining it up front. This is why access and governance have to advance together rather than sequentially.

Federation gets the query to the data. It does not define what your business means by revenue, which customer table is authoritative, or who is permitted to see either. Those remain decisions your organization has to make, and the engine’s job is to enforce them consistently once you have.

When you should still centralize data

Federation does not make centralizing obsolete. What it changes is the status of the decision, turning consolidation from a prerequisite into a choice you make for a specific workload. The useful question stops being whether to centralize and becomes which subset genuinely warrants it.

These often include: 

• A governed persistent table that multiple teams depend on and that should not change without review.
• A dashboard with an SLA behind it, where query latency against a slow operational source is not acceptable.
• Training data that must stay fixed while a model is developed and evaluated.
• Historical facts accumulating over years, which belong in a data lakehouse table rather than being repeatedly pulled from an operational system.

What these share is a workload that benefits from a stable, governed copy. Everything outside that set can be queried where it already lives, which is the difference between centralizing deliberately and centralizing by default.

Getting started with SQL on anything

How do you begin? Start with a question your organization has postponed because it spans two systems. Most teams have several of these. Connect both sources as catalogs and write the join, which will usually take less time than scoping the pipeline that was going to answer it.

Audit your connector coverage against the sources that matter most. Where a connector exists, federation replaces integration work immediately. Where one does not, you know which sources still require copying and can plan accordingly.

Finally, put the governance layer in place before you need it rather than after. Access control, views, and shared definitions are what keep broad query access from becoming a liability as usage grows across teams.

Want to see it in practice? Check out Starburst Galaxy and connect your first two sources.

FAQs

Is SQL on anything the same as data federation?

Federation is the mechanism that executes a query across sources without copying them into one repository. SQL on anything describes the resulting capability, which is one SQL dialect reaching across all of those sources. 

Which data sources can I query?

Any source with a connector that maps it to tables, columns, and rows. Starburst supports more than 50 connectors across data lakes, data warehouses, streaming systems, relational databases, and SaaS applications. What matters as much as coverage is connector depth, since a connector that supports pushdown lets the source do the filtering while a shallow one pulls raw data into the engine. Where no connector exists yet, you are back to copying data or building a custom plugin.

Do I still need a data warehouse?

You may still run one, and plenty of organizations should. What changes is that you no longer have to copy every other source into it before you can join across them. The warehouse becomes one participant in a federated architecture rather than the mandatory destination for all data, which also means you are no longer paying to store copies of data whose system of record lives elsewhere.

How does this stay governed?

Virtualization, access control, and catalog policy have to travel with the query rather than being enforced separately at each source. That means row filters, column masks, and role-based or attribute-based access control applied consistently across every catalog the engine can reach. 

What is the difference between Galaxy and Enterprise?

Starburst Galaxy is built for cloud workloads, and is the easiest way to operate data federation using SQL the cloud. Starburst Enterprise is self-managed, and should be chosen when data has to remain inside a private, hybrid, or on-premises environment. Both implement federated SQL on Trino and both give you the same SQL interface, so the choice comes down to who runs the infrastructure and what your data residency requirements are.

Does SQL on anything mean I never copy data?

No. You copy a subset when you need persistence, a stable dashboard grain, or training data that must not shift underneath you, and those are legitimate architectural decisions. What changes is that copying becomes a choice you make for a specific reason rather than a prerequisite you complete before anyone in the organization is able to ask a question.

For example, a grocery store may want to know: “was my sale on eggs last week profitable?” Obviously, this is not a trivial question to answer. It cannot be answered by simply looking at egg sales that week vs other weeks. To answer this question correctly, higher level questions need to be answered. Did this sale bring in customers who would not otherwise have come? Did it bring in new customers who have never entered the store before (and have they come back again since that sale)? What else was bought in the same transaction as eggs? Are they physically located near the eggs in the store? How profitable are those other items? 

It is precisely these types of non-trivial questions — questions that may have a series of followup questions and many rounds of analysis of one or more datasets — that agents have the potential to provide an enormous amount of value. Instead of a human having to attack these questions from all of the relevant angles, the agent can take control and perform a lot of the grunt work, with some amount of guidance and direction from a human. 

Everybody knows that AI tools in general — and agents are no exception — are only as good as the humans who are guiding them. If you don’t know what to ask or to request, or how to optimally phrase the request, the result might fall short of what the tool is able to accomplish. Some people are better at using AI than others, and those people get the most out of these tools.

Aside from the way the request is made, there is another, and perhaps more important factor in the quality of the output: the quality (and amount) of the input data to the request. Most are familiar with the concept of attaching input data or files to AI prompts. For enterprise users, this may involve handing over keys to one or more enterprise database systems to an AI agent to use in answering specific questions. In our example above, the agent will need access to the relevant transaction and user data to answer the question of “was my sale on eggs last week profitable?”

In most non-trivial scenarios, more than one database may contain relevant data for a particular interaction. For example, the transaction data for our grocery store application may be stored in one database, and the customer loyalty (e.g. loyalty card information that contains information about the name, address, and other demographic information) data may be stored in an entirely different database. This demographic information is crucial to answer questions about the profitability of a particular sale — especially when breaking down the data across different demographics for the purposes of targeted campaigns). 

In these scenarios, the key question is how to pass the data in these different database systems to the AI agent. Whether this is done well or poorly will have strong implications on the experience of the users across three critical dimensions: the correctness/quality of the output, the speed/latency of getting this output, and the cost (e.g. in tokens) of the tool in responding to user prompts. 

A comparison of two (or four) alternative architectures

Option 0: Give the agent a dump of the raw data

One potential option for getting the data to the agent is to simply extract the data from the source and then send this extracted data to the agent. For example, the database system can be asked to write the data from the relevant tables to a file (or set of files). This file or files can then be sent as an attachment to the agent. Alternatively, the data could be piped more directly from the database to the agent. Either way: the point is that the raw data gets sent to the agent, and any subsequent data processing that is required, it will do itself.

This option can potentially get costly. To the extent that agents charge per data item received/processed, sending terabytes of data to the agent can get prohibitively expensive. Furthermore, there are significant performance disadvantages relative to option 1 that I will discuss shortly.

These disadvantages of option 0 reduce its practicality to the point where it is not a good option in practice. This is why I’m calling it “option 0” — it is not something that I would recommend outside of specific unusual circumstances. 

Option 1: Give the agent the keys to your database system and let it drive

The simplest practical way to pass data to the agent is simply by giving the agent direct access to the database system(s). The agent then writes its own SQL queries (or queries in any other language that is supported by the database system) to access data as it needs it.

By giving the agent direct access to the database, it can iteratively refine its requests over the data. Instead of giving the agent a giant dump of the entire database and being forced to locally perform advanced query operations over it (as in option 0 above), the agent acts as a user of the database — allowing the database execution engine to do what it does best:  process its own local data using carefully optimized query plans. The agent merely oversees this process, sending successive (or parallel) data processing requests to the database system until it reaches a satisfactory conclusion that it can return to its client.  

It has been noted in the literature that this type of usage can sometimes be a challenge for the database system. Agents can be far more demanding than humans, and can sometimes overwhelm the database system with speculative queries as it iteratively refines its focus on behalf of a user request. Nonetheless, the database industry is laser-focused on supporting agentic workloads moving forward, and modern database systems are increasingly capable of handling this type of scalable data processing efficiently. 

Option 1 yields better performance than option 0

Agents are very powerful and only getting more advanced over time. However, it will be a while before they can process terabytes of tabular/structured data at the same level of optimization as the elite database systems on the market today that have hyper-optimized code based on decades of research (not all of which may be in the public domain). Until that point, a major efficiency gain can be achieved by subcontracting the scalable data processing to the database system instead of trying to do everything itself.

Moreover, even if the agent was just as good as the database system in processing data, there is still an overhead in writing and passing the raw data to the agent. Data is often stored in a proprietary format inside the database system. To get the raw data out of the system, this format needs to be deserialized (and often decompressed) before being written to a file or being passed to the agent. This overhead can be avoided if the database system processes its own data, but must be paid if it is extracted from the system (as in option 0 above). 

Furthermore, if the data is larger than what fits in memory at the agent, it then has to be written to storage again once it reaches the agent, and serialized to the data format that the agent prefers (or the format used by the external system that the agent writes the data to). This write overhead is also avoided if the data is simply processed at its source, rather than being passed in raw form to the agent. 

The bottom line is that letting the agent function as a user of the database system is generally preferable to extracting data into the agent to perform processing there. Only if the database system is incapable of handling the scale or type of requests from the agent should a full data extraction be considered. 

We discussed above that in most non-trivial scenarios, more than one database may contain relevant data for a particular interaction. In these scenarios, the keys to each of these database systems would need to be passed to the agent. The agent is only as good as the data it has access to. If the agent is not given access to a database that may contain potentially relevant data, this data cannot be considered by the agent when being given a question to answer.  

Option 2: Give the agent the keys to a virtualization layer that includes access to potentially many database systems

Instead of giving the agent access to each database system (that may contain relevant data) individually, an alternative architecture is to first unify them using a virtualization (or federation) layer, and then give the agent access to the virtualization layer instead of each database individually. 

The difference between option 1 and option 2 is illustrated in the figure above. At first blush, options 1 and 2 appear very similar — the AI agent is issuing queries to three different databases and each one is returning query results. The only difference is that in option 1 the AI agent issues queries to the database systems directly, while in option 2, these queries first go through a virtualization layer. The virtualization layer is then in charge of rewriting these queries so that they can be sent to the underlying databases, and collecting the results before returning them to the agent.

There are, however, a surprising number of practical differences between these options, that impact the quality, speed, and cost of the results. I will discuss these differences below. But first I want to briefly mention a third option (that is not the focus of this post). 

Option 3: Use the agent that comes with your database system or virtualization layer

This whole post is assuming that you have an AI agent that you want to use that is a distinct piece of software from the database systems or virtualization tools that store and manage your data. However, it is important to remember that many of these systems have their own AI agents and tools that are specifically designed to work with the data managed by them. 

For example, Starburst’s AIDA tool is tightly integrated with its query engine and is capable of answering the questions you might ask an AI agent. This tight integration along with native knowledge of the data managed by Starburst and how Starburst optimizes its access of this data can often lead to a better quality set of results, and often at higher speeds.

Option 3 is so simple, there is no real need to discuss it in this post. When the AI agent you want to use is tightly integrated with your data architecture, this option is often a no-brainer. 

The goal of this post however, is to discuss the architectural options outside of option 3. For example, you might have an enterprise licence for a particular AI agent, or have other reasons why that agent is preferred (e.g. it might be better at accessing data outside of that which is managed by the database or virtualization system). In those scenarios, what is the optimal data architecture?

What are the downsides to having the virtualization/federation layer?

The main difference between options 1 and 2 discussed above is that option 1 gives the agent direct access to all the relevant database systems, while option 2 places a virtualization or federation layer between the agent and the database systems. Option 2 thus requires additional software that is not required by option 1. To the extent that this software is separate from the underlying database systems, this adds some additional cost and complexity to the data infrastructure.

The reason why this adds cost is obvious. If none of the existing database systems being used support this type of global data virtualization that manages access to its own data along with data stored by other database systems, then a separate software vendor needs to be contracted to provide this service. Unless a free and open source option is used (such as Trino) the software vendor must generally be paid for its services. Even if a free and open source option is used — the hardware and engineering costs of deploying the software still have to be paid. 

The addition of complexity comes from the need to configure the virtualization layer to interface with the underlying database systems. It needs to be given access to every underlying database system whose data it will be accessing. It needs basic information such as the location and port of the underlying database systems, the username and password (or other key information) that it should use to access this data, etc. It also may need metadata that describes the data inside those databases (to whatever extent the data schemas are insufficient). 

Furthermore, if the virtualization layer is a separate piece of software, there may be some effort involved in learning how to use it. Increasingly, vendors are reducing these efforts by providing cloud-based software-as-a-service options (such as Starburst Galaxy) that greatly reduce the training required to use the software. Nonetheless, everything requires at least some effort in getting familiar with its interface and capabilities.

There is also the complexity of choosing what virtualization software to use in the first place. Data virtualization is far from a commodity, and each software solution works differently. 

For example some data virtualization products only work with a limited set of database systems. If you are using a database that is not supported by the virtualization product, that database will necessarily be left out of the data being virtualized. Other products (such as Starburst) work with a large number of underlying database systems. 

There can also be a significant performance difference between virtualization products. Some products are not designed for massively parallel processing and have issues with scenarios in which it receives large amounts of data from the underlying database systems. Others (again — Starburst would be an example) are designed for scalability from the outset. The bottom line is that research must be performed when selecting a virtualization product, and this research adds complexity to the process. 

One other downside (although usually not significant) is the extra hop required to get requests to the database systems and results back. When the agent interfaces directly with the database systems, there is no extra hop. However, when there is a virtualization layer between them, queries and results must be routed through this layer. Usually this extra hop of communication is not noticeable relative to the latency of the actual query processing and the agent computation, but at least in theory this is a slight disadvantage. 

What are the advantages to having the virtualization/federation layer?

Joins across distinct datasets

The biggest advantage of having a virtualization layer is when datasets stored by different database systems need to be joined together. For example, in the scenario discussed above where the transaction data is stored in one database system and the customer loyalty data stored in a different database system, it is easy to envision scenarios where joining these two datasets together would be useful. For example, breaking down profits by demographic requires the profit data from the transaction dataset and and the demographic data from the customer loyalty database. 

Without a virtualization layer, the agent needs to pull both datasets from the respective underlying systems and perform the join there. Both datasets need to be in one place in order for the join to occur (I’m ignoring unusual scenarios in which the databases are able to communicate with each directly via semi-join style of algorithms). Since the datasets being joined are currently managed by separate database systems, the raw data being joined will need to be extracted and performed by the agent. 

All of the issues mentioned above that come up when agents deal with raw data thus reemerge. It can be costly (since the agent needs to perform computation as it reads the raw data). It can be slow (because agents are not optimized for joining raw, structured data). It requires additional deserialization and serialization costs. In general, performing the join in the agent is far less efficient than performing it in a database system.  

Moreover, many modern agents do not in practice perform the join locally. Instead, they write the data being joined to an external database system and perform the join there. This further reduces performance, since load costs need to be incurred when writing each table being joined to the external database.

In contrast, performing joins across external datasets are a major use case for data virtualization systems. Almost every data virtualization system supports such joins — in many cases they implement multiple different algorithms and choose the optimal one on the fly based on the datasets being joined and the source systems storing them (I discuss different algorithms in my O’Reilly book: “Data virtualization in the Cloud Era”). 

Scalable data virtualization systems — such as Trino and Starburst — are designed to allow multiple independent machines to perform such joins in parallel. For example, they route tuples to different machines based on a hash of the join key and then perform joins of each hash partition locally. 

Generally speaking, data virtualization systems are able to perform joins of external datasets far more efficiently (and at lower cost) than modern agents. In some cases, agents will avoid trying the join in the first place due to the costs involved unless it can be outsourced in this way. Thus, the mere presence of the data virtualization system allows the agent to explore a larger space of possibilities in answering a query, and thus may lead to higher quality results. 

This is the type of thing that an end user may not notice. The same agent with the same query may return different quality results depending on the data infrastructure operating underneath the agent. Most agents will not let the user know — “I might have returned better results if I had tried out an alternative path involving a join of two disparate datasets”. The bottom line is that the user needs to set up the agent for the highest probability of success prior to using it. 

Tidiness

The entire point of a data virtualization or federation system is to provide a single interface for users to access all the data in the enterprise (or at least all the data that the system has been set up and given permissions to access). Instead of needing to access each data source individually, users have a single place to go in order to access everything they need. 

In most scenarios, the agent is not the only user that will need to access data at the enterprise. Humans (and other agents) will also be accessing data. 

After the data virtualization software has been installed and set up, the same system can serve as the interface to the data — both for the agent(s) and human users. When new datasets are created or added to the enterprise, only a single update needs to be made to the data virtualization system (to set up access to it). Once this is done, all users — agents and humans — will be able to benefit from its existence. 

In contrast, without the data virtualization system, someone has to realize that this new dataset may be useful to the agent and give the agent direct access to it, separately from any human that may already have access to it. There are separate actions and setup steps that need to be taken for each user that needs access to the new dataset. And the very existence of this dataset may not be easy to communicate if there is no centralized catalog across the enterprise. 

In such scenarios, it often happens that useful and valuable datasets get ignored and neglected — merely from the inertia of the additional steps required to alert the agent to their existence. Once again, this leads to lower quality results from agent queries in a way that is essentially impossible to detect. 

Just like things run more smoothly in a tidy house, so too things run smoothly when using a tidy data architecture.

Result caching

Agents are notorious for sending many similar queries with overlapping subqueries or access patterns. Many data virtualization systems use result caching to handle such workloads — where results of subqueries are cached and reused when reappearing in subsequent agent queries. This can reduce the burden on the underlying systems from handling the large scale of requests that agents can often generate. 

Native API knowledge

Data virtualization systems are designed to work with many different types of underlying database systems. For example, Starburst comes with dozens of connectors to different underlying systems. Each connector is designed to optimize access of that system, often taking advantage of particular features in the design of that system. 

While agents are also often broad in their knowledge of how to communicate with a large variety of systems, this knowledge can often be more superficial. This may result in a reduction of efficiency when communicating with these systems — which can reduce the quality, performance, or inexpensiveness of the results. 

Conclusion

There are a lot of advantages to using a data virtualization system to interface between the agent and underlying database systems. Data virtualization systems are more capable of performing scalable joins across large external datasets. They help give the agent a global view of all data that may be useful, and are more knowledgeable in leveraging the particular technical details of the underlying systems when communicating with them. Overall, they improve the efficiency of the communication with the data sources, reduce cost, and improve quality of the results.

The main disadvantages are upfront complexity and cost. However, it is important to note that there are a lot of reasons to use data virtualization even outside the context of using AI agents. This upfront complexity and cost can thus be amortized across other use cases.

Furthermore, many organizations are already running data virtualization software without realizing it. For example, many of Starburst’s customers originally set up Starburst to build a “lakehouse architecture” over a data lake in which Starburst accesses data stored in Parquet files and other open data formats in the lake. In many cases, they are unaware that Starburst is actually a data virtualization product. However, it is quite trivial to set up Starburst to access other systems as well, and then give the agent access to Starburst. 

The bottom line is that if you are not tied to a particular agent or AI tool, using one that is tightly integrated with your data management platform (such as Starburst AIDA) is often best. But even if you are tied to a given agent, it is still important to leverage data virtualization technology in order to maximize efficiency in the way that the agent accesses and leverages enterprise data.