Databricks Introduces New Products to Address AI Agent Data Pipeline Challenges
Databricks has announced two new products, Lakehouse//RT and LTAP (Lake Transactional/Analytical Processing), designed to solve the decades-old problem of managing operational and analytical databases in a unified, low-latency manner. Unveiled at the Data + AI Summit, these solutions aim to provide millisecond query latency directly on governed Delta and Iceberg tables, eliminating complex ETL pipelines and dedicated real-time serving tiers. This streamlined data stack is intended to enable AI agents to reason continuously and act on live data without performance degradation.
Databricks introduced Lakehouse//RT and LTAP (Lake Transactional/Analytical Processing) at the Data + AI Summit on Tuesday. These new products are aimed at addressing the long-standing challenge data professionals have faced in unifying operational and analytical databases without introducing latency.
The company states that the continuous reasoning and live data interaction required by AI agents necessitate a simpler data infrastructure. Reynold Xin, co-founder of Databricks, described a simpler data stack as "the holy grail for agents," noting that it allows them to operate faster.
Lakehouse//RT delivers millisecond query latency directly on governed Delta and Iceberg tables. It is designed to eliminate the need for a separate real-time serving tier that enterprises have traditionally maintained alongside their lakehouses. Key capabilities of Lakehouse//RT include the Reyden compute engine, which queries Delta and Iceberg tables directly for high-concurrency, low-latency serving. It offers sub-100ms latency at 12,000 queries per second, with response times as low as 10ms on smaller datasets, and integrates with Unity Catalog's governance framework.
LTAP focuses on storing Postgres-native transactional data directly in Delta and Iceberg format from the point of write, thereby removing ETL pipelines that have connected operational and analytical systems. This approach unifies data at the storage layer, in contrast to earlier Hybrid Transactional/Analytical Processing (HTAP) attempts at engine convergence. LTAP utilizes Databricks' Lakebase architecture, a serverless cloud-based PostgreSQL database service that became generally available in February.
Databricks addresses the latency challenge of object storage, which typically has response times in the seconds range, by employing a caching layer within Lakebase. This layer, positioned between Postgres compute instances and object storage, performs row-to-column conversion using idle CPU capacity, compressing data and reducing network costs for OLTP workloads requiring sub-millisecond performance.
Analysts have noted the "agentic AI framing" as a key differentiator. Stephanie Walter, Practice Leader for AI Stack at HyperFRAME Research, highlighted that agents require live operational data, historical context, governance, retrieval, and write-back within the same workflow. Mike Leone, an analyst at Moor Insights and Strategy, suggested that allowing transactional writes to land in open formats is a significant move, offering a credible case for retiring specialized systems.
Enterprises evaluating their data stacks for agentic workloads face a shift from best-of-breed tools to integrated solutions. Gaps between separate operational databases, real-time serving tiers, and analytical lakehouses, previously seen as maintenance burdens, are now viewed as operational risks for AI agents. Market trends, including a tripling of hybrid retrieval intent and a decline in standalone vector database adoption, suggest a consolidation logic impacting real-time serving tiers.
According to VentureBeat, the traditional architecture built for human-speed analytical consumption is not suitable for agent workloads, which do not tolerate the copying and syncing required between separate operational and analytical systems.
