Data Warehouses aren't enough for the era
Today, data warehouses aren’t enough. Just connect it to BI tools, and voila, it works! Unfortunately, this model is struggling in the modern world. It assumes that the data…
Today, data warehouses aren’t enough.
Just connect it to BI tools, and voila, it works! Unfortunately, this model is struggling in the modern world. It assumes that the data sources and the data query needs would remain largely static. Neither are, and data warehouse maintainers are stuck in an endless cycle of building connectors and reshaping data structures. When the world around a data warehouse moves faster than the data warehouse can adapt, its usefulness falls apart.
This issue is getting worse with AI. On a weekly basis, AI agents are adding new tools and AI coding agents are modifying data schemas. That means rapidly changing sources. Simultaneously, AI agents are tweaking BI tools and fetching data for RAG/context. That means changing use cases. It’s why your average warehouse maintainer and warehouse user is frustrated: everything keeps breaking because we’re wrangling with a static system in a dynamic world.
How do we avoid a warehouse that's perpetually out-of-sync? One option is to let AI agents own the warehouse's configuration and automate maintenance. But if we've learned anything from the last few years, agents tend to degrade code quality over time. Instead, we need a system that runs parallel to the warehouse and can adapt to constant change.
That’s what we’re building at Peaka. The goal of Peaka isn't to replace a data warehouse. Warehouses have their place: they excel at OLAP workloads when sources and schemas are stable enough to amortize the cost of pipeline maintenance. Instead, we're building a dynamic layer on top of your existing data infrastructure that queries and joins data at runtime, handling all the performance sub-problems under the hood. It's not as fast as a fully-tuned data warehouse, but it's significantly faster than a warehouse that needs hours of engineering time just to stay in-sync.
Today, we want to discuss our vision at length.
Why warehouses struggle to keep up
At the risk of over-categorizing, I want to break the problems with data warehouses into three buckets.
The Shaping Problem
A data warehouse’s underlying data shape is a complex problem. A warehouse isn’t just a massive USB drive. It de-normalize data into an ordered structure so that diverse sources could be joined and collectively queried. Without this work, information would be stored in a crooked fashion and be fundamentally useless at query-time.
This prompts a lot of sub-problems for engineers. How do you join an application’s Postgres state with a customer’s Stripe records? How should the web-app’s PostHog events be joined with an Amplitude instance? This is the purpose of the “T” in ETL: Transform.
To add to the chaos, many data sources produce redundant or out-of-order events: CDC pipelines emitting duplicate change events, retried syncs pushing the same rows twice, webhooks firing more than once. Engineers end up writing dedup logic to enforce idempotency into nearly every pipeline they own.
The moving target problem
The shaping problem is complicated by schema drift, the constantly changing inputs and outputs of warehouses.
[[Image Removed]]
Sources change and use cases change. New SaaS tools are added, new data stores are created by AI agents, and operators constantly are asking new questions. This puts engineers in a never-ending spiral of maintaining their data warehouse. Even well-funded data teams with mature Snowflake or BigQuery deployments feel this. The warehouse is always catching up.
The separation of duties problem
For many companies, the biggest issue is the separation of duties.
The data warehouse’s users are typically distinct from the data warehouse’s maintainers. Operators and product teams need to query data to make decisions. But they don’t build the data warehouse—that responsibility falls on data engineers.
This invariably leads to a lag: a warehouse's configurations trail behind real-time demand as engineers get bogged down by other tickets (especially since a warehouse is rarely considered an urgent issue). Even worse, indecisiveness and poor communication from operators slows down the necessary changes.
AI raises the stakes
Thus far, the answer has been more workarounds. ETL layers on top of ETL layers, pipelines slapped on top of pipelines, trying to create clean views out of already-messy data. There are entire schools of thought around navigating this complexity (the lambda vs. kappa architecture debate being a prime example).
Enter AI. AI needs data. Not just data, but metadata. It wants to know what tables and columns mean, it needs the relationships between them, and it requires statistics. For RAG, it needs recent information.
Broadly speaking, we’re entering an age of copious prompts that are data hungry. Queries that multiply into dozens of queries. A warehouse that can't keep up with these demands becomes a bottleneck for AI adoption.
Speaking candidly, it's a tricky situation. On one end, leaders are yelling "go, go, go" to integrate AI while builders are rigging things together haphazardly. On the other end, data teams are struggling to build a data operation that can keep up, relying on Claude Code to plug the holes.
I'd like to offer an alternative: what if, in addition to your existing warehouse, you add a dynamic layer on top that queries information from the root sources at runtime — picking up where the warehouse falls short? That is the design of Peaka.
Consolidation at the query level
Peaka might seem like a “yeah, but…” product. Yeah, that sounds straightforward, but it’s incredibly slow in practice.
The skepticism isn’t unwarranted, but there are significant optimizations that can happen under-the-hood to deliver near-data warehouse efficiency while keeping joins to runtime. Simultaneously, you’ll net massive benefits by using a product like Peaka:
Federated querying. Peaka connects directly to databases, SaaS APIs (Stripe, HubSpot, etc.), and your existing warehouses, sitting on top of them without mandatory data movement. This is an optimal design: data is accessible whenever, yet doesn’t need to be ported over prematurely.
Selective caching. With federation as a default, Peaka still supports optional caching. With incremental sync on configurable schedules (down to 1-minute intervals), Peaka users can achieve data warehouse performance through materialized views for BI and RAG use cases despite the data layer being virtualized. The short interval window also ensures cached content is minutes, not hours, old. This piece of the puzzle is critical when understanding Peaka alongside data warehouses: it's a dynamic layer that delivers warehouse-like performance while remaining flexible and up-to-date.
Semantic layer. Peaka's users compose their own views from modular building blocks without waiting on the data team. They can also publish or discover reusable data products in a shared marketplace, assembling their own data space à la carte. Especially now with AI tooling, anyone can construct a complex query without worrying about the underlying data shape.
Observability. All queries are sent to Datadog, Grafana, OTel or a built-in observability layer.
All of these benefits couple with a massive security one: Peaka is one place to enforce masking, row/column filtering, and user permissions across all connected sources.
A closing thought: the evolution of software
We believe Peaka is part of the natural evolution of software infrastructure.
Years ago, teams hand-orchestrated servers on EC2 before Kubernetes ate the world. Now, most teams push to an orchestration platform and let it handle the complexity. Peaka applies that same philosophy to data.
There is no mandatory migration, and therefore no rip-and-replace in the not-so-distant future.
