Shared Lakehouse or Separate Real-Time Stack?

Shared Lakehouse or Separate Real-Time Stack?

When we work with Azure data teams, event volume is often less revealing than the workload behind it. One Event Hubs throughput unit supports up to 1 MB/s ingress, so latency, state, and recovery requirements must shape the architecture decision before tool count does.

We recommend a shared lakehouse when batch, BI, machine learning, and streaming workloads can meet their latency target while using the same governed tables and storage. Keep a separate real-time path when live decisions need specialised state, replay, ordering, or independent scaling. Centralize data and metadata before retiring processors.

We compare both patterns, identify the costs hidden in siloed tools, and give you a phased way to consolidate without disrupting working workloads. We also show where a separate real-time engine remains the responsible choice.

How Do We Decide Lakehouse vs Separate Real-Time Stack?

For us, the right comparison is not “one platform versus many.” It is whether one governed data foundation can serve the required outcomes without compromising a live operational workload. Azure guidance distinguishes micro-batch, event-based, stateful, and checkpointing capabilities across streaming choices, which is why a decision matrix is more useful than a blanket consolidation mandate.

We score the options below by fit for the stated requirement. A score of five means the architecture is the stronger default for that dimension, not that it eliminates design work.

Choose the shared option when the same data can arrive, be refined, and be consumed through different compute interfaces. Choose the split option when the live processor has a clearly measured requirement that curated-table processing cannot meet. Our unified platform overview explains the shared-foundation pattern in more detail.

Where Do Silo Costs Accumulate?

Silos rarely begin as a strategy. They appear when a BI team creates its own extract, an engineering team builds another copy for transformations, and an ML team materializes features elsewhere. The result is more connectors, permission paths, schemas, and handoffs than anyone owns end to end. A shared data lifecycle reduces movement because data can remain in one foundation across ingestion, refinement, and reporting, as the lifecycle guidance describes.

We do not promise a universal savings percentage because the cost depends on your storage tier, contracts, engineering time, and recovery obligations. Instead, we recommend turning duplication into an inventory that your team can measure and own.

This inventory is especially useful during an inherited-pipeline migration. We use it alongside an SSIS migration guide to separate essential operational interfaces from copies that exist only because previous teams lacked a shared contract.

How Can Batch and Streaming Share Infrastructure Without Duplicating Durable Data?

Batch and streaming can share infrastructure when they converge on a governed table contract, not when they are forced through identical compute. We want a durable raw or Bronze layer, validated Silver assets, and business-ready Gold assets with clear owners. Azure’s data layer model frames those layers as raw, cleansed, and curated.

Use One Durable Table Contract

We define an authoritative table with a schema version, quality expectations, retention policy, classification, owner, and intended consumers. Batch backfills and continuously arriving records can both write to that contract, while BI, ML, and engineering teams choose the interface that suits their work.

For Azure teams building this foundation, our Azure lakehouse guide helps connect the storage and governance choices to practical engineering work.

Separate Replay from Analytical Duplication

A replayable event log is not automatically a harmful duplicate. It is justified when the real-time processor needs a time-bounded recovery source. The problem starts when that log becomes a second, unmanaged analytical record with separate transformations and access rules.

Open table formats help prevent that split. Delta tables support ACID transactions, schema enforcement, upserts, time travel, and unified batch and streaming reads and writes, as covered in the Delta documentation. We still document which asset is authoritative, because a format alone cannot resolve ownership.

Let Each Team Use Its Own Compute Interface

Engineering teams can use pipelines and notebooks, BI teams can query curated SQL assets and semantic models, and ML practitioners can train from versioned curated data. They do not need separate physical copies simply because they work differently.

That separation of interfaces from storage is the useful part of a shared architecture. Our shared-platform architecture shows how teams can use one governed foundation without pretending every workload has the same runtime needs.

When Should Streaming Stay Separate?

We keep a separate real-time stack when the live path has a requirement that cannot be softened into a scheduled or micro-batch target. This is not a failure of consolidation. It is a boundary that protects the live workload while letting its durable outputs remain governed and reusable.

When Latency Is an Operational Requirement

Start with an action deadline, then measure p50, p95, and p99 ingest-to-action latency. A dashboard that refreshes in minutes is a different workload from an alert that must react while an event is still operationally relevant. If the deadline is consistently very low, isolated event processing is usually the safer fit.

When State, Ordering, and Replay Matter

Stateful processing becomes distinct when a job needs keyed history, temporal joins, event-time windows, late-event handling, or strict ordering within a partition. Fault recovery also needs a replayable source and a transactional or idempotent sink for end-to-end exactly-once results, according to the recovery guide.

A real-time path should publish durable, governed outputs back to the lakehouse. That keeps live operational decisions separate without leaving their history stranded in a runtime that BI and ML teams cannot reliably consume.

When Independent Scaling Protects Other Teams

A sudden event surge can create backpressure, checkpoint delays, or recovery work that should not degrade scheduled transformations or reporting. Azure Stream Analytics supports stateful temporal operations, but its windows have a documented seven-day limit. We validate those limits against the actual project before collapsing the runtime boundary.

Teams that need to build capability around these choices can compare Fabric team options before standardizing a delivery model.

How Do We Consolidate Without Ripping Out What Works?

We consolidate in phases because the most valuable early win is often shared data and governance, not an immediate processor replacement. That sequence lowers risk, proves parity, and lets us retain specialised components when they continue to earn their operational cost.

Inventory Interfaces and Dependencies

List every source, connector, physical copy, transformation, schema, access rule, semantic definition, lineage gap, owner, and service-level target. Capture event rate, payload size, peak behavior, current latency, and recovery expectations before proposing a destination architecture.

Standardize Formats and Contracts

Define the authoritative table contract before moving consumers. We specify schema-evolution rules, data quality checks, retention, ownership, classification, and replay behavior. Where external open-format data can be referenced safely, shortcuts can expose it without an extra copy, as OneLake shortcuts demonstrate.

Centralize Governance and Consumption

Centralize the catalog, lineage, permissions, ownership, and semantic definitions even if a live runtime stays independent. Governance systems can stitch metadata from processing, storage, and reporting systems, although teams should document unsupported paths and manual lineage. Our Azure learning paths help teams decide which skills must accompany that operating model.

Retire Duplication Selectively

Retire a connector, copy, or transformation only after parity testing, rollback planning, owner sign-off, and a cost comparison. Keep a real-time path when the acceptance criteria below show it has a justified job.

• Event Rate: Record average and peak events per second, payload size, key distribution, and expected burst behavior.
• Latency Target: Define p50, p95, and p99 ingest-to-action and ingest-to-dashboard targets.
• Recovery Objective: Set maximum recovery time, accepted loss or duplication semantics, replay source, and sink idempotency.
• Retention Period: Document replay retention, regulatory retention, curated-table retention, and deletion rules.
• State Requirements: Specify joins, windows, ordering scope, late-event tolerance, and expected state size.
• Governance Ownership: Name the asset owner, permission owner, schema approver, and semantic-definition owner.

A practical curriculum audit can help ensure that the team learning plan covers these decisions, not only tool navigation.

Build the Decision with Vision Board

At Vision Board, we help Azure data teams turn architecture choices into skills that hold up in real projects. We do not treat a lakehouse as a reason to abandon every existing stream processor. Instead, we help teams map their ingestion paths, define durable table contracts, set governance ownership, and test latency, state, replay, and recovery needs before changing production workloads. Our learning paths connect data engineering practice with the architecture decisions that BI, ML, and real-time teams must make together. Whether you are modernising inherited pipelines or establishing standards for a new platform, we can help your team ask better questions, sequence migration safely, and build confidence with hands-on Azure work, role-aware exercises, implementation reviews, and a shared vocabulary for decisions that otherwise create expensive handoffs across teams. Explore our training and practical guidance at Vision Board.

FAQs on Lakehouse vs Separate Real-time Stack

We answer by workload. Not preference.

Can Batch and Machine Learning Share Infrastructure?

Yes. We can use one governed table layer for historical backfills, continuously arriving records, BI queries, and ML training when workload latency and state requirements fit.

When Should Streaming Stay Separate from a Lakehouse?

Keep it separate when event-time ordering, long-lived state, replay guarantees, independent scaling, or consistently ultra-low latency matters more than minimizing platform count for a specific operational decision.

How Do We Reduce Duplicate Storage Across Analytics Tools?

Record every physical copy, connector, transformation, schema, permission model, and manual handoff, then remove only those with proven parity, ownership, and rollback coverage after testing.

What Must We Measure Before Consolidating?

Start with event rate, peak payload, latency percentile target, recovery objective, retention period, ordering scope, state size, and the business action triggered by an event.