Dataflows - Performance: Optimized Writes from Staging Warehouse Compute to Lakehouse Destinations
Data Factory · General availability · Planned
Description
The Optimized Lakehouse Writes capability for Dataflow Gen2 is now generally available, improving performance for a common pattern where data is processed using Staging Warehouse Compute and written to Lakehouse destinations, including scenarios such as persisting Computed Entity outputs.This enhancement reduces data movement overhead between Dataflow Gen2's Warehouse compute and Lakehouse data destinations, improving throughput and lowering end-to-end refresh times for large and complex dataflows. Customers benefit from faster data availability in their Lakehouses, more consistent refresh performance at scale, and improved efficiency when using warehouse-backed compute for intermediate transformation and shaping workloads.This capability is especially valuable for organizations standardizing on Lakehouse architectures and building scalable analytics and AI solutions on Microsoft Fabric.
Change History
-
2026-06-06
Feature Description We are continuing to expand the significant performance gains delivered by Dataflow Gen2 with a new set of targeted performance and latency improvements across the end-to-end experience. These enhancements focus on the most critical execution paths and user interactionsfurther differentiating Dataflow Gen2 from previous dataflow generations and raising the bar for low-code data transformations in Microsoft Fabric.This round of improvements addresses performance holistically--spanning data movementauthoringexecutionmonitoringand downstream consumption--to deliver a fastermore responsiveand more predictable experience at scale.Key enhancements include:* Optimized data movement from Data Warehouse compute to Lakehouse destinationsreducing end-to-end refresh times when landing transformed data into Fabric Lakehouse* Reduced metadata refresh latencyimproving responsiveness when working with schemas and downstream consumers* Lower latency when creating and editing Dataflow Gen2 artifactsmaking authoring and iteration faster and more fluid* Improved refresh history performanceenabling quicker inspection of run statusdurationand failures* Reduced Dataflow connector latencya critical improvement for downstream consumption and a key enabler for customers migrating from Dataflow Gen1* The ability to turn off Vertiparquet compressiongiving advanced users greater control over performance and cost trade-offs for specific workloadsTogetherthese enhancements further extend the performance advantages of Dataflow Gen2 compared to earlier dataflow implementationsreducing latency across both runtime execution and day-to-day interactions. By streamlining the most common and time-sensitive operationsDataflow Gen2 continues to deliver faster refreshessmoother authoringand more predictable behavior as data volumes and usage scale.At the same timethese improvements materially enhance the customer experience around low-code data transformations in Fabric. Faster feedback loopsmore responsive monitoringand greater control over execution characteristics make it easier for both business users and data engineers to designoperateand reuse dataflows with confidence--reinforcing Dataflow Gen2 as a high-performanceproduction-ready transformation layer within Microsoft Fabric.These performance and latency improvements will become generally available and are designed to be ready for production usage. With GAcustomers can rely on these enhancements for business-critical workloadsbenefiting from improved stabilitypredictable behaviorand the operational maturity required to run Dataflow Gen2 at scale across developmenttestand production environments. -> The Optimized Lakehouse Writes capability for Dataflow Gen2 is now generally availableimproving performance for a common pattern where data is processed using Staging Warehouse Compute and written to Lakehouse destinationsincluding scenarios such as persisting Computed Entity outputs.This enhancement reduces data movement overhead between Dataflow Gen2's Warehouse compute and Lakehouse data destinationsimproving throughput and lowering end-to-end refresh times for large and complex dataflows. Customers benefit from faster data availability in their Lakehousesmore consistent refresh performance at scaleand improved efficiency when using warehouse-backed compute for intermediate transformation and shaping workloads.This capability is especially valuable for organizations standardizing on Lakehouse architectures and building scalable analytics and AI solutions on Microsoft Fabric.Name Dataflows - Performance and Latency Improvements -> Dataflows - Performance: Optimized Writes from Staging Warehouse Compute to Lakehouse Destinations -
2026-06-02
Roadmap Item Added
Workload: Data Factory