Dataflows - Performance: Optimized Writes from Staging Warehouse Compute to Lakehouse Destinations

Description

Change History

1. 2026-10-07
   Release Status Planned -> Shipped
2. 2026-06-16
   Restored to Roadmap
3. 2026-06-16
   Removed from Roadmap
4. 2026-06-06
   Feature Description We are continuing to expand the significant performance gains delivered by Dataflow Gen2 with a new set of targeted performance and latency improvements across the end-to-end experience. These enhancements focus on the most critical execution paths and user interactions further differentiating Dataflow Gen2 from previous dataflow generations and raising the bar for low-code data transformations in Microsoft Fabric.This round of improvements addresses performance holistically--spanning data movement authoring execution monitoring and downstream consumption--to deliver a faster more responsive and more predictable experience at scale.Key enhancements include:* Optimized data movement from Data Warehouse compute to Lakehouse destinations reducing end-to-end refresh times when landing transformed data into Fabric Lakehouse* Reduced metadata refresh latency improving responsiveness when working with schemas and downstream consumers* Lower latency when creating and editing Dataflow Gen2 artifacts making authoring and iteration faster and more fluid* Improved refresh history performance enabling quicker inspection of run status duration and failures* Reduced Dataflow connector latency a critical improvement for downstream consumption and a key enabler for customers migrating from Dataflow Gen1* The ability to turn off Vertiparquet compression giving advanced users greater control over performance and cost trade-offs for specific workloadsTogether these enhancements further extend the performance advantages of Dataflow Gen2 compared to earlier dataflow implementations reducing latency across both runtime execution and day-to-day interactions. By streamlining the most common and time-sensitive operations Dataflow Gen2 continues to deliver faster refreshes smoother authoring and more predictable behavior as data volumes and usage scale.At the same time these improvements materially enhance the customer experience around low-code data transformations in Fabric. Faster feedback loops more responsive monitoring and greater control over execution characteristics make it easier for both business users and data engineers to design operate and reuse dataflows with confidence--reinforcing Dataflow Gen2 as a high-performance production-ready transformation layer within Microsoft Fabric.These performance and latency improvements will become generally available and are designed to be ready for production usage. With GA customers can rely on these enhancements for business-critical workloads benefiting from improved stability predictable behavior and the operational maturity required to run Dataflow Gen2 at scale across development test and production environments. -> The Optimized Lakehouse Writes capability for Dataflow Gen2 is now generally available improving performance for a common pattern where data is processed using Staging Warehouse Compute and written to Lakehouse destinations including scenarios such as persisting Computed Entity outputs.This enhancement reduces data movement overhead between Dataflow Gen2's Warehouse compute and Lakehouse data destinations improving throughput and lowering end-to-end refresh times for large and complex dataflows. Customers benefit from faster data availability in their Lakehouses more consistent refresh performance at scale and improved efficiency when using warehouse-backed compute for intermediate transformation and shaping workloads.This capability is especially valuable for organizations standardizing on Lakehouse architectures and building scalable analytics and AI solutions on Microsoft Fabric. Name Dataflows - Performance and Latency Improvements -> Dataflows - Performance: Optimized Writes from Staging Warehouse Compute to Lakehouse Destinations
5. 2026-06-02
   Roadmap Item Added