Dataflows - Dataflows Gen2 Partitioned Compute
Data Factory · General availability · Planned
Description
Partitioned Compute is a capability of Dataflow Gen2 that enables parts of a dataflow to run in parallel, reducing the time to finish its evaluations.Partitioned compute targets scenarios where the Dataflow engine can efficiently fold operations that can partition the data source and process each partition in parallel. For example, in a scenario where you're connecting to multiple files stored in an Azure Data Lake Storage Gen2, you can partition the list of files from your source, efficiently retrieve the partitioned list of files using query folding, use the combine files experience, and process all files in parallel.By moving to GA, Partitioned Compute becomes a stable, supported foundation for performant and highly scalable data transformation in Fabric.
Change History
-
2026-04-16
Feature Description Users want a flexible way to define the logic of their Dataflow Gen2 transformations and parallelize the execution with different arguments. Today they need to create multiple dataflows or multiple queries within their single dataflow in order to have a logic that can be reused with different arguments.As part of this enhancementwe will enable ways for users to set a ""foreach"" loop for their entire dataflow item driven from a standalone query that acts as the list of parameter values to iterate over and drive this containerized approach for parallelized and dynamic execution. -> Partitioned Compute is a capability of Dataflow Gen2 that enables parts of a dataflow to run in parallelreducing the time to finish its evaluations.Partitioned compute targets scenarios where the Dataflow engine can efficiently fold operations that can partition the data source and process each partition in parallel. For examplein a scenario where you're connecting to multiple files stored in an Azure Data Lake Storage Gen2you can partition the list of files from your sourceefficiently retrieve the partitioned list of files using query foldinguse the combine files experienceand process all files in parallel.By moving to GAPartitioned Compute becomes a stablesupported foundation for performant and highly scalable data transformation in Fabric.Name Dataflows - Dataflows Gen2 Parallelized Execution -> Dataflows - Dataflows Gen2 Partitioned Compute -
2026-04-01
Release Date 2026-06-30 -> 2026-06-02 -
2026-02-24
Release Date 2026-03-16 -> 2026-06-30 -
2025-11-18
Roadmap Item Added
Workload: Data Factory