Wiki source code of Fabric Dataflows
Last modified by Dimitri Rupp on 2026/06/11 09:30
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | {{box cssClass="floatinginfobox" title="**On this page**"}} | ||
| 2 | {{toc/}} | ||
| 3 | {{/box}} | ||
| 4 | |||
| 5 | = Fabric Dataflows = | ||
| 6 | |||
| 7 | {{info}} | ||
| 8 | **What:** Dataflows Gen2 — modern, scalable ETL via Power Query in Fabric. | ||
| 9 | **Strength:** No-/low-code, native Lakehouse integration, incremental refresh. | ||
| 10 | {{/info}} | ||
| 11 | |||
| 12 | == What are Dataflows Gen2? == | ||
| 13 | |||
| 14 | **Dataflows Gen2** are a **modern, scalable way** to **ingest, transform, and load data** into Microsoft Fabric. They allow users to build **reusable ETL pipelines** using familiar **Power Query** experiences, while integrating natively with **Lakehouse**, **Warehouse**, and other Fabric destinations. | ||
| 15 | |||
| 16 | == 🔑 Key Features == | ||
| 17 | |||
| 18 | |=Feature|=What it gives you | ||
| 19 | |**Power Query Interface**|Build transformations using a no-code or low-code experience, with full support for **M language**. | ||
| 20 | |**Direct Lakehouse Integration**|Write output directly to Lakehouse tables in **Parquet** format, enabling seamless analytics and reporting. | ||
| 21 | |**Incremental Refresh**|Filter out previously processed data — only **new** or **changed** rows are loaded on each refresh. | ||
| 22 | |||
| 23 | == 🧩 Common Use Cases == | ||
| 24 | |||
| 25 | * Loading **structured data from SharePoint** into Lakehouse | ||
| 26 | * Transforming **Excel-based reports** into analytics-ready formats | ||
| 27 | * **Filtering** and **deduplicating** data before ingestion | ||
| 28 | * Automating **daily refreshes** for operational dashboards | ||
| 29 | |||
| 30 | == Dataflow Gen2 vs. Pipelines vs. Notebooks == | ||
| 31 | |||
| 32 | |=Orchestration type|=Strength|=When to use | ||
| 33 | |**Dataflow Gen2**|Power-Query-Low-Code; Lakehouse-native|Strukturierte Ingestion aus SharePoint/Excel/APIs, einfache Transformationen | ||
| 34 | |**Data Pipeline**|Native Fabric copy + transform; parametrierbar|Standard-DB-zu-Lakehouse-Bewegung, getriggerte Workflows | ||
| 35 | |**Notebook**|PySpark / Python — volle Kontrolle|Custom-Logik, ML, komplexe File-Parsing, Spezialfaelle | ||
| 36 | |||
| 37 | {{mermaid}} | ||
| 38 | flowchart LR | ||
| 39 | SP[SharePoint Files / Lists] -->|Power Query| DF[Dataflow Gen2<br/>M-language] | ||
| 40 | XLS[Excel Reports] --> DF | ||
| 41 | API[REST API / Board] --> DF | ||
| 42 | DF -->|Parquet| LH[(Lakehouse<br/>Bronze / Silver)] | ||
| 43 | DF -->|incremental refresh| LH | ||
| 44 | |||
| 45 | classDef src fill:#e3f2fd,stroke:#1976d2,stroke-width:1px | ||
| 46 | classDef df fill:#fff3cd,stroke:#b85c00,stroke-width:1px | ||
| 47 | classDef lh fill:#f4a261,stroke:#b85c00,stroke-width:1px,color:#000 | ||
| 48 | class SP,XLS,API src | ||
| 49 | class DF df | ||
| 50 | class LH lh | ||
| 51 | {{/mermaid}} | ||
| 52 | |||
| 53 | == Real-world examples in SPL == | ||
| 54 | |||
| 55 | * (% style="font-family:monospace" %)get_Board_API(%%) — feeds (% style="font-family:monospace" %)board_bronze_lh(%%) (siehe [[Board Bronze Lakehouse>>doc:IT-Wiki.01-Data.Ingestion.Bronze-Lakehouse.Board-Bronze-Lakehouse.WebHome]]) | ||
| 56 | * SharePoint-File-Ingestion ergaenzt durch Notebooks fuer [[SharePoint Bronze Lakehouse>>doc:IT-Wiki.01-Data.Ingestion.Bronze-Lakehouse.SharePoint-Bronze-Lakehouse.WebHome]] | ||
| 57 | |||
| 58 | == Related pages == | ||
| 59 | |||
| 60 | * [[Fabric (Hub)>>doc:IT-Wiki.02-Tools.Fabric.WebHome]] | ||
| 61 | * [[Fabric Data Pipelines>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Data-Pipelines.WebHome]] | ||
| 62 | * [[Fabric Notebooks>>doc:IT-Wiki.02-Tools.Fabric.Fabric-Notebooks.WebHome]] | ||
| 63 | * [[Fabric Lakehouse Hub>>doc:IT-Wiki.01-Data.Processing.Lakehouse.WebHome]] | ||
| 64 | |||
| 65 | ---- | ||
| 66 | |||
| 67 | == Verwandte Themen == | ||
| 68 | |||
| 69 | //Kuratiert — kann direkt im Editor ergänzt werden:// | ||
| 70 | |||
| 71 | * [[Fabric Data Pipelines>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Data-Pipelines.WebHome]] | ||
| 72 | * [[Fabric Excel Ingestion>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Excel-Ingestion.WebHome]] | ||
| 73 | * [[Bronze Lakehouse>>doc:IT-Wiki.01-Data.Ingestion.Bronze-Lakehouse.WebHome]] | ||
| 74 | |||
| 75 | {{include reference="IT-Wiki.Makros.Verwandte-Themen.WebHome"/}} | ||
| 76 | |||
| 77 | ---- | ||
| 78 | **Status:** Migriert — Team-Review ausstehend · **Owner:** //(festlegen)// · **Letzter Review:** 2026-06-11 |