Skip to Content

Wiki source code of Fabric Data Pipelines

Version 2.1 by Dimitri Rupp on 2026/06/11 09:16
Hide last authors
Dimitri Rupp 1.1 1 {{box cssClass="floatinginfobox" title="**On this page**"}}
2 {{toc/}}
3 {{/box}}
4
5 = Fabric Data Pipelines =
6
7 {{info}}
8 **What:** Strukturierte Sequenz von Schritten, die Daten von einem System ins andere bewegt — transformiert, validiert, anreichert.
9 {{/info}}
10
11 == Overview ==
12
13 A **data pipeline** is a **structured sequence of steps** that moves data from one system to another — often **transforming**, **validating**, or **enriching** it along the way.
14
15 In **Microsoft Fabric**, a data pipeline typically:
16
17 * **Connects** to a source system (database, API, file store)
18 * **Extracts** data using queries or connectors
19 * Optionally **transforms** the data
20 * **Loads** it into a target (Lakehouse, Warehouse, etc.)
21 * Can be **scheduled** or **triggered** by events / other pipelines
22
23 == Types of pipelines ==
24
25 Based on the **purpose**, data pipelines can be classified as:
26
27 |=Type|=Function
28 |**Ingestion pipeline**|Brings raw data into a central platform or storage layer (Data Lake / Warehouse / Lakehouse). Often involves **connecting** to source systems, **extracting** data, and **landing** it as-is (Bronze).
29 |**Orchestration pipeline**|Coordinates **downstream activities** — data transformation, validation, enrichment, or triggering analytics workflows. May involve **chaining multiple pipelines** or **notebooks**. Also called **execution pipelines** in some contexts.
30
31 {{mermaid}}
32 flowchart LR
33 subgraph SRC[Sources]
34 S1[(BMD DWH)]
35 S2[(Rimo DWH)]
36 S3[(H3A DWH)]
37 end
38
39 subgraph INGEST[Ingestion pipelines]
40 I1[getBMDDWH_BMD]
41 I2[getRimoDWH]
42 I3[geth3aDWH]
43 end
44
45 subgraph BRONZE[Bronze Lakehouses]
46 B1[(bmd_bronze_lh)]
47 B2[(rimo_bronze_lh)]
48 B3[(h3a_bronze_lh)]
49 end
50
51 subgraph ORCH[Orchestration pipelines]
52 O1[0445_Daily_bmd<br/>04:35]
53 O2[0335_Daily_rimo_new<br/>03:35]
54 O3[0400_Daily_h3a<br/>04:00]
55 end
56
57 S1 --> I1 --> B1
58 S2 --> I2 --> B2
59 S3 --> I3 --> B3
60 O1 --> I1
61 O2 --> I2
62 O3 --> I3
63 O1 -.->|notebooks| GOLD[spl_gold_lh]
64 O2 -.->|notebooks| GOLD
65 O3 -.->|notebooks| GOLD
66
67 classDef src fill:#e3f2fd,stroke:#1976d2,stroke-width:1px
68 classDef ing fill:#fff3cd,stroke:#b85c00,stroke-width:1px
69 classDef brz fill:#f4a261,stroke:#b85c00,stroke-width:1px,color:#000
70 classDef orc fill:#bbdefb,stroke:#1976d2,stroke-width:1px
71 classDef gold fill:#ffd54f,stroke:#b45f06,stroke-width:1px,color:#000
72
73 class S1,S2,S3 src
74 class I1,I2,I3 ing
75 class B1,B2,B3 brz
76 class O1,O2,O3 orc
77 class GOLD gold
78 {{/mermaid}}
79
80 == List of all data pipelines in telefabric ==
81
82 Shows all the pipelines across the **Fabric** and **Fabric_Dev** workspaces under **telefabric** capacity. The (% style="font-family:monospace" %)Location(%%) column indicates **Workspace name / {folder name}**.
83
84 |=Name of pipeline|=Type|=📂 Location|=Purpose
85 |(% style="font-family:monospace" %)0335_Daily_rimo_new(%%)|Orchestration|(% style="font-family:monospace" %)Fabric/00_Execution(%%)|Executes all data-movement activities of **Rimo** tables from on-premise DWH to the gold layer. Includes ingestion pipeline and notebooks.
86 |(% style="font-family:monospace" %)0400_Daily_h3a(%%)|Orchestration|(% style="font-family:monospace" %)Fabric/00_Execution(%%)|Executes all data-movement activities of **h3a** tables from on-premise DWH to the gold layer. Includes ingestion pipeline and notebooks.
87 |(% style="font-family:monospace" %)0445_Daily_bmd(%%)|Orchestration|(% style="font-family:monospace" %)Fabric/00_Execution(%%)|Executes all data-movement activities of **bmd** tables from on-premise DWH to the gold layer. Includes ingestion pipeline and notebooks.
88 |(% style="font-family:monospace" %)getRimoDWH(%%)|Ingestion|(% style="font-family:monospace" %)Fabric/01_Dataingestion(%%)|Copy tables from Rimo DWH to (% style="font-family:monospace" %)rimo_bronze_lh(%%)
89 |(% style="font-family:monospace" %)geth3aDWH(%%)|Ingestion|(% style="font-family:monospace" %)Fabric/01_Dataingestion(%%)|Copy tables from h3a DWH to (% style="font-family:monospace" %)h3a_bronze_lh(%%)
90 |(% style="font-family:monospace" %)getBMDDWH_BMD(%%)|Ingestion|(% style="font-family:monospace" %)Fabric/01_Dataingestion(%%)|Copy tables from BMD DWH to (% style="font-family:monospace" %)bmd_bronze_lh(%%)
91 |(% style="font-family:monospace" %)test(%%)|-|(% style="font-family:monospace" %)Fabric/archived_items(%%)|Pipeline for learning and testing
92 |(% style="font-family:monospace" %)dp_rimo_dev(%%)|-|(% style="font-family:monospace" %)Fabric_Dev(%%)|Data pipeline for developmental activities
93
94 == Naming pattern ==
95
96 * (% style="font-family:monospace" %)HHMM_Daily_<source>(%%) — Orchestration pipeline, daily run at the indicated time (04:35 = 04:35 AM)
97 * (% style="font-family:monospace" %)get<Source>DWH(%%) — Ingestion pipeline for that source
98
99 == Change log ==
100
101 |=Date|=Description|=Person
102 |25.11.2025|Changed the (% style="font-family:monospace" %)geth3aDWH(%%) pipeline documentation|Sreejith
103
104 == Related pages ==
105
106 * [[Fabric (Hub)>>doc:IT-Wiki.02-Tools.Fabric.WebHome]]
107 * [[Fabric Dataflows>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Dataflows.WebHome]]
108 * [[Fabric Notebooks>>doc:IT-Wiki.02-Tools.Fabric.Fabric-Notebooks.WebHome]]
109 * [[Bronze Lakehouse Hub>>doc:IT-Wiki.01-Data.Ingestion.Bronze-Lakehouse.WebHome]]
110
111 ----
112
113 {{info}}
114 **Migration footer**
115 **Source:** [[Fabric Data Pipelines (SharePoint)>>url:https://spltele.sharepoint.com/sites/WikiPowerBI/SitePages/Fabric-Data-Pipelines.aspx]]
116 **Original author:** Bukkerji Sreejith
117 **Original last-modified:** 2025-11-25
118 **Migration date:** 2026-05-24
119 **Migrated by:** Dimitri Rupp via ClaudeAI
120 **Notes:** EN-Original-Texte 1:1 (Overview/Types/Pipeline-Liste/Change-Log); Mermaid-Diagramm fuer Source→Ingestion→Bronze + Orchestration-Beziehung ergaenzt; Naming-Pattern-Section als Wissensbasis-Hilfe; technische Pipeline-Namen 1:1.
121 {{/info}}
122
123
124 ----
125
126 == Verwandte Themen ==
127
128 //Kuratiert — kann direkt im Editor ergänzt werden://
129
130 * [[Fabric Dataflows>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Dataflows.WebHome]]
131 * [[Fabric Excel Ingestion>>doc:IT-Wiki.01-Data.Ingestion.Fabric-Excel-Ingestion.WebHome]]
132 * [[Bronze Lakehouse>>doc:IT-Wiki.01-Data.Ingestion.Bronze-Lakehouse.WebHome]]
133
134 {{include reference="IT-Wiki.Makros.Verwandte-Themen.WebHome"/}}
135
136 ----
137 **Status:** Migriert — Team-Review ausstehend · **Owner:** //(festlegen)// · **Letzter Review:** 2026-06-11
138