TY - JOUR A1 - Schoenenwald, Alexander A1 - Kern, Simon A1 - Viehhauser, Josef A1 - Schildgen, Johannes T1 - Collecting and visualizing data lineage of Spark jobs T2 - Datenbank-Spektrum N2 - Metadata management constitutes a key prerequisite for enterprises as they engage in data analytics and governance. Today, however, the context of data is often only manually documented by subject matter experts, and lacks completeness and reliability due to the complex nature of data pipelines. Thus, collecting data lineage—describing the origin, structure, and dependencies of data—in an automated fashion increases quality of provided metadata and reduces manual effort, making it critical for the development and operation of data pipelines. In our practice report, we propose an end-to-end solution that digests lineage via (Py‑)Spark execution plans. We build upon the open-source component Spline, allowing us to reliably consume lineage metadata and identify interdependencies. We map the digested data into an expandable data model, enabling us to extract graph structures for both coarse- and fine-grained data lineage. Lastly, our solution visualizes the extracted data lineage via a modern web app, and integrates with BMW Group’s soon-to-be open-sourced Cloud Data Hub. Y1 - 2021 UR - https://opus4.kobv.de/opus4-oth-regensburg/frontdoor/index/index/docId/2606 N1 - Corresponding author: Johannes Schildgen VL - 21 IS - 3 SP - 179 EP - 189 PB - Springer Nature ER -