Uncovering the evolution history of data lakes
- Data accumulating in data lakes can become inaccessible in the long run when its semantics are not available. The heterogeneity of data formats and the sheer volumes of data collections prohibit cleaning and unifying the data manually. Thus, tools for automated data lake analysis are of great interest. In this paper, we target the particular problem of reconstructing the schema evolution history from data lakes. Knowing how the data is structured, and how this structure has evolved over time, enables programmatic access to the lake. By deriving a sequence of schema versions, rather than a single schema, we take into account structural changes over time. Moreover, we address the challenge of detecting inclusion dependencies. This is a prerequisite for mapping between succeeding schema versions, and in particular, detecting nontrivial changes such as a property having been moved or copied. We evaluate our approach for detecting inclusion dependencies using the MovieLens dataset, as well an adaption of a dataset containing botanical descriptions, to cover specific edge cases.
Verfasserangaben: | Meike Klettke, Hannes Awolin, Uta StörlORCiDGND, Daniel Müller, Stefanie ScherzingerORCiD, Uta StorlORCiDGND, Daniel Muller |
---|---|
DOI: | https://doi.org/10.1109/BigData.2017.8258204 |
Titel des übergeordneten Werkes (Englisch): | 2017 IEEE International Conference on Big Data (Big Data),,11-14 Dec. 2017, Boston, MA, USA |
Verlag: | IEEE |
Dokumentart: | Konferenzveröffentlichung (Beitrag) |
Sprache der Veröffentlichung: | Englisch |
Jahr der Veröffentlichung: | 2017 |
Datum der Freischaltung: | 05.05.2022 |
Freies Schlagwort / Tag: | Data mining; Grippers; Lakes; NoSQL databases; Protocols; evolution operations; history; inclusion dependencies; integrity constraints; schema version extraction |
Erste Seite: | 2462 |
Letzte Seite: | 2471 |
Fakultäten / Institute / Einrichtungen: | Fakultät Informatik und Mathematik |
Begutachtungsstatus: | peer-reviewed |
Forschungsschwerpunkt: | Digitalisierung |
Lizenz (Deutsch): | Keine Lizenz - Es gilt das deutsche Urheberrecht: § 53 UrhG |