Uncovering the evolution history of data lakes
- Data accumulating in data lakes can become inaccessible in the long run when its semantics are not available. The heterogeneity of data formats and the sheer volumes of data collections prohibit cleaning and unifying the data manually. Thus, tools for automated data lake analysis are of great interest. In this paper, we target the particular problem of reconstructing the schema evolution history from data lakes. Knowing how the data is structured, and how this structure has evolved over time, enables programmatic access to the lake. By deriving a sequence of schema versions, rather than a single schema, we take into account structural changes over time. Moreover, we address the challenge of detecting inclusion dependencies. This is a prerequisite for mapping between succeeding schema versions, and in particular, detecting nontrivial changes such as a property having been moved or copied. We evaluate our approach for detecting inclusion dependencies using the MovieLens dataset, as well an adaption of a dataset containing botanical descriptions, to cover specific edge cases.
Author: | Meike Klettke, Hannes Awolin, Uta StörlORCiDGND, Daniel Müller, Stefanie ScherzingerORCiD, Uta StorlORCiDGND, Daniel Muller |
---|---|
DOI: | https://doi.org/10.1109/BigData.2017.8258204 |
Parent Title (English): | 2017 IEEE International Conference on Big Data (Big Data),,11-14 Dec. 2017, Boston, MA, USA |
Publisher: | IEEE |
Document Type: | conference proceeding (article) |
Language: | English |
Year of first Publication: | 2017 |
Release Date: | 2022/05/05 |
Tag: | Data mining; Grippers; Lakes; NoSQL databases; Protocols; evolution operations; history; inclusion dependencies; integrity constraints; schema version extraction |
First Page: | 2462 |
Last Page: | 2471 |
Institutes: | Fakultät Informatik und Mathematik |
Begutachtungsstatus: | peer-reviewed |
research focus: | Digitalisierung |
Licence (German): | Keine Lizenz - Es gilt das deutsche Urheberrecht: § 53 UrhG |