Refine
Year of publication
- 2017 (3) (remove)
Document Type
Is part of the Bibliography
- no (3)
Keywords
- Data mining (1)
- Forschung (1)
- Forschungsbericht (1)
- Grippers (1)
- Lakes (1)
- NoSQL databases (1)
- Protocols (1)
- evolution operations (1)
- history (1)
- inclusion dependencies (1)
Institute
Begutachtungsstatus
- peer-reviewed (1)
Data accumulating in data lakes can become inaccessible in the long run when its semantics are not available. The heterogeneity of data formats and the sheer volumes of data collections prohibit cleaning and unifying the data manually. Thus, tools for automated data lake analysis are of great interest. In this paper, we target the particular problem of reconstructing the schema evolution history from data lakes. Knowing how the data is structured, and how this structure has evolved over time, enables programmatic access to the lake. By deriving a sequence of schema versions, rather than a single schema, we take into account structural changes over time. Moreover, we address the challenge of detecting inclusion dependencies. This is a prerequisite for mapping between succeeding schema versions, and in particular, detecting nontrivial changes such as a property having been moved or copied. We evaluate our approach for detecting inclusion dependencies using the MovieLens dataset, as well an adaption of a dataset containing botanical descriptions, to cover specific edge cases.
Forschungsbericht 2017
(2017)
NoSQL databases are popular in agile software development, where a frequently changing database schema imposes challenges for the production database. In this demo, we present Darwin, a middleware for systematic, tool-based support specifically designed for NoSQL database systems. Darwin carries out schema evolution and data migration tasks. To the best of our knowledge, Darwin is the first tool of its kind that supports both eager and lazy NoSQL data migration.