TY - CHAP A1 - Klettke, Meike A1 - Störl, Uta A1 - Shenavai, Manuel A1 - Scherzinger, Stefanie A1 - Storl, Uta T1 - NoSQL schema evolution and big data migration at scale T2 - 2016 IEEE International Conference on Big Data (Big Data), 5-8 Dec. 2016, Washington, DC N2 - This paper explores scalable implementation strategies for carrying out lazy schema evolution in NoSQL data stores. For decades, schema evolution has been an evergreen in database research. Yet new challenges arise in the context of cloud-hosted data backends: With all database reads and writes charged by the provider, migrating the entire data instance eagerly into a new schema can be prohibitively expensive. Thus, lazy migration may be more cost-efficient, as legacy entities are only migrated in case they are actually accessed by the application. Related work has shown that the overhead of migrating data lazily is affordable when a single evolutionary change is carried out, such as adding a new property. In this paper, we focus on long-term schema evolution, where chains of pending schema evolution operations may have to be applied. Chains occur when legacy entities written several application releases back are finally accessed by the application. We discuss strategies for dealing with chains of evolution operations, in particular, the composition into a single, equivalent composite migration that performs the required version jump. Our experiments with MongoDB focus on scalable implementation strategies. Our lineup further compares the number of write operations, and thus, the operational costs of different data migration strategies. KW - Big data KW - Context KW - Data Migration Strategies KW - Data models KW - Databases KW - Incremental Migration KW - Lazy Composite Migration KW - Lazy Migration KW - NoSQL databases KW - Predictive Migration KW - Production KW - Runtime KW - schema evolution KW - Software Y1 - 2016 U6 - https://doi.org/10.1109/BigData.2016.7840924 SP - 2764 EP - 2774 PB - IEEE ER - TY - CHAP A1 - Störl, Uta A1 - Müller, Daniel A1 - Tekleab, Alexander A1 - Tolale, Stephane A1 - Stenzel, Julian A1 - Klettke, Meike A1 - Scherzinger, Stefanie A1 - Storl, Uta A1 - Muller, Daniel T1 - Curating Variational Data in Application Development T2 - 2018 IEEE 34th International Conference on Data Engineering, 16-19 April 2018, Paris, France N2 - Building applications for processing data lakes is a software engineering challenge. We present Darwin, a middleware for applications that operate on variational data. This concerns data with heterogeneous structure, usually stored within a schema-flexible NoSQL database. Darwin assists application developers in essential data and schema curation tasks: Upon request, Darwin extracts a schema description, discovers the history of schema versions, and proposes mappings between these versions. Users of Darwin may interactively choose which mappings are most realistic. Darwin is further capable of rewriting queries at runtime, to ensure that queries also comply with legacy data. Alternatively, Darwin can migrate legacy data to reduce the structural heterogeneity. Using Darwin, developers may thus evolve their data in sync with their code. In our hands-on demo, we curate synthetic as well as real-life datasets. KW - data migration KW - Data mining KW - Evolution (biology) KW - history KW - NoSQL databases KW - query rewriting KW - schema evolution KW - schema management KW - Software KW - Task analysis KW - variational data Y1 - 2018 U6 - https://doi.org/10.1109/ICDE.2018.00187 SP - 1605 EP - 1608 PB - IEEE ER -