TY - JOUR A1 - Schildgen, Johannes A1 - Jörg, Thomas A1 - Dossinger, Manuel A1 - Deßloch, Stefan T1 - Marimba: A framework for making MapReduce jobs incremental JF - Proceedings - 2014 IEEE International Congress on Big Data, BigData Congress 2014, 27 June 2014 - 02 July 2014, Anchorage, AK, USA N2 - Many MapReduce jobs for analyzing Big Data require many hours and have to be repeated again and again because the base data changes continuously. In this paper we propose Marimba, a framework for making MapReduce jobs incremental. Thus, a recomputation of a job only needs to process the changes since the last computation. This accelerates the execution and enables more frequent recomputations, which leads to results which are more up-to-date. Our approach is based on concepts that are popular in the area of materialized views in relational database systems where a view can be updated only by aggregating changes in base data upon the previous result. KW - Aggregates KW - Rhythm KW - Computational modeling KW - Programming KW - Google KW - Big data KW - Relational databases Y1 - 2014 U6 - https://doi.org/10.1109/BigData.Congress.2014.27 PB - IEEE ER - TY - CHAP A1 - Ringlstetter, Andreas A1 - Scherzinger, Stefanie A1 - Bissyandé, Tegawendé F. T1 - Data Model Evolution Using Object-NoSQL Mappers: Folklore or State-of-the-Art? T2 - 2016 IEEE/ACM 2nd International Workshop on Big Data Software Engineering (BIGDSE), 16 May 2016, Austin, TX, USA N2 - In big data software engineering, the schema flexibility of NoSQL document stores is a major selling point: When the document store itself does not actively manage a schema, the data model is maintained within the application. Just like object-relational mappers for relational databases, object-NoSQL mappers are part of professional software development with NoSQL document stores. Some mappers go beyond merely loading and storing Java objects: Using dedicated evolution annotations, developers may conveniently add, remove, or rename attributes from stored objects, and also conduct more complex transformations. In this paper, we analyze the dissemination of this technology in Java open source projects. While we find evidence on GitHub that evolution annotations are indeed being used, developers do not employ them so much for evolving the data model, but to solve different tasks instead. Our observations trigger interesting questions for further research. KW - Big data KW - data model evolution KW - Data models KW - Java KW - Loading KW - Object-NoSQL mappers KW - Software KW - Software engineering KW - Transient analysis Y1 - 2016 U6 - https://doi.org/10.1145/2896825.2896827 SP - 33 EP - 36 PB - ACM ER - TY - CHAP A1 - Cerqueus, Thomas A1 - Cunha de Almeida, Eduardo A1 - Scherzinger, Stefanie T1 - Safely Managing Data Variety in Big Data Software Development T2 - 2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering, 23-23 May 2015, Florence, Italy N2 - We consider the task of building Big Data software systems, offered as software-as-a-service. These applications are commonly backed by NoSQL data stores that address the proverbial Vs of Big Data processing: NoSQL data stores can handle large volumes of data and many systems do not enforce a global schema, to account for structural variety in data. Thus, software engineers can design the data model on the go, a flexibility that is particularly crucial in agile software development. However, NoSQL data stores commonly do not yet account for the veracity of changes when it comes to changes in the structure of persisted data. Yet this is an inevitable consequence of agile software development. In most NoSQL-based application stacks, schema evolution is completely handled within the application code, usually involving object mapper libraries. Yet simple code refactorings, such as renaming a class attribute at the source code level, can cause data loss or runtime errors once the application has been deployed to production. We address this pain point by contributing type checking rules that we have implemented within an IDE plug in. Our plug in ControVol statically type checks the object mapper class declarations against the code release history. ControVol is thus capable of detecting common yet risky cases of mismatched data and schema, and can even suggest automatic fixes. KW - Big data KW - history KW - Java KW - Loading KW - NoSQL data stores KW - object mapping KW - Production KW - Runtime KW - schema evolution KW - Software KW - type checking Y1 - 2015 U6 - https://doi.org/10.1109/BIGDSE.2015.9 SP - 4 EP - 10 PB - IEEE ER - TY - CHAP A1 - Klettke, Meike A1 - Störl, Uta A1 - Shenavai, Manuel A1 - Scherzinger, Stefanie A1 - Storl, Uta T1 - NoSQL schema evolution and big data migration at scale T2 - 2016 IEEE International Conference on Big Data (Big Data), 5-8 Dec. 2016, Washington, DC N2 - This paper explores scalable implementation strategies for carrying out lazy schema evolution in NoSQL data stores. For decades, schema evolution has been an evergreen in database research. Yet new challenges arise in the context of cloud-hosted data backends: With all database reads and writes charged by the provider, migrating the entire data instance eagerly into a new schema can be prohibitively expensive. Thus, lazy migration may be more cost-efficient, as legacy entities are only migrated in case they are actually accessed by the application. Related work has shown that the overhead of migrating data lazily is affordable when a single evolutionary change is carried out, such as adding a new property. In this paper, we focus on long-term schema evolution, where chains of pending schema evolution operations may have to be applied. Chains occur when legacy entities written several application releases back are finally accessed by the application. We discuss strategies for dealing with chains of evolution operations, in particular, the composition into a single, equivalent composite migration that performs the required version jump. Our experiments with MongoDB focus on scalable implementation strategies. Our lineup further compares the number of write operations, and thus, the operational costs of different data migration strategies. KW - Big data KW - Context KW - Data Migration Strategies KW - Data models KW - Databases KW - Incremental Migration KW - Lazy Composite Migration KW - Lazy Migration KW - NoSQL databases KW - Predictive Migration KW - Production KW - Runtime KW - schema evolution KW - Software Y1 - 2016 U6 - https://doi.org/10.1109/BigData.2016.7840924 SP - 2764 EP - 2774 PB - IEEE ER - TY - CHAP A1 - Wostmann, Rene A1 - Schlunder, Philipp A1 - Temme, Fabian A1 - Klinkenberg, Ralf A1 - Kimberger, Josef A1 - Spichtinger, Andrea A1 - Goldhacker, Markus A1 - Deuse, Jochen T1 - Conception of a Reference Architecture for Machine Learning in the Process Industry T2 - 2020 IEEE International Conference on Big Data (Big Data): 10.12.2020 - 13.12.2020 Atlanta, GA, USA N2 - The increasing global competition demands continuous optimization of products and processes from companies in the process industry. Where conventional methods of Lean Management and Six Sigma reach their limits, new opportunities and challenges arise through increasing connectivity in the Industrial Internet of Things and machine learning. The majority of industrial projects do not reach the deployment or are isolated solutions, as the structures for data integration, training, deployment and maintenance of models are not established. This paper presents the conception of a reference architecture for machine learning in the process industry to support companies in implementing their own specific structures. The focus is on the development process and an exemplary implementation in the brewing industry. KW - Big data KW - Companies KW - Industrial Internet of Things KW - Industries KW - Machine learning KW - Optimization KW - process industry KW - Production KW - Reference architecture KW - Six sigma KW - Training Y1 - 2020 SN - 978-1-7281-6251-5 U6 - https://doi.org/10.1109/bigdata50022.2020.9378290 SP - 1726 EP - 1735 PB - IEEE ER -