TY - CHAP A1 - Schmidtke, Robert A1 - Laubender, Guido A1 - Steinke, Thomas T1 - Big Data Analytics on Cray XC Series DataWarp using Hadoop, Spark and Flink T2 - CUG Proceedings Y1 - 2016 ER - TY - GEN A1 - Scheffler, Daniel A1 - Sips, Mike A1 - Behling, Robert A1 - Dransch, Doris A1 - Eggert, Daniel A1 - Fajerski, Jan A1 - Freytag, Johann-Christoph A1 - Griffiths, Patrick A1 - Hollstein, André A1 - Hostert, Patrick A1 - Köthur, Patrick A1 - Peters, Mathias A1 - Pflugmacher, Dirk A1 - Rabe, Andreas A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Segel, Karl T1 - GeoMultiSens – Scalable Multisensoral Analysis of Satellite Remote Sensing Data T2 - ESA Living Planet Symposium, EO Open Science Posters Y1 - 2016 ER - TY - CHAP A1 - Scheffler, Daniel A1 - Sips, Mike A1 - Behling, Robert A1 - Dransch, Doris A1 - Eggert, Daniel A1 - Fajerski, Jan A1 - Freytag, Johann-Christoph A1 - Griffiths, Patrick A1 - Hollstein, André A1 - Hostert, Patrick A1 - Köthur, Patrick A1 - Peters, Mathias A1 - Pflugmacher, Dirk A1 - Rabe, Andreas A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Segel, Karl T1 - Geomultisens – a common automatic processing and analysis system for multi-sensor satellite data T2 - Advancing Horizons for Land Cover Services Entering the Big Data Era, Second joint Workshop of the EARSeL Special Interest Group on Land Use & Land Cover and the NASA LCLUC Program Y1 - 2016 SP - 18 EP - 19 ER - TY - JOUR A1 - Seibert, Felix A1 - Peters, Mathias A1 - Schintke, Florian T1 - Improving I/O Performance Through Colocating Interrelated Input Data and Near-Optimal Load Balancing JF - Proceedings of the IPDPSW; Fourth IEEE International Workshop on High-Performance Big Data, Deep Learning, and Cloud Computing (HPBDC) N2 - Most distributed file systems assign new files to storage servers randomly. While working well in some situations, this does not help to optimize the input performance for most MapReduce computations’ data access patterns. In this work, we consider an access pattern where input files are partitioned into groups of heterogeneous size. Each group is accessed by exactly one process. We design and implement a data placement strategy that places these file groups together on the same storage server. This colocation approach is combined with near-optimal storage load balancing. To do so, we use a classical scheduling approximation algorithm to solve the NP hard group assignment problem. We argue that local processing is not only beneficial because of reduced network traffic, but especially because it imposes an even resource schedule. Our experiments, based on the parallel processing of remote sensing images, reveal an enormous reduction of network traffic and up to 39 % faster input read times. Further, simulations show that our approximate assignments limit storage server imbalances to less than 5 % above the theoretical minimum, in contrast to more than 85 % with random assignment. Y1 - 2018 U6 - https://doi.org/10.1109/IPDPSW.2018.00081 N1 - Best Paper Award VL - 2018 SP - 448 EP - 457 ER -