TY - GEN A1 - Schintke, Florian A1 - Reinefeld, Alexander A1 - Haridi, Seif A1 - Schütt, Thorsten T1 - Enhanced Paxos Commit for Transactions on DHTs N2 - Key/value stores which are built on structured overlay networks often lack support for atomic transactions and strong data consistency among replicas. This is unfortunate, because consistency guarantees and transactions would allow a wide range of additional application domains to benefit from the inherent scalability and fault-tolerance of DHTs. The Scalaris key/value store supports strong data consistency and atomic transactions. It uses an enhanced Paxos Commit protocol with only four communication steps rather than six. This improvement was possible by exploiting information from the replica distribution in the DHT. Scalaris enables implementation of more reliable and scalable infrastructure for collaborative Web services that require strong consistency and atomic changes across multiple items. T3 - ZIB-Report - 09-28 KW - Paxos KW - Transaktionen KW - DHT KW - strenge Konsistenz KW - Paxos KW - transactions KW - DHT KW - strong consistency Y1 - 2009 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-11448 SN - 1438-0064 ER - TY - THES A1 - Schütt, Thorsten T1 - Synchronisation von verteilten Verzeichnisstrukturen N2 - Das Management von großen Datenmengen spielt eine immer wichtigere Rolle, wie aktuelle Entwicklungen in der Hochenergiephysik [2] zeigen. Für das DataGrid-Projekt zum Beispiel ist es notwendig, große Datenmengen auf mehrere Rechenzentren in Europa zu verteilen und die Daten untereinander zu synchronisieren. Auch innerhalb von Clustern gewinnen mit zunehmender Anzahl der Knoten Werkzeuge zur effizienten Synchronisation und Verteilung von Daten an Bedeutung. Im Rahmen dieser Arbeit wurde ein effizientes Verfahren zur Synchronisation von verteilten Verzeichnisstrukturen entwickelt und implementiert. Mit diesem Verfahren ist es möglich, unabhängige Änderungen an beliebigen Repositories gleichzeitig durchzuführen. Das Verfahren benötigt keine zentrale Instanz, wodurch eine gegenüber vielen existierenden Verfahren verbesserte Skalierbarkeit erreicht werden konnte. Dabei wurden Erkenntnisse aus der Graphentheorie eingesetzt und weiterentwickelt, um die Netzwerktopolo-gie und -handbreiten zwischen den Rechnern zu berücksichtigen. Durch die Verwendung einer Offline-Synchronisation werden Änderungen erst dann an andere Rechner propagiert, wenn der Nutzer dies anstößt. Das kann zum Beispiel nach einer abgeschlossenen Transaktion, die Änderungen an mehreren Dateien beinhaltet, angemessen sein. N2 - Current developments in high energy physics [2] show that the management of large datasets plays an important role. For the DataGrid project it is necessary to distribute large datasets over several computing centers all over Europe and to synchronize these datasets. Within clusters tools for efficient synchronization and distribution of data become more important, too. In this thesis, a method to synchronize distributed directory structures was developed and implemented which makes it possible to perform independent changes to arbitrary repositories simultaneously. This method needs no central instance and therefore the presented system achieves a better scalability than many existing systems. Knowledge from graph theory was used and improved to take the network topology and the network bandwidth between the computers into account. By using offline synchronization, changes will only be propagated when the user initiates it. This can be reasonable after a completed transaction which consists of changes on several files. Y1 - 2002 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-10071 ER - TY - GEN A1 - Moser, Monika A1 - Haridi, Seif A1 - Shafaat, Tallat A1 - Schütt, Thorsten A1 - Högqvist, Mikael A1 - Reinefeld, Alexander T1 - Transactional DHT Algorithms N2 - We present a framework for transactional data access on data stored in a DHT. It allows to atomically read and write items and to run distributed transactions consisting of a sequence of read and write operations on the items. Items are symmetrically replicated in order to achieve durability of data stored in the SON. To provide availability of items despite the unavailability of some replicas, operations on items are quorum-based. They make progress as long as a majority of replicas can be accessed. Our framework processes transactions optimistically with an atomic commit protocol that is based on Paxos atomic commit. We present algorithms for the whole framework with an event based notation. Additionally we discuss the problem of lookup inconsistencies and its implications on the one-copy serializability property of the transaction processing in our framework. T3 - ZIB-Report - 09-34 KW - Distributed System KW - DHT KW - Paxos KW - Algorithms KW - Storage Y1 - 2009 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-11532 SN - 1438-0064 ER - TY - THES A1 - Schütt, Thorsten T1 - Range queries in distributed hash tables Y1 - 2010 UR - http://www.dr.hut-verlag.de/978-3-86853-370-5.html ER - TY - CHAP A1 - Döbbelin, Robert A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - An Analysis of SMP Memory Allocators T2 - Proceedings of the 41st International Conference on Parallel Processing Workshops (Fifth International Workshop on Parallel Programming Models and Systems Software for High-End Computing (P2S2)) Y1 - 2012 U6 - https://doi.org/10.1109/ICPPW.2012.10 SP - 48 EP - 54 PB - IEEE Computer Society ER - TY - JOUR A1 - Kruber, Nico A1 - Högqvist, Mikael A1 - Schütt, Thorsten T1 - The Benefits of Estimated Global Information in DHT Load Balancing JF - Cluster Computing and the Grid, IEEE International Symposium on Y1 - 2011 U6 - https://doi.org/10.1109/CCGrid.2011.11 VL - 0 SP - 382 EP - 391 PB - IEEE Computer Society CY - Los Alamitos, CA, USA ER - TY - CHAP A1 - Schintke, Florian A1 - Reinefeld, Alexander A1 - Haridi, Seif A1 - Schütt, Thorsten T1 - Enhanced Paxos Commit for Transactions on DHTs T2 - CCGRID Y1 - 2010 U6 - https://doi.org/10.1109/CCGRID.2010.41 SP - 448 EP - 454 PB - IEEE ER - TY - CHAP A1 - Schütt, Thorsten A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Hennig, C. T1 - Self-Adaptation in Large-Scale Systems: A Study on Structured Overlays Across Multiple Datacenters T2 - Architectures and Languages for Self-Managing Distributed Systems (SelfMan@SASO) Y1 - 2009 U6 - https://doi.org/10.1109/SASOW.2010.65 ER - TY - CHAP A1 - Schütt, Thorsten A1 - Moser, Monika A1 - Plantikow, Stefan A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - A Transactional Scalable Distributed Data Store T2 - 1st IEEE International Scalable Computing Challenge, co-located with CCGrid’08 Y1 - 2008 ER - TY - CHAP A1 - Schütt, Thorsten A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Hoffmann, Marie T1 - Gossip-based Topology Inference for Efficient Overlay Mapping on Data Centers T2 - Peer-to-Peer Computing Y1 - 2009 U6 - https://doi.org/10.1109/P2P.2009.5284553 SP - 147 EP - 150 ER - TY - CHAP A1 - Reinefeld, Alexander A1 - Schütt, Thorsten T1 - Out-of-Core Parallel Frontier Search with MapReduce T2 - HPCS Y1 - 2009 U6 - https://doi.org/10.1007/978-3-642-12659-8_24 VL - 5976 SP - 323 EP - 336 PB - Springer ER - TY - CHAP A1 - Reinefeld, Alexander A1 - Schütt, Thorsten A1 - Döbbelin, Robert ED - Hariri, Salim ED - Keahey, Kate T1 - Very large pattern databases for heuristic search T2 - HPDC Y1 - 2010 U6 - https://doi.org/10.1145/1851476.1851592 SP - 803 EP - 809 PB - ACM ER - TY - CHAP A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Haridi, Seif ED - Tselentis, G. ED - Domingue, J. ED - Galis, A. ED - Gavras, A. ED - Hausheer, D. ED - Krco, S. ED - Lotz, V. ED - Zahariadis, T. T1 - A Scalable, Transactional Data Store for Future Internet Services T2 - Towards the Future Internet - A European Research Perspective Y1 - 2009 U6 - https://doi.org/10.3233/978-1-60750-007-0-148 SP - 148 EP - 159 PB - IOS Press ER - TY - CHAP A1 - Costa, Paolo A1 - Pierre, Guillaume A1 - Reinefeld, Alexander A1 - Schütt, Thorsten A1 - van Steen, Maarten T1 - Sloppy Management of Structured P2P Services T2 - Proceedings of the 3$^rd$ International Workshop on Hot Topics in Autonomic Computing (HotAC III), co-located with IEEE ICAC’08 Y1 - 2008 CY - Chicago, IL, USA ER - TY - CHAP A1 - Högqvist, Mikael A1 - Haridi, Seif A1 - Kruber, Nico A1 - Reinefeld, Alexander A1 - Schütt, Thorsten T1 - Using Global Information for Load Balancing in DHTs T2 - Workshop on Decentralized Self Management for Grids, P2P, and User Communities Y1 - 2008 ER - TY - GEN A1 - Andrzejak, Artur A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Mastroianni, Carlo A1 - Fragopoulou, Paraskevi A1 - Kondo, Derrick A1 - Malecot, Paul A1 - Cosmin Silaghi, Gheorghe A1 - Moura Silva, Luis A1 - Trunfio, Paolo A1 - Zeinalipour-Yazti, Demetris A1 - Zimeo, Eugenio T1 - Grid Architectural Issues: State-of-the-art and Future Trends Y1 - 2008 UR - http://coregrid.ercim.eu//mambo/images/stories/WhitePapers/whp-0004.pdf IS - WHP-0004 PB - Institute on Architectural Issues: Scalability, Dependability, Adaptability, CoreGRID - Network of Excellence ER - TY - CHAP A1 - Shafaat, Tallat A1 - Moser, Monika A1 - Ghodsi, Ali A1 - Schütt, Thorsten A1 - Haridi, Seif A1 - Reinefeld, Alexander T1 - Key-Based Consistency and Availability in Structured Overlay Networks T2 - International ICST Conference on Scalable Information Systems Y1 - 2008 ER - TY - CHAP A1 - Shafaat, Tallat A1 - Moser, Monika A1 - Ghodsi, Ali A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - On Consistency of Data in Structured Overlay Networks T2 - Coregrid Integration Workshop Y1 - 2008 ER - TY - JOUR A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - Range Queries on structured overlay networks JF - Computer Communications Y1 - 2008 U6 - https://doi.org/10.1016/j.comcom.2007.08.027 VL - 31 IS - 2 SP - 280 EP - 291 PB - Elsevier ER - TY - CHAP A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - Scalaris: Reliable Transactional P2P Key/Value T2 - ERLANG '08: Proceedings of the 7th ACM SIGPLAN workshop on ERLANG Y1 - 2008 U6 - https://doi.org/10.1145/1411273.1411280 SP - 41 EP - 47 ER - TY - CHAP A1 - Schütt, Thorsten A1 - Döbbelin, Robert A1 - Reinefeld, Alexander T1 - Forward Perimeter Search with Controlled Use of Memory T2 - International Joint Conference on Artificial Intelligence, IJCAI-13, Beijing Y1 - 2013 ER - TY - JOUR A1 - Schütt, Thorsten A1 - Reinefeld, Alexander A1 - Döbbelin, Robert T1 - MR-search: massively parallel heuristic search JF - Concurrency and Computation: Practice and Experience Y1 - 2011 U6 - https://doi.org/10.1002/cpe.1833 VL - 25 IS - 1 SP - 40 EP - 54 ER - TY - JOUR A1 - Reinefeld, Alexander A1 - Schütt, Thorsten A1 - Döbbelin, Robert T1 - Analyzing the performance of SMP memory allocators with iterative MapReduce applications JF - Parallel Computing Y1 - 2013 U6 - https://doi.org/10.1016/j.parco.2013.09.007 VL - 39 IS - 12 SP - 879 EP - 889 PB - Elsevier ER - TY - CHAP A1 - Kleineweber, Christoph A1 - Reinefeld, Alexander A1 - Schütt, Thorsten T1 - QoS-aware Storage Virtualization for Cloud File Systems T2 - Proceedings of the 1st ACM International Workshop on Programmable File Systems Y1 - 2014 U6 - https://doi.org/10.1145/2603941.2603944 SP - 19 EP - 26 ER - TY - CHAP A1 - Weinhold, Carsten A1 - Lackorzynski, Adam A1 - Bierbaum, Jan A1 - Küttler, Martin A1 - Planeta, Maksym A1 - Härtig, Hermann A1 - Shiloh, Amnon A1 - Levy, Ely A1 - Ben-Nun, Tal A1 - Barak, Amnon A1 - Steinke, Thomas A1 - Schütt, Thorsten A1 - Fajerski, Jan A1 - Reinefeld, Alexander A1 - Lieber, Matthias A1 - Nagel, Wolfgang T1 - FFMK: A Fast and Fault-tolerant Microkernel-based System for Exascale Computing T2 - SPPEXA Symposium 2016 Y1 - 2016 U6 - https://doi.org/10.1007/978-3-319-40528-5_18 ER - TY - CHAP A1 - Fajerski, J. A1 - Noack, Matthias A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Steinke, Thomas T1 - Fast In-Memory Checkpointing with POSIX API for Legacy Exascale-Applications T2 - SPPEXA Symposium 2016 Y1 - 2016 U6 - https://doi.org/10.1007/978-3-319-40528-5_19 ER - TY - CHAP A1 - Coutinho, Jose A1 - Stillwell, Mark A1 - Argyraki, Katerina A1 - Ioannidis, George A1 - Iordache, Anca A1 - Kleineweber, Christoph A1 - Koliousis, Alexandros A1 - McGlone, John A1 - Pierre, Guillaume A1 - Ragusa, Carmelo A1 - Sanders, Peter A1 - Schütt, Thorsten A1 - Yu, Teng A1 - Wolf, Alexander T1 - The HARNESS Platform: A Hardware- and Network-Enhanced Software System for Cloud Computing T2 - Software Architecture for Big Data and the Cloud Y1 - 2017 SN - 9780128054673 SP - 323 EP - 351 PB - Morgan Kaufmann ER - TY - CHAP A1 - Döbbelin, Robert A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Building Large Compressed PDBs for the Sliding Tile Puzzle T2 - Workshop on Computer Games Y1 - 2013 ER - TY - JOUR A1 - Allen, Gabrielle A1 - Davis, K. A1 - Goodale, Tom A1 - Hutanu, Andrei A1 - Kaiser, Hartmut A1 - Kielmann, Thilo A1 - Merzky, Andre A1 - Nieuwpoort, R. A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Seidel, Edward A1 - Ullmer, Brygg T1 - The Grid Application Toolkit: Toward Generic and Easy Appliction Programming Interfaces for the Grid JF - Procceedings of the IEEE Y1 - 2005 U6 - https://doi.org/10.1109/JPROC.2004.842755 VL - 93 IS - 3 SP - 534 EP - 550 PB - IEEE Computer Society Press ER - TY - CHAP A1 - Schütt, Thorsten A1 - Merzky, André A1 - Hutanu, Andrei A1 - Schintke, Florian T1 - Remote partial file access using compact pattern descriptions T2 - 4th IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2004), April 19-22, 2004, Chicago, Illinois, USA Y1 - 2004 U6 - https://doi.org/10.1109/CCGrid.2004.1336604 SP - 482 EP - 489 PB - IEEE Computer Society ER - TY - CHAP A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Scalable and Self-Optimizing Data Grids T2 - Annual Review of Scalable Computing Y1 - 2004 U6 - https://doi.org/10.1142/9789812794727_0002 VL - 6 IS - chapter 2 SP - 30 EP - 60 PB - World Scientific ER - TY - GEN A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Vorrichtung und Verfahren zum Abrufen / Speichern von elektronischen Daten in einem System mit mehreren Datenverarbeitungseinheiten Y1 - 2007 PB - Europäisches Patent EP 1 744 257 A1 ER - TY - GEN A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Vorrichtung und Verfahren zum Speichern / Abrufen von Objekten mit mehrdimensional adressierten, elektronischen Daten Y1 - 2007 PB - Europäische Patentanmeldung Nr. 06012030.0 vom 12.06.2006, Patent Nr: 1868114 ER - TY - CHAP A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - A Framework for Self-Optimizing Grids Using P2P Components T2 - 14th International Workshop on Database and Expert Systems Applications (DEXA’03), September 1-5, 2003, Prague, Czech Republic Y1 - 2003 U6 - https://doi.org/10.1109/DEXA.2003.1232101 SP - 689 EP - 693 PB - IEEE Computer Society ER - TY - CHAP A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander ED - Sloot, Peter ED - Abramson, David ED - Bogdanov, Alexander ED - Dongarra, Jack ED - Zomaya, Albert ED - Gorbachev, Yuri T1 - Efficient Synchronization of Replicated Data in Distributed Systems T2 - Computational Science - ICCS 2003, International Conference, Melbourne, Australia and St. Petersburg, Russia, June 2-4, 2003. Proceedings, Part I Y1 - 2003 U6 - https://doi.org/10.1007/3-540-44860-8_28 VL - 2657 SP - 274 EP - 283 PB - Springer ER - TY - CHAP A1 - Andrzejak, Artur A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten ED - Getov, Vladimir ED - Laforenza, Domenico ED - Reinefeld, Alexander T1 - On Adaptability in Grid Systems T2 - Future Generation Grids: Proceedings of the Workshop on Future Generation Grids, November 1-5, 2004, Dagstuhl, Germany Y1 - 2006 U6 - https://doi.org/10.1007/978-0-387-29445-2_2 SP - 29 EP - 46 PB - Springer ER - TY - CHAP A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - Structured Overlay without Consistent Hashing: Empirical Results T2 - Sixth Workshop on Global and Peer-to-Peer Computing (GP2PC’06) at Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGrid 2006), 16-19 May 2006, Singapore Y1 - 2006 UR - http://doi.ieeecomputersociety.org/10.1109/CCGRID.2006.175 U6 - https://doi.org/10.1109/CCGRID.2006.175 PB - IEEE Computer Society ER - TY - CHAP A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander ED - Kermarrec, Anne-Marie T1 - A Structured Overlay for Multi-dimensional Range Queries T2 - Euro-Par Conference Y1 - 2007 UR - http://dx.doi.org/10.1007/978-3-540-74466-5_54 U6 - https://doi.org/10.1007/978-3-540-74466-5_54 VL - 4641 SP - 503 EP - 513 PB - Springer ER - TY - GEN A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Device and method for retrieving / storing electronic data in a system with a plurality of data processing units Y1 - 2007 PB - United States Patent Application Publication No. US 2007/0165619 A1 ER - TY - CHAP A1 - Andersson, Stefan A1 - Sachs, Stephen A1 - Tuma, Christian A1 - Schütt, Thorsten T1 - DataWarp: First Experiences T2 - Cray User Group Y1 - 2015 ER - TY - CHAP A1 - Skrzypczak, Jan A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Linearizable State Machine Replication of State-Based CRDTs without Logs T2 - Proceedings of the 2019 ACM Symposium on Principles of Distributed Computing, PODC 2019 N2 - General solutions of state machine replication have to ensure that all replicas apply the same commands in the same order, even in the presence of failures. Such strict ordering incurs high synchronization costs due to the use of distributed consensus or a leader. This paper presents a protocol for linearizable state machine replication of conflict-free replicated data types (CRDTs) that neither requires consensus nor a leader. By leveraging the properties of state-based CRDTs—in particular the monotonic growth of a join semilattice—synchronization overhead is greatly reduced. In addition, updates just need a single round trip and modify the state ‘in-place’ without the need for a log. Furthermore, the message size overhead for coordination consists of a single counter per message. While reads in the presence of concurrent updates are not wait-free without a coordinator, we show that more than 97 % of reads can be handled in one or two round trips under highly concurrent accesses. Our protocol achieves high throughput without auxiliary processes such as command log management or leader election. It is well suited for all practical scenarios that need linearizable access on CRDT data on a fine-granular scale. Y1 - 2019 U6 - https://doi.org/10.1145/3293611.3331568 SP - 455 EP - 457 ER - TY - JOUR A1 - Skrzypczak, Jan A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - RMWPaxos: Fault-Tolerant In-Place Consensus Sequences JF - IEEE Transactions on Parallel and Distributed Systems Y1 - 2020 U6 - https://doi.org/10.1109/TPDS.2020.2981891 SN - 1045-9219 VL - 31 IS - 10 SP - 2392 EP - 2405 ER - TY - CHAP A1 - Hartung, Marc A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Pinpoint Data Races via Testing and Classification T2 - 2019 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW); 3rd International Workshop on Software Faults (IWSF 2019) Y1 - 2019 U6 - https://doi.org/10.1109/ISSREW.2019.00100 SP - 386 EP - 393 ER - TY - CHAP A1 - Schmidtke, Robert A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - From Application to Disk: Tracing I/O Through the Big Data Stack T2 - High Performance Computing ISC High Performance 2018 International Workshops, Frankfurt/Main, Germany, June 24 - 28, 2018, Revised Selected Papers, Workshop on Performance and Scalability of Storage Systems (WOPSSS) N2 - Typical applications in data science consume, process and produce large amounts of data, making disk I/O one of the dominating — and thus worthwhile optimizing — factors of their overall performance. Distributed processing frameworks, such as Hadoop, Flink and Spark, hide a lot of complexity from the programmer when they parallelize these applications across a compute cluster. This exacerbates reasoning about I/O of both the application and the framework, through the distributed file system, such as HDFS, down to the local file systems. We present SFS (Statistics File System), a modular framework to trace each I/O request issued by the application and any JVM-based big data framework involved, mapping these requests to actual disk I/O. This allows detection of inefficient I/O patterns, both by the applications and the underlying frameworks, and builds the basis for improving I/O scheduling in the big data software stack. Y1 - 2018 U6 - https://doi.org/10.1007/978-3-030-02465-9_6 SP - 89 EP - 102 ER - TY - JOUR A1 - Skrzypczak, Jan A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Linearizable State Machine Replication of State-Based CRDTs without Logs JF - arXiv N2 - General solutions of state machine replication have to ensure that all replicas apply the same commands in the same order, even in the presence of failures. Such strict ordering incurs high synchronization costs caused by distributed consensus or by the use of a leader. This paper presents a protocol for linearizable state machine replication of conflict-free replicated data types (CRDTs) that neither requires consensus nor a leader. By leveraging the properties of state-based CRDTs - in particular the monotonic growth of a join semilattice - synchronization overhead is greatly reduced. In addition, updates just need a single round trip and modify the state `in-place' without the need for a log. Furthermore, the message size overhead for coordination consists of a single counter per message. While reads in the presence of concurrent updates are not wait-free without a coordinator, we show that more than 97% of reads can be handled in one or two round trips under highly concurrent accesses. Our protocol achieves high throughput without auxiliary processes like command log management or leader election. It is well suited for all practical scenarios that need linearizable access on CRDT data on a fine-granular scale. Y1 - 2019 ER - TY - JOUR A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Skrzypczak, Jan T1 - Transactions on Red-black and AVL trees in NVRAM JF - arXiv Y1 - 2020 ER - TY - GEN A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - P2P Routing of Range Queries in Skewed Multidimensional Data Sets N2 - We present a middleware to store multidimensional data sets on Internet-scale distributed systems and to efficiently perform range queries on them. Our structured overlay network \emph{SONAR (Structured Overlay Network with Arbitrary Range queries)} puts keys which are adjacent in the key space on logically adjacent nodes in the overlay and is thereby able to process multidimensional range queries with a single logarithmic data lookup and local forwarding. The specified ranges may have arbitrary shapes like rectangles, circles, spheres or polygons. Empirical results demonstrate the routing performance of SONAR on several data sets, ranging from real-world data to artificially constructed worst case distributions. We study the quality of SONAR's routing information which is based on local knowledge only and measure the indegree of the overlay nodes to find potential hot spots in the routing process. We show that SONAR's routing table is self-adjusting, even under extreme situations, keeping always a maximum of $\lceil \log N \rceil$ routing entries. T3 - ZIB-Report - 07-23 KW - structured overlays KW - range queries KW - routing KW - multidimensional data sets Y1 - 2007 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-9671 SN - 1438-0064 ER - TY - GEN A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - Chord#: Structured Overlay Network for Non-Uniform Load-Distribution N2 - \newcommand{\chordsharp}{Chord$^\##$} Data lookup is a fundamental problem in peer-to-peer systems: Given a key, find the node that stores the associated object. Chord and other P2P algorithms use distributed hash tables (DHTs) to distribute the keys and nodes evenly across a logical ring. Using an efficient routing strategy, DHTs provide a routing performance of $O (\log N)$ in networks of $N$ nodes. While the routing performance has been shown to be optimal, the uniform key distribution makes it impossible for DHTs to support range queries. For range queries, consecutive keys must be stored on lo gically neighboring nodes. In this paper, we present an enhancement of Chord that eliminates the hash function while keeping the same routing performance. The resulting algorithm, named \chordsharp{}, provides a richer function ality while maintaining the same complexity. In addition to Chord, \chordsharp{} adapts to load imbalance. T3 - ZIB-Report - 05-40 KW - DHT KW - P2P KW - Range Queries Y1 - 2005 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-8736 ER - TY - GEN A1 - Döbbelin, Robert A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Building Large Compressed PDBs for the Sliding Tile Puzzle N2 - The performance of heuristic search algorithms depends crucially on the effectiveness of the heuristic. A pattern database (PDB) is a powerful heuristic in the form of a pre-computed lookup table. Larger PDBs provide better bounds and thus allow more cut-offs in the search process. Today, the largest PDB for the 24-puzzle is a 6-6-6-6 PDB with a size of 486 MB. We created 8-8-8, 9-8-7 and 9-9-6 PDBs that are three orders of magnitude larger (up to 1.4 TB) than the 6-6-6-6 PDB. We show how to compute such large PDBs and we present statistical and empirical data on their efficiency. The largest single PDB gives on average an 8-fold improvement over the 6-6-6-6 PDB. Combining several large PDBs gives on average an 12-fold improvement. T3 - ZIB-Report - 13-21 KW - heuristic search KW - pattern databases Y1 - 2013 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-18095 SN - 1438-0064 ER - TY - CHAP A1 - Wende, Florian A1 - Noack, Matthias A1 - Schütt, Thorsten A1 - Sachs, Stephen A1 - Steinke, Thomas T1 - Application Performance on a Cray XC30 Evaluation System with Xeon Phi Coprocessors at HLRN-III T2 - Cray User Group Y1 - 2015 ER - TY - CHAP A1 - Salem, Farouk A1 - Schütt, Thorsten A1 - Schintke, Florian A1 - Reinefeld, Alexander T1 - Scheduling Data Streams for Low Latency and High Throughput on a Cray XC40 Using Libfabric T2 - CUG Conference Proceedings N2 - Achieving efficient many-to-many communication on a given network topology is a challenging task when many data streams from different sources have to be scattered concurrently to many destinations with low variance in arrival times. In such scenarios, it is critical to saturate but not to congest the bisectional bandwidth of the network topology in order to achieve a good aggregate throughput. When there are many concurrent point-to-point connections, the communication pattern needs to be dynamically scheduled in a fine-grained manner to avoid network congestion (links, switches), overload in the node’s incoming links, and receive buffer overflow. Motivated by the use case of the Compressed Baryonic Matter experiment (CBM), we study the performance and variance of such communication patterns on a Cray XC40 with different routing schemes and scheduling approaches. We present a distributed Data Flow Scheduler (DFS) that reduces the variance of arrival times from all sources at least 30 times and increases the achieved aggregate bandwidth by up to 50%. Y1 - 2019 ER - TY - JOUR A1 - Salem, Farouk A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Scheduling data streams for low latency and high throughput on a Cray XC40 using Libfabric JF - Concurrency and Computation Practice and Experience Y1 - 2020 U6 - https://doi.org/10.1002/cpe.5563 VL - 32 IS - 20 SP - 1 EP - 14 ER - TY - JOUR A1 - Salem, Farouk A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Improving the throughput of a scalable FLESnet using the Data-Flow Scheduler JF - CBM Progress Report 2018 Y1 - 2019 SN - 978-3-9815227-6-1 U6 - https://doi.org/10.15120/GSI-2019-01018 SP - 149 EP - 150 ER - TY - JOUR A1 - Salem, Farouk A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Supporting various interconnects in FLESnet using Libfabric JF - CBM Progress Report 2016 Y1 - 2017 UR - https://repository.gsi.de/record/201318 SN - 978-3-9815227-4-7 SP - 159 EP - 160 ER - TY - JOUR A1 - Salem, Farouk A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Data-flow scheduling for a scalable FLESnet JF - CBM Progress Report 2017 Y1 - 2018 SN - 978-3-9815227-5-4 U6 - https://doi.org/10.15120/GSI-2018-00485 SP - 130 EP - 131 ER - TY - CHAP A1 - Gholami, Masoud A1 - Schintke, Florian A1 - Schütt, Thorsten T1 - Checkpoint Scheduling for Shared Usage of Burst-Buffers in Supercomputers T2 - Proceedings of the 47th International Conference on Parallel Processing Companion; SRMPDS 2018: The 14th International Workshop on Scheduling and Resource Management for Parallel and Distributed Systems N2 - User-defined and system-level checkpointing have contrary properties. While user-defined checkpoints are smaller and simpler to recover, system-level checkpointing better knows the global system's state and parameters like the expected mean time to failure (MTTF) per node. Both approaches lead to non-optimal checkpoint time, intervals, sizes, or I/O bandwidth when concurrent checkpoints conflict and compete for it. We combine user-defined and system-level checkpointing to exploit the benefits and avoid the drawbacks of each other. Thus, applications frequently offer to create checkpoints. The system accepts such offers according to the current status and implied costs to recalculate from the last checkpoint or denies them, i.e., immediately lets continue the application without checkpoint creation. To support this approach, we develop economic models for multi-application checkpointing on shared I/O resources that are dedicated for checkpointing (e.g. burst-buffers) by defining an appropriate goal function and solving a global optimization problem. Using our models, the checkpoints of applications on a supercomputer are scheduled to effectively use the available I/O bandwidth and minimize the failure overhead (checkpoint creations plus recalculations). Our simulations show an overall reduction in failure overhead of all nodes of up to 30% for a typical supercomputer workload (HLRN). We can also derive the most cost effective burst-buffer bandwidth for a given node's MTTF and application workload. Y1 - 2018 U6 - https://doi.org/10.1145/3229710.3229755 SP - 44:1 EP - 44:10 ER - TY - CHAP A1 - Weinhold, Carsten A1 - Lackorzynski, Adam A1 - Bierbaum, Jan A1 - Küttler, Martin A1 - Planeta, Maksym A1 - Weisbach, Hannes A1 - Hille, Matthias A1 - Härtig, Hermann A1 - Margolin, Alexander A1 - Sharf, Dror A1 - Levy, Ely A1 - Gak, Pavel A1 - Barak, Amnon A1 - Gholami, Masoud A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander A1 - Lieber, Matthias A1 - Nagel, Wolfgang T1 - FFMK: A Fast and Fault-Tolerant Microkernel-Based System for Exascale Computing T2 - Software for Exascale Computing - SPPEXA 2016-2019 Y1 - 2019 U6 - https://doi.org/10.1007/978-3-030-47956-5_16 SP - 483 EP - 516 PB - Springer ER - TY - GEN A1 - Gholami, Masoud A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Reinefeld, Alexander T1 - Modeling Checkpoint Schedules for Concurrent HPC Applications T2 - CoSaS 2018 International Symposium on Computational Science at Scale Y1 - 2018 ER -