High performance scalable message processing with distributed in-memory cache of object state

  • Within this bachelor thesis, the possibility and efficiency of using a log-based message broker as an object state store is examined. The main goal is to show the high performance a distributed in-memory local state store can achieve and why a log based message broker is a useful complement to create a reliable and scalable object store. The project of this work is designed as state store for current device data of an IoT system based on the microservice architecture. As log-based message broker Kafka is used in this thesis with its additional Streams DSL, which simultaneously perform the functions of a data consumer, producer and a processor. Kafkas’ Streams DSL is the perfect fit for this project, because it already provides an automatic state store handling. Firstly this work describes the implementation how to replace an object database with a distributed local state store. Secondly it shows the providing of the state to other services over a changelog topic. And thirdly the project illustrates how the changelog can be materialized into a view, which makes the data accessible to the outside via a REST API. In the performance analysis of this paper, the comparison between the local state store implementation and a predecessor implementation, with MongoDB as object store, is made. Also, the differences in performance and resource utilization between the two types of key-value stores, that Kafka Streams provides, are investigated. The two types are the only in-memory and the persistent key-value store using RocksDB. The thesis successfully shows that the implementation of such a distributed local state store can be efficiently and easily integrated into a microservice architecture. Furthermore the service is capable to process much more messages per second than the MongoDB implementation. And in addition the resource utilization is much lower in such a system that does not use an external database. Moreover, it is shown that the latency, which is measured by the time the service needs to process one message, does not depend on the number of messages per second. This cannot be achieved with the implementation in MongoDB.

Download full text files

Export metadata

Additional Services

Share in Twitter Search Google Scholar
Metadaten
Author:Robin Jägers
URN:urn:nbn:de:bvb:m347-opus-4695
Publisher:Hochschule für angewandte Wissenschaften München
Place of publication:München
Advisor:Johannes Ebke
Document Type:Bachelor Thesis
Language:English
Date of Publication (online):2023/07/31
Year of first Publication:2023
Publishing Institution:Hochschule für angewandte Wissenschaften München
Granting Institution:Hochschule für angewandte Wissenschaften München
Date of final exam:2023/07/09
Contributing Corporation:Schwarm Technologies Germany GmbH
Release Date:2023/08/01
Tag:Kommunikationsserver; Zustandsspeicher
GND Keyword:DatenstromGND; Verteiltes SystemGND; Internet der DingeGND; Apache KafkaGND
Page Number:vi, 51
Departments and Institutes:Fakultäten der Hochschule München / FK 07 Fakultät für Informatik und Mathematik / Bachelorstudiengang Informatik
Dewey Decimal Classification:0 Informatik, Informationswissenschaft, allgemeine Werke / 00 Informatik, Wissen, Systeme / 004 Datenverarbeitung; Informatik
Illustrations:Illustration/en
Illustrations:grafische Darstellung/en
Licence (German):Creative Commons - CC BY-NC-ND - Namensnennung - Nicht kommerziell - Keine Bearbeitungen 4.0 International
Einverstanden
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.