TY - JOUR A1 - Hutzelmann, Thomas A1 - Mauksch, Dominik A1 - Petrovska, Ana A1 - Pretschner, Alexander T1 - Generation of Tailored and Confined Datasets for IDS Evaluation in Cyber-Physical Systems JF - IEEE Transactions on Dependable and Secure Computing N2 - The state-of-the-art evaluation of an Intrusion Detection System (IDS) relies on benchmark datasets composed of the regular system's and potential attackers’ behavior. The datasets are collected once and independently of the IDS under analysis. This paper questions this practice by introducing a methodology to elicit particularly challenging samples to benchmark a given IDS. In detail, we propose (1) six fitness functions quantifying the suitability of individual samples, particularly tailored for safety-critical cyber-physical systems, (2) a scenario-based methodology for attacks on networks to systematically deduce optimal samples in addition to previous datasets, and (3) a respective extension of the standard IDS evaluation methodology. We applied our methodology to two network-based IDSs defending an advanced driver assistance system. Our results indicate that different IDSs show strongly differing characteristics in their edge case classifications and that the original datasets used for evaluation do not include such challenging behavior. In the worst case, this causes a critical undetected attack, as we document for one IDS. Our findings highlight the need to tailor benchmark datasets to the individual IDS in a final evaluation step. Especially the manual investigation of selected samples from edge case classifications by domain experts is vital for assessing the IDSs. UR - https://doi.rog/10.1109/TDSC.2023.3341211 Y1 - 2023 UR - https://doi.rog/10.1109/TDSC.2023.3341211 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-54670 SN - 1941-0018 VL - 21 IS - 4 SP - 3948 EP - 3962 PB - IEEE CY - New York ER - TY - JOUR A1 - Petrovska, Ana A1 - Kugele, Stefan A1 - Hutzelmann, Thomas A1 - Beffart, Theo A1 - Bergemann, Sebastian A1 - Pretschner, Alexander T1 - Defining adaptivity and logical architecture for engineering (smart) self-adaptive cyber–physical systems JF - Information and Software Technology UR - https://doi.org/10.1016/j.infsof.2022.106866 Y1 - 2022 UR - https://doi.org/10.1016/j.infsof.2022.106866 SN - 0950-5849 VL - 2022 IS - 147 PB - Elsevier CY - Amsterdam ER - TY - CHAP A1 - Zieglmeier, Valentin A1 - Kacianka, Severin A1 - Hutzelmann, Thomas A1 - Pretschner, Alexander T1 - A Real-Time Remote IDS Testbed for Connected Vehicles T2 - Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing UR - https://doi.org/10.1145/3297280.3297465 Y1 - 2019 UR - https://doi.org/10.1145/3297280.3297465 SN - 978-1-4503-5933-7 SP - 1898 EP - 1905 PB - ACM CY - New York ER - TY - CHAP A1 - Lipp, Stephan A1 - Elsner, Daniel A1 - Hutzelmann, Thomas A1 - Banescu, Sebastian A1 - Pretschner, Alexander A1 - Böhme, Marcel T1 - FuzzTastic: A Fine-grained, Fuzzer-agnostic Coverage Analyzer T2 - Proceedings, 2022 ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings N2 - Performing sound and fair fuzzer evaluations can be challenging, not only because of the randomness involved in fuzzing, but also due to the large number of fuzz tests generated. Existing evaluations use code coverage as a proxy measure for fuzzing effectiveness. Yet, instead of considering coverage of all generated fuzz inputs, they only consider the inputs stored in the fuzzer queue. However, as we show in this paper, this approach can lead to biased assessments due to path collisions. Therefore, we developed FuzzTastic, a fuzzeragnostic coverage analyzer that allows practitioners and researchers to perform uniform fuzzer evaluations that are not affected by such collisions. In addition, its time-stamped coverage-probing approach enables frequency-based coverage analysis to identify barely tested source code and to visualize fuzzing progress over time and across code. To foster further studies in this field, we make FuzzTastic, together with a benchmark dataset worth ~12 CPU-years of fuzzing, publicly available; the demo video can be found at https://youtu.be/Lm-eBx0aePA. UR - https://doi.org/10.1145/3510454.3516847 Y1 - 2022 UR - https://doi.org/10.1145/3510454.3516847 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-54624 SN - 978-1-4503-9223-5 SP - 75 EP - 79 PB - ACM CY - New York ER - TY - JOUR A1 - Hutzelmann, Thomas A1 - Banescu, Sebastian A1 - Pretschner, Alexander T1 - A Comprehensive Attack and Defense Model for the Automotive Domain JF - SAE International Journal of Transportation Cybersecurity and Privacy UR - https://doi.org/10.4271/11-02-01-0001 Y1 - 2019 UR - https://doi.org/10.4271/11-02-01-0001 SN - 2572-1054 VL - 2 IS - 1 PB - SAE International CY - Warrendale ER - TY - INPR A1 - Ognawala, Saahil A1 - Pretschner, Alexander A1 - Hutzelmann, Thomas A1 - Psallida, Eirini A1 - Amato, Ricardo Nales T1 - Reviewing KLEE’s Sonar-Search Strategy in Context of Greybox Fuzzing N2 - Automatic test-case generation techniques of symbolic execution and fuzzing are the most widely used methods to discover vulnerabilities in, both, academia and industry. However, both these methods suffer from fundamental drawbacks that stop them from achieving high path coverage that may, consequently, lead to discovering vulnerabilities at the numerical scale of static analysis. In this presentation, we examine systems-under-test (SUTs) at the granularity level of functions and postulate that achieving higher function coverage (execution of functions in a program at least once) than, both, symbolic execution and fuzzing may be a necessary condition for discovering more vulnerabilities than both. We will start this presentation with the design of a targeted search strategy for KLEE, sonar-search, that prioritizes paths leading to a target function, rather than maximizing overall path coverage in the program. Then, we will show that examining SUTs at the level of functions (compositional analysis) leads to discovering more vulnerabilities than symbolic execution from a single entry point. Using this finding, we will, then, demonstrate a greybox fuzzing method that can achieve higher function coverage than symbolic execution. Finally, we will present a framework to effectively manage vulnerabilities and assess their severities. UR - https://doi.org/10.48550/arXiv.1803.04881 Y1 - 2018 UR - https://doi.org/10.48550/arXiv.1803.04881 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-54646 PB - arXiv CY - Ithaca ER - TY - CHAP A1 - Ognawala, Saahil A1 - Hutzelmann, Thomas A1 - Psallida, Eirini A1 - Pretschner, Alexander T1 - Improving Function Coverage with Munch: A Hybrid Fuzzing and Directed Symbolic Execution Approach T2 - Proceedings of the 33rd Annual ACM Symposium on Applied Computing UR - https://doi.org/10.1145/3167132.3167289 Y1 - 2018 UR - https://doi.org/10.1145/3167132.3167289 SN - 978-1-4503-5191-1 SP - 1475 EP - 1482 PB - ACM CY - New York ER - TY - CHAP A1 - Hutzelmann, Thomas A1 - Mauksch, Dominik A1 - Pretschner, Alexander ED - Muccini, Henry ED - Avgeriou, Paris ED - Buhnova, Barbora ED - Camara, Javier ED - Caporuscio, Mauro ED - Franzago, Mirco ED - Koziolek, Anne ED - Scandurra, Patrizia ED - Trubiani, Catia ED - Weyns, Danny ED - Zdun, Uwe T1 - How to Conduct Experiments with a Real Car? Experiences and Practical Guidelines T2 - Software Architecture: 14th European Conference, ECSA 2020, Tracks and Workshops, L'Aquila, Italy, September 14–18, 2020, Proceedings UR - https://doi.org/10.1007/978-3-030-59155-7_37 Y1 - 2020 UR - https://doi.org/10.1007/978-3-030-59155-7_37 SN - 978-3-030-59155-7 SP - 518 EP - 526 PB - Springer CY - Cham ER -