Experiences With Replicable Experiments and Replication Kits for Software Engineering Research

  • Replications and replicable research are currently gaining traction in the software engineering research community. Our research group made an effort in the recent years to make our own research accessible for other researchers, through the provision of replication kits that allow rerunning our experiments. Within this chapter, we present our experiences with replication kits. We first had to learn which contents are required, how to structure them, how to document them, and also how to best share them with other researchers. While this sounds very straightforward, there are many small potential mistakes, which may have a strong negative impact on the usefulness and long-term availability of replication kits. We derive best practices for the content and the sharing of replication kits based on our experiences. Moreover, we outline how platforms for replicable research may further help our community, especially with problems related to the external validity of results. Finally, we discuss the lack of integration of replication kitsReplications and replicable research are currently gaining traction in the software engineering research community. Our research group made an effort in the recent years to make our own research accessible for other researchers, through the provision of replication kits that allow rerunning our experiments. Within this chapter, we present our experiences with replication kits. We first had to learn which contents are required, how to structure them, how to document them, and also how to best share them with other researchers. While this sounds very straightforward, there are many small potential mistakes, which may have a strong negative impact on the usefulness and long-term availability of replication kits. We derive best practices for the content and the sharing of replication kits based on our experiences. Moreover, we outline how platforms for replicable research may further help our community, especially with problems related to the external validity of results. Finally, we discuss the lack of integration of replication kits into most current review processes at conferences and journals. We also give one example for a review process into which replication kits were well-integrated. Altogether, this chapter demonstrates that making research replicable is a challenging task and there is a long road ahead until our community has a generally accepted and enforced standard of replicability.show moreshow less

Export metadata

Additional Services

Search Google Scholar
Metadaten
Author:Steffen Herbold, Fabian Trautsch, Patrick HarmsORCiD, Verena Herbold, Jens Grabowski
DOI:https://doi.org/10.1016/bs.adcom.2018.10.003
Parent Title (English):Advances in Computers
Publisher:Elsevier
Document Type:Article
Language:English
Date of first Publication:2019/01/01
Release Date:2025/02/05
Volume:113
Pagenumber:315 - 344
institutes:Fakultät für Elektrotechnik Feinwerktechnik Informationstechnik
Ohm User Experience Center (Ohm-UX)
Research Themes:Digitalisierung & Künstliche Intelligenz
Verstanden ✔
Diese Webseite verwendet technisch erforderliche Session-Cookies. Durch die weitere Nutzung der Webseite stimmen Sie diesem zu. Unsere Datenschutzerklärung finden Sie hier.