TY - CHAP A1 - Hoess, Nicole A1 - Paradis, Carlos A1 - Kazman, Rick A1 - Mauerer, Wolfgang T1 - Does the Tool Matter? Exploring Some Causes of Threats to Validity in Mining Software Repositories T2 - 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), Montreal, QC, Canada, March, 4-7, 2025 N2 - Software repositories are an essential source of information for software engineering research on topics such as project evolution and developer collaboration. Appropriate mining tools and analysis pipelines are therefore an indispensable precondition for many research activities. Ideally, valid results should not depend on technical details of data collection and processing. It is, however, widely acknowledged that mining pipelines are complex, with a multitude of implementation decisions made by tool authors based on their interests and assumptions. This raises the questions if (and to what extent) tools agree on their results and are interchangeable. In this study, we use two tools to extract and analyse ten large software projects, quantitatively and qualitatively comparing results and derived data to better understand this concern. We analyse discrepancies from a technical point of view, and adjust code and parametrisation to minimise replication differences. Our results indicate that despite similar trends, even simple metrics such as the numbers of commits and developers may differ by up to 500%. We find that such substantial differences are often caused by minor technical details. We show how tool-level and data post-processing changes can overcome these issues, but find they may require considerable efforts. We summarise identified causes in our lessons learned to help researchers and practitioners avoid common pitfalls, and reflect on implementation decisions and their influence in ensuring obtained data meets explicit and implicit expectations. Our findings lead us to hypothesise that similar uncertainties exist in other analysis tools, which may limit the validity of conclusions drawn in tool-centric research. KW - Codes KW - Collaboration KW - Data collection KW - Data mining KW - Developer Networks KW - Empirical Software Engineering KW - Market research KW - Measurement KW - Mining Software Repositories KW - Pipelines KW - Research Software KW - Software engineering KW - Uncertainty Y1 - 2025 SN - 979-8-3315-3510-0 U6 - https://doi.org/10.1109/SANER64311.2025.00067 SP - 645 EP - 656 PB - IEEE ER - TY - CHAP A1 - Ramsauer, Ralf A1 - Lohmann, Daniel A1 - Mauerer, Wolfgang T1 - The List is the Process: Reliable Pre-Integration Tracking of Commits on Mailing Lists T2 - 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 25-31 May 2019, Montreal, QC, Canada N2 - A considerable corpus of research on software evolution focuses on mining changes in software repositories, but omits their pre-integration history. We present a novel method for tracking this otherwise invisible evolution of software changes on mailing lists by connecting all early revisions of changes to their final version in repositories. Since artefact modifications on mailing lists are communicated by updates to fragments (i.e., patches) only, identifying semantically similar changes is a non-trivial task that our approach solves in a language-independent way. We evaluate our method on high-profile open source software (OSS) projects like the Linux kernel, and validate its high accuracy using an elaborately created ground truth. Our approach can be used to quantify properties of OSS development processes, which is an essential requirement for using OSS in reliable or safety-critical industrial products, where certifiability and conformance to processes are crucial. The high accuracy of our technique allows, to the best of our knowledge, for the first time to quantitatively determine if an open development process effectively aligns with given formal process requirements. KW - commits KW - Data mining KW - Electronic mail KW - Linux KW - mailing lists KW - mining software repositories KW - Open source software KW - patches KW - Software engineering KW - Software maintenance KW - Software reliability Y1 - 2019 U6 - https://doi.org/10.1109/ICSE.2019.00088 SP - 807 EP - 818 PB - IEEE ER -