Open Access

Latest methods for visual counterfactual explanations (VCE) harness the power of deep generative models to synthesize new examples of high-dimensional images of impressive quality. However, it is currently difficult to compare the performance of these VCE methods as the evaluation procedures largely vary and often boil down to visual inspection of individual examples and small scale user studies. In this work, we propose a framework for systematic, quantitative evaluation of the VCE methods and a minimal set of metrics to be used. We use this framework to explore the effects of certain crucial design choices in the latest diffusion-based generative models for VCEs of natural image classification (ImageNet). We conduct a battery of ablation-like experiments, generating thousands of VCEs for a suite of classifiers of various complexity, accuracy and robustness. Our findings suggest multiple directions for future advancements and improvements of VCE methods. By sharing our methodology and our approach to tackle the computational challenges of such a study on a limited hardware setup (including the complete code base), we offer a valuable guidance for researchers in the field fostering consistency and transparency in the assessment of counterfactual explanations.

Deep Learning models are trained to detect humans, cars, and other large objects which are centered in the images. The same models struggle with detecting small and tiny objects because of architecture design decisions that reduce the entropy of small and tiny objects during training. These small and tiny objects are essential for damage identification and maintenance including inspection and documentation of aeroplanes, constructions, offshore structures, and forests. Our work defines the terms tiny and small in context of deep learning models to evaluate possible approaches to resolve the issue of low accuracy in detecting these objects. We analyse the currently applied common datasets Common Objects in Context, ImageNet and Tiny Object Detection Challenge dataset. In addition we compare these datasets and present the differences in terms of object instance size. The COCO dataset, ImageNet dataset and TinyObjects dataset are analysed regarding size categorization and relative object size. The results show the large differences between the size ratios of the three chosen datasets, with ImageNet having by far the largest object instances, COCO being in the middle and TinyObjects having the smallest objects as its name would indicate. Since the objects themselves are larger in terms of total pixel width and height, they therefore make up a bigger percentage on the superordinate picture. Looking at the size categories of the COCO dataset and our extension of the tiny and very small category, the results confirm the size hierarchy of the datasets. With ImageNet having most of its objects in the large category, COCO respectively in the medium category and TinyObjects in the very small category. By taking these results into account, the reader is able to choose a fitting dataset for their tasks.We expect our analysis to help and improve future research in the area of small and tiny object detection.