Refine
Year of publication
Document Type
- Article (22) (remove)
Keywords
- DEAL (2)
- DFG Publikationskosten (2)
- HSD Publikationsfonds (2)
- ADAS (1)
- AI performance metrics (1)
- Artificial Intelligence (1)
- Computer Vision (1)
- DOAJ (1)
- Data integrity (1)
- Data quality (1)
Department/institution
Spatial precision and recall indices to assess the performance of instance segmentation algorithms
(2022)
Refractive power measurements serve as the primary quality standard in the automotive glazing industry. In the light of autonomous driving new optical metrics are becoming more and more popular for specifying optical quality requirements for the windshield. Nevertheless, the link between those quantities and the refractive power needs to be established in order to ensure a holistic requirement profile for the windshield. As a consequence, traceable high-resolution refractive power measurements are still required for the glass quality assessment. Standard measurement systems using Moiré patterns for refractive power monitoring in the automotive industry are highly resolution limited, wherefore they are insufficient for evaluating the camera window area. Consequently, there is a need for more sophisticated refractive power measurement systems that provide a higher spatial resolution. In addition, a calibration procedure has to be developed in order to guarantee for comparability of the measurement results. For increasing the resolution, a measurement setup based on an auto-correlation algorithm is tested in this paper. Furthermore, a calibration procedure is established by using a single reference lens with a nominal refractive power of 100 km-1. For the calibration of the entire measurement range of the system, the lens is tilted by an inclination angle orthogonal to the optical axis. The effective refractive power is then given by the Kerkhof model. By adopting the measurement and calibration procedure presented in this paper, glass suppliers in the automotive industry will be able to detect relevant manufacturing defects within the camera window area more accurately paving the way for a holistic quality assurance of the windshield for future advanced driver-assistance system (ADAS) functionalities. Concurrently, the traceability of the measurement results is ensured by establishing a calibration chain based on a single reference lens, which is traced back to international standards.
Abstract The modulation-transfer function (MTF) is a fundamental optical metric to measure the optical quality of an imaging system. In the automotive industry it is used to qualify camera systems for ADAS/AD. Each modern ADAS/AD system includes evaluation algorithms for environment perception and decision making that are based on AI/ML methods and neural networks. The performance of these AI algorithms is measured by established metrics like Average Precision (AP) or precision-recall-curves. In this article we research the robustness of the link between the optical quality metric and the AI performance metric. A series of numerical experiments were performed with object detection and instance segmentation algorithms (cars, pedestrians) evaluated on image databases with varying optical quality. We demonstrate with these that for strong optical aberrations a distinct performance loss is apparent, but that for subtle optical quality differences – as might arise during production tolerances – this link does not exhibit a satisfactory correlation. This calls into question how reliable the current industry practice is where a produced camera is tested end-of-line (EOL) with the MTF, and fixed MTF thresholds are used to qualify the performance of the camera-under-test.
Artificial intelligence methods based on machine learning or artificial neural networks have become indispensable in camera-based driver assistance systems, and also represent an essential building block for future autonomous driving. However, the great successes of these evaluation methods in environment perception and also driving planning are accompanied by equally great challenges in the validation and verification of these systems. One of the essential aspects for this is the required guaranteed safety of the functions under mass production conditions of the vehicles. This article explains this point of view using a detailed example from the field of camera-based driver assistance systems: the determination of inspection limits at the end of the production line. The camera is one of the most important sensor modalities for vehicle environment sensing and as such, the quality of the camera systems plays a key role in the safety argumentation of the overall system. Several illustrative application examples (role of simulations, calibration, influence of the windshield) will be presented. The basic ideas presented can be well transferred to the other sensor modalities (lidar, radar, ToF, etc.). The investigations/evidence show that doubts are allowed whether or how fast autonomous driving on level L4/5 will take hold as robotaxis or – even more challenging – in private ownership on a larger scale.
Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design
(2021)
Most modern commodity imaging systems we use directly for photography‐-or indirectly rely on for downstream applications‐-employ optical systems of multiple lenses that must balance deviations from perfect optics, manufacturing constraints, tolerances, cost, and footprint. Although optical designs often have complex interactions with downstream image processing or analysis tasks, today’s compound optics are designed in isolation from these interactions. Existing optical design tools aim to minimize optical aberrations, such as deviations from Gauss’ linear model of optics, instead of application-specific losses, precluding joint optimization with hardware image signal processing (ISP) and highly parameterized neural network processing. In this article, we propose an optimization method for compound optics that lifts these limitations. We optimize entire lens systems jointly with hardware and software image processing pipelines, downstream neural network processing, and application-specific end-to-end losses. To this end, we propose a learned, differentiable forward model for compound optics and an alternating proximal optimization method that handles function compositions with highly varying parameter dimensions for optics, hardware ISP, and neural nets. Our method integrates seamlessly atop existing optical design tools, such as Zemax. We can thus assess our method across many camera system designs and end-to-end applications. We validate our approach in an automotive camera optics setting‐-together with hardware ISP post processing and detection‐-outperforming classical optics designs for automotive object detection and traffic light state detection. For human viewing tasks, we optimize optics and processing pipelines for dynamic outdoor scenarios and dynamic low-light imaging. We outperform existing compartmentalized design or fine-tuning methods qualitatively and quantitatively, across all domain-specific applications tested.
We present a novel metric Spatial Recall Index to assess the performance of machine-learning (ML) algorithms for automotive applications, focusing on where in the image which performance occurs. Typical metrics like intersection-over-union (IoU), precisionrecallcurves or average precision (AP) quantify the performance over a whole database of images, neglecting spatial performance variations. But as the optics of camera systems are spatially variable over the field of view, the performance of ML-based algorithms is also a function of space, which we show in simulation: A realistic objective lens based on a Cooke-triplet that exhibits typical optical aberrations like astigmatism and chromatic aberration, all variable over field, is modeled. The model is then applied to a subset of the BDD100k dataset with spatially-varying kernels. We then quantify local changes in the performance of the pre-trained Mask R-CNN algorithm. Our examples demonstrate the spatial dependence of the performance of ML-based algorithms from the optical quality over field, highlighting the need to take the spatial dimension into account when training ML-based algorithms, especially when looking forward to autonomous driving applications.