TY - THES A1 - Köllner, Dario Kevin T1 - Optimization of convolutional neural networks and transformer neural networks using post-training integer quantization N2 - Neural networks are getting more complex and therefore need more computation power. This leads to the optimization which makes it possible to reduce this costs. One method of optimization is the post-training integer quantization. Quantization reduces the data precision from 32-bit floating-point to 8-bit integer. This can be applied on neural networks, which then makes it possible to deploy the neural network on edge platforms that can not deal with high precision values. The goal of this thesis is to evaluate the post-training quantization on integer precision using different networks, platforms and criteria. It leads to the question how the quantization can be performed and what are the differences when using different toolkits. Another aspect is to provide an answer about how different types of neural networks can be quantized. As well as which impact quantization on various properties of the neural network have. To address the research questions an intensive testing on two different kinds of neural networks is performed. The convolutional neural network VGG-16 and the transformer neural network Vision Transformer. Both networks are getting quantized and inference performed using the frameworks TensorFlow and TensorRT. The corresponding platforms are the NVIDIA Tesla V100, NVIDIA RTX 2080 TI, NVIDIA Jetson AGX Xavier and NVIDIA Jetson AGX Orin. The quantization happens in 16-bit floating-point as well as 8-bit integer precision on each platform. Each network, including the quantized ones, are evaluated using the accuracy, file size, average throughput and the power consumption. In this thesis the post-training integer quantization was only successful for the VGG-16 not for the Vision Transformer (ViT) using TensorRT. Nevertheless the 16-bit quantization worked also for the ViT and with 8-bit integer quantization using TensorFlow Lite (TF Lite), but with a near to zero accuracy. The post-training quantization had a significant accuracy drop for 8-bit integer precision where it was for the VGG-16 from around 65 % with 32-bit floating-point precision to approximately 55 percent. The 16-bit floating-point quantized models had nearly no accuracy drop for the VGG-16 and a drop around 73 % to 70 % for the ViT using TensorRT. The average throughput is increasing when using TensorFlow-TensorRT and TensorRT but decreasing when inference is run using the TensorFlow Lite converted models. TensorFlow-TensorRT performed as nearly as good as TensorRT, but did not quantize the models fully. TensorFlow Lite had the lowest throughput, since it is primarily optimized for mobile devices, where TensorRT performed overall the best. It was also possible to reduce the power consumption up to 50 %. Future research in can be the full 8-bit integer quantization for the ViT with own operator support as well as having a higher calibration data amount for both networks. Y1 - 2023 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-36606 CY - Ingolstadt ER -