46E35 Sobolev spaces and other spaces of \smooth" functions, embedding theorems, trace theorems
The present cumulative thesis is concerned with theoretical properties of (deep) fully connected neural networks. It is based on five publications and can be divided into two parts.
In the first part, we establish bounds on the ℓᵖ-Lipschitz constants of deep ReLU neural networks at initialization, that is, networks with random weights and biases. More precisely, we assume that the weights and biases are drawn according to a generalization of the popular He initialization. In the zero-bias case, we prove upper and lower high-probability bounds for wide networks that differ only by a logarithmic factor in the network width and a polynomial factor in the network depth. We then extend the analysis to symmetric bias distributions, for which similar bounds are established.
The second part of the thesis investigates the approximation capabilities of complex-valued neural networks (CVNNs), that is, networks with complex-valued weights and biases and with activation functions mapping from ℂ to ℂ. We prove sharp quantitative bounds for the worst-case approximation error when approximating Cʳ-functions by shallow CVNNs, both under continuous weight selection and without any assumptions on the weight selection. As part of this analysis, we generalize well-known results on the approximation properties of univariate ridge functions to the multivariate setting. Moreover, we study the universality of deep, narrow CVNNs, that is, classes of CVNNs with restricted width but arbitrary depth.
The study of nonlocal energy-based models has proliferated in the last decades
due to their relevance in applications and the novel mathematical challenges that they provide. Within this context, the thesis addresses a wide variety of aspects concerning variational problems involving nonlocal gradients. These have been proposed for modeling the emergence of cracks and cavitations in materials, and can be used in image denoising applications to preserve sharp features. The contributions of the thesis are concerned with several families of nonlocal gradients and can be divided into the following overarching themes:
(i) Development of nonlocal Sobolev spaces;
(ii) existence theories for minimizers of nonlocal integral functionals;
(iii) asymptotic analysis of parameter-dependent problems.
In the context of (i), we extend and unify the results established for the Riesz and finite-horizon fractional gradient by considering more general nonlocal gradients with radial kernels. Precisely, based on a delicate analysis of the Fourier symbol of the nonlocal gradient, we present minimal assumptions on the kernel function such that Poincaré inequalities and compact embeddings hold. Moreover, we provide sharp embeddings into Orlicz spaces and spaces with prescribed modulus of continuity, that refine the fractional Sobolev and Morrey inequalities.
With these tools at hand, the goal in (ii) is to establish rigorous existence results for minimizers of integral functionals depending on nonlocal gradients. We develop a translation mechanism that connects the nonlocal gradients with their local counterpart and use this to characterize the lower semicontinuity of the nonlocal functionals in terms of the classical notion of quasiconvexity. Based on this, we establish the existence of minimizers via the direct method under Dirichlet or novel Neumann-type boundary conditions. The latter conditions necessitate the study of functions with zero nonlocal gradient, which surprisingly constitute an infinite-dimensional vector space, and we employ recent results regarding pseudo-differential boundary-value problems to characterize them. We also tackle linear growth functionals, which lack the usual coercivity requirement, and provide an explicit formula for their relaxation in the space of functions with bounded fractional variation. This reveals insights about the behavior of minimizing sequences and is based on a careful analysis of the concentration effects of the fractional gradient by using tools from Young measure theory.
Building upon the existence results, the focus in (iii) lies on studying the dependence of the minimizers on parameters in the problem, specifically, the fractional parameter s and the interaction range δ. We establish using Gamma-convergence that the minimizers depend continuously on s and converge to the local solutions as s → 1. In parallel, we prove that localization also occurs in the vanishing horizon limit δ → 0, while the diverging horizon regime δ → ∞ leads to the models based on the Riesz fractional gradient. A crucial ingredient for the proofs of these results are uniform compactness statements, which are established by studying the dependence of the Fourier symbols on the respective parameters. In terms of applications, we develop an abstract framework for the well-posedness of bi-level optimization problems for parameter learning in image denoising. The results of the thesis show that the models involving nonlocal gradients fit into this framework, where the fractional parameter is tuned in order to obtain the optimal degree of smoothing.
