Open Access

Current supercomputers often have a heterogeneous architecture using both conventional Central Processing Units (CPUs) and Graphics Processing Units (GPUs). At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on conventional CPUs. Two characteristic model situations involving dense and dilute particle systems are used as benchmark scenarios. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based Eulerian fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale Eulerian flow simulation results in only moderate slowdowns. This is due to the efficiency of the CUDA-aware MPI communication, combined with the use of communication hiding techniques. On 1024 A100 GPUs, an overall parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck, especially when simulating the dense particle system. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers.

Fully-resolved micromechanical simulations coupling the Lattice Boltzmann Method (LBM) with the Discrete Element Method (DEM) provide high-fidelity insights into granular fluidization. However, the substantial computational demands of such simulations require efficient implementations on supercomputing architectures. This work presents a comprehensive performance analysis of a fully-resolved LBM-DEM model to study granular fluidization, i.e., piping erosion, during the installation of an offshore caisson foundation. The performance evaluation focuses on real-world workloads rather than simplified benchmark problems to obtain realistic performance insights. The study considers a diverse range of state-of-the-art HPC hardware architectures, namely the LUMI and MareNostrum 5 EuroHPC supercomputers, both CPU and GPU partitions. The results demonstrate that GPU-based systems generally outperform the CPU-based systems. Strong and weak scaling analyses were conducted with up to 512 nodes, and parallel efficiencies reached up to 92%. Nonetheless, the results also indicate that atomic add operations on GPUs can become a bottleneck for the parallel efficiency at large scales. Moreover, the study reveals a close link between physical variations of the setup and significant scaling implications, underscoring the need to consider physical model characteristics in scaling assessments.