TY - JOUR A1 - Suffa, Philipp A1 - Kemmler, Samuel A1 - Koestler, Harald A1 - Ruede, Ulrich T1 - Large-scale simulations of fully resolved complex moving geometries with partially saturated cells N2 - We employ the Partially Saturated Cells Method to model the interaction between the fluid flow and solid moving objects as an extension to the conventional lattice Boltzmann method. We introduce an efficient and accurate method for mapping complex moving geometries onto uniform Cartesian grids suitable for massively parallel processing. A validation of the physical accuracy of the solid–fluid coupling and the proposed mapping of complex geometries is presented. The implementation is integrated into the code generation pipeline of the waLBerla framework so that highly optimized kernels for Central Processing Unit (CPU) and Graphical Processing Unit (GPU) architectures become available. We study the node-level performance of the automatically generated solver routines. 71% of the theoretical peak performance can be achieved on CPU nodes and 86% on GPU accelerated nodes. Only a moderate overhead is observed for the processing of the solid–fluid coupling when compared to the fluids simulations without moving objects. Finally, a counter-rotating open rotor is presented as a prototype industrial scenario, resulting in a mesh size involving up to 4.3 × 109 fluid grid cells. For this scenario, excellent parallel efficiency is reported in a strong scaling study on up to 32 768 CPU cores on the LUMI-C supercomputer and on up to 1024 NVIDIA A100 GPUs on the JUWELS Booster system. KW - Computational fluid dynamics KW - Lattice Boltzmann methods KW - Fluid coupling KW - Turbulent flows KW - Supercomputer PY - 2025 DO - https://doi.org/10.1063/5.0268021 SN - 1070-6631 VL - 37 IS - 5 SP - 1 EP - 13 PB - AIP Publishing AN - OPUS4-63188 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER - TY - JOUR A1 - Kemmler, Samuel A1 - Rettinger, Christoph A1 - Rüde, Ulrich A1 - Cuéllar, Pablo A1 - Köstler, Harald T1 - Efficiency and scalability of fully-resolved fluid-particle simulations on heterogeneous CPU-GPU architectures N2 - Current supercomputers often have a heterogeneous architecture using both conventional Central Processing Units (CPUs) and Graphics Processing Units (GPUs). At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on conventional CPUs. Two characteristic model situations involving dense and dilute particle systems are used as benchmark scenarios. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based Eulerian fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale Eulerian flow simulation results in only moderate slowdowns. This is due to the efficiency of the CUDA-aware MPI communication, combined with the use of communication hiding techniques. On 1024 A100 GPUs, an overall parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck, especially when simulating the dense particle system. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers. KW - Discrete element method KW - Hybrid implementation KW - High-performance computing KW - Particulate flow KW - Lattice Boltzmann method PY - 2025 UR - https://nbn-resolving.org/urn:nbn:de:kobv:b43-623958 DO - https://doi.org/10.1177/10943420241313385 SN - 1741-2846 SP - 1 EP - 19 PB - SAGE Publications AN - OPUS4-62395 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER - TY - JOUR A1 - Kemmler, Samuel A1 - Cuéllar, Pablo A1 - Artinov, Antoni A1 - Luu, Li-Hua A1 - Farhat, Abbas A1 - Philippe, Pierre A1 - Rettinger, Christoph A1 - Köstler, Harald T1 - A fully-resolved micromechanical simulation of piping erosion during a suction bucket installation N2 - Granular fluidization phenomena such as piping erosion represent a challenge to the delicate installation process of offshore suction bucket foundations. A detailed analysis of the complex conditions in terms of soil composition, soil state, and foundation installation parameters that may lead to piping can be very demanding, if at all possible, solely by experimental means or using macroscopic continuum-based seabed models. The present paper presents a fully-resolved fluid-coupled micromechanical approach for a three-dimensional numerical simulation of the installation process of a suction bucket using the lattice Boltzmann method and discrete element method. The developed model is validated using well-established benchmarks and calibrated by means of experimental data from physical model tests on relevant scenarios focusing on the local fluidization of fixed embedded suction buckets as well as on the suction-driven installation of unrestrained buckets. The qualitative and quantitative agreement with the experimental data both endorse the proposed methodology and highlight the physical soundness of the obtained results. Thereby, the paper shows that three-dimensional analyses of relevant local scenarios at a real scale with little macromechanical model assumptions are feasible. KW - Micromechanical simulation KW - Fluid–solid coupling KW - Piping erosion KW - Suction bucket foundation KW - Offshore wind support structure KW - High-performance computing PY - 2025 UR - https://nbn-resolving.org/urn:nbn:de:kobv:b43-633790 DO - https://doi.org/10.1016/j.compgeo.2025.107375 SN - 0266-352X VL - 186 SP - 1 EP - 17 PB - Elsevier B.V. AN - OPUS4-63379 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER - TY - JOUR A1 - Faghih-Naini, Sara A1 - Kuckuk, Sebastian A1 - Zint, Daniel A1 - Kemmler, Samuel A1 - Köstler, Harald A1 - Aizinger, Vadym T1 - Discontinuous Galerkin method for the shallow water equations on complex domains using masked block-structured grids N2 - We evaluate masked block-structured grids for ocean domains which allow to represent small-scale geometric features without resorting to very small blocks or excessive mesh resolution. The considered approach aims to combine the geometric flexibility of unstructured meshes with the computational efficiency of stencil-based discretizations and is implemented and tested in a quadrature-free discontinuous Galerkin shallow water solver. We investigate the accuracy and the computational performance of the scheme on a range of realistic ocean domains meshed with blocks of different size and provide some comparisons to results obtained on unmasked block-structured grids and unstructured meshes. KW - Water Science and Technology PY - 2023 DO - https://doi.org/10.1016/j.advwatres.2023.104584 VL - 182 SP - 1 EP - 11 PB - Elsevier AN - OPUS4-58884 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER -