TY - JOUR A1 - Kemmler, Samuel A1 - Rettinger, Christoph A1 - Rüde, Ulrich A1 - Cuéllar, Pablo A1 - Köstler, Harald T1 - Efficiency and scalability of fully-resolved fluid-particle simulations on heterogeneous CPU-GPU architectures N2 - Current supercomputers often have a heterogeneous architecture using both conventional Central Processing Units (CPUs) and Graphics Processing Units (GPUs). At the same time, numerical simulation tasks frequently involve multiphysics scenarios whose components run on different hardware due to multiple reasons, e.g., architectural requirements, pragmatism, etc. This leads naturally to a software design where different simulation modules are mapped to different subsystems of the heterogeneous architecture. We present a detailed performance analysis for such a hybrid four-way coupled simulation of a fully resolved particle-laden flow. The Eulerian representation of the flow utilizes GPUs, while the Lagrangian model for the particles runs on conventional CPUs. Two characteristic model situations involving dense and dilute particle systems are used as benchmark scenarios. First, a roofline model is employed to predict the node level performance and to show that the lattice-Boltzmann-based Eulerian fluid simulation reaches very good performance on a single GPU. Furthermore, the GPU-GPU communication for a large-scale Eulerian flow simulation results in only moderate slowdowns. This is due to the efficiency of the CUDA-aware MPI communication, combined with the use of communication hiding techniques. On 1024 A100 GPUs, an overall parallel efficiency of up to 71% is achieved. While the flow simulation has good performance characteristics, the integration of the stiff Lagrangian particle system requires frequent CPU-CPU communications that can become a bottleneck, especially when simulating the dense particle system. Additionally, special attention is paid to the CPU-GPU communication overhead since this is essential for coupling the particles to the flow simulation. However, thanks to our problem-aware co-partitioning, the CPU-GPU communication overhead is found to be negligible. As a lesson learned from this development, four criteria are postulated that a hybrid implementation must meet for the efficient use of heterogeneous supercomputers. KW - Discrete element method KW - Hybrid implementation KW - High-performance computing KW - Particulate flow KW - Lattice Boltzmann method PY - 2025 UR - https://nbn-resolving.org/urn:nbn:de:kobv:b43-623958 DO - https://doi.org/10.1177/10943420241313385 SN - 1741-2846 SP - 1 EP - 19 PB - SAGE Publications AN - OPUS4-62395 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER - TY - JOUR A1 - Benseghier, Z. A1 - Cuéllar, Pablo A1 - Luu, L.-H. A1 - Delenne, J.-Y. A1 - Bonelli, S. A1 - Philippe, P. T1 - Relevance of Free Jet Model for Soil Erosion by Impinging Jets N2 - The surface erosion of soil samples caused by an impinging jet can be analyzed using the jet erosion test (JET), a standard experimental test to characterize the erosion resistance of soils. This paper specifically addresses the flow characteristics of a laminar impinging jet over the irregular surface of granular beds to discuss the pertinence and relevance of commonly used empirical estimations based on a selfsimilar model of a free jet. The JET is here investigated at the microscale with a coupled fluid-particle flow numerical odel featuring the lattice Boltzmann method (LBM) for the fluid phase combined with the discrete element method (DEM) for the mechanical behavior of the solid particles. The hydrodynamics of a laminar plane free jet are confronted with the results from a parametric study of jet impingement, both on solid smooth and fixed granular surfaces, that take into account variations in particle size, distance from jet origin, and jet Reynolds number. The flow characteristics at the bed surface are here quantified, including the maximal values in tangential velocity and wall shear stress, which can be regarded as the major cause of particle detachments under hydrodynamic solicitation. It is shown that the maximal velocity at the impinged surface can be described by the free jet self-similar model, provided that a simple empirical coefficient is introduced. Further, an expression is proposed for the maximal shear stress in laminar conditions, including a Blasius-like friction coefficient that is inversely proportional to the square root of the jet Reynolds number. To conclude, finally, the JET erosion of different cohesionless granular samples is analyzed, confirming that the threshold condition at the onset of granular motion is consistent with the Shields diagram and in close agreement with previous experimental results. KW - Lattice Boltzmann method KW - Soil erosion KW - Discrete element method KW - Laminar flow KW - Jet impingement PY - 2020 DO - https://doi.org/10.1061/(ASCE)HY.1943-7900.0001652 VL - 146 IS - 1 SP - 04019047 PB - ASCE AN - OPUS4-49491 LA - eng AD - Bundesanstalt fuer Materialforschung und -pruefung (BAM), Berlin, Germany ER -