TY - CHAP A1 - Fajerski, J. A1 - Noack, Matthias A1 - Reinefeld, Alexander A1 - Schintke, Florian A1 - Schütt, Thorsten A1 - Steinke, Thomas T1 - Fast In-Memory Checkpointing with POSIX API for Legacy Exascale-Applications T2 - SPPEXA Symposium 2016 Y1 - 2016 U6 - https://doi.org/10.1007/978-3-319-40528-5_19 ER - TY - GEN A1 - Kramer, Tobias A1 - Noack, Matthias A1 - Baum, Daniel A1 - Hege, Hans-Christian A1 - Heller, Eric J. T1 - Dust and gas emission from cometary nuclei: the case of comet 67P/Churyumov-Gerasimenko N2 - Comets display with decreasing solar distance an increased emission of gas and dust particles, leading to the formation of the coma and tail. Spacecraft missions provide insight in the temporal and spatial variations of the dust and gas sources located on the cometary nucleus. For the case of comet 67P/Churyumov-Gerasimenko (67P/C-G), the long-term obser- vations from the Rosetta mission point to a homogeneous dust emission across the entire illuminated surface. Despite the homogeneous initial dis- tribution, a collimation in jet-like structures becomes visible. We propose that this observation is linked directly to the complex shape of the nucleus and projects concave topographical features into the dust coma. To test this hypothesis, we put forward a gas-dust description of 67P/C-G, where gravitational and gas forces are accurately determined from the surface mesh and the rotation of the nucleus is fully incorporated. The emerging jet-like structures persist for a wide range of gas-dust interactions and show a dust velocity dependent bending. T3 - ZIB-Report - 17-78 Y1 - 2017 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-66338 SN - 1438-0064 ER - TY - CHAP A1 - Noack, Matthias T1 - OpenCL in Scientific High Performance Computing: The Good, the Bad, and the Ugly T2 - Proceedings of the 5th International Workshop on OpenCL N2 - For writing a new scientific application, portability across existing and future hardware should be the major design goal, as there is a multitude of different compute devices, and programme codes typically outlive systems by far. Unlike other programming models that address parallelism or heterogeneity, OpenCL does provide practical portability across a wide range of HPC-relevant architectures. Other than that, it has a range of further advantages like being a library-only implementation, and using runtime kernel-compilation. We present experiences with utilising OpenCL alongside C++, MPI, and CMake in two real-world scientific codes. Our targets are a Cray XC40 supercomputer with multi- and many-core (Xeon Phi) CPUs, as well as multiple smaller systems with Nvidia and AMD GPUs. We shed light on practical issues arising in such a scenario, like the interaction between OpenCL and MPI, discuss solutions, and point out current limitations of OpenCL in the domain of scientific HPC from an application developer's and user's point of view. Y1 - 2017 U6 - https://doi.org/10.1145/3078155.3078170 SP - 12:1 EP - 12:3 ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias A1 - Baum, Daniel A1 - Hege, Hans-Christian A1 - Heller, Eric J. T1 - Dust and gas emission from cometary nuclei: the case of comet 67P/Churyumov-Gerasimenko JF - Advances in Physics: X N2 - Comets display with decreasing solar distance an increased emission of gas and dust particles, leading to the formation of the coma and tail. Spacecraft missions provide insight in the temporal and spatial variations of the dust and gas sources located on the cometary nucleus. For the case of comet 67P/Churyumov-Gerasimenko (67P/C-G), the long-term obser- vations from the Rosetta mission point to a homogeneous dust emission across the entire illuminated surface. Despite the homogeneous initial dis- tribution, a collimation in jet-like structures becomes visible. We propose that this observation is linked directly to the complex shape of the nucleus and projects concave topographical features into the dust coma. To test this hypothesis, we put forward a gas-dust description of 67P/C-G, where gravitational and gas forces are accurately determined from the surface mesh and the rotation of the nucleus is fully incorporated. The emerging jet-like structures persist for a wide range of gas-dust interactions and show a dust velocity dependent bending. Y1 - 2018 U6 - https://doi.org/10.1080/23746149.2017.1404436 VL - 3 IS - 1 SP - 1404436 ER - TY - GEN A1 - Kramer, Tobias A1 - Noack, Matthias T1 - On the origin of inner coma structures observed by Rosetta during a diurnal rotation of comet 67P/Churyumov-Gerasimenko. N2 - The Rosetta probe around comet 67P/Churyumov-Gerasimenko (67P) reveals an anisotropic dust distribution of the inner coma with jet-like structures. The physical processes leading to jet formation are under debate, with most models for cometary activity focusing on localised emission sources, such as cliffs or terraced regions. Here we suggest, by correlating high-resolution simulations of the dust environment around 67P with observations, that the anisotropy and the background dust density of 67P originate from dust released across the entire sunlit surface of the nucleus rather than from few isolated sources. We trace back trajectories from coma regions with high local dust density in space to the non-spherical nucleus and identify two mechanisms of jet formation: areas with local concavity in either two dimensions or only one. Pits and craters are examples of the first case, the neck region of the bilobed nucleus of 67P for the latter one. The conjunction of multiple sources in addition to dust released from all other sunlit areas results in a high correlation coefficient (∼0.8) of the predictions with observations during a complete diurnal rotation period of 67P. T3 - ZIB-Report - 16-26 KW - comet KW - 67p Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-59225 SN - 1438-0064 ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias A1 - Baum, Daniel A1 - Hege, Hans-Christian A1 - Heller, Eric J. T1 - Homogeneous dust emission and jet structure near active cometary nuclei: the case of 67P/Churyumov-Gerasimenko N2 - We compute trajectories of dust grains starting from a homogeneous surface activity-profile on a irregularly shaped cometary nucleus. Despite the initially homogeneous dust distribution a collimation in jet-like structures becomes visible. The fine structure is caused by concave topographical features with similar bundles of normal vectors. The model incorporates accurately determined gravitational forces, rotation of the nucleus, and gas-dust interaction. Jet-like dust structures are obtained for a wide range of gas-dust interactions. For the comet 67P/Churyumov-Gerasimenko, we derive the global dust distribution around the nucleus and find several areas of agreement between the homogeneous dust emission model and the Rosetta observation of dust jets, including velocity-dependent bending of trajectories. Y1 - 2015 ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias T1 - On the origin of inner coma structures observed by Rosetta during a diurnal rotation of comet 67P/Churyumov-Gerasimenko JF - The Astrophysical Journal Letters N2 - The Rosetta probe around comet 67P/Churyumov–Gerasimenko (67P) reveals an anisotropic dust distribution of the inner coma with jet-like structures. The physical processes leading to jet formation are under debate, with most models for cometary activity focusing on localized emission sources, such as cliffs or terraced regions. Here we suggest, by correlating high-resolution simulations of the dust environment around 67P with observations, that the anisotropy and the background dust density of 67P originate from dust released across the entire sunlit surface of the nucleus rather than from few isolated sources. We trace back trajectories from coma regions with high local dust density in space to the non-spherical nucleus and identify two mechanisms of jet formation: areas with local concavity in either two dimensions or only one. Pits and craters are examples of the first case; the neck region of the bi-lobed nucleus of 67P is an example of the latter case. The conjunction of multiple sources, in addition to dust released from all other sunlit areas, results in a high correlation coefficient (~0.8) of the predictions with observations during a complete diurnal rotation period of 67P. Y1 - 2016 U6 - https://doi.org/10.3847/2041-8205/823/1/L11 VL - 823 IS - 1 SP - L11 ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias T1 - Prevailing Dust-transport Directions on Comet 67P/Churyumov–Gerasimenko JF - The Astrophysical Journal Letters N2 - Dust transport and deposition behind larger boulders on the comet 67P/Churyumov–Gerasimenko (67P/C–G) have been observed by the Rosetta mission. We present a mechanism for dust-transport vectors based on a homogeneous surface activity model incorporating in detail the topography of 67P/C–G. The combination of gravitation, gas drag, and Coriolis force leads to specific dust transfer pathways, which for higher dust velocities fuel the near-nucleus coma. By distributing dust sources homogeneously across the whole cometary surface, we derive a global dust-transport map of 67P/C–G. The transport vectors are in agreement with the reported wind-tail directions in the Philae descent area. Y1 - 2015 U6 - https://doi.org/10.1088/2041-8205/813/2/L33 VL - 813 IS - 2 SP - L33 ER - TY - CHAP A1 - Noack, Matthias A1 - Focht, Erich A1 - Steinke, Thomas T1 - Heterogeneous Active Messages for Offloading on the NEC SX-Aurora TSUBASA T2 - 2019 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), Heterogeneity in Computing Workshop (HCW 2019) N2 - The NEC SX-Aurora TSUBASA is a new generation of vector processing architectures that combines a standard Intel Xeon host with the newly developed NEC Vector Engine co-processor cards. One way to use these co-processors is offloading suitable parts of the program from the host to the Vector Engines. Currently, the only vendor-provided offloading solutions are the low-level Vector Engine Offloading (VEO) library, and a builtin reverse-offloading mechanism named VHcall. In this work, we extend the portable Heterogeneous Active Messages (HAM) based HAM-Offload framework with support for the NEC SX-Aurora TSUBASA. Therefore, we design, implement, and evaluate two messaging protocols aimed at minimising offloading cost. This sheds some light on how to achieve fast communication between host CPU and the Vector Engines of the NEC SX-Aurora TSUBASA. Compared with VEO, the DMA-based protocol reduces offloading overhead by a factor of 13×. The resulting framework enables users to write portable offload applications with low overhead, that do neither require a language extension like OpenMP, nor a special language like OpenCL. Existing HAM-Offload applications are now ready to run on the NEC SX-Aurora TSUBASA. Y1 - 2019 ER - TY - CHAP A1 - Noack, Matthias T1 - Heterogeneous Active Messages (HAM) — Implementing Lightweight Remote Procedure Calls in C++ T2 - Proceedings of the 5th International Workshop on OpenCL, The Distributed & Heterogeneous Programming in C/C++ (DHPCC++ 2019) Conference N2 - We present HAM (Heterogeneous Active Messages), a C++-only active messaging solution for heterogeneous distributed systems.Combined with a communication protocol, HAM can be used as a generic Remote Procedure Call (RPC) mechanism. It has been used in HAM-Offload to implement a low-overhead offloading framework for inter- and intra-node offloading between different architectures including accelerators like the Intel Xeon Phi x100 series and the NEC SX-Aurora TSUBASA Vector Engine. HAM uses template meta-programming to implicitly generate active message types and their corresponding handler functions. Heterogeneity is enabled by providing an efficient address translation mechanism between the individual handler code addresses of processes running different binaries on different architectures, as well a hooks to inject serialisation and deserialisation code on a per-type basis. Implementing such a solution in modern C++ sheds some light on the shortcomings and grey areas of the C++ standard when it comes to distributed and heterogeneous environments. Y1 - 2019 U6 - https://doi.org/10.1145/3318170.3318195 ER - TY - CHAP A1 - Noack, Matthias A1 - Reinefeld, Alexander A1 - Kramer, Tobias A1 - Steinke, Thomas T1 - DM-HEOM: A Portable and Scalable Solver-Framework for the Hierarchical Equations of Motion T2 - 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW), 19th IEEE Int. Workshop on Parallel and Distributed Scientific and Engineering Computing (PDSEC 2018) N2 - Computing the Hierarchical Equations of Motion (HEOM) is by itself a challenging problem, and so is writing portable production code that runs efficiently on a variety of architectures while scaling from PCs to supercomputers. We combined both challenges to push the boundaries of simulating quantum systems, and to evaluate and improve methodologies for scientific software engineering. Our contributions are threefold: We present the first distributed memory implementation of the HEOM method (DM-HEOM), we describe an interdisciplinary development workflow, and we provide guidelines and experiences for designing distributed, performance-portable HPC applications with MPI-3, OpenCL and other state-of-the-art programming models. We evaluated the resulting code on multi- and many-core CPUs as well as GPUs, and demonstrate scalability on a Cray XC40 supercomputer for the PS I molecular light harvesting complex. Y1 - 2018 SN - 978-1-5386-5555-9 U6 - https://doi.org/10.1109/IPDPSW.2018.00149 N1 - Best Paper Award SP - 947 EP - 956 ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias A1 - Reinefeld, Alexander A1 - Rodríguez, Mirta A1 - Zelinskyi, Yaroslav T1 - Efficient calculation of open quantum system dynamics and time-resolved spectroscopy with Distributed Memory HEOM (DM-HEOM) JF - Journal of Computational Chemistry N2 - Time- and frequency resolved optical signals provide insights into the properties of light harvesting molecular complexes, including excitation energies, dipole strengths and orientations, as well as in the exciton energy flow through the complex. The hierarchical equations of motion (HEOM) provide a unifying theory, which allows one to study the combined effects of system-environment dissipation and non-Markovian memory without making restrictive assumptions about weak or strong couplings or separability of vibrational and electronic degrees of freedom. With increasing system size the exact solution of the open quantum system dynamics requires memory and compute resources beyond a single compute node. To overcome this barrier, we developed a scalable variant of HEOM. Our distributed memory HEOM, DM-HEOM, is a universal tool for open quantum system dynamics. It is used to accurately compute all experimentally accessible time- and frequency resolved processes in light harvesting molecular complexes with arbitrary system-environment couplings for a wide range of temperatures and complex sizes. Y1 - 2018 U6 - https://doi.org/doi:10.1002/jcc.25354 VL - 39 IS - 22 SP - 1779 EP - 1794 PB - Wiley Periodicals, Inc. ER - TY - JOUR A1 - Kramer, Tobias A1 - Noack, Matthias A1 - Reimers, Jeffrey R. A1 - Reinefeld, Alexander A1 - Rodríguez, Mirta A1 - Yin, Shiwei T1 - Energy flow in the Photosystem I supercomplex: comparison of approximative theories with DM-HEOM JF - Chemical Physics N2 - We analyze the exciton dynamics in PhotosystemI from Thermosynechococcus elongatus using the distributed memory implementation of the hierarchical equation of motion (DM-HEOM) for the 96 Chlorophylls in the monomeric unit. The exciton-system parameters are taken from a first principles calculation. A comparison of the exact results with Foerster rates and Markovian approximations allows one to validate the exciton transfer times within the complex and to identify deviations from approximative theories. We show the optical absorption, linear, and circular dichroism spectra obtained with DM-HEOM and compare them to experimental results. Y1 - 2018 U6 - https://doi.org/10.1016/j.chemphys.2018.05.028 VL - 515 SP - 262 EP - 271 PB - Elsevier B.V. ER - TY - CHAP A1 - Baumann, Tobias A1 - Noack, Matthias A1 - Steinke, Thomas T1 - Performance Evaluation and Improvements of the PoCL Open-Source OpenCL Implementation on Intel CPUs T2 - IWOCL'21: International Workshop on OpenCL N2 - The Portable Computing Language (PoCL) is a vendor independent open-source OpenCL implementation that aims to support a variety of compute devices in a single platform. Evaluating PoCL versus the Intel OpenCL implementation reveals significant performance drawbacks of PoCL on Intel CPUs – which run 92 % of the TOP500 list. Using a selection of benchmarks, we identify and analyse performance issues in PoCL with a focus on scheduling and vectorisation. We propose a new CPU device-driver based on Intel Threading Building Blocks (TBB), and evaluate LLVM with respect to automatic compiler vectorisation across work-items in PoCL. Using the TBB driver, it is possible to narrow the gap to Intel OpenCL and even outperform it by a factor of up to 1.3× in our proxy application benchmark with a manual vectorisation strategy. Y1 - 2021 U6 - https://doi.org/10.1145/3456669.3456698 ER - TY - GEN A1 - Noack, Matthias A1 - Wende, Florian A1 - Zitzlsberger, Georg A1 - Klemm, Michael A1 - Steinke, Thomas T1 - KART – A Runtime Compilation Library for Improving HPC Application Performance N2 - The effectiveness of ahead-of-time compiler optimization heavily depends on the amount of available information at compile time. Input-specific information that is only available at runtime cannot be used, although it often determines loop counts, branching predicates and paths, as well as memory-access patterns. It can also be crucial for generating efficient SIMD-vectorized code. This is especially relevant for the many-core architectures paving the way to exascale computing, which are more sensitive to code-optimization. We explore the design-space for using input-specific information at compile-time and present KART, a C++ library solution that allows developers to compile, link, and execute code (e.g., C, C++ , Fortran) at application runtime. Besides mere runtime compilation of performance-critical code, KART can be used to instantiate the same code multiple times using different inputs, compilers, and options. Other techniques like auto-tuning and code-generation can be integrated into a KART-enabled application instead of being scripted around it. We evaluate runtimes and compilation costs for different synthetic kernels, and show the effectiveness for two real-world applications, HEOM and a WSM6 proxy. T3 - ZIB-Report - 16-48 Y1 - 2016 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-60730 SN - 1438-0064 ER - TY - CHAP A1 - Noack, Matthias A1 - Wende, Florian A1 - Steinke, Thomas A1 - Cordes, Frank T1 - A Unified Programming Model for Intra- and Inter-Node Offloading on Xeon Phi Clusters T2 - SC '14: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. SC14, November 16-21, 2014, New Orleans, Louisiana, USA N2 - Standard offload programming models for the Xeon Phi, e.g. Intel LEO and OpenMP 4.0, are restricted to a single compute node and hence a limited number of coprocessors. Scaling applications across a Xeon Phi cluster/supercomputer thus requires hybrid programming approaches, usually MPI+X. In this work, we present a framework based on heterogeneous active messages (HAM-Offload) that provides the means to offload work to local and remote (co)processors using a unified offload API. Since HAM-Offload provides similar primitives as current local offload frameworks, existing applications can be easily ported to overcome the single-node limitation while keeping the convenient offload programming model. We demonstrate the effectiveness of the framework by using it to enable a real-world application from the field of molecular dynamics to use multiple local and remote Xeon Phis. The evaluation shows good scaling behavior. Compared with LEO, performance is equal for large offloads and significantly better for small offloads. Y1 - 2014 UR - http://dl.acm.org/citation.cfm?id=2683616 U6 - https://doi.org/10.1109/SC.2014.22 ER - TY - GEN A1 - Noack, Matthias T1 - HAM - Heterogenous Active Messages for Efficient Offloading on the Intel Xeon Phi N2 - The applicability of accelerators is limited by the attainable speed-up for the offloaded computations and by the offloading overheads. While GPU programming models like CUDA and OpenCL only allow to optimise the application code and its speed-up, the available low-level APIs for the Intel Xeon Phi provide opportunity to address the overheads, too. This work presents an Heterogeneous Active Message (HAM) layer that minimises software overheads for offloading on Intel’s Xeon Phi. It provides the basis for an offload API with similar semantics as the Intel Language Extensions for Offload (LEO). In contrast to LEO, HAM works within the C++ language and needs no additional compiler support. We evaluated HAM on top of SCIF and MPI as communication backends. While the SCIF backend offers the best performance, the MPI backend allows for inter-node offloads which are not possible with other offload solutions. Benchmark results show that the cost for offloading a function call can be decreased by a factor up to 18 compared with LEO. T3 - ZIB-Report - 14-23 KW - Intel Xeon Phi KW - MIC KW - Active message KW - Offloading KW - SCIF KW - MPI KW - LEO KW - HAM Y1 - 2014 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-50610 SN - 1438-0064 ER - TY - CHAP A1 - Noack, Matthias A1 - Wende, Florian A1 - Zitzlsberger, Georg A1 - Klemm, Michael A1 - Steinke, Thomas T1 - KART - A Runtime Compilation Library for Improving HPC Application Performance T2 - High Performance Computing: ISC High Performance 2017 International Workshops, DRBSD, ExaComm, HCPM, HPC-IODC, IWOPH, IXPUG, P^3MA, VHPC, Visualization at Scale, WOPSSS, Frankfurt, Germany, June 18-22, 2017, Revised Selected Papers N2 - The effectiveness of ahead-of-time compiler optimization heavily depends on the amount of available information at compile time. Input-specific information that is only available at runtime cannot be used, although it often determines loop counts, branching predicates and paths, as well as memory-access patterns. It can also be crucial for generating efficient SIMD-vectorized code. This is especially relevant for the many-core architectures paving the way to exascale computing, which are more sensitive to code-optimization. We explore the design-space for using input-specific information at compile-time and present KART, a C++ library solution that allows developers tocompile, link, and execute code (e.g., C, C++ , Fortran) at application runtime. Besides mere runtime compilation of performance-critical code, KART can be used to instantiate the same code multiple times using different inputs, compilers, and options. Other techniques like auto-tuning and code-generation can be integrated into a KART-enabled application instead of being scripted around it. We evaluate runtimes and compilation costs for different synthetic kernels, and show the effectiveness for two real-world applications, HEOM and a WSM6 proxy. Y1 - 2017 U6 - https://doi.org/10.1007/978-3-319-67630-2_29 N1 - Best Paper Award VL - 10524 SP - 389 EP - 403 PB - Springer International Publishing ER - TY - CHAP A1 - Wende, Florian A1 - Noack, Matthias A1 - Steinke, Thomas A1 - Klemm, Michael A1 - Zitzlsberger, Georg A1 - Newburn, Chris J. ED - Dutot, Pierre-Francois ED - Trystram, Denis T1 - Portable SIMD Performance with OpenMP* 4.x Compiler Directives N2 - Effective vectorization is becoming increasingly important for high performance and energy efficiency on processors with wide SIMD units. Compilers often require programmers to identify opportunities for vectorization, using directives to disprove data dependences. The OpenMP 4.x SIMD directives strive to provide portability. We investigate the ability of current compilers (GNU, Clang, and Intel) to generate SIMD code for microbenchmarks that cover common patterns in scientific codes and for two kernels from the VASP and the MOM5/ERGOM application. We explore coding strategies for improving SIMD performance across different compilers and platforms (Intel® Xeon® processor and Intel® Xeon Phi™ (co)processor). We compare OpenMP* 4.x SIMD vectorization with and without vector data types against SIMD intrinsics and C++ SIMD types. Our experiments show that in many cases portable performance can be achieved. All microbenchmarks are available as open source as a reference for programmers and compiler experts to enhance SIMD code generation. Y1 - 2016 SN - 978-3-319-43659-3 U6 - https://doi.org/10.1007/978-3-319-43659-3_20 VL - Euro-Par 2016: Parallel Processing: 22nd International Conference on Parallel and Distributed Computing PB - Springer International Publishing ER - TY - CHAP A1 - Wende, Florian A1 - Noack, Matthias A1 - Schütt, Thorsten A1 - Sachs, Stephen A1 - Steinke, Thomas T1 - Application Performance on a Cray XC30 Evaluation System with Xeon Phi Coprocessors at HLRN-III T2 - Cray User Group Y1 - 2015 ER - TY - CHAP A1 - Noack, Matthias A1 - Wende, Florian A1 - Oertel, Klaus-Dieter ED - Reinders, James ED - Jeffers, Jim T1 - OpenCL: There and Back Again T2 - High Performance Parallelism Pearls Y1 - 2015 SN - 978-0-12-803819-2 VL - 2 SP - 355 EP - 378 PB - Morgan Kaufman, Elsevier ER - TY - JOUR A1 - Knoop, Helge A1 - Gronemeier, Tobias A1 - Sühring, Matthias A1 - Steinbach, Peter A1 - Noack, Matthias A1 - Wende, Florian A1 - Steinke, Thomas A1 - Knigge, Christoph A1 - Raasch, Siegfried A1 - Ketelsen, Klaus T1 - Porting the MPI-parallelized LES model PALM to multi-GPU systems and many integrated core processors: an experience report JF - International Journal of Computational Science and Engineering. Special Issue on: Novel Strategies for Programming Accelerators N2 - The computational power and availability of graphics processing units (GPUs), such as the Nvidia Tesla, and Many Integrated Core (MIC) processors, such as the Intel Xeon Phi, on high performance computing (HPC) systems is rapidly evolving. However, HPC applications need to be ported to take advantage of such hardware. This paper is a report on our experience of porting the MPI+OpenMP parallelised large-eddy simulation model (PALM) to multi-GPU as well as to MIC processor environments using the directive-based high level programming paradigm OpenACC and OpenMP, respectively. PALM is a Fortran-based computational fluid dynamics software package, used for the simulation of atmospheric and oceanic boundary layers to answer questions linked to fundamental atmospheric turbulence research, urban modelling, aircraft safety and cloud physics. Development of PALM started in 1997, the project currently entails 140 kLOC and is used on HPC farms of up to 43,200 cores. The main challenges we faced during the porting process are the size and complexity of the PALM code base, its inconsistent modularisation and the complete lack of a unit-test suite. We report the methods used to identify performance issues as well as our experiences with state-of-the-art profiling tools. Moreover, we outline the required porting steps in order to properly execute our code on GPUs and MIC processors, describe the problems and bottlenecks that we encountered during the porting process, and present separate performance tests for both architectures. These performance tests, however, do not provide any benchmark information that compares the performance of the ported code between the two architectures. Y1 - 2017 PB - Inderscience ET - Special Issue on: Novel Strategies for Programming Accelerators ER -