TY - CHAP A1 - Gorlatch, Sergei A1 - Dünnweber, Jan ED - Getov, Vladimir ED - Laforenza, Domenico ED - Reinefeld, Alexander T1 - From Grid Middleware to Grid Applications BT - Bridging the Gap with HOCs T2 - Future Generation Grids N2 - This paper deals with the problem of application programming for grid systems that combine heterogeneous data and computational resources via the Internet. We argue that grid programming is still too complex because of the big gap between the currently used and anticipated grid middleware, (e.g., Globus or WSRF) and the application level. We suggest that this gap needs to be closed in future-generation grids and propose a novel approach to bridging the gap by using Higher-Order Components (HOCs) — recurring patterns of parallel behaviour that are provided to the user as program building blocks with pre-packaged implementation and middleware setup. The presentation is illustrated with a simple case study of computing fractal images. Our experiments demonstrate that HOCs can simplify grid application programming significantly, without serious performance loss. KW - components KW - Grid services KW - Globus toolkit KW - mobile code Y1 - 2005 SN - 978-0-387-27935-0 U6 - https://doi.org/10.1007/978-0-387-29445-2_13 SP - 241 EP - 261 PB - Springer Verlag CY - Berlin ER - TY - BOOK A1 - Dünnweber, Jan A1 - Gorlatch, Sergei T1 - Higher-Order Components for Grid Programming BT - Making Grids More Usable N2 - A major challenge in grid computing remains the application software development for this new kind of infrastructure. Grid application programmers have to take into account several complicated aspects: distribution of data and computations, parallel computations on different sites and processors, heterogeneity of the involved computers, load balancing, etc. Grid programmers thus demand novel programming methodologies that abstract over such technical details while preserving the beneficial features of modern grid middleware. For this purpose, the authors introduce Higher-Order Components (HOCs). HOCs implement generic parallel/distributed processing patterns, together with the required middleware support, and they are offered to users via a high-level service interface. Users only have to provide the application-specific pieces of their programs as parameters, while low-level implementation details, such as the transfer of data across the grid, are handled by the HOCs. HOCs were developed within the CoreGRID European Network of Excellence and have become an optional extension of the popular Globus middleware. The book provides the reader with hands-on experience, describing a broad collection of example applications from various fields of science and engineering, including biology, physics, etc. The Java code for these examples is provided online, complementing the book. The expected application performance is studied and reported for extensive performance experiments on different testbeds, including grids with worldwide distribution. The book is targeted at graduate students, advanced professionals, and researchers in both academia and industry. Readers can raise their level of knowledge about methodologies for programming contemporary parallel and distributed systems, and, furthermore, they can gain practical experience in using distributed software. Practical examples show how the complementary online material can easily be adopted in various new projects. KW - Component-Based Development KW - Distributed Computing KW - Distributed Resource Management KW - Globus KW - Grid and Cloud Computing KW - Java KW - Middleware KW - Web Services KW - distributed systems KW - grid computing KW - load balancing KW - programming Y1 - 2009 SN - 978-3-642-00841-2 U6 - https://doi.org/10.1007/978-3-642-00841-2 PB - Springer CY - Berlin, Heidelberg ER - TY - CHAP A1 - Dünnweber, Jan A1 - Lüdeking, Philipp A1 - Gorlatch, Sergei ED - Danelutto, Marco ED - Fragopoulou, Paraskevi ED - Getov, Vladimir T1 - A Higher-Order Component for Efficient Genome Processing on the Grid T2 - Making Grids Work N2 - Computational grids combine computers in the Internet for distributed data processing and are an attractive platform for the data-intensive applications of bioinformatics. We present an extensible genome processing software for the grid and evaluate its performance. Our software was able to discover previously unknown circular permutations (CP) in the ProDom database containing more than 70MB of protein data. A specific feature of our software is its design as a component: the Alignment HOC, a Higher-Order Component that makes use of the latest Globus toolkit as grid middleware. Besides genome data, the Alignment HOC accepts plugin code for processing this data as its input, and contains all the required configuration to run the component on top of Globus, thus, freeing the non-grid-expert user from dealing with grid middleware. Instead of writing data distribution procedures and configuring the middleware appropriately for every new algorithm, Alignment HOC users reuse the existing component and only write application-specific plugins. To maintain plugins persistently in a reusable manner, we built a web-accessible plugin database with a comfortable administration GUI. The flexible component-based implementation makes it easy to study CPs in other databases (e.g. UniProt/Swiss-Prot) or to use an alignment algorithm different than the standard Needleman-Wunsch. For the efficient distribution of workload, we developed a library of group communication operations for HOCs. KW - higher-order components KW - genome processing KW - sequence alignment KW - middleware Y1 - 2007 SN - 978-0-387-78447-2 U6 - https://doi.org/10.1007/978-0-387-78448-9_28 SP - 351 EP - 362 PB - Springer CY - Boston, MA ER - TY - CHAP A1 - Dünnweber, Jan A1 - Tomasoni, Johannes A1 - Gorlatch, Sergei A1 - Claßen, Michael A1 - Claßen, Philipp A1 - Lengauer, Christian ED - Gorlatch, Sergei ED - Fragopoulou, Paraskevi ED - Priol, Thierry T1 - A Grid Component with Embedded Loop Parallelization T2 - Grid Computing N2 - This work integrates two distinct research areas of parallel and distributed computing, (1) automatic loop parallelization, and (2) component-based Grid programming. The latter includes technologies developed within CoreGRID for simplifying Grid programming: the Grid ComponentModel (GCM) and Higher- Order Components (HOCs). Components support developing applications on the Grid without taking all the technical details of the particular platform type into account (network communication, heterogeneity, etc.). The GCMenables a hierarchical composition of programpieces and HOCs enable the reuse of component code in the development of new applications by specifying application-specific operations in a program via code parameters. When a programmer is provided, e. g., with a compute farm HOC, only the independent worker tasks must be described. But, once an application exhibits data or control dependences, the trivial farm is no longer sufficient. Here, the power of loop parallelization tools, like LooPo, comes into play: by embedding LooPo into a HOC, we show that these two technologies in combination facilitate the automatic transformation of a sequential loop nest with complex dependences (supplied by the user as a HOC parameter) into an ordered task graph, which can be processed on the Grid in parallel. This technique can significantly simplify GCM-based systems which combine multiple HOCs and other components. We use an equation system solver based on the successive overrelaxation method (SOR) as our motivating application example and for performance experiments. KW - Higher-Order Components (HOCs) KW - Loop Parallelization KW - GCM KW - Grid Programming Y1 - 2007 SN - 978-0-387-09456-4 U6 - https://doi.org/10.1007/978-0-387-09457-1_22 SP - 261 EP - 273 PB - Springer CY - Berlin ER - TY - JOUR A1 - Müller-Iden, Jens A1 - Dünnweber, Jan A1 - Gorlatch, Sergei A1 - Alt, Martin A1 - Fujita, Hamido A1 - Funyu, Yutaka T1 - Clayworks: Toward user-oriented software for collaborative modeling JF - Knowledge-Based Systems N2 - We consider the development of software systems that integrate collaborative real-time modeling and distributed computing. Our main goal is user-orientation: we need a collaborative workspace for geographically dispersed users with a seamless access of every user to high-performance servers. This paper presents a particular prototype, Clayworks, that allows modeling of virtual clay objects and running computation-intensive deformation simulations for objects crashing into each other. In order to integrate heterogeneous computational resources, we adopt modern Grid middleware and provide the users with an intuitive graphical interface. Simulations are parallelized using a higher-order component (HOC) which abstracts over the web service resource framework (WSRF) used to interconnect our worksuite to the computation server. Clayworks is a representative of a large class of demanding systems which combine collaborative, user-oriented modeling with performance-critical computations, e.g., crash-tests or simulations of biological population evolution. KW - User-oriented software development KW - Collaborative environments KW - Real-time modeling KW - High-performance simulation Y1 - 2008 U6 - https://doi.org/10.1016/j.knosys.2008.12.001 VL - 22 IS - 3 SP - 209 EP - 215 PB - Elsevier CY - Oxford, UK ER - TY - CHAP A1 - Dünnweber, Jan A1 - Parlavatzas, Nikos A1 - Gorlatch, Sergei A1 - Baude, Francoise A1 - Legrand, Virginie ED - Gorlatch, Sergei ED - Danelutto, Marco T1 - Automatic Creation of Web Services for Grid Components T2 - Integrated Research in GRID Computing N2 - While high-level software components simplify the programming of grid applications and Web services increase their interoperability, developing such components and configuring the interconnecting services is a demanding task. In this paper, we consider the combination of Higher-Order Components (HOCs) with the Fractal component model and the ProActive library. HOCs are parallel programming components, made accessible on the grid via Web services that use a special class loader enabling code mobility: executable code can be uploaded to a HOC, allowing one to customize the HOC. Fractal simplifies the composition of components and the ProActive library offers a generator for automatically creating Web services from components composed with Fractal, as long as all the parameters of these services have primitive types. Taking all the advantages of HOCs, ProActive and Fractal together, the obvious conclusion is that composing HOCs using Fractal and automatically exposing them as Web services on the grid via ProActive minimizes the required efforts for building complex grid systems. In this context, we solved the problem of exchanging code-carrying parameters in automatically generated Web services by integrating the HOC class loading mechanism into the ProActive library. KW - CoreGRID Component Model (GCM) & Fractal KW - Higher-Order Components Y1 - 2006 SN - 978-0-387-47656-8 U6 - https://doi.org/10.1007/978-0-387-47658-2_3 SP - 31 EP - 42 PB - Springer CY - Sophia Antipolis, France ER - TY - CHAP A1 - Dünnweber, Jan A1 - Aldinucci, Marco A1 - Gorlatch, Sergei A1 - Campa, Sonia A1 - Danelutto, Marco ED - Gorlatch, Sergei ED - Danelutto, Marco T1 - Adaptable Parallel Components for Grid Programming T2 - Integrated Research in GRID Computing N2 - We suggest that parallel software components used for grid computing should be adaptable to application-specific requirements, instead of developing new components from scratch for each particular application. As an example, we take a parallel farm component which is “embarrassingly parallel”, i. e., free of dependencies, and adapt it to the wavefront processing pattern with dependencies that impact its behavior. We describe our approach in the context of Higher-Order Components (HOCs), with the Java-based system Lithium as our implementation framework. The adaptation process relies on HOCs’ mobile code parameters that are shipped over the network of the grid. We describe our implementation of the proposed component adaptation method and report first experimental results for a particular grid application — the alignment of DNA sequence pairs, a popular, time-critical problem in computational molecular biology. KW - Grid Components KW - Adaptable Code KW - Wavefront Parallelism KW - Java KW - Web Services Y1 - 2006 SN - 978-0-387-47656-8 U6 - https://doi.org/10.1007/978-0-387-47658-2_4 SP - 43 EP - 57 PB - Springer Verlag CY - Boston, MA ER - TY - CHAP A1 - Müller, Jens A1 - Alt, Martin A1 - Dünnweber, Jan A1 - Gorlatch, Sergei T1 - Clayworks BT - A System for Collaborative Real-Time Modeling and High-Performance Simulation T2 - IEEE International Conference on e-Science and Grid Computing (e-Science'06), 2nd, 4-6 Dec 2006, Amsterdam, Netherlands N2 - Clayworks is a software system which integrates collaborative real-time modeling and distributed computing. It addresses the challenge of developing a collaborative workspace with a seamless access to high-performance servers. Clayworks allows modeling of virtual clay objects and running computation-intensive deformation simulations for objects crashing into each other. To integrate heterogeneous computational resources, we adopted modern Grid middleware and provided the users with an intuitive graphical interface. We parallelized the computation of simulations using a Higher-Order Component (HOC) which abstracts over the Globus Web service resource framework (WSRF) used to interconnect our worksuite to the computation server. Clayworks is a representative of a large class of demanding systems which combine collaborative modeling with performance-critical computations, e.g., crash-tests or simulations for biological population evolution. KW - Collaboration KW - Real time systems KW - Computational modeling KW - Collaborative work KW - Biology computing KW - Biological system modeling KW - Deformable models KW - Computer crashes KW - Software systems KW - Concurrent computing Y1 - 2006 SN - 0-7695-2734-5 U6 - https://doi.org/10.1109/E-SCIENCE.2006.261188 ER - TY - CHAP A1 - Dünnweber, Jan A1 - Gorlatch, Sergei A1 - Benoit, Anne A1 - Cole, Murray T1 - Integrating MPI-Skeletons with Web Services T2 - ParCo'05 - International Conference on Parallel Computing, 2005, Malaga, Spain N2 - Interoperating components, implemented in multiple programming languages, are one of the key requirements of grid computing that operates over the borders of individual hardware and software platforms. Modern grid middleware like WSRF facilitates interoperability through service-orientation but it also increases software complexity. We show that Higher-Order Components (HOCs) provide a service-oriented programming abstraction over middleware technology. By offering the pipeline skeleton from the MPI-based eSkel library as a HOC, we show how machine-oriented technologies can be made available via Web Services on grids. We bind a Java-based Web application to the HOC to demonstrate its connectivity: user defined input can be transformed in a highly performant manner by running wavelet computations remotely on parallel machines. Y1 - 2005 UR - http://homepages.inf.ed.ac.uk/mic/Pubs/malaga.pdf SP - 787 EP - 794 ER - TY - CHAP A1 - Dünnweber, Jan A1 - Gorlatch, Sergei T1 - HOC-SA BT - A Grid Service Architecture for Higher-Order Components T2 - IEEE International Conference on Services Computing (SCC 2004), 15-18 Sept. 2004, Shanghai, China N2 - The current efforts on programming grid applications often rely on service-oriented approaches like grid services. This work presents HOC-SA -a service architecture for higher-order components, which provides the programmer with reusable and composable patterns of parallelism and is interoperable with the latest Globus toolkit implementations. We describe our implementation of HOC-SA using OGSA-DAI, a framework for integrating grids with distributed databases. We present a simple example application and report first measurements on our grid testbed. KW - Web services KW - Programming profession KW - Parallel processing KW - Grid computing KW - Service oriented architecture KW - Testing KW - Distributed computing KW - Concurrent computing KW - Image segmentation KW - Electronic mail Y1 - 2004 SN - 0-7695-2225-4 U6 - https://doi.org/10.1109/SCC.2004.1358017 ER - TY - CHAP A1 - Dünnweber, Jan A1 - Gorlatch, Sergei A1 - Parlavantzas, Nikos A1 - Baude, Francoise A1 - Legrand, Virginie T1 - Towards automatic creation of web services for grid component composition T2 - Integrated Research in GRID Computing : CoreGRID Integration Workshop 2005 (Selected Papers), November 28-30, Pisa, Italy N2 - While high-level software components simplify the programming of grid applications and Web services increase their interoperability, developing such components and configuring the interconnecting services is a demanding task. In this paper, we consider the combination of Higher-Order Components (HOCs) with the Fractal component model and the ProActive library. HOCs are parallel programming components, made accessible on the grid via Web services that use a special class loader enabling code mobility: executable code can be uploaded to a HOC, allowing one to customize the HOC. Fractal simplifies the composition of components and the ProActive library offers a generator for automatically creating Web services from components composed with Fractal, as long as all the parameters of these services have primitive types. Taking all the advantages of HOCs, ProActive and Fractal together, the obvious conclusion is that composing HOCs using Fractal and automatically exposing them as Web services on the grid via ProActive minimizes the required efforts for building complex grid systems. In this context, we solved the problem of exchanging code-carrying parameters in automatically generated Web services by integrating the HOC class loading mechanism into the ProActive library. KW - CoreGRID Component Model (GCM) & Fractal KW - Higher-Order Components Y1 - 2007 SN - 978-0-387-47658-2 SN - 978-0-387-47656-8 U6 - https://doi.org/10.1007/978-0-387-47658-2_3 SP - 31 EP - 42 PB - Springer Science+Business Media CY - Boston, MA ER -