TY - GEN A1 - Benger, Werner A1 - Hege, Hans-Christian A1 - Merzky, Andre A1 - Radke, Thomas A1 - Seidel, Edward T1 - Efficient Distributed File I/O for Visualization in Grid Environments N2 - Large scale simulations running in metacomputing environments face the problem of efficient file I/O. For efficiency it is desirable to write data locally, distributed across the computing environment, and then to minimize data transfer, i.e.\ reduce remote file access. Both aspects require I/O approaches which differ from existing paradigms. For the data output of distributed simulations, one wants to use fast local parallel I/O for all participating nodes, producing a single distributed logical file, while keeping changes to the simulation code as small as possible. For reading the data file as in postprocessing and file based visualization, one wants to have efficient partial access to remote and distributed files, using a global naming scheme and efficient data caching, and again keeping the changes to the postprocessing code small. However, all available software solutions require the entire data to be staged locally (involving possible data recombination and conversion), or suffer from the performance problems of remote or distributed file systems. In this paper we show how to interface the HDF5 I/O library via its flexible Virtual File Driver layer to the Globus Data Grid. We show, that combining these two toolkits in a suitable way provides us with a new I/O framework, which allows efficient, secure, distributed and parallel file I/O in a metacomputing environment. T3 - ZIB-Report - SC-99-43 KW - Grid KW - Data Grid KW - Globus KW - HDF5 KW - Cactus KW - CCTK KW - Amira KW - VFD KW - Storage Systems Y1 - 1999 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-4326 ER - TY - GEN A1 - Hege, Hans-Christian A1 - Hutanu, Andrei A1 - Kähler, Ralf A1 - Merzky, André A1 - Radke, Thomas A1 - Seidel, Edward A1 - Ullmer, Brygg T1 - Progressive Retrieval and Hierarchical Visualization of Large Remote Data N2 - \noindent The size of data sets produced on remote supercomputer facilities frequently exceeds the processing capabilities of local visualization workstations. This phenomenon increasingly limits scientists when analyzing results of large-scale scientific simulations. That problem gets even more prominent in scientific collaborations, spanning large virtual organizations, working on common shared sets of data distributed in Grid environments. In the visualization community, this problem is addressed by distributing the visualization pipeline. In particular, early stages of the pipeline are executed on resources closer to the initial (remote) locations of the data sets. \noindent This paper presents an efficient technique for placing the first two stages of the visualization pipeline (data access and data filter) onto remote resources. This is realized by exploiting the ``extended retrieve'' feature of GridFTP for flexible, high performance access to very large HDF5 files. We reduce the number of network transactions for filtering operations by utilizing a server side data processing plugin, and hence reduce latency overhead compared to GridFTP partial file access. The paper further describes the application of hierarchical rendering techniques on remote uniform data sets, which make use of the remote data filtering stage. T3 - ZIB-Report - 03-40 KW - remote data access KW - large files KW - visualization pipeline KW - hdf5 KW - grid Y1 - 2003 U6 - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:0297-zib-7623 ER -