High Performance Computing in Remote Sensing - Chapter 10

Chapter 10 Remote Sensing Grids: Architecture and Implementation Samuel D. Gasster, The Aerospace Corporation Craig A. Lee, The Aerospace Corporation James W. Palko, The Aerospace Corporation Contents 10.1 Introduction .......................................................... 204 10.1.1 Remote Sensing and Sensor Webs ............................ 204 10.1.2 Remote Sensing System Architectures and Grid Computing ... 205 10.2 Remote Sensing Grids: Architecture ................................... 209 10.2.1 Weather Forecasting and Climate Science Grid ................ 209 10.2.2 Derived Functional Requirements and Resources .............. 214 10.2.3 WFCS Grid Services Architecture ............................ 217 10.3 Remote Sensing Grids: Implementation ............................... 220 10.3.1 Management and Policy Considerations ....................... 220 10.3.2 Data Management ............................................ 222 10.3.3 Job Management and Workflow ............................... 223 10.3.4 Grid Monitoring and Testing .................................. 224 10.4 Remote Sensing Grids: Examples ..................................... 226 10.4.1 Example 1: Linked Environments for Atmospheric Discovery (LEAD) ........................................... 226 10.4.2 Example 2: Landsat Data Continuity Mission (LDCM) Grid Prototype (LGP) Project ................................. 228 10.4.3 Example 3: NOAA National Operational Model Archive and Distribution System (NOMADS) ......................... 229 10.5 Remote Sensing Grids: Adoption ...................................... 230 10.6 Acronyms ............................................................ 231 References .................................................................. 232 203 © 2008 by Taylor & Francis Group, LLC 204 High-Performance Computing in Remote Sensing 10.1 Introduction In the last 40 years, remote sensing of the Earth has seen a continuous growth in the capabilitiesoftheinstrumentation(satellites,airborne,andground-basedsensorsthat monitor and measure the environment) that provides the fundamental data sets and an increase in the complexity of the data analyses and modeling that these data sets support. The rate of increase in the remote sensing data volume continues to grow. Additionally, the number of organizations and users is also expanding and is now a worldwidecommunitystrugglingtosharedataandresources.Thesetrendsnecessitate a shift in the way in which remote sensing systems are designed and implemented in order to manage and process these massive data sets and support users worldwide. Grid computing, as originally described by Foster et al. [1, 2], provides a new and rich paradigm with which to describe and implement various distributed computing system architectures. The promise of grid computing for science users is a shared environment that will facilitate their scientific research and provide them access to an unprecedented range of resources: instrumentation, data, high-performance compute engines, models, and software tools. Ultimately this access will be two-way, one in whichtheremotesensingscientistwillnotonlyreceiveandprocessdatafromremote sensors and instrumentation, but will be able to reconfigure them as well. In a 2002 GridToday [3] article, Ian Foster further clarifies the definition of a grid as a system with these three elements: (1) coordination of resources not subject to centralized control; (2) utilizes standard, open, and general-purpose protocols and interfaces; (3) delivery of non-trivial qualities of service. We will show that these elements map nicely into the remote sensing domain. The goal of this chapter is to apply the grid computing paradigm to the domain of EarthRemoteSensingSystems.Thesesystemsinvolvethecollection,processing,and distribution of large amounts of data and often require massive computing resources to generate the data products of interest to users. Current systems, such as the NASA EOS mission, generate hundreds of gigabytes of raw sensor data per day that must be ingested and processed to produce the mission data products. The trend in both scientificandoperational1 weatherforecastingisasteadyincreaseintheamount,and types, of data necessary to support all the applications in this problem domain. 10.1.1 Remote Sensing and Sensor Webs There are many definitions of remote sensing in the literature [4, 5, 6]. The common theme in all of these definitions is the measurement of some physical property associ-atedwiththeobjectunderobservationwithoutbeinginphysicalcontactwiththeobject 1The term operational as used here is intended to convey the fact that the capability is required on a con-tinuous and reliable basis and that the performance meets some minimum quality of service requirements. In the case of weather forecasting, the capability to continuously generate weather forecasts with specific latency and accuracy is the key requirement. © 2008 by Taylor & Francis Group, LLC Remote Sensing Grids: Architecture and Implementation 205 (hencethetermremote).Thisformofremotesensinggenerallyinvolvesthedetection ofsomeformofelectromagneticenergyeitherreflectedby,oremittedfrom,theobject (e.g.,visiblelightreflectedfromtheoceansurfaceorinfraredradiationemittedfroma cloudtop),whichisthenfedtoaninversionalgorithmtoretrievespecificgeophysical parameters such as sea surface temperature or atmospheric vertical moisture profile. For the sake of the present discussion, we generalize this definition to include not onlythetraditionalconceptofremotesensingbutalsothatofin-situsensing,wherethe measurements are made by instrumentation in contact with, or close proximity to, the objectofinterest.Ineithercase,theresultsofthesemeasurementsaresenttoalocation remote from the source for further processing and distribution. Thus we include a wide range of sensors and instrumentation that perform their measurements either remotely or in-situ and then transmit those observations to remote data collection, processing, and distribution sites. We also consider the concept of a sensor web as defined by Higgins, et al., in their report to the NASA Earth Science Technology Office (ESTO) [7]: A sensor web is a distributed, organized system of nodes, interconnected by a communications fabric that behaves as a single, coherent instru-ment.Throughtheexchangeofmeasurementdataandotherinformation, produced and consumed by its sensing and non-sensing nodes, the sen-sor web dynamically reacts causing subsequent sensor measurements and node information processing states to be appropriately modified to continually ensure optimal science return. The sensor web concept considers a highly distributed system that includes feed-backbetweenvariousnodes.Suchaconceptisconsistentwiththefundamentalnotions of grid computing. By combining the sensor web and grid computing paradigms we create what we term a remote sensing grid (RSG). This is a highly distributed system that includes resources that support the collection, processing, and utilization of the remote sensing data. Many of the resources are not under a single central control, yet we have the ability to coordinate the activities of any of these resources. It is possi-ble to construct a remote sensing grid using standard, open, protocols and interfaces. Finally,manyoftheoperationalremotesensingsystemsarerequiredtosupporthighly non-trivial quality of service requirements, such as availability. 10.1.2 Remote Sensing System Architectures and Grid Computing The goal of this chapter is to provide a description of remote sensing grids by com-bining the concepts of remote sensing or sensor web systems with those of grid computing. In order to do this one needs to understand how remote sensing systems are described and specified, and similarly how grids are described and specified, and how these two approaches are merged to describe and specify remote sensing grids. Wedrawontheapproachtraditionallyemployedbysystemsengineeringtospecify a system as consisting of various elements and subsystems [8]. The remote sensing systemisdesignedtobeanorganizedassemblyofresourcesandproceduresunitedand © 2008 by Taylor & Francis Group, LLC 206 High-Performance Computing in Remote Sensing GEO LEO LEO UAV Ground Station Ground Station NCAR FNMOC CAL/VAL NEXRAD Site Ground Station AFWA NESDIS NWS Field Users Figure 10.1 High-level architectural view of a remote sensing system. regulated by interaction or interdependence to accomplish a set of specific functions, which are performed by the various elements and subsystems. The system, as com-posed of these elements and subsystems, is described by specifying, at various levels of detail, the system architecture. The term architecture refers to a formal description of a system, defining its purpose, functions, externally visible properties, and inter-faces. It also includes the description of the system’s internal components and their relationships,alongwiththeprinciplesgoverningitsdesign,operation,andevolution. In order to provide a specific example and context for discussing remote sensing grids, we consider the design of a notional weather forecasting and climate science (WFCS) grid. This notional system is motivated by several projects including LEAD [13]andNOMADS[17](discussedinSection10.4.1),andTheAerospaceHighReso-lutionForecastPrototype[20].Thissystemisintendedtosupportweatherforecasting and climate studies and is described in more detail in Section 10.2.1. A high-level view of the WFCS system architecture is shown in Figure 10.1. This figure describes the architecture primarily in terms of the various elements and sub-systems, and in later sections of this chapter we will work towards a description in terms of grid computing concepts. We envision the WFCS system as being made up of resources from a variety of organizations (the notional system is illustrated with examples that include FNMOC, AFWA, NCAR, NESDIS/NWS, and others). The organizations provide specific ca-pabilities necessary to implement such a system, but there may also be additional capabilities required for the realization of a grid architecture. Figure 10.1 illustrates the following elements of the WFCS architecture: r Observing Elements: in-situ sensors, ground based, airborne, and spaceborne instruments that collect the basic environmental measurements © 2008 by Taylor & Francis Group, LLC Remote Sensing Grids: Architecture and Implementation 207 r Data Management Elements: data transport, storage, archive discovery, and distribution r Data Processing and Utilization Elements: user applications, modeling and assimilation, forecasting, etc. r Communications, Command, and Control Elements: the resources that allow all elements to work together; includes interaction and feedback between any of the sensor web elements r Core Infrastructure: underlying resources needed to tie all elements together, including networks, communication links, etc. In order to map the system engineering view of the WFCS system into grid com-puting terms, we need to understand how grid architectures are described. The grid computing community has developed an approach to describing grids known as the OpenGridServicesArchitecture(OGSA)[9],usingtheconceptsofaservice-oriented architecture (SOA). It is therefore prudent to define some of the terminology used in OGSA and SOA. The Global Grid Forum (GGF) and others [10, 11] provide the following definitions: Service is a software component that can be accessed via a network to provide functionality to a service requester. Service Oriented Architecture refers to a style of building reliable distributed sys-tems that deliver functionality as services, with the additional emphasis on loose coupling between interacting services. Workflow is the structured organization of a set of activities, their relative ordering and synchronization, and data needs necessary to accomplish a specific task or goal. The workflow may also specify any necessary participants. Given these definitions, we see that SOA refers to the design of a system and not how it is implemented. It is also possible to describe, at least in part, various workflows using a service-oriented description. Following Srinivasan and Treadwell [11], we employ SOA as an architectural style that utilizes components as modular services (generally considered to be atomic in that they provide one service) that can bediscoveredandusedtobuildworkflowsbyclients.Wefurtherassumethatservices have the following characteristics [11]: Composition — may be used alone or combined with other services. Communication via Messages — communicates with clients or other services by exchanging messages. Workflow Participation — services may be aggregated to participate in a specified workflow. Interaction — services may perform their key functions as stand-alone entities or require interactions with other services to fulfill their functions. Advertise — services advertise their capabilities, interfaces, polices, etc., to clients. © 2008 by Taylor & Francis Group, LLC ... - tailieumienphi.vn