If you are using Navigator 4.x or Internet Explorer 4.x, this site will not render correctly!
Global Organization for Earth System Science Portals (GO-ESSP) 2008 Workshop
Phil Bentley
Data Applications Manager
Met Office Hadley Centre
My interests in attending this year's GO-ESSP meeting in Seattle reflect my participation in, and contribution to, a number of pertinent activities.
Under the CF Conventions theme I shall be looking to engage with other delegates to discuss:
- Follow-on work arising from my revisions to the CF specification in connection with grid mapping and coordinate reference system attributes.
- Formulation of additional CF attributes which can be used to facilitate greater netCDF-GIS integration / interoperability (I have a potential presentation in mind along these lines; I can submit this separately if desired - please let me know. It's too long to include here!).
- Development of the common concept idea and related vocabulary mapping initiatives (I'm engaged in implementing interim solutions here at the Met Office).
Under the Semantically-enabled Environments theme:
- As a member of the Met Office team participating in the EU METAFOR project we have a keen interest in understanding how other organisations are tackling the issue of discovering, describing, and comparing climate model datasets across distributed data archives. This looks like it will be a hot topic at this year's meeting.
Under the Federation theme:
- I have responsibilities for facilitating the dissemination of Met Office climate model data to a number of external partners. Naturally we are looking ahead to the potential data distribution requirements arising from any IPCC AR5 programme. This meeting hopefully should provide good opportunities for sharing ideas on appropriate technical solutions for achieving this.
Jennifer M. Adams
IGES/COLA
Ensemble Handling in GrADS and GDS: Working with TIGGE Data
Recent improvements to GrADS and the GrADS Data Server (GDS) include the implementation
of the 5th grid dimension for ensemble handling and a new interface for data in
the GRIB2 format. These enhancements make it much easier for users to work with
modern multi-member and multi-model data sets. GrADS and GDS are widely used tools
for distribution, access, analysis, and visualization of data, and they are ready
to meet the technological challenges brought on by recent advances in weather and
climate forecasting research and operations. GrADS and GDS are at the core of NOMADS,
and are also poised to take on the formidable collection of forecast data in the
TIGGE project. The TIGGE data archive on disk at NCAR will be behind a dedicated
GDS; the format translation (from GRIB2 to NetCDF), file aggregation, subsetting,
and server-side analysis capabilities of the GDS will revolutionize how TIGGE data
will be used by the international community.
Joe Sirott
NOAA/PMEL
Adventures in providing web services for large geophysical datasets
------------------------------------------------------------------
The amount of geophysical data is increasing exponentially. For example, it is anticipated
that the IPCC AR5 climate model simulations will require 100 times the storage of
the IPCC AR4 model runs. However, many of the tools that have traditionally been
used by data providers to make geophysical data available to scientists and the
public will not scale to handle terabytes or petabytes of data.
Dapper/DChart (http://dapper.pmel.noaa.gov) is an OPeNDAP server with an AJAX-based
user interface that allows users to visualize or download in-situ or gridded geophysical
data. This talk discusses how some of the challenges encountered in scaling Dapper/DChart
to serve terabytes of model data from the IPCC AR4 model runs were overcome as well
as some approaches that are being considering in providing access to even larger
datasets.
Dennis Heimbigner
Unidata
Integrating OPeNDAP into netCDF-4
The SDCI project is a joint NSF-funded effort between Unidata and OpeNDAP.org. The
goal is to integrate the client-side OpeNDAP protocol into the standard netCDF distribution.
The result will be a version of netCDF version 4 capable of seamless access to networked
data through the OpeNDAP protocol. Format translation will be based on an extension
of the existing libnc-dap translations. The approach for this integration will be
discussed as well as some issues that must be addressed.
Alexander Pletzer
Tech-X
MoDAVE: a plug-in for CDAT to read, interpret and visualize Mosaic datasets
Mosaic defines a grid specification that encompasses structured, unstructured, as well as unstructured assemblies of structured grids, including the cubed sphere grid as a special case. The Mosaic specification has been submitted to the Climate Forecasting (CF) conventions as an extension. Unstructured assemblies of structured grids are attractive as a means to overcome numerical Courant time step restrictions associated with conventional longitude/latitude based grids at the poles, while at the same time leveraging all the advantages of structured grids in terms of numerical accuracy and efficiency. Here we present progress on the design and development of the Mosaic Data Analysis and Visualization Extension (MoDAVE), which will be offered as a plug-in to one of the leading community data analysis tool (CDAT). Live demos of 3D visualization/parallel data processing of cubed-sphere data will be presented.
Stephan Kindermann (DKRZ)
Michael Lautenschlager(MPI-M/M&D)
The C3Grid Federated System for Climate Data Handling
Within the last 3 years C3Grid established a federated, grid-based infrastructure for the German Climate Community as part of the national D-Grid e-Science initiative. It is planed to convey the developments into a sustainable inter-operable production system in the next project phase, which could then act as a national contribution to the trans-national effort of IPCC AR5 data handling.
In the talk we want to present and discuss the main technological design decisions with respect to:
- AA: The requirement was to combine a federated identity management at the individual home institutes with a PKI based grid security environment and single sign on at a central portal. This resulted in a security architecture based on Shibboleth, a national Short Lived Certificate Service and GridShibb based Authorization at resource providers based on SAML assertions embedded in proxy certificates.
- Uniform data discovery: Standard technology was adopted to enable uniform data discovery in a central portal. OAI-PMH harvesting of ISO 19139 profile descriptions into a central (lucene based) search engine was implemented. No specific extensions were made to define the C3Grid ISO profile, but the use of standard ISO tags was restricted to carry the necessary information e.g. CF names based on controlled vocabularies.
- Data access: In the first phase standard Web Service technology was adopted to implement asynchronous data access. Data access requests in general contain spatial, temporal subsetting requirements as well CF variable selections, whereas data delivery is based on GridFTP. Because the data access interface is an integral part of the C3Grid middleware a transition to adopt WSRF based web services was necessary. This also enables the implementation of authorization based on GridShibb.
Possible contribution to "Show and Tell" Session:
Title: Overview of C3Grid Portal and C3Grid Infrastructure
Description: An overview of the GridSphere based C3Grid Portal is shown along with details about the status of the technological implementation of C3Grid metadata harvesting, data access and data analysis job scheduling.
Luca Cinquini
CISL, NCAR
The ESG-CET Gateway: a distributed and federated architecture for data search, access
and analysis in the geo-sciences.
Abstract: This talk will present a general introduction to the ESG-CET Science Gateway Framework, which is being developed to support the upcoming IPCC AR5 data management needs as well as other geo-informatics projects. The talk will focus on 3 selected aspects of the framework:
- An overview of the ESG-CET architecture, with particular emphasis on the Gateway component
- The use of OpenID to achieve web Single-Sign-On among Gateways
- An RDF-based framework for data search and discovery
Katherine A. Bouton/Lois Steenman-Clark
NCAS CMS METAFOR
METAFOR - Common Metadata for Climate Modelling Digital Repositories
The main objective of METAFOR is to develop a Common Information Model (CIM) to describe the models that produce climate data by building on existing standards, such as NMM and Curator, to ensure that key data and model distinctions can be shared and understood between users of different scientific repositories.
METAFOR will develop, deploy, and evaluate a prototype infrastructure that will allow key data and model types to be found and compared between holdings at partner sites.
The impact of this project will not only be on the European contribution for planning for AR5, but also on existing scientific activities which exploit the partner repositories.
Dr. Roberto De Almeida
The best of worlds: pydap as a proxy to other Opendap servers
In this talk I would like to show the use of pydap as a proxy to other Opendap servers, which allows the use of pydap-only features (like Google Earth integration or WMS output) with conventional servers. I also plan to discuss the implementation of server-side functions on pydap, and how it is abstracted in order to simplify the development of new data handlers.
Roland Schweitzer, Ansley Manke and Steve Hankin
Server-side OPeNDAP Analysis – Concrete steps toward a generalized framework via a reference implementation using F-TDS
The GrADS Data Server (GDS) was an early implementation of an OPeNDAP server that introduced a rich set of server-side analysis functions based on the GrADS scripting language. The advantages of data reduction performed on fast servers local to the data and transmitting the much smaller results are clear. The TMAP group at PMEL is currently using a second generation implementation of a server-side analysis capability based on Ferret as the underlying analysis engine. Our current implementation is based on plug-ins available from the THREDDS Data Server.
At the 2007 OPeNDAP developer’s meeting in Boulder, CO we reported on the current state of the F-TDS server. We also made an appeal to the community for input on a plan for developing a common syntax for server-side functions. The community processed launched with an official OPeNDAP working group. We quickly developed survey of existing implementations, a general requirements list of a way to discover the capabilities of the server along with several proposals for a generalized syntax for invoking the server-side functions.
There are several of the difficulties inherent to building a consensus and a set of specifications even if the developers are committed to the general goals. One difficulty we faced with the specification was how quickly complicated requirements (like nesting and composition of function calls) were proposed from use cases without regard to the implementation details. Developers also found it difficult to commit to specifics without the opportunity to explore possible implementations. Even so, several concrete proposals emerged from the community process though none of them were ever implemented.
We as the F-TDS developers decided to take a step back from the community process and see if we could implement some of the proposals for an extremely limited set of functions and input types. By considering only transformations which collapsed one or more axes of a rectilinear geo-spatial, time grid and transformations which interpolated one rectilinear geo-spatial, time grid onto another we were able to create concrete implementations of a generalized server-side function syntax using our F-TDS server as the platform. This paper will present the results of that work and attempt to re-energize the community process for developing a more complete specification with one or more reference implementations.
Ethan Davis
UCAR Unidata
THREDDS Data Server, OGC WCS, and CF
The OGC WCS 1.0 implementation in the THREDDS Data Server (TDS) has been improved to better deal with selection of temporal and vertical dimensions. As part of the WCS 1.0 Plus project, an experimental WCS server has been added that keep some of the simplicity of WCS 1.0 while adding some higher dimensional features of WCS 1.1. Improvements to the handling of Coordinate Reference Systems (CRS) in the TDS WCS implementations have fed into suggestions to recent CF Grid Mapping/CRS changes.
I would be happy to give a brief presentation on the above. However, my main interest is in taking part in further discussions with others involved in implementing OGC WCS servers and/or clients. As well as further discussions on CF Grid Mapping and CRS (including vertical datum, etc).
Eric Nienhouse
CISL, NCAR
The Challenges of Building Rich Web Clients Geo-Spatial Applications – Lessons Learned.
New strategies and technologies for web-based user interface development.
Many scientific research organizations are developing application components in
support of geo-spatial data discovery, presentation and analysis. These solutions
vary widely in terms of functionality, quality and level of integration with other
products and components. In addition, a number of technical trends are enabling
scientific web applications to provide “richer”, more fully featured and desktop-like
functionality. This talk will present a number of the challenges encountered
developing web browser based geo-spatial application user interfaces within the
earth science domain. The talk will focus on the following areas:
An overview and comparison of several existing solutions, including the technologies
and frameworks supporting them.
A technical case study of an in-development web based geo-spatial user interface.
Lessons learned and thoughts on emerging “best practices” from this experience.
Roy Mendelssohn, Lynn deWitt and Cara Wilson
NOAA/NMFS/SWFSC/ERD
Environmental Data Connector (EDC) – A Tool for Interoperability Between the Metocean Community and the GIS Community.
We will discuss the EDC, and ArcGIS plugin that provides a graphical interface to read THREDDS catalogs and OPENDAP servers, and subset and import the data into ArcGIS. The plugin has a timeslider that aligns the times and provides automatic animation of the data. “Features” can be used to draw vector data.
Bob Simons (bob.simons@noaa.gov) and Roy Mendelssohn
(Roy.Mendelssohn@noaa.gov)
NOAA / NMFS / SFSC / Environmental Research Division
ERDDAP - A RESTful, Resource-Oriented Web Service for Simplified Client Access to Scientific Data
ERDDAP is a new open-source, all-Java, RESTful, resource-oriented, web-based service that provides simplified client access to scientific data. ERDDAP aggregates over existing web services (OPeNDAP grids, OPeNDAP sequences (DRDS), OPeNDAP nested sequences (Dapper), NOS SOAP service, OOStehtys SOS, microWFS, DiGIR (OBIS)) and returns standardized data in a variety of formats more directly useful to clients. Because ERDDAP is RESTful, the URI uniquely defines the data request, and the URI can be used in any application that can send a URI and receive a file. This allows ERDDAP to be used in mashups with other web services. ERDDAP makes all datasets available via standard (and enhanced) DAP queries, with standard .das, .dds, .html, and .dods responses. It could be extended to support other protocols. Other output formats include csv, htmlTable, XHTML, .mat, netCDF, .kml, .png, and .pdf. Examples will be shown where ERDDAP is used in Matlab, R, browsers, Google Earth, Google Gadgets, and in user-defined widgets that can be used in a web page. Thus ERDDAP makes data from diverse sources available to diverse clients. ERDDAP has a hub and spoke architecture which simplifies adding support for new types of data sources and new types of clients. ERDDAP includes catalog services, metadata support, and options to make graphs and maps. See http://coastwatch.pfel.noaa.gov/erddap/index.html .
Seiya NISHIZAWA, Takeshi HORINOUCHI, Akinori TOMOBAYASHI,
and Chiemi WATANABE
Depertment of Geophysics, Kyoto University
Gfdnavi: its design and implementation with Ajax and Ruby-On-Rails
Gfdnavi is a Web-based database and analysis tool for geophysical fluid data.
The basic technology which is used to develop it is Ruby-On-Rails (Rails). Rails
is an open-source web application framework based on the Model-View-Controller architecture
and is written in the Ruby programming language. The Gfdnavi has some relational
data base tables, which have information about directory tree structure, metadata
(keyword and value pairs) and information about spatial and temporal region of scientific
data, information on users and groups, and etc. The nodes table, which is for the
tree structure, has some types, such as directory, variable and figure. Data file
on a file system is stored as the directory type node in the data base, and variable
in the file is as the variable type node. Metadata which are added to a directory
node are applied to all the variable nodes which are contained in the directory
and its sub-directories. The node has an owner, groups and permission mode like
the inode for access control (It can have plural groups, while the inode has only
one).
The Gfdnavi provides a GUI interface on a web-browser to search, analyze and visualize
data. We have tried to design the interface as user can use it easily without referring
manuals: For examples, the directory tree view looks like the Microsoft Explore,
and the Google Map can be used to specify the spatial region. Using Ajax enables
seamless operation.
Jeremy Malczyk
JISAO
Contrasting styles of Web UI development: Google Web Toolkit (GWT) versus the JavaScript API and Firebug
In this presentation we present the distinct styles of browser UI development that are inherent in the Google Web Toolkit (GWT) versus JavaScript toolkits and Firebug. We will outline the requirements for an ambitious UI that can handle 4D scientific visualization, transformations and differencing among many datasets. We will overview the history of UI development in the LAS project, and highlight some lessons learned. And will compare and contrast the merits of GWT versus native JavaScript from the standpoint of development effort, debugging environment, browser compatibility, maintainability of code and integration with other JavaScript toolkits.
Alison Pamment
British Atmospheric Data Centre
The Current Status and Future Development of CF Standard Names.
The current status of the CF standard name table will be summarized and a formal procedure for the submission of new standard names proposals will be presented. This will include a discussion of timescales for the discussion and acceptance of new standard names. There has been much recent debate surrounding the purpose of standard names in CF metadata, the way in which the names should be constructed and the manner in which standard names can be utilized to provide services to data users. The current state of the debate will be summarized and the key areas in which decisions are needed from the CF community will be highlighted.
Stephen Pascoe, D.Lowe, B.Lawernce, A.Stephens
Centre for Atmospheric Data Archival, Rutherford Appleton Laboratory
Integrated OGC Web Service delivery with the COWS framework
The CEDA OGC Web Services framework (COWS) takes much of the the burden of the OWS protocol infrastructure away from the developer, handling service metadata, request/response encoding and exception handling whilst maintaining maximum flexibility in the choice of data and visualisation backends. COWS currently provides implementation layers for WMS, WCS and WPS. Here we demonstrate several bespoke data services built on COWS. (1) An integrated WCS/WMS exposing RAPID climate model data using the Climate Science Markup Language (CSML) as the data backend. (2) A WMS of Met. stations linked to an integrated plotting service using a GeoServer WFS as a backend. (3) An asynchronous geospatial and statistical plotting service implemented as WPS.
Matt Pritchard
NEODC / CEDA / RAL
1) Development & deployment of services within the ESA Service Support Environment
(SSE)
Steve Donegan & Matt Pritchard, CEDA/RAL
The Centre for Environmental Data Archival (CEDA) at the Rutherford Appleton Laboratory,
UK, manages both the British Atmospheric Data Centre and the NERC Earth Observation
Data Centre on behalf of the UK Natural Environment Research Council. In addition
to providing user access to a 100+ Tb archive of environmental data and supporting
a wide range of users from many science disciplines, CEDA undertakes development
work aimed at improving discovery and delivery of data and services to users.
The Service Support Environment operated by the European Space Agency (ESA), is
a framework in which service providers can develop and deploy services in support
of the delivery, processing and exploitation of Earth Observation (EO) data. The
SSE defines a core set of service interfaces which service providers are able to
extend and use in either synchronous or asynchronous service operations. The SSE
framework also facilitates service chaining to enable re-use of component services.
The NEODC has recently deployed a new suite of services aimed at aiding users of
its archive of sea- and land-surface temperature observations from the (A)ATSR series
of instruments, and to provide novice users of airborne EO data with a user interface
to essential processing software.
2) CSML : current status and roadmap
Dominic Lowe & Andrew Woolf, CEDA/RAL
Jon Blower, University of Reading E-science Centre
Climate Science Modelling Language (CSML) comprises a standards-based data model
that defines thirteen “Feature Types” that describe a very large proportion of atmospheric
and oceanic data. By mapping heterogeneous data onto common Feature Types, intercomparison
of data and reuse of common software components is achieved. The BADC has built
a software stack in Python to mediate between CSML, legacy data and standard interfaces
(such as OGC Web Map and Coverage services).
Additionally we have now defined, in conjunction with Reading e-Science centre,
an initial set of Java interfaces to CSML Feature Types. Under this collaboration,
the CSML data model is to be deployed in a series of EU projects; in the ECOOP project,
a decision-support system is being developed that allows coastal-ocean model and
observational data from all around Europe to be shared and intercompared. In GENESI-DR2,
satellite data is used to diagnose and improve numerical models of atmospheric circulation.
Finally, the Java framework will be applied to the validation of state-of-the-art
global ocean models against a large database of historical observations; this will
be used, for example, to search for signals of climate change in decadal simulations
of ocean circulation.
Jon Blower, University of Reading, UK
Pauline Mak, TPAC/ARCS
Ethan Davis, Unidata, US
John Caron, Unidata, US
Integrating a Web Map Service into the THREDDS Data Server
The Open Geospatial Consortium’s Web Map Service specification (WMS) is becoming
increasingly popular for the visualization of all kinds of geospatial data.
Many scientific data providers are now providing WMS interfaces to their data holdings
(e.g. NASA, NSIDC, NERC Data Grid). This presentation describes the integration
of ncWMS (a Web Map Service for NetCDF data, http://ncwms.sf.net)
with the THREDDS Data Server. We shall discuss the conceptual issues involved
in mapping scientific data onto OGC concepts, the technical issues involved in the
development of the software, the implications for performance and scalability and
the application of the new software to a number of major projects, including the
European MyOcean programme.
Note: I also have an interest in the development of rich internet applications for earth science portals and would be able to participate in discussions and tutorials relating to technologies such as HTML, Javascript, DOM, Google Web Toolkit, OpenLayers and more.
Note 2: If JB is unable to attend the meeting in person Ethan Davis will present this work, possibly in combination with related work on THREDDS-WCS.
M. Benno Blumenthal
IRI of Columbia University
Connecting netcdf/CF to a semantic framework
The standards underlying the Semantic Web -- Resource Description Framework (RDF) and Web Ontology Language (OWL), among others -- show great promise in addressing some of the basic problems in earth science metadata. In particular they provide a single framework that allows us to describe datasets according to multiple standards, creating a more complete description than any single standard can support, and avoiding the difficult problem of creating a super-standard that can describe everything about everything.
One step on that road for us is to write down the CF metadata standard in a semantic framework, i.e. RDF/OWL. This happens on two levels: a literal level which explains which attributes are available to be attached to datasets/variables, and a more semantic level, which gives explicit expression to concepts like Coordinate and Non-Coordinate variables, and how a Non-Coordinate Variable can be geolocated.
Writing down the CF standard on a semantic level then allows interoperability with other standards, e.g. other ways of marking geolocated Non-Coordinate variables. A rule-based framework for creating such mappings is demonstrated.
Some additional issues are touched upon, like how to less-ambiguously tag metadata in netcdf files so that software can more easily determine which attribute belongs to which metadata standard, and how to better register netcdf metadata standards in general and CF concepts in particular, so that interoperability (or indeed operability) can move forward.
Jianfu Pan (jianfu.pan-1@nasa.gov)
Adnet Systems, Inc.
GSFC-NASA
Giovanni III - From Concept to Reality
Web services for scientific data systems
With the technological advancement in recent years, especially in mass storage and
high speed internet, data archive and distribution is no longer a commodity.
GES DISC of NASA has been developing an innovative data analysis, visualization
and download system with its Giovanni III (G3) project. G3 evolves from DISC's
earlier versions of Giovanni's, which served as quick checks of data mostly through
the Grads software tool. G3 adopts a service-oriented architecture (SOA) and
offers a range of sophisticated functionality from data algorithms to service orchestration.
This presentation will discuss the conceptual model of the system and some of its
design philosophies. The presentation will also provide an overview of the
system's services and functionality, including virtual parameter using Math ML,
which allows scientists to implement science algorithms by writing a recipe in Math
ML, an XML standard for mathematics. Further, the presentation will demonstrate
a typical work flow by online users as well as WCS-type of remote access to the
G3 services.
Steve Ansari, Rich Baldwin, Stephen Del Greco, Neal Lott, Glenn Rutledge
Web Services at the National Oceanic and Atmospheric Administration (NOAA) National Climatic Data Center (NCDC)
NOAA’s National Climatic Data Center (NCDC) currently archives over 1.5 petabytes of climatological data from various networks and sources including in-situ, numerical models, radar and satellite. Access to these datasets is evolving from interactive web interfaces utilizing database technology to standardized web services in a Service Oriented Architecture (SOA).
NCDC is currently offering several web services using XML over Representational State Transfer (REST/XML), KML and Shapefile REST Services, Open Geospatial Consortium (OGC) Web Map Service (WMS) / Web Feature Service (WFS) / Web Coverage Service (WCS) and OPeNDAP web service protocols. These services offer users a direct connection between their client applications and NCDC data servers. In addition, users may embed access to the services in custom applications to efficiently navigate and subset data in an automated fashion.
NCDC currently provides gridded numerical model data through a THREDDS Data Server and GrADS Data Server which offers OPeNDAP and WCS access. In-situ network metadata are available through WMS and WFS while the corresponding time-series data are accessible through SOAP and REST web services. These in-situ services are a part of the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI) WaterOneFlow services, a consolidated access system for hydrologic data, and comply with the WaterOneFlow specifications. NCDC’s Severe Weather Data Inventory (SWDI), which provides user access to archives of several datasets critical to the detection and evaluation of severe weather, is also accessible through XML, CSV, Shapefile and KMZ REST services.
Providing cataloging, access and search capabilities for many of NCDC’s datasets using community driven standards is a top priority for the ever increasing data volumes being archived at NCDC. Providing interoperable access is critical to supporting data stewardship across multiple scientific disciplines and user types. This presentation will showcase NCDC’s latest work towards standardized web services with both server and client examples
Russ Rew
UCAR Unidata
Update on Unidata technologies relevant to GO-ESSP
Unidata's mission is to provide the data services, tools, and cyberinfrastructure leadership that advance Earth system science, enhance educational opportunities, and broaden participation. Unidata's support, maintenance, and development of software relevant to the GO-ESSP community is progressing along several fronts. We provide a brief overview of progress and plans for C- and Java-based netCDF libraries and utilities, libcf support for the CF Conventions, work with OPeNDAP on improving netCDF and OPeNDAP integration, IDV visualization and analysis software, and data services associated with the THREDDS Data Server. In addition we outline plans and progress for more explicit CF support at the Unidata Program Center.
Rich Signell
U.S Geological Survey
Woods Hole Science Center
The Model Interoperability Experiment in the Gulf of Maine: a success story made possible by NetCDF, CF, NcML, NetCDF-Java, THREDDS, OPeNDAP
The Gulf of Maine Ocean Data Partnership Modeling Committee has been developing a Model Interoperability Experiment in the Gulf of Maine built around the CF standard. The goal was to allow scientists using Matlab to issue a single identical command to retrieve geospatially located data, regardless of what type of model they are accessing. Our starting point was output from six different models (four ocean circulation models, a meteorological model and an ocean-wave model) with six different grid conventions produced at six different institutions. All groups produced NetCDF files, used Matlab for visualization and analysis, and had a standard HTTP 1.1 web server. Only one group used CF-conventions, however, and as a result each group had their own set of analysis and visualization routines to perform nearly identical tasks.
The goal is to achieve interoperability with a minimum of effort on the part of the data providers and data users. To provide data for the interoperability experiment, participants need only place their existing NetCDF files on their own web sites. A layer of XML (NcML) is used to achieve CF standardization without altering the existing NetCDF files, and to provide virtual aggregation of data. This functionality is made possible by the NetCDF-Java library, which understands NcML and has the ability to access a byte range of a remote file on a web site. The final component of the distributed system is the THREDDS Data Server, which uses NcML & NetCDF-Java behinds the scenes, but allows for central cataloging of the datasets and access via the OpenDAP service. For uniformly spaced grids, the datasets are available via THREDDS using the OGC Web Coverage Service (WCS).
The CF-standard data is accessed with the CF Toolkit for Matlab, a toolbox we are actively developing using NetCDF-Java. The advantage of these tools is that the work on any system without compiling, and they work for any structured grid model that can be described by the CF standard, eliminating the need for custom toolkits for individual models. We need to work to develop CF conventions that specify staggered grids, masked regions, velocity components and unstructured grid data so that toolboxes like this one can address an even broader range of user needs.
John Caron
Unidata
Point Observations : Data models and possible encodings.
We present current work within Unidata on a comprehensve data model for point observations (aka in-situ data). We will try to characterize various encodings of point data, such as DAPPER, ERDDAP, netCDF, and BUFR. We will discuss tradeoffs, with the goal of proposing a "CF Convention for Point Observation Data".
Roy Lowry
BODC
The NERC DataGrid Vocabulary Server: an operational system with distributed ontology potential
The NERC DataGrid Vocabulary Server has now developed into a fully operational term registry holding over 100,000 terms organised into over 100 vocabulary lists linked together by over 75,000 SKOS mapping triples. It is receiving hundreds of thousands of hits per month from robots mining the semantic network and about 1% of this activity level from human-driven requests. It has been integrated into ‘Smart Discovery’ option in a NERC DataGrid portal and has been adopted for operational use by the EU SeaDataNet project.
The server is based on an operationally-hardened Oracle database back end and the service-based front end provides multiple ways to access content:
- Term and list URLs
- An HTTP-POX (pseudo-RESTful) URL API interface
- A SOAP API
- Clients delivering human-readable (XHTML) documents
System development is continuing with short term plans to fully implement versioned serving, label vocabularies with their content governance, enrich the predicate vocabulary used and serve ontologies as OWL documents. Longer term developments under consideration are mapping support for external URIs and the incorporation of an inference engine.
Eric Kihn, NGDC NOAA, Boulder
Mikhail Zhizhin, Space Research Institute, Russian Academy of Sciences
NOAA CLASS API and Portal Prototype
The NOAA Comprehensive Large Array Stewardship System (CLASS) is a very large collection of the Earth Science (ES) datasets in different data models, including geolocated time series (grids and stations), remote sensing images and satellite telemetry. We have prototyped a CLASS data selection and visualization portal from several building blocks, which can be of interest and re-use in other ES portals. For dataset and user metadata we utilize VxOware toolkit. It is developed by NOAA and Russian Academy of Sciences (RAS) for federating of ES virtual observatories and Web 2.0 collaboration. For time series data selection, processing and mining in the UNIDATA Common Data Model (CDM) we use Environmental Scenario Search Engine (ESSE), which is a set of grid-services hosted by the OGSA-DAI container. ESSE is developed by NOAA, RAS and Microsoft Research. For navigating through satellite orbits in the NASA ECHO data model we also use OGSA-DAI grid-services developed by NOAA. Data request web forms on the portal are built by XSLT transform directly from the grid-service metadata, if it contains a special Ordering Extensions XML section. For the interactive visualization we create mashups of the climate data time series and remote sensing images using a special plugin in the NASA World Wind and in the MS Virtual Earth.
Bruce Wright
UK Met Office
CF-netCDF in WMO
The WMO Expert Team on the Assessment of Data Representation Systems (ET-ADRS) was established to assess advantages and disadvantages of different data formats for use in real-time operational international exchanges within WMO and for transmission of information to outside users. An initial meeting in Washington in April 2008, considered a range of file formats, including GRIB, BUFR, netCDF, HDF5 and XML, in the context of future WMO usage.
The meeting report proposed that WMO should look to adopt netCDF-4 (rather than netCDF-3, as this offers improved performance and convergence with HDF5), where appropriate. NetCDF was seen as particularly important as part of the Future WMO Information System (FWIS), and for sharing data with non-WMO organisations.
Further, it recommended that WMO should engage with the netCDF and CF community to:
- Agree on a coordination mechanism to drive forward the netCDF format (e.g. enhanced packing/compression) and the CF metadata standards to be fit for operational meteorology (WMO requirement);
- Consider the question of the resources (e.g. at National Meteorological & Hydrological Service level) required to support the data format and metadata standard development;
- Develop a model for the governance.
This presentation will report on progress ahead of a formal ET-ADRS recommendation, which goes to the WMO in November 2008, and provide an opportunity to discuss the related issues.
Daniel J. Crichton, Chris A. Mattmann, Amy Braverman
NASA Jet Propulsion Laboratory
California Institute of Technology
Facilitating Distributed Climate Modeling Research and Analysis via the Climate Data eXchange
Climate change research requires the ability to rapidly find, access, manipulate, and combine massive, heterogeneous, physically distributed model output and satellite remote sensing data sets. Current data and analysis systems are inadequate: they are isolated from one another, and deliver only static products in their native, disparate forms. Such systems do not permit interactive combination, exploration and analysis of the data and model outputs. To address these needs, we are constructing the Climate Data eXchange (CDX) system at NASA’s Jet Propulsion Laboratory, leveraging our past experience building large-scale data systems for planetary science (NASA’s Planetary Data System) and for biomedicine (the NIH Early Detection Research Network). CDX is a science data analysis environment that employs grid computing technology to share and analyze climate data by a) connecting isolated sub-networks, across NASA and externally to, e.g., the DOE; and by b) pushing as much computation as possible to nodes where data reside, minimizing movement of data. We envision an organic network of networks around the country, all using open source CDX technology to interconnect. We discuss our plans to erect core infrastructure to support CDX, including establishing a two-way connection between JPL’s CDX services and PCMDI, making critical observational data sets (e.g., AIRS Level 2 data products) and PCMDI model outputs seamlessly available to NASA and the DOE.
Masayoshi Ishii
Frontier Research Center for Global Change of Japan Agency for Marine-Earth Science
and Technology (FRCGC/JAMSTEC)
We, Japanese community as a whole, are willing to build a distributed CMIP5 data server which provides data produced through Japanese research activities on CMIP5 and IPCC AR5. I will introduce several ongoing Japanese projects for IPCC AR5 and researches on data integration and anslysis.
WCSplus: Making the Web Coverage Service work for the Fluid Earth Sciences
community
Bruce Wright, UK Met Office
The Open Geospatial Consortium Web Coverage Service is an important standard for providing access to "coverages" of data (most often grids) over the internet (or other networks) through a standard interface, promoting interoperability both within and between disciplines. However, WCS version 1.0 failed to support many important capabilities required by the Fluid Earth Sciences community (e.g. it is limited to regular grids). The WCS 1.1 standard attempted to satisfy all possible use cases, resulting in a standard that is bulky and unmanageable in practice.
The WCSplus initiative was established to address this, starting with WCS 1.0, for which a substantial number of working implementations exist, it aims to take forward the FES requirements (with working demonstrations), including:
- CF-netCDF binary encoding;
- 3 spatial dimensions;
- Multiple time dimensions;
- Time relative to the present;
- Non-spatial coordinates;
- Non-gridded coverages;
- Asynchronous response.
The outcome will feed into the WCS 1.2 standard, which is likely to consist of a rather limited core (possibly only 2d) and a set of extensions. Thus, the WCSplus "capability" will probably be realised through an FES Profile and a number of relevant programmatic extensions.
This presentation will review progress of the WCSplus initiative, as well as providing some practical examples and insights from Met Office (and others) experiences with the WCS (e.g. Met Office Visual Weather WCS and Unidata THREDDS WCS).
