High Performance Computing and Four-Dimensional Data Assimilation: The Impact on Future and Current Problems

Objective: The goal of Four Dimensional (space and time) Data Assimilation is to incorporate actual observations (satellite, ship, land surface, balloon) into mathematical and computational models in order to create a unified, complete description of the atmosphere. This can be used by community-wide scientists to study important phenomena associated with the environmental global change; such as greenhouse-gas research, ozone-loss research, or atmospheric pollution.

Approach: At NASA Goddard's Data Assimilation Office (DAO) research is being performed using existing techniques of 4DDA, as well as work on prototype techniques that will make use of the new observations in the Earth Observing System (EOS) project. Work on High Performance Computing and Communications (HPCC) is a part of the DAO's present task. This is broken down into the following categories:

(1) Porting present day operational algorithms (GEOS-DAS-1.2) for 4DDA to high performance computers (Intel, Cray, Thinking Machines, and networks of workstations). The DAO is also ensuring that future improvements to the operational algorithms, such as the PSAS methodology, are compatible with High Performance Computers.

Accomplishments: To date the GEOS-DAS-1.2 analysis code has been ported to parallel systems of workstations and Intel parallel computers (Delta, and Paragon). At present the ported version is in relatively crude form. It is scientific code that is the product of years of research and many scientists. Hence work is being done to bring it up to current standards of software (fortran 90 and ultimately high performance fortran). Once this phase has been completed in the summer of 1994, the algorithm will be improved in terms of parallel computing issues such as load balancing and turn-around speed.

The DAO is also in the planning phase for parallelizing the improved version of GEOS-DAS-1.2, namely the Physical space Statistical Analysis System (PSAS) code. This involves the potentially difficult global matrix solve -- corresponding to a simultaneous analysis for all observations across the numerical grid. Members of the HPCC team at JPL will be visiting Goddard in the early Fall to help in this work.

(2) Parallelizing the semi-lagrangian general circulation model at the DAO. This model is a research project; a parallel version of a one layer model has taken place and it is being used to test the feasibility of obtaining a true scalable three dimensional (multilayer) model.

(3) Research on Kalman Filtering. The Kalman filter uses the principles of Estimation Theory to formulate a consistent approach to 4DDA. Ultimately not only are accurate datasets produced, but strongly-bound errorbars are calculated. The latter are important to scientists who use the data, and who need to know how reliable those data are. The present GEOS-DAS-1.2 produces such errorbars, while the Kalman filter is intended to be a substantial improvement on current methodology.

The main problem with the Kalman filter is that a fully three dimensional (in space) version would demand very large computer memory (terabyte) and CPU resources (at least teraFLOPS). The DAO is a world leader in Kalman Filter research. A tractable two-dimensional (latitude-longitude in space) algorithm is being formulated, the scientific and technical work is being done entirely on massively parallel computer code.

At present the basic two-dimensional algorithm has been built and tested on Intel massively parallel computers (Delta, and Paragon). This code produces results with acceptable turn-around time, as predicted when the project was submitted. In the coming year the scientific capabilities of this code will be assessed. Actual GEOS-DAS-1.2 assimilated wind data along with UARS satellite retreived profiles of one of ozone, nitrous oxide, or methane will be incorporated. The purpose of this is to test the viability of the Kalman filter to assimilate single trace constituents and produce globally accurate datasets. By February of 1995 the first integrated algorithm will have been completed. After that, scientists at the DAO will use the algorithm to study constituent assimilation and to develop simplified (suboptimal) Kalman filter that may be implemented in three spatial dimensions.

(4) Parallelizing current transport codes at the DAO. These codes are used by scientists at Goddard. The assimilated wind datasets of the DAO's GEOS-1 runs are used as input to these transport codes, which then study transport of important trace chemical constituents such as ozone, nitrous ozide, and methane.

We have developed a transport code that uses van Leer algorithm for horizontal advection and Prather method for vertical advection. It is written in a high-level data parallel language (High Performance Fortran) which makes it portable between various parallel and distributed architectures. As of the beginning of summer 1994 we have two high performance computing versions of the van Leer (horizontal) code: message passing, and data parallel. In theccoming year we will engage in a study of these two approaches to assess their suitability for the van leer and similar Eurerian schemes.

We have also developed a graphical user interface for our transport codes using the AVS package.

We are collaborating with John Baumgardner of LANL on the development of a semi-lagrangian transport algorithm on icosahedral grid in High Performance Fortran. The semi-lagrangian algorithm is notoriously difficult to parallelize due to its irregular and dynamic computational requirements. Our goal is to develop a more efficient, portable and transparent code by combining physically motivated data structure (icosahedral geometry that is approximately homogeneous across the whole three dimensional sphere) and High Performance Fortran.

Significance: These experiments are an important first step in solving one of the Grand Challenge problems, a TeraFlop scalable GCM. Furthermore, our parallel version of the Dynamical Core executes efficiently on several parallel architectures from a single source code. The programming techniques and algorithms we are developing can be applied to a wide range of scientific codes that will need to be ported to parallel architectures as such platforms become more generally available. Finally, we have compared our general parallel code on the C98 against our current C98 optimized production code, with the parallel code executing at approximately 70% of the machine specific production run. If the same holds true for other architectures, as initial hardware optimizations to the T3D also indicate, then these codes are well suited for measuring the relative performance of scientific codes on various parallel architectures.

Status/Plans: Ultimately is expected that High Performance Computing will aid in the speed, resolution, and scientific capabilities of codes and algorithms within Goddard's DAO. The speed is important because the output of scientists is improved by their not having to wait days for results to appear. The resolution is important because the ultimate believability of the results is improved by the resolution of smaller spatial scales (of the order of kilometers) of the weather and climate phenomena. The present day codes are by no means perfect, hence work on the Kalman filter is an important aspect of the DAO's plans to improve the scientific capabilities of the assimilation.

More than porting and improving scientific algorithms, the DAO is involved in assessing the High Performance Computers and their software (for example parallel fortran 90 and high performance fortran). This will be useful information that will ultimately be fed back to the manufacturers and policy makers. The DAO is also strongly coupled to the general parallel computing community, providing policy advice and absorbing new developments in the field.

Principal Investigator Progress Metric(s)

Point of Contact:

Return to the PREVIOUS PAGE

curator: Larry Picha (lpicha@cesdis.gsfc.nasa.gov)