Parallel Multiscale Algorithms for Astrophysical Fluid Dynamics Simulations

Objective: Our goal is to overcome the multiple length-scale barrier that characterizes gravitationally-driven astrophysical and cosmological fluid flows through the use of adaptive mesh refinement (AMR) implemented on parallel computers. The driving application is cosmological structure formation, which demands a spatial dynamic range of at least 10,000 in 3 spatial dimensions. This rules out uniform grids, as 10^12 cells would be required to achieve the required dynamic range.

Approach: Adaptive mesh refinement works by first defining a uniform coarse mesh covering the region of interest. As structure develops, subgrids are introduced locally and adaptively in order to maintain adequate resolution. The process is recursive, so that regions of very high contrast are resolved by a hierarchy of successively refined grids, where each grid is refined by an integral factor R with respect to its parent grid. The dynamic range of an AMR hierarchy with base grid dimension N and L levels of refinement is N*R**(L-1); thus, N=150, R=4, and L=4 gives the desired dynamic range.

Accomplishments: We have developed a 3D cosmological AMR code incorporating ideal gas dynamics, collisionless dark matter, gravity, and cosmological expansion. Gas dynamics is evolved using the piecewise parabolic method (PPM) adapted to cosmological flows. Dark matter is evolved using the particle-particle particle-mesh (P3M) algorithm. Gravity is computed using FFT's to solve the Poisson equation in an expanding universe. The code is currently implemented for shared memory multiprocessors in C++ and Fortran77. The use of AMR results in a vast savings of CPU and memory over equivalently resolved uniform grid calculations. Figure 1 (at left) shows a 2D test problem that achieves a dynamic range of 512 in the two level-3 subgrids, which are centered on high-density peaks representing galaxy clusters. Here, N=32, R=4, and L=3, for a memory savings of approximately 50 and a CPU savings of over 100.

Significance: All gravitationally driven flows in astrophysics and cosmology span a vast range of length- and time-scales. AMR allows us to efficiently attack these problems in full 3D, thus making tractable with current computers problems that were previously intractable. Our cosmological simulations will allow us to model the formation and evolution of volume-limited samples of X-ray clusters while at the same time resolving the internal structure of individual clusters.

Status/Plans: Year 2 milestones have been reached; we have a functioning sequential object-oriented AMR code for cosmological applications. In Year 3, we will explore parallel object-oriented implementations using pC++ (Gannon et al.), so that we may take advantage of the memory and speed afforded by these systems. The pC++ language has been ported to all available high performance systems, so our code will be completely portable. As an early experiment in pC++ code development, we have implemented an object parallel domain decomposed version of our PPM code called CMHOG++. Figure 2 (below) shows a simulation of a 3D supersonic jet carried out on the Thinking Machines Corp. CM-5 massively parallel computer using a grid of 512 x 128 x 128 cells.

Point of Contact:

Michael L. Norman
National Center for Supercomputing Applications
norman@ncsa.uiuc.edu
217-244-6099