Back to the top of page
In February our team met it's first performance milestone with the ST3D code which achieved floating point speeds in excess of 10 Gigaflops/sec on the NASA Goddard Cray T3D Seymour. This code evolved a spacetime containing gravitational waves using the BMSS hyperbolic form of the Einstein field equations. The code was documented and delivered to the NASA ESS HPCC team for distribution on the National High Performance Software Exchange (NHSE).
In September our team, with the help of Tom Clune of Cray Research, was able to achieve a floating point speed of 50 Gigaflops/sec with the GR3D code developed in collaboration between the University of Illinois, Washington University of St. Louis, and the Albert Einstein Institute of the Max Planck Society in Potsdam Germany. This code combines a relativistic hydrodynamics staggered mesh algorithm together with a MacCormack scheme for evolving the Einstein equations in the BMSS form. A performance milestone of approximately 66 Gigaflops/sec was met with this code on a Cray T3E. The test problem was the evolution of a Friedman-Robertson-Walker (FRW) cosmology. Further work is being carried out on this code in order to increase the floating point performance. The code is being documented and will be delivered to NASA in the spring of 1998 for distribution on the NHSE.
Back to the top of page
The most difficult aspects of the binary neutron star problem result from the fact that general relativity must be used to determine the evolution of the system. The spacetime evolution of our 2nd milestone code is based on the Cactus'' code, the main part of which is written by members of the Potsdam group (Joan Masso and Paul Walker) and developed together with collaborators at Potsdam and Wash U. In this code, the full 3D Einstein equations are written in an explicitly hyperbolic, first order, flux-conservative form. This allows conservative finite difference techniques developed in fluid dynamics to be used on the Einstein equations. The use of operator splitting techniques allows one to treat the principal part of the system as a flux conservative first order system.
The space-time evolution code runs on massively parallel machines through it's messaging layer PUGH. PUGH is not only responsible for the high parallel efficiency, but also enables the spacetime code to conveniently interface with various modules important for the neutron star merger simulation, including the hydrodynamics solver, the gravitational waveform extraction, the coordinate condition solver, the constraint enforcement and others. Other computer science modules like check-pointing, interpolation routines etc. are also coupled in through it. Many of these modules are presently under development by our groups at Potsdam, Wash U and UI; and a relativistic hydro evolution module has been coupled through PUGH to form the 2nd milestone code with the FLOP rate given above. The interface with the hydro evolution module is non-trivial as the artificial viscosity based hydro code uses staggered grids, different from the main spacetime evolution code. The elliptic solver PETSc developed by the Argonne Lab is also coupled in through PUGH. The construction and optimization of PUGH were due to members in the NCSA/Potsdam/Wash U collaboration (mainly Paul Walker, Joan Masso, and Mark Miller), and Tom Clune at Cray Research, with help from members of our group at Argonne Lab.
One of the main goals of the project is to study the gravitational waves generated by the coalescences of neutron star binaries. It is crucial to be able to simulate gravitational waves accurately. We have carried out various wave propagation testbeds, seeing satisfactory results by comparing to perturbation theory and results obtained with previous generations of our spacetime code. In the following we show a sequence of snapshots of a quadrupole gravitational wave (the Teukolsky wave) exploding outward. The function being plotted is the equal-potential surfaces of one of the metric function (gxx). The first octant is made transparent to reveal the internal structure.
The wave study was carried out mainly by Malcolm Tobias, Joan Masso and Peter Anninos.
The following are movies of other aspects of the evolution of the quadrupole gravitational waves, and a movie on some aspects of the evolution of a black hole, produced by Joan Masso and Paul Walker.
 |
Teukolsky Wave Evolution (1.9 Mb Quicktime) We show isosurfaces of grr for a linearized l=2 m=2 wave packet. The parameter file for this run is available |
 |
Teukolsky Wave Evolution (460K Quicktime) We show a slice of the same evolution. The similarity to quadrupolar radiation in black hole spacetimes is noticeable. |
 |
Maximally Sliced Black Hole (552K Quicktime) We show the growth of the radial metric function and collapse of the lapse for a single schwarzschild black hole evolved with maximal slicing. |
A long standing problem in numerical simulations of strongly gravitating systems is the setting up of initial data. There are constraints in the Einstein theory, which restrict the structure of the initial data. At present it is not clear how to construct initial data that solves the constraints while at the same time accurately represents the physical system we want to simulate. We take the approach of first constructing the initial data that best represents the system through the post-Newtonian formulation, and then modify it to satisfy the constraint. As the first few steps in this approach, we have constructed newtonian initial data with gravitational radiation reaction, and newtonian and post-newtonian initial data representing neutron star binaries in equilibrium or quasi-equilibrium. We have also shown that post-newtonian (P1N) initial data can be matched to full general relativistic initial data on a spatial boundary without producing instability, providing certain class of time slicing is used. This work is carried out mainly by Hisaaki Shinkai, together with other members in the Wash U group.
A substantial amount of work has been carried out by Doug Swesty and Ed Wang on Newtonian and post-Newtonian simulations of neutron star coalescence as well as the testing of numerical algorithms for this problem. A number of surprising results have emerged from our work. First, we have learned that the radiation reaction forces that result from the loss of energy and momentum due to gravitational radiation dominate tidal instabilities during the merger. That is the tidal instability in stiff polytropes first uncovered by Rasio \& and Shapiro (1994) is dominated by the radiation losses as described by the 2.5PN corrections to the Newtonian hydrodynamics equations as described by Blanchet, Damour, and Schaeffer (1990). This can be clearly seen in the following series of snapshots from a set of simulations carried out by Doug Swesty and Ed Wang (1997). The images show a logarithmic color map of density in the orbital plane of the system. The four images on the left show the Newtonian evolution which is driven by purely tidal forces while the three images on the left illustrate the same initial data evolved including radiation reaction effects.
A comparison of the images at 0.002 seconds reveals that the dumbbell shaped cores are 90 degrees out of phase with the post-Newtonian model (right image) further advanced relative to the Newtonian model (left image). The difference in evolution is due to the angular momentum loss in the post-Newtonian model. This indicates that the post-Newtonian effects dominate the tidal instabilities and further motivates the need for fully relativistic calculations. A color movie (10 Mbytes) of the post-Newtonian simulation depicted above can be found here
Additionally, recent work by Swesty and Wang, along with Alan Calder, who has been hired as a postdoc under this project, has centered on convergence studies and a comparison of numerical methods for this problem. Swesty, Wang, & Calder have been able to uncover strong differences between Calculations done in fixed and rotating frames of reference in the Newtonian case. The evidence seems to point to problems with angular momentum non-conservation in the fixed frame case due to spurious numerical diffusion. Such problems were anticipated because of the difficulty of advecting neutron stars across a fixed grid. The results strongly indicate that where possible numerical models of this phenomena need to be carried out in a rotating frame.
Swesty and Wang have also developed the relativistic hydrodynamics code, GRH3D, that was coupled to the space-time evolution code. The code employs a staggered mesh artificial viscosity scheme similar to that of Hawley, Smarr, and Wilson (1984). The advection is second order and is carried out by the monotonized Van Leer method. Swesty and Wang have extensively validated the code on a number of relativistic and non-relativistic test problems. The code has also been extensively optimized for single processor performance on a number of superscalar architectures.
Back to the top of page