ÌÇÐÄÊÓƵ

To probe the mysteries of how galaxies evolve over time, scientists needed a supercomputer with out-of-this-world computational power.

The results of a conducted on the Frontier supercomputer at the Department of Energy's Oak Ridge National Laboratory and published in The Astrophysical Journal offer the clearest portrait so far of how some galaxies regulate the energy produced by supermassive black holes at their cores.

These black holes—which range up to billions of times the sun's mass—power phenomena known as active galactic nuclei, which constantly pump heat, dust and gas into their environments. Some of that material orbits the nuclei in luminous regions known as accretion disks, while some wanders far beyond galactic bounds.

"Fundamentally, we set out to understand how these galaxies regulate themselves over the age of the universe," said Brian O'Shea, a computational astrophysicist at Michigan State University and co-author of the study.

"These galaxy clusters are the biggest things in the universe, millions of light-years across, with black holes at their centers that are bigger than our entire solar system. All the heat and material are being ejected and recycled from each nucleus as jets into the galactic atmosphere.

"We wanted to find out: Where do the energy and material end up? How does that energy and turbulence feed into the galactic structures and their formation? And how do these systems keep radiating all this heat for billions of years without collapsing on themselves? We've never been able to even attempt to answer these questions before at this level of fidelity because we never had a machine like Frontier."

The Frontier study used as its test case a black hole of about 10⁹ solar masses, or a billion times the size of the sun, at the center of a galaxy cluster of about 10¹⁵ solar masses, or a quadrillion times the size of the sun—and about a thousand times the mass of the Milky Way galaxy that's home to Earth. The study simulated the cluster's evolution over billions of years, modeling each step of the various jet activity cycles.

"These jets are extremely fast, so fast that even with Frontier's power we had to artificially limit their speed in the simulation to about 5% of the speed of light," said Philipp Grete, a computational astrophysicist at the Hamburg Observatory in Germany. "That still resulted in the simulations taking about 2 million steps to complete."

The team ran a series of simulations ( on GitHub) that tested a wide range of model assumptions. The process required 700,000 node hours and 17,088 GPUs using , an open-source, astrophysical magnetohydrodynamics code based on the framework. The number of steps, the long timespan and the sheer amount of detail revealed by the simulations couldn't have been accomplished on any other machine.

"One of the long-standing questions we've had is whether these systems can stay stable across billions of years," Grete said. "But a big challenge for us has always been just keeping the system evolving long enough to observe everything and understand the details.

"The only way to do this at all was on a machine not just with lots of memory plus storage to host the data but with enough GPU computing power to deliver a fast turnover. That's why we could only do these kinds of simulations on Frontier."

Frontier's nearly 2-exaflop speeds—equal to 2 quintillion calculations per second—enabled the team to zero in on key details never before simulated, including the gradual formation of gas filaments. Similar filaments surround such well-known astronomical features as the Perseus galaxy cluster, a group of more than 1,000 galaxies about 240 million light-years from Earth.

"We're the first study ever to reproduce this phenomenon," O'Shea said. "Because these galaxy clusters are so big, we couldn't watch these filaments evolve before. Their formation had been a mystery, but now we know how they come to be: through the turbulence created by the interaction of these cold gases with the hot intergalactic plasma—some of it as hot as 100 million Kelvins—and the magnetic fields that surround them."

The results ultimately showed galaxy clusters rely on those magnetic fields to regulate their energy and remain stable over time. The team hopes to expand on the study to incorporate additional physics such as cosmic rays and other plasma phenomena.

"We're just beginning to unpack the role played by the interaction between these fields and the turbulent plasma," O'Shea said. "As we increase our understanding of these phenomena, there could be lessons learned that apply not just to galaxy clusters but to supernovae and even to the turbulence found in fusion tokamaks. The more closely we can analyze this data, the more secrets of these complex systems we can unravel."

Besides Grete and O'Shea, the research team included Mark Voit and Benjamin Wibking of Michigan State University, Deovrat Prasad of Cardiff University, Forrest Glines of NVIDIA, and Marcus Brüggen and Martin Fournier of the University of Hamburg. resulting from this research were published in Astronomy & Astrophysics.