Exascale Software Lays Foundation for Discovery of New Materials

Computing is in Livermore’s DNA. From its founding in  1952, Livermore has depended on computers to help answer extremely difficult technical questions. As physicist George Maenchen said early in Lawrence Livermore’s history, “We are not like Thomas Edison. We don’t test a thousand light bulbs to figure out what is the best filament. Here we model everything first on a computer, then we do a test to confirm what we already know from our modeling.”

Fast forward to 2020. Lawrence Livermore is leading a new research center responsible for creating high-performance computing (HPC) software to aid the development of ground-breaking functional materials. These materials respond to stimuli such as voltages, light, or magnetic fields and could be used in quantum computers, photovoltaic solar cells, or solid-state data storage devices.

Funded by the Department of Energy’s (DOE’s) Basic Energy Science (BES) Program, the Center for Non-Perturbative Studies of Functional Materials under Non-Equilibrium Conditions (NPNEQ) is working to develop software that can simulate movements of electrons and ions at the quantum mechanical level. Based on a method called real-time time-dependent density functional theory (RT-TDDFT), the software allows researchers to simulate the response of quantum mechanical electrons and ions in a material in response to strong external stimuli such as a high-intensity laser. This might trigger an instantaneous change of crystal structures, leading to a dramatic change of electronic properties. Such nonequilibrium phenomena taking place in a billionth of a second have long been sought to make new types of opto-electronic devices that source, detect, and control invisible light such as gamma rays, x rays, ultraviolet, and infrared.

For functional materials—such as the those used in transistors and diodes—the static picture of the state of electrons calculated by DFT was good enough for explaining their functionalities. Now, the RT-TDDFT software developed at the NPNEQ Center, combined with the DOE leadership-class HPC systems, opens the possibility of simulating nonequilibrium states of materials in the nonperturbative limit where standard approaches that rely on small corrections to exact model systems no longer apply. The software developed by the Center is expected to open up a new era of ab-initio material design, meaning it will allow researchers to go beyond material modeling to building custom compounds from the ground up with specifically designed properties.

The NPNEQ Center brings together three national laboratories with complementary resources and expertise. Lawrence Livermore, an internationally recognized leader in HPC architecture and software design, is contributing its supercomputing facilities and expertise in software development for exascale HPC. Researchers from SLAC National Accelerator Laboratory are using their cutting-edge experimental and theory programs in advanced materials science to conduct ultrafast experiments and simulations. SLAC also assists in the implementation and testing of additional software functionalities. Researchers from Lawrence Berkeley National Laboratory (LBNL) are expert in spectroscopy simulations and assisting with software validation through interpretation of experimental results.

Tadashi Ogitsu, the NPNEQ Center director and a physicist specializing in computational physics in Livermore’s Physical and Life Sciences Directorate, says, “Now that we have the computing power we need, our vision is to develop a first-principles, nonperturbative simulation framework that provides material-specific predictions and guidance for experimental efforts. We are using an experiment–theory approach. Our goal isn’t to develop the materials themselves, although that is an exciting prospect. Our goal is to help many industries, educational institutions, and other organizations working to design, synthesize, and characterize new materials by providing easy-to-use tools to handle the complex, convoluted data that comes from material experiments.”

Powerful Computers Matched to New Software

The software developed by the NPNEQ Center is specifically designed to take advantage of the ongoing transition to exascale computing. This powerful combination of software and exascale supercomputers will allow researchers to perform theoretical modeling at extreme scales and complexity, and to use these models to guide and interpret corresponding experimental efforts. Data from experiments will be fed back into the software to refine and validate the simulation approaches to continually improve the software.

The Center researchers are currently tapping into the power of Lawrence Livermore’s Lassen HPC system for this effort. Most recently ranked number 17 on the November 2020 Top500 list of most powerful computers, Lassen has a peak performance of 23 petaflops (23 quadrillion calculations per second). The Laboratory is also home to the Sierra HPC system, ranked number 3 with a peak performance of 125 petaflops, and will continue to dominate the Top500 list when El Capitan starts up in 2023, with an anticipated performance of 1.5 exaflops (1.5 quintillion calculations per second).

Livermore’s most current HPC systems use a combination of central processing units (CPUs) and graphics processing units (GPUs). HPC designers looked at earlier systems, optimized by visionary computer scientist Francois Gygi, that used CPUs and then borrowed GPU technology traditionally used to accelerate graphics. The result was hybrid CP–GP HPC systems that enabled a nearly hundred-fold increase in computing power.

But to get the most out of the hardware and to run at full capacity, Livermore scientists need optimized codes that can take advantage of the complex system configuration to process very complicated data. Lawrence Livermore’s experts in software development for exascale, Xavier Andrade and Alfredo Correa, bring unique expertise in code optimization and RT-TDDFT software development. While RT-TDDFT software was available in the past, it was not commonly used in part because it took more time and expense than DFT. But now the NPNEQ Center is working to broaden the use of TDDFT software, making it faster and easier to use.

Supporting BES Goals in Tandem with Other Centers

The NPNEQ Center supports BES interest in fundamental investigations in the fields of material and chemical sciences by creating software that will work for a variety of applications on a range of potential materials that optimizes the full power of HPC systems. With code that is easier to run, users will be able to prepare simulations and make predictions about complex systems for a number of applications. The Laboratory’s BES Lab Coordinator, Eric Schwegler, says, “The BES Computational Materials and Chemical Sciences program was established with the goal of developing user-friendly, open-source software that can benefit many researchers. The NPNEQ Center is directly contributing to this goal by enabling the use of sophisticated TDDFT codes that take full advantage of the latest HPC platforms.”

The NPNEQ Center is one of several BES centers that complement one another. Livermore Laboratory also participates in the Center for Predictive Simulation of Fundamental Materials, led by Oak Ridge National Laboratory, which is focused on the development of quantum Monte Carlo-based approaches. The Laboratory also participates in the SPARC-X Center, led by Georgia Institute of Technology, which focuses on a computational framework for performing highly scalable DFT calculations.

With easy-to-use software tools, the power of HPC will become more accessible to researchers, paving the way to transformative technologies such as novel quantum materials and the design of materials with specific functionality. Software designed by these centers can be expanded to support new investigations in nanomaterials, photoelectric science, photovoltaics, hydrogen production, and condensed matter physics.

—Karen Rath

Key Words: Center for Non-Perturbative Studies of Functional Materials under Non-Equilibrium Conditions (NPNEQ), time-dependent-density functional theory (TDDFT), quantum materials.

For further information contact Tadashi Ogitsu at (925) 422-8511 (ogitsu1@llnl.gov).