Lawrence Livermore National Laboratory

Back to top

Back to top

Livermore researchers develop a system to help analysts find evidence of illicit nuclear weapons proliferation using open-source information.
In 2009, VisIt ran on 12,000 processing cores to render a data set consisting of 1 trillion grid points—a feat that was extraordinary for its time and is still not commonplace today. Executing this large-scale mesh visualization is one example of VisIt’s ability to meet the scalability demands of Livermore’s simulation codes.

Visualization Software Stands the Test of Time

In the decades since Lawrence Livermore’s founding, the  technology used in pursuit of the Laboratory’s national security mission has changed over time. For example, studying scientific phenomena and predicting their behaviors require increasingly robust, high-resolution simulations. These crucial tasks compound the demands on high-performance computing (HPC) hardware and software, which must continually be maintained, upgraded, or replaced.

Perhaps no other software project has adapted to such evolving needs as well as Livermore’s VisIt, which has become a trusted tool since its development began in 2000. VisIt—a portmanteau of visualize and it—is open-source software that enables researchers to rapidly visualize, animate, and analyze scientific simulations. It is the Laboratory’s go-to software for scalable visualization as well as graphical and numerical analysis of both simulated and experimental data sets.

Project leader Eric Brugger, who has worked on VisIt since its inception, states, “Our software is used broadly throughout the Department of Energy (DOE) complex, at universities, and at other supercomputing centers. It’s running on some of the world’s most powerful machines. VisIt strikes a balance between application support and capability development that appeals to the larger HPC community.”

Diagram of computational neural network
Rayleigh–Taylor instability occurs when fluids of different densities mix, such as in inertial confinement fusion. Livermore’s hydrodynamics codes depend on visualization software like VisIt to reveal contours, flows, eddies, and other physical processes on large-scale meshes. These images were generated with VisIt on Livermore’s Blue Gene/L (2006) and Sierra (2018) supercomputers and demonstrate VisIt’s resolution improvements during the 12-year period. The 2018 Sierra calculation (bottom) was 60x the size of the 2006 calculation (top), revealing greater detail.

A Dynamic History

Initially developed for the DOE’s Advanced Simulation and Computing program, VisIt is in many ways a microcosm of the Laboratory’s tradition of technological advancement through collaboration. It grew from a prototype in 2000 to its first major release in 2002. The project was soon released as open source and won an R&D 100 Award in 2005 (S&TR October 2005, pp. 10–11). Incremental feature releases followed regularly to enhance VisIt’s performance and capabilities.

The Livermore team originally shared project-hosting responsibilities with Oak Ridge National Laboratory (ORNL), the National Energy Research Scientific Computing Center (NERSC), and commercial company Intelligent Light. While Livermore handled VisIt’s developer documentation and website, nightly testing, and user hotline, ORNL managed issue tracking and the user email listserv, and Intelligent Light contributed the user documentation, forum, and an anonymous Subversion mirror. In addition to hosting VisIt’s nightly testing dashboard, NERSC hosted the source code, as it did for much of the DOE Office of Science’s open code development in the early 2000s.

As VisIt’s source code grew in size and functionality, the team moved it from a ClearQuest version-control platform behind Livermore’s firewalls to Subversion, hosted at NERSC. Today, most of the software’s services reside on the open-source platform GitHub—a move that enables more community development, interaction, and visibility than ever before. According to team member Mark Miller, the GitHub migration led to many workflow enhancements in continuous integration testing and issue tracking. “GitHub has improved the way we develop,” he says.

Brugger emphasizes that VisIt’s development has always been a team effort. He notes, “The Laboratory’s environment makes stability and collaboration possible.” Alongside Brugger and Miller, the current team of computer scientists and software developers includes Kathleen Biagas, Cyrus Harrison, Matt Larsen, Alister Maguire, and Eddie Rusu. Livermore alumni include Hank Childs (original VisIt software architect and now a contributor affiliated with the University of Oregon), Sean Ahern (formerly of ORNL, now with ANSYS), Jeremy Meredith (formerly of ORNL, now with Google), Brad Whitlock (currently at Intelligent Light), and Kevin Griffin (formerly of Lawrence Livermore, now at NVidia). Other collaborators include Dave Pugmire (ORNL), Allen Sanderson (University of Utah), and Gunther Weber (Lawrence Berkeley National Laboratory).

The team’s expertise, passion for customer service, and commitment to excellence have ensured VisIt remains relevant to DOE goals, which in turn has earned sustained funding. Miller points out, “Visualization tools and the people who support them are at the nexus of vastly different data-handing requirements coming together in a common tool to help advance science. We have to be creative in our solutions. There’s never a dull moment.”

Livermore researchers develop a system to help analysts find evidence of illicit nuclear weapons proliferation using open-source information.
Livermore researchers use VisIt with a multiphysics code to simulate propellant flow in a firearm suppression device. In 2019, the Lassen supercomputer produced this image during a run of 1.1 billion grid points. After refining the suppressor’s design through iterative simulations, the Laboratory created a prototype that performed as required. For U.S. soldiers using automatic weapons in training and combat situations, the suppressor design minimizes blowback and reduces the hearing loss associated with operation near the ear.

Foresight and Luck

From the outset, VisIt was designed to support future evolution. Harrison says, “No software solution is perfect, but a solid foundation was established early on.” Miller adds, “VisIt’s long-term success is due to a mix of foresight and luck regarding how technology would change.” For example, the team has kept pace with the latest versions of programming models, as well as containerization technology that modularizes and automates the software’s continuous integration testing and deployment.

Extensibility and scalability are also key reasons for VisIt’s longevity. The tool’s capabilities can quickly and easily expand through the use of plug-ins, which allow new features to be added without rewriting the entire code base. The team has built more than 250 plug-ins to handle a wide range of data formats, operators, and plots. “Plug-ins have given VisIt flexibility and widespread applicability,” explains Miller, who notes that careful management of third-party dependencies has also been an important part of the decision process. For instance, Brugger states, “We leverage the open-source VTK (Visualization Toolkit) and OSPRay software libraries to maximize VisIt’s modeling and image-rendering features.”

Not every development choice is obvious or easy. In the early 2000s, developing simulations in the C++ programming language was challenging due to wide variation in language features supported in compiler programs, which caused issues with applications’ portability and performance. Miller recalls, “We heavily constrained how VisIt used C++ to minimize the chances that a compiler wouldn’t be able to compile VisIt. We built a fallback implementation if the compiler’s exception handling was problematic.” By the mid-2000s, C++ had matured and become more prevalent, so VisIt’s compatibility with it evolved accordingly. In general, Miller summarizes, “We have been conservative about introducing new dependencies and features, making room for them without requiring them. VisIt must work on Linux, macOS, Windows, and DOE HPC platforms, so we carefully manage any choices that can impact any of those environments.”

Ultimately, Brugger says, “We’ve made some good technology decisions that have paid off.” VisIt’s development scope broadened as the software gained traction among other researchers, both at the Laboratory and externally. Harrison, who joined the team in 2007, adds, “Seeing VisIt used in new ways helps us determine where to go next. We learn a lot from users.” The team’s experience and customer dedication have helped guide development of another Livermore software project called Ascent, which processes visualizations while a simulation code is running, also known as in situ.

Livermore researchers develop a system to help analysts find evidence of illicit nuclear weapons proliferation using open-source information.

The Proof Is in the Picture

VisIt’s feature development is driven by the DOE’s mission, for which scalable, high-resolution computer simulations on HPC systems are of paramount importance. For instance, VisIt has long enabled hydrodynamics code teams to explore computationally intensive phenomena. One such phenomenon is the Rayleigh–Taylor instability that occurs when fluids mix—a key process in inertial confinement fusion. To visualize it, VisIt’s development focuses on parallelization for large data sets, implementation of custom data analysis routines, handling of non-standard data models, and a flexible, easy-to-use interface.

Among its many features, VisIt enables users to analyze structured and unstructured meshes, animate two- and three-dimensional data, and manipulate a range of geometric objects. The tool’s visualization algorithms easily process large data sets serially or in parallel, and users can query different points and variables within the results. VisIt’s latest major release included compatibility with new file formats, upgrades to the user interface and documentation, and support for graphics processing units, which accelerate a machine’s computational performance.

Users can run VisIt on personal laptops, supercomputers, and everything in-between—including the more than two dozen HPC systems the Laboratory has brought online since 2000. The software is running on Livermore’s Sierra and ORNL’s Summit supercomputers. “Prioritizing portability to as many platforms as possible has come with certain tradeoffs,” states Brugger. “But it was the right decision given the increasing number of groups we work with and their diverse applications.”

As HPC processing power turns the corner toward a new era, VisIt is a key post-processing tool in the DOE’s Exascale Computing Project and Livermore’s RADIUSS project (Rapid Application Development via an Institutional Universal Software Stack) (S&TR, February 2021, The Exascale Software Portfolio). “We are excited to participate in these project portfolios,” says Harrison. When the Laboratory’s first exascale supercomputer, El Capitan, comes online in 2023, he adds, “VisIt will be ready.”

With 20 years of service behind it, VisIt continues to offer meaningful capabilities to the scientific community and remains Livermore’s flagship visualization software. Back in 2000, Brugger did not imagine he would still be developing VisIt today. He says, “The work continues to be challenging and interesting. Technology always changes, so there’s no time to get bored.”

—Holly Auten

Key Words: C++, data analysis, Department of Energy (DOE), high-performance computing (HPC), inertial confinement fusion, Intelligent Light, National Energy Research Scientific Computing Center (NERSC), Oak Ridge National Laboratory (ORNL), open-source software, programming, Rayleigh–Taylor instability, scientific code, simulation, VisIt, visualization.

For further information contact Eric Brugger at (925) 423-1293 (