High-performance computing (HPC) is a relatively small, niche market in the global computing landscape, but its importance to scientific discovery is outsized. Over the years, models and simulations run on HPC systems at the national laboratories and in industry have led to breakthroughs in areas ranging from national security and physics to biology and engineering. With the advent of exascale computing, including Lawrence Livermore’s El Capitan system, one of the world’s fastest verified supercomputers, additional paths to scientific discovery have emerged, providing an unprecedented capability for studying complex physical phenomena.
The Department of Energy (DOE) helped pave the road to exascale. DOE’s more than $1.2 billion CORAL2 procurement led to the delivery of three DOE exascale machines: Frontier at Oak Ridge National Laboratory, Aurora at Argonne National Laboratory, and El Capitan at Livermore. Concurrently, DOE’s 8-year, $1.8 billion Exascale Computing Project (ECP) partnered the National Nuclear Security Administration and Office of Science national laboratories with external collaborators in academia and industry to develop the applications and essential software technologies needed to make exascale systems realizable, functional, and integrated with the hardware. (See S&TR, January/February 2021, The Exascale Software Portfolio.)
ECP ushered in a transformative era in HPC, and although the technical work on ECP ended in March 2024, researchers and software engineers have continued to pursue avenues for building, enhancing, and sustaining the software ecosystem that is key to executing exascale, and, more broadly, advanced HPC applications and workflow. The effort required to create and maintain such a software network is nontrivial. Once software is deployed, developers have an ever-growing to-do list that includes evaluating performance, identifying and fixing bugs, addressing security vulnerabilities, managing version control, making improvements, and updating documentation. The idea that one developer will experience every issue and provide every possible fix for a software tool is unrealistic—use cases are varied as are the computing environments. Yet, many hands make light work, which is why open-source software, enabling developers anywhere to use, explore, and contribute to a software’s source code, has become a lifeline for ensuring software viability.
“Open”-ing Collaborative Doors
Livermore has long supported open-source software, realizing that communities built around a shared commitment to software development can dramatically improve foundational tools. (See S&TR, January/February 2018, Ambassadors of Code.) The Laboratory’s open-source software offerings include math libraries, compilers, and operating systems, among others. “The value from software is that people use it, and as more users adopt it for their specific work, the more likely they will want to improve it and make it better,” says Livermore computer scientist and Distinguished Member of Technical Staff Todd Gamblin, who along with HPC expert Christian Trott, a Distinguished Member of Technical Staff at Sandia National Laboratories, set out to build an open-source community to both sustain and grow the collection of critical HPC software post-ECP.
Gamblin and Trott both led large projects within the ECP software technologies portfolio: the Spack package manager for automation of building, testing, and deploying software for HPC systems (Gamblin); and the Kokkos performance portability programming library, which enables programming code to be written once but run on existing and future hardware architectures (Trott). Through ECP, both software technologies, available through GitHub, matured as part of the HPC software ecosystem. “Spack and Kokkos were widely adopted projects within ECP. We established the most active developer communities and received many contributions from outside the laboratories, so we felt we could use this approach to motivate other projects,” says Gamblin.
Using Spack as an example, the project now contains more than 8,900 package “recipes” that describe how to build and install a piece of software. Each recipe represents accumulated knowledge from contributors who have tried, struggled, and succeeded to build the package on different platforms. At this point, Livermore maintains less than 10 percent of the packages. “We can leverage open source to help carry the maintenance burden, which we could not do alone,” says Gamblin. (See S&TR, March 2023, Expediting Research with Spack) Wanting to continue this momentum and encourage future adoption and support for all HPC projects, Gamblin and Trott took steps to establish an open-source software foundation, one specific to HPC.
A Neutral Solution Reduces Risk
Software foundations have existed for decades as a way to increase collaboration around open-source products. In addition to providing legal, outreach, and overall logistical support, foundations provide a neutral home for projects and facilitate neutral governance. Gamblin says, “Foundations don’t fund all the development effort that goes into software, but they provide a structure to facilitate people becoming involved and being a part of the decision making strategy.” Trott adds, “Foundations give everyone equal footing, which makes it easier for users to contribute and to justify funding from their home organizations to work on the software, since it is no longer tied to a single entity.”
The Linux Foundation, a nonprofit technology consortium, is one of the largest software foundations of its kind, providing an overarching umbrella organization for many subfoundations, including those related to AI, cloud computing, edge computing, and now HPC. To establish the High Performance Software Foundation (HPSF), Gamblin and Trott took an approach that other, thriving Linux subfoundations, such as the Cloud Native Computing Foundation (CNCF), used to get started. “We took Spack and Kokkos and rallied other projects around them, similar to how CNCF originated with Kubernetes (an open-source container orchestration system created by Google), but at a much smaller scale,” says Gamblin. “For more than a year, we worked with colleagues at the national laboratories and with the National Nuclear Security Administration and the Office of Science legal teams to understand the complexities of joining a foundation, gathering consensus, and establishing best practices and precedent.”
Officially launched in 2024, HPSF is its own legal entity—a 501(c)(6) industry consortium—wherein the Linux Foundation maintains the trademark policy for all the projects and holds each project’s assets, such as the GitHub account or the website, but where copyright for each project is retained by contributors. HPSF was started with nine initial projects and has since increased to 16. “Allowing projects to be open source and part of a foundation makes each one more stable, more sustainable, and higher quality because more total resources are working to improve and maintain them. It also helps us increase the scope of each product,” says Trott. “In turn, contributing organizations are also much more likely to suggest additional projects to meet the particular needs of their customers, and that helps grow the user base and gives more visibility to the HPSF community.”
HPSF Takes Shape
HPSF membership falls within three categories—premier, general, and associate—and includes academic and industry partners such as AWS (Amazon Web Services), HPE (Hewlett Packard Enterprise), and NVIDIA. Each premier member organization has a single representative who serves on the HPSF governing board, which provides general oversight for the entire foundation. Livermore and Sandia plus industry partners AWS and HPE were founding premier members and helped get the organization and its governance off the ground. Microsoft joined as a premier member in 2025. In conjunction with the board, the foundation’s Technical Advisory Council, made up of representatives from different general member organizations and external stakeholders, coordinates the technical direction and priorities for all HPSF projects.
“We also have individual working groups,” says Gamblin, “which are the heart of HPSF in that these specific teams collaborate on tools and resources that then can become useful for all the other projects in the community.” HPSF has three established working groups. The benchmarking working group helps establish and track metrics for measuring the performance and functionality of all HPSF projects over time. The outreach working group focuses on ways to bring visibility to HPSF through events and communications that raise awareness and promote engagement. Finally, the CI/CD (continuous integration/continuous delivery) working group enables dependable automation and helps sustain software quality across large-scale development efforts. Gamblin adds, “Continuous integration was considered one of the top needs for all the projects, and this group has developed tools that allow projects to be securely run on our HPC machines and those of partnering institutions, which has traditionally been a challenge when using open-source software.”
HPSF membership increased to 23 organizations in 2025 and includes a greater mix of national laboratories, academia, and industry. New projects are also in process to become part of the foundation. Livermore’s Flux software—a scalable, next-generation workflow management tool that maximizes HPC resource utilization—was accepted in June 2026. (See S&TR,
July/August 2022, Optimizing Workflow with Flux) “Making Flux part of HPSF shows that we are serious about wanting others to use the software, which ultimately can benefit the entire community,” says Gamblin.
In 2025, HPSF was recognized by HPCwire with the “Readers’ Choice Award for Best HPC Collaboration Between Government and Industry.” Presented at the SC25 conference, held in St. Louis, Missouri, the award acknowledges the goal of HPSF—to facilitate the productive use of HPC systems now and in the future by bringing together a community that can use, improve, and sustain fundamental software. The HPSF annual conference is one way the foundation fosters this community, and it also recently contributed to the Sustainable Horizons Institute, which runs a program to match interns to projects in STEM (science, technology, engineering, and mathematics), as a way of broadening the HPC workforce pipeline.
HPC is an evolving field, and an essential aspect of its continued utility will be the sustainability of its software base. The introduction of AI and machine learning, advancements in heterogenous system architectures, and the increasing pressures of national needs, demand that HPC and the requisite software remain agile and capable. “The whole point of this effort was to facilitate partnerships with non-DOE entities, including with industry, as outside contributors are critical to sustainability and innovation,” says Trott. “Joining forces to address key challenges, including developments in AI and hardware evolution, is critical to keeping up with the changing world and improving the quality and the maturity of HPC projects long term. Making software projects we rely on more stable, more widely used, and better supported, even by AI tools, creates better products for everyone.”
—Caryn Meissner
For further information contact Todd Gamblin (925) 422-9319 (gamblin2 [at] llnl.gov (gamblin2[at]llnl[dot]gov)).
