S&TR | August 2020

Contents

The Laboratory in the News

Working toward a World Free of Chemical Weapons

Commentary by Huban A. Gowadia

The Worldwide Effort to Ban Chemical Weapons

The Laboratory’s Forensic Science Center is an important ally in the international endeavor to prohibit chemical warfare agents.

The Sierra Era

Lawrence Livermore’s fastest supercomputer runs mission-driven simulations on specially optimized hardware and software.

Tapping the Earth’s Heat for Clean Energy

Working in a deep underground tunnel, researchers are learning how to turn heat from impermeable rock into clean, low-carbon-emitting power.

Drop by Drop: A Promising Method for Printing Metal Parts

A novel additive manufacturing technique applies tiny droplets of molten metal layer by layer.

Awards

Patents

Staff

About S&TR

The Laboratory in the News

Experiment Improves Predictions of Uranium Dispersion

Researchers from Lawrence Livermore and the University of Illinois at Urbana–Champaign have demonstrated that the behavior of uranium after a nuclear incident is incompletely predicted by computational fallout models, which approximate the physical and chemical processes occurring when the nuclear fireball condenses. In particular, uranium oxide is assumed to condense in its most stable form after cooling below its boiling temperature. However, the study, published in the April 1, 2020, edition of Analytical Chemistry, finds that kinetically driven processes in a system of rapidly decreasing temperature can result in substantial deviations from chemical equilibrium.

Funded by the National Nuclear Security Administration’s Office of Defense Nuclear Nonproliferation Research and the Defense Threat Reduction Agency, the team synthesized uranium-oxide nanoparticles using a plasma flow reactor under controlled conditions of temperature, pressure, and oxygen concentration. The team also developed a laser-based diagnostic to detect uranium-oxide particles as they formed inside the flow reactor. Using this approach, the researchers gathered direct experimental evidence for a change in the molecular composition of uranium-oxide condensates as a function of oxygen concentration.

Livermore nuclear scientist and principal investigator Kim Knight, says, “This work provides the first, detailed experimental insights that help explain the longstanding problem of why uranium can exhibit variations in volatile behavior during nuclear fireball condensation.” Livermore’s Batikan Koroglu, lead author of the research paper, adds, “This study will improve our ability to predict uranium’s multiphase transport in nuclear incident scenarios.”

Contact: Batikan Koroglu (925) 422-1867 (koroglu1@llnl.gov).

Identifying COVID-19 Antibody Sequences

Using the Laboratory’s advanced supercomputing resources and a machine-learning computational platform, researchers have computationally designed antibodies targeting SARS-CoV-2, which have been successfully synthesized and have shown promising activity in in-vitro experiments. The initial research results appeared online in the April 10, 2020, edition of BioRxiv.

In just 22 days, using the SARS-CoV-2 protein sequence and known antibody structures for SARS-CoV-1 (a similar coronavirus that causes Severe Acute Respiratory Syndrome), the Laboratory team, led by data scientists Dan Faissol and Thomas Desautels, used a computational platform combining machine learning, bioinformatics, experimental data, structural biology, and molecular simulations to drastically narrow down the possible antibody designs predicted to target SARS-CoV-2. The Laboratory’s Corona and Catalyst supercomputers performed nearly 180,000 free-energy calculations of candidate antibodies with the SARS-CoV-2 Receptor Binding Domain.

The team’s first designs were subsequently synthesized and evaluated, with one showing promising activity. A second iteration of computational designs has now yielded additional and improved molecules. Several of these have demonstrated binding activity in in-vitro SARS-CoV-2 assays, including one that has also shown neutralization activity. This design-first approach to antibody discovery could lead to a fully computational and rapid design of targeted antibody therapeutics for pandemic response. For more on the Laboratory’s COVID-19 research, visit llnl.gov/coronavirus.

Contact: Dan Faissol (925) 423-2544 (faissol1@llnl.gov).

Second Skin Protects against Multiple Agents

A multi-institutional team led by Lawrence Livermore scientist Francesco Fornasiero has developed a smart, breathable fabric designed to protect the wearer against biological and chemical warfare agents. In addition to protecting military personnel, this material could also be useful in clinical and medical settings. The work was published online in the April 27, 2020, edition of Advanced Functional Materials and represents the successful completion of a key milestone for a project funded by the Defense Threat Reduction Agency.

Personnel safety garments must contain materials that provide protective qualities to the wearer, but those same qualities also limit the materials’ breathability. Fornasiero says, “We made our smart material both breathable and protective by combining two key elements: a base membrane layer made from trillions of aligned carbon nanotube pores and a threat-responsive polymer layer grafted onto the membrane surface.”

As part of the research, the team demonstrated that the moisture vapor transport rate through carbon nanotubes (graphitic cylinders with diameters more than 5,000 times smaller than a human hair) is high and increases with decreasing tube diameter. In addition, these tubes are small enough to block biological threats. To add protection against chemical hazards, which are smaller in size than biological ones, a layer of polymer chains was grown on the membrane surface that reversibly collapses when in contact with the chemical threat, temporarily blocking the pores.

The enhanced properties of this material could improve the thermal comfort of the user and greatly extend the wear time of protective clothing, whether in a hospital or on the battlefield. In the next phase of the project, the team aims to incorporate on-demand protection against additional chemical threats and make the material stretchable for a better body fit, thus more closely mimicking the human skin.

Contact: Francesco Fornasiero (925) 422-0089 (fornasiero1@llnl.gov).

Commentary: Working toward a World Free of Chemical Weapons

Huban A. Gowadia

Principal Associate Director for Global Security

In recent years, international incidents involving chemical weapons have made headlines. Kim Jong Nam, the half-brother of current North Korean leader Kim Jong Un, was murdered in 2017 with VX nerve agent at Kuala Lumpur International Airport in Malaysia. In 2018, former Russian military officer Sergei Skripal and his daughter were poisoned with a Novichok nerve agent in the United Kingdom. In Mideast conflicts, the use of chlorine, sulfur mustard, and sarin as weapons in the Syrian Civil War has been confirmed by the United Nations. Such incidents illustrate the need for outstanding forensic expertise and capabilities in determining what chemical was involved; what group, individual, or nation was responsible; where it came from; and how it was manufactured.

As described in the feature article titled "The Worldwide Effort to Ban Chemical Weapons," Lawrence Livermore’s Forensic Science Center (FSC) provides valuable support to the worldwide effort to eliminate chemical weapons through the Chemical Weapons Convention (CWC) and its Organisation for the Prohibition of Chemical Weapons (OPCW). Founded in 1991, the FSC drives research and development as well as real-world sample analysis activities. The center supplies decades of forensic analytical expertise and state of-the-art instrumentation to counter terrorism, aid domestic and international law enforcement agencies, and verify compliance with international treaties, including CWC.

The urgent need for an international agreement banning the use of chemical weapons was illustrated during the Iran–Iraq war (1980–1988) when nerve agents and mustard gas employed by Iraqi forces caused tens of thousands of casualties. CWC entered into force in 1997, outlawing the development, production, acquisition, storage, and use of chemical weapons, which include choking, blister, blood, and nerve agents, as well as other toxic chemicals. One hundred ninety-three nations have ratified the treaty, and 98 percent of chemical weapon stockpiles—more than 71,000 metric tons—have been verifiably destroyed. This tremendous progress has been achieved through inspections of both government and commercial facilities and a worldwide network of analytical laboratories certified by OPCW for identifying chemical warfare agents contained in virtually every kind of evidence.

Since becoming an OPCW-certified laboratory for environmental samples in 2003, the FSC has made important contributions with extremely sensitive analytical techniques and decision-making processes required for quick and effective response to any incident involving toxic chemicals. In 2017, the FSC gained an additional OPCW certification for analyzing blood and urine, and the center’s researchers continue to develop techniques for identifying chemical warfare agents in biomedical samples. The FSC is one of only 21 environmental laboratories and 18 biomedical laboratories certified by OPCW.

Another critical task for the FSC is training personnel in at-risk nations on response strategies for incidents involving toxic chemicals, whether the result of a chemical weapon or an industrial accident—a derailed train car transporting chemicals, for example. As described in the article, the training combines lectures, demonstrations, and practical exercises that cover a broad range of topics. Advanced training at specialized OPCW facilities includes simulated chemical incidents with participants wearing protective clothing and using detection equipment.Strengthening the available technical capabilities for detecting and characterizing activities indicative of chemical weapons production and use are critical to national and global security.

At Lawrence Livermore, we also apply advances in science and technology to counter nuclear proliferation and biological threats to human health. Threats to the nation are very real and can come as a surprise, as the COVID-19 pandemic makes clear. Our scientists and engineers are engaged in many responsive efforts to accelerate scientific discovery of the virus, develop improved detection technologies, and advance medical countermeasures as part of the national effort. (Visit llnl.gov/coronavirus for more information.)

The Worldwide Effort to Ban Chemical Weapons

For thousands of years, chemical weapons have been used in warfare. In 600 B.C., the Athenian military poisoned the water supply of the besieged city of Kirrha. During World War I, an estimated 124,000 tons of chlorine, phosgene, mustard, and other chemical warfare agents (CWAs) were released by German forces, killing more than 90,000 soldiers and injuring nearly a million people. As part of the Cold War, both the United States and the Soviet Union maintained stockpiles of CWAs. More recently, events in the Mideast have shown that the threat of chemical weapons continues.

Through an international effort to permanently eliminate existing chemical weapons and prevent their re-emergence, the Chemical Weapons Convention (CWC) treaty entered into force in 1997 and prohibits the development, production, acquisition, stockpiling, or transfer of CWAs. The treaty is implemented through the Organisation for the Prohibition of Chemical Weapons (OPCW), headquartered in The Hague, Netherlands. To support its mission, OPCW relies on a global network of accredited laboratories to analyze samples from suspected chemical weapons production facilities. OPCW laboratories are located in Finland, Sweden, Spain, Singapore, the United Kingdom, the United States, Germany, India, Russia, the People’s Republic of China, South Korea, and France, among others.

OPCW requires that samples taken by its inspectors be analyzed by two OPCW-designated laboratories, and U.S. legislation requires that all samples collected in the United States be analyzed within the country. Only two laboratories in the United States are certified for the analysis of environmental samples collected by OPCW: the Laboratory’s Forensic Science Center (FSC) and the U.S. Army Combat Capabilities Development Command Chemical Biological Center (formerly known as the U.S. Army Edgewood Chemical Biological Center). The FSC has also earned its OPCW certification for biomedical materials and can thus analyze samples containing any mixture of both sample types.

In addition to supporting OPCW, the FSC performs assessments for homeland security, law enforcement, and intelligence agencies. For example, the FSC partners with the Federal Bureau of Investigation for chemical, radiological, nuclear, and explosive materials analysis. “The FSC has two intertwined missions,” says the center’s Director Audrey Williams. “The first is performing cutting-edge research, the second is responding to the urgent needs of our sponsors.”

A Dynamic Environment

CWC’s controls on toxic chemicals and their precursors are listed on three schedules according to their toxicity, military and commercial utility, and risk. Many chemicals of concern have legitimate civilian uses, so industrial facilities as well as government sites are subject to OPCW inspections.

When OPCW’s laboratory accreditation program began in 2001, the U.S. State Department requested that the FSC seek certification because of the Laboratory’s strong physical security and environmental controls, as well as the center’s recognized technical experience with CWA and expertise in analyzing trace levels of unknown substances, especially nuclear materials. Indeed, the FSC pioneered nuclear forensic analysis and remains the principal U.S. laboratory for analyzing possible nuclear and radiological contraband. (See S&TR, July/August 2018, The Case for Modern Forensic Science)

Armando Alcaraz, a chemist in Lawrence Livermore’s Forensic Science Center (FSC) prepares samples for analysis using gas chromatography–mass spectrometry. (Photo by George Kitrinos.)

FSC staff are experts in organic, inorganic, analytical, and nuclear chemistry, environmental and biological sample analysis, and medical countermeasures. Williams acknowledges the challenge in finding people with the needed mix of skills. She says, “Working at the FSC requires not only strong technical expertise, but also an ability to think on one’s feet under severe time constraints and in a team environment.” As part of their duties, the staff of approximately 25 works to improve detection methods. FSC analysts, for example, sometimes discover that standard sample preparation and analysis methods are lacking, and so must develop new identification protocols. These efforts have included novel techniques to analyze blood and urine and complex extraction protocols for difficult soiled materials.

To maintain OPCW accreditation, laboratories are required to participate in extremely challenging annual proficiency tests. The environmental tests typically contain dilute amounts of CWAs, precursor chemicals, degradation materials, and compounds that can hinder successful analysis. For each test, one OPCW-designated laboratory volunteers to formulate the samples and another grades the findings of participating laboratories. FSC staff spend long hours during 15 consecutive days each year isolating possible reportable compounds. “Testing is an immensely stressful time, but it is also intellectually satisfying and rewarding,” says Williams. Each laboratory must maintain a three-year rolling average of at least two “A” grades and one “B.” Recently, the FSC earned its 10th consecutive “A” grade.

Extending FSC Expertise

Biologist Todd Corzett notes that several years ago, in light of growing evidence of CWAs being used in Mideast conflicts, OPCW recognized the importance of finding biomedical signs of exposure to a CWA. OPCW assembled a worldwide laboratory network modeled after the one for environmental samples but focused this time on analyzing blood and urine. In addition to the FSC, the U.S. Army Combat Capabilities Development Command Chemical Biological Center, Centers for Disease Control and Prevention, and U.S. Army Medical Research Institute of Chemical Defense are OPCW biomedical-designated laboratories. All certified OPCW laboratories equipped for biomedical sample analysis must pass an annual proficiency test, similar to that for environmental accreditation.

Corzett explains that exposure to CWAs leaves behind unique biomarkers that are quickly excreted in urine but can remain in blood plasma for up to 90 days. Concentrations of biomarkers are often lower than CWA concentrations found in environmental samples—5 to 10 parts per billion instead of several parts per million, respectively.

FSC experts work with biomedical samples in a Biosafety Level-2 laboratory, which features strong safety and environmental engineering systems as well as personnel protection. When analyzing blood plasma, the FSC team uses liquid chromatography–mass spectrometry to search for biomarkers called adducts, molecular complexes that form when a CWA binds to a particular protein or enzyme. For example, butyrylcholinesterase (BChE) is a protein found in blood plasma that when exposed to one of a dozen nerve agents produces a persistent adduct. Corzett uses magnetic beads coated in antibodies to extract BChE from blood plasma for biomarker analysis.

FSC biologist Todd Corzett prepares a solution for detecting any of a dozen nerve agents in blood plasma. (Photo by George Kitrinos.)

Other proteins form adducts upon exposure to different CWAs, and FSC scientists look for those, as well. Corzett notes that knowing the clinical symptoms (for instance, twitching or skin blistering) of the individuals from whom the blood or urine samples were taken can help guide detection protocols, but adds that this information is not always available. “We have to be ready for anything,” he says. “Unlike proficiency tests, real-world samples are often collected and transported in less than ideal conditions, making subsequent analysis more difficult.”

Advanced Equipment Is Critical

FSC staff take advantage of the latest advances in analytical instrumentation that can isolate and identify increasingly more minute quantities of CWAs and other compounds. According to FSC Deputy for Operations Carolyn Koester, an ever-widening family of instrumentation includes gas, liquid, and ion chromatographs coupled with a variety of mass spectrometers; nuclear magnetic resonance (NMR); and infrared and Raman spectrometers.

A recently acquired best-in-class liquid chromatographic tandem mass spectrometer has proven essential for analyzing biomedical samples. The instrument achieves unprecedented sensitivity and fast identification of adducts and other biomolecules. “We’re lucky we work with sponsors who understand the benefits of state-of-the-art instrumentation,” says FSC analytical chemist Brian Mayer. “They recognize that the best equipment helps us get the answers they need. Five years ago, I never imagined we could see down beyond the picogram (10–12 grams) level, but it’s now routine.”

On the other hand, instruments’ heightened sensitivity can pose a serious challenge. “We can now detect tens of thousands of different compounds in a single sample, which can be overwhelming,” says Mayer. Williams adds that detection limits are now so low “we can detect nefarious activity that took place years ago, as a trace amount of the target chemical is still there.”

The search for butyrylcholinesterase (BChE) adducts starts with adding magnetic beads coated in BChE antibodies to blood plasma for biomarker analysis. The BChE adducts are then extracted with a magnet, “digested” with pepsin, and analyzed with liquid chromatography–mass spectrometry (LC–MS).

Clues for Chemical Attribution

The FSC has made significant advances not only in detecting CWAs, but also in chemical attribution signatures—trace amounts of synthesis precursors and byproducts, impurities, degradation products, and metabolites, which can provide clues to the likely source of reagents, type of synthesis equipment, or production method used. Williams says, “Even if we don’t see a listed chemical warfare agent in a sample, the compounds we identify could be linked to its preparation.”

Some CWAs can be made on a small scale with methods that would be difficult to scale up. Likewise, simpler methods typically require chemicals that are strictly controlled, so a terrorist group might choose a more complex method that uses widely available compounds. FSC chemists can often determine which synthesis pathway was implemented, including starting reagents.

In a project sponsored by the U.S. Department of Homeland Security, FSC chemists, in collaboration with the Swedish Research Defence Agency, developed a model to attribute samples containing sulfur mustard to 1 of 11 possible synthesis methods. Samples made with all 11 routes were analyzed to extract chemical attribution signatures as the basis for the model. “We developed a machine-learning tool that could give analysts significant insight into how a sample was made. Computers can identify important features in the data that an analyst would simply never notice,” says Mayer.

Williams notes that a sample may contain a toxic chemical that is not on a threat list. As an example, in 2018 Sergei Skripal, a former Russian military officer and double agent for the United Kingdom’s intelligence service, and his daughter were poisoned in Salisbury, England, with a Novichok nerve agent, which was not on OPCW’s list of CWAs. Both recovered, as did a police officer hospitalized after exposure to the remnants of the toxic agent at Skripal’s residence. The U.S. State Department concluded that Russia was behind the poisoning. Four months later, two British citizens were hospitalized with Novichok poisoning and one died. British police hypothesized that both individuals had come across the original chemical that had been disposed of haphazardly.

Developing Countermeasures

An important FSC research thrust is developing countermeasures to stop the often-lethal effects of CWAs. Much of the research is funded by the U.S. Department of Defense, whose soldiers may be thrust into theaters of war where they could encounter CWAs. One concern is that an enemy could potentially vaporize an easily manufactured opioid and thereby incapacitate soldiers. Mayer points to a 2002 incident when Chechnyan terrorists took over a Moscow theater and held 850 hostages. Russian special forces vaporized what was believed to be an opioid to subdue the insurgents. All 40 of the insurgents were killed, as were more than 200 hostages.

With funding from Livermore’s Laboratory Directed Research and Development Program, FSC researchers in 2016 launched an integrated experimental and computational approach for developing antidotes to nerve agents and incapacitating agents such as fentanyl, a synthetic opioid pain medication that is highly addictive, as well as toxic at low doses. The challenge for FSC scientists is to develop medical countermeasures that can be injected intravenously or intramuscularly, are more effective than current countermeasures with fewer side effects, and have potential as a prophylaxis for soldiers entering combat.

A team led by Mayer and chemist Carlos Valdez used a variety of techniques including NMR to screen candidate compounds identified and then down-select options based on computational chemistry results. They arrived at a promising molecule called subetadex. The U.S. Department of Defense is currently funding the FSC to develop an advanced version of subetadex that would possess enhanced affinities for fentanyl and other synthetic opioids.

In a related effort, a team of scientists from the FSC developed LLNL-02, the first molecule that effectively crosses the protective blood–brain barrier to prevent long-lasting effects on the brain from nerve agents (see S&TR, June 2019, Nerve-Agent Antidote Shows Great Potential). A significant challenge for researchers developing nerve-agent antidotes is creating a drug that simultaneously protects both the body’s central nervous system and peripheral nervous system. The effort, headed by Valdez, was aided by a computer model that simulates the efficacy of potential antidotes. The speedy penetration of LLNL-02 into the brain has been demonstrated in guinea pigs.

First-responder trainees participate in a staged exercise wherein a smoke bomb mimics a vaporized nerve agent.

No Letup in Effort

In 2019, 98 percent of the world’s population was considered under CWC’s protection. As of August 2020, 98 percent of all chemical weapon stockpiles declared by possessor nations have been destroyed. However, the threat posed from the acquisition and use of chemical weapons by nations that are non-CWC signatories or terrorists remains. Clandestine production of CWAs has occurred in repurposed facilities (for example, pesticide manufacturing plants); specially built laboratories; and small, makeshift laboratories. Information on how to manufacture and weaponize CWAs has also become more accessible.

With their exceptional analytical expertise, FSC scientists have played a significant role in helping to enforce the tough tenets of CWC through OPCW’s aggressive inspection regimes and its worldwide laboratory network. FSC researchers are always mindful of the dangers of chemical weapons—witness Saddam Hussein’s killing of 5,000 Iraqis with chemical weapons in 1988, and a Japanese cult’s release of a nerve agent in a Tokyo subway that killed 13 people and injured thousands in 1995. Together with the recent incidents in the Mideast, these events have become an urgent reminder that the knowledge to make and weaponize CWAs cannot be taken back.

Two Decades of Enhancing Response to Chemical Weapons Use

In 2016, Islamic State fighters launched two chemical attacks involving chlorine and sulfur mustard near the city of Kirkuk in northern Iraq, killing a three-year-old girl and wounding nearly 600 people. The extremist group was thought to have set up a special chemical weapons unit composed of Iraqi scientists, who had worked for Saddam Hussein, and foreign experts. Afterwards, first responders from Iraq’s Ministry of the Interior were able to collect valuable evidence related to the attack—thanks in part to training from scientists at Lawrence Livermore’s Forensic Science Center (FSC).

As part of an effort to train personnel in at-risk nations to effectively respond to incidents involving toxic chemicals, FSC scientists travel to other countries, especially those in high-priority areas, to instruct first responders and medical personnel on essential protocol. The work is supported by the U.S. Department of State’s Chemical Security Program and Biosecurity Engagement Program. Through the Department of State, FSC staff collaborate with foreign government agencies to identify needs and deliver training, including “train-the-trainer” programs that enable local experts to pass on their knowledge.

Since 2013, FSC chemist Armando Alcaraz and colleagues have conducted first-responder training in Iraq, Lebanon, Kenya, Turkey, Malaysia, Jordan, and Yemen. The FSC has conducted eight training sessions with Iraqi civil defense experts, the latest in 2019. In the 2016 sulfur mustard attacks, Iraqi first responders had earlier participated in a mock field-training scenario where a sulfur mustard munition contaminated an area. The same team of first responders responded to the real attack in Kirkuk. The combination of classroom training and realistic field exercises provided the Iraqi responders with the skills they needed to effectively address the situation. The FSC received recognition from the Iraqi Ministry of the Interior for their efforts. “We potentially saved lives and kept responders safe from exposure,” says Alcaraz. “They were very appreciative of the training.”

In 2019, first responders from Latin American and the Caribbean attended the first regional training course for chemical warfare agent (CWA) sampling and analysis in contaminated areas. The five-day class, run by the Organisation for the Prohibition of Chemical Weapons (OPCW), was held in Bogota, Columbia, for participants from Argentina, Barbados, Belize, Brazil, Chile, Colombia, Costa Rica, the Dominican Republic, Guatemala, Ecuador, Mexico, Panama, Peru, and Saint Lucia. FSC scientists supported the course through the Instructor Development and Exchange Program administered by the OPCW Assistance and Protection Branch.

Alcaraz says, “We teach the basics: explaining what chemical warfare agents are and their effects, what methods are available for detection and decontamination, how to collect evidence, and how to use and remove chemical protective clothing.” He notes that the training, which sometimes includes the establishment of a forensic laboratory, helps to ensure governments remain committed to the nonproliferation of CWAs.

A key part of training involves realistic field exercises. During these activities, participants practice with detection equipment; collect samples for forensic laboratories; don protective equipment such as respirators, face shields, and chemical-resistant clothing; and follow procedures to prevent contamination. The most realistic training occurs at specialized facilities in the Netherlands, Czech Republic, and Belgium. Training at these sites features staged incidents, in which participants must identify the agent involved and respond appropriately.

Back at home, FSC staff have been providing incident response support for more than a decade. Since 2008, the FSC has been the Chemical Agent Reference Laboratory for the Environmental Protection Agency (EPA) for developing reliable, accurate, and extremely sensitive analytical methods and providing standards (milliliter quantities of 10 parts per million CWAs). EPA is the federal agency responsible for environmental cleanup following acts of terrorism involving CWAs as well as toxic industrial compounds.

Livermore analytical chemist and FSC Deputy for Operations Carolyn Koester is the principal investigator for the partnership with EPA. As part of their duties, FSC scientists work to improve detection methods in partnership with EPA. “We want to help EPA laboratories ensure that all public areas are safe after an incident involving chemical weapons,” she says. Following an attack on U.S. soil, EPA’s Environmental Response Laboratory Network (ERLN), which includes the FSC, would quickly become involved. Response teams, using Livermore-developed techniques, would determine the nature and extent of the contamination and then help monitor decontamination and restoration activities. As part of ERLN, the FSC could also provide overflow chemical analysis capacity, if requested.

The EPA work is part of a larger Livermore effort to help federal and state agencies plan for efficient recovery from a chemical weapons release and reduce the remediation effort by days or weeks. (S&TR, March 2010, Responding to a Terrorist Attack Involving Chemical Warfare Agents.) Toward that goal, Livermore scientists have also strengthened scientific understanding of how CWAs interact with different building and office materials.

Iraqi first responders conduct a training exercise during a simulated chemical warfare agent attack. The same first responders, trained by FSC staff, later performed admirably when terrorists launched an actual attack in the city of Kirkuk in 2016.

Furthermore, OPCW’s confirmation of a Novichok nerve agent being used in the 2018 Salisbury attack resulted in the organization recently adding four more compound families, including the Novichoks, to the list of tightly restricted chemicals under the CWC. This revision marked the first time any class of chemical had been added to the CWC schedules since the treaty came into force more than two decades ago. As a result, the FSC is actively investigating the characteristics of the four additional compound families and their precursors, environmental degradation, and decontamination products. Says Williams, “One or more chemicals from the newly added chemical families could be spiked in a future OPCW environmental proficiency test sample.”

In this training exercise, a first responder wearing protective clothing examines a human mannequin covered with skin blisters at a makeshift laboratory. The blisters are indicative of exposure to sulfur mustard.

Although FSC researchers continue to acquire state-of-the-art instrumentation and develop more effective protocols for detecting CWAs, FSC staff, including chemist Armando Alcaraz, conduct further work with national ministries to provide training for detecting and responding to chemical threats (see Two Decades of Enhancing Response to Chemical Weapons Use, above). He notes OPCW is hoping to launch training programs in countries such as Morocco, Algeria, Nigeria, and several Latin American nations. OPCW is also desirous of establishing accredited laboratories in Africa, Latin America, and the Mideast. Clearly, the fight to eliminate chemical weapons must be a worldwide endeavor.

—Arnie Heller

Key Words: Biosafety Level-2, butyrylcholinesterase (BChE), Biosecurity Engagement Program, Chemical Security Program, chemical warfare agent (CWA), Chemical Weapons Convention (CWC), Environmental Protection Agency (EPA), Forensic Science Center (FSC), Organisation for the Prohibition of Chemical Weapons (OPCW), subetadex, sulfur mustard, Swedish Research Defence Agency, U.S. Army Combat Capabilities Development Command Chemical Biological Center.

For further information contact Armando Alcaraz (925) 423-6889 (alcaraz1@llnl.gov).

The Sierra Era

This three-dimensional simulation of an idealized inertial confinement fusion implosion shows turbulent mixing in a spherical geometry. Livermore’s Sierra supercomputer makes high-fidelity calculations like this routine, yielding crucial insights into physical phenomena.

Lawrence Livermore’s high-performance computing (HPC) facilities house some of the fastest supercomputers in the world, including the flagship Sierra machine. Online for more than a year, Sierra primarily runs simulations for the National Nuclear Security Administration’s (NNSA’s) Advanced Simulation and Computing (ASC) Program. Sierra substantially increases the Laboratory’s ability to support ASC’s stockpile stewardship work by providing more accurate, predictive simulations.

Sierra was formally dedicated in October 2018 and opened to NNSA users for classified work in the spring of 2019. Leading up to those milestones was a yearslong effort by CORAL—a collaboration of Oak Ridge, Argonne, and Lawrence Livermore national laboratories—and Livermore’s Sierra Center of Excellence to prepare applications for the first major heterogeneous system at the Laboratory (see S&TR, March 2015, Gearing Up for the Next Challenge in High-Performance Computing; and March 2017, A Center of Excellence Prepares for Sierra). Through CORAL, the Department of Energy (DOE) requested proposals for extreme-scale computing systems. After hardware vendors were selected, the Center of Excellence provided resources to modify Livermore’s large code base for execution on the new machine’s architecture.

Likewise, Sierra had to be optimized for DOE workloads. “We have been continuously engaged with HPC vendors to help steer development of future computing systems,” explains Chris Clouse, Livermore’s acting program director for Weapons Simulation and Computing. “Collaborating with other DOE laboratories helps capture vendors’ attention so they can appreciate the larger context of the country’s scientific computing needs.”

This lengthy preparation—designing advanced computing hardware, shifting the software programming paradigm, and pushing the industry standard—has culminated in unprecedented simulation capabilities at Livermore. As NNSA’s most powerful supercomputer, Sierra ushers in a new era of computing architectures, software development, and scientific applications.

Processing Power

When the Sequoia supercomputer came online in 2012, it put Lawrence Livermore squarely in the petaflop (10^15 floating-point operations per second) era of computing (see S&TR, July/August 2013, Reaching for New Computational Heights with Sequoia). It had taken a decade of computing innovations to achieve the factor of 1,000 gains in performance from the first teraflop (10^12 floating-point operations per second) systems. All that extra computing power, and resulting computing gains over the last decade, have come at the expense of increased power demands. Looking ahead to the next generation of computers, Laboratory researchers knew a more energy-efficient approach would be needed. “Any extreme-scale machine must balance operating costs with computing performance gains,” says Clouse. “More powerful regimes cannot be achieved by simply scaling out current, heavyweight core technology. We need a new approach that offers an affordable electricity bill.”

Sierra’s advanced heterogeneous, or hybrid, architecture uses more than one type of processor or core. It combines 17,280 NVIDIA Tesla V100 (volta) graphics processing units (GPUs), which increase parallel processing power, and 8,640 IBM Power9 central processing units (CPUs). Clouse says, “Some parts of our large multiphysics applications simply will not run well on a solely GPU-based system.” Sierra’s GPU-to-CPU ratio helps balance the machine’s application workload. Sierra’s 100-gigabit-per-second network swiftly transfers data between these processors, and its memory capacity for data-intensive calculations reaches 1.38 petabytes. In comparison to other GPU-based machines at the Laboratory, Sierra also has significantly more high-bandwidth memory per processor. “Performance is heavily tied to the high-bandwidth memory associated with the GPUs. The more data we can fit into that type of memory, the better our codes will perform,” states Clouse.

Advanced computer architectures such as Sierra (above) combine graphics processing units with central processing units and require a new programming paradigm to ensure that simulation codes run efficiently and effectively.

Sierra’s sophisticated architecture enables the machine to register a peak performance of 125 petaflops using only 11 megawatts of electricity. In other words, Sierra is six times faster than Sequoia but uses only one-third more wattage. Terri Quinn, associate program director for Livermore Computing, explains, “GPUs sip energy compared to traditional server processors. With leading HPC systems incorporating tens of thousands of processors and consuming multiple megawatts of power each year, GPUs keep power consumption in check, require a smaller footprint, and cost less than a CPU-only system of comparable performance—if you can take advantage of them.” (See Global Distinction below.)

Abstraction and Memory Solutions

According to Quinn, pursuing Sierra’s hybrid architecture was a difficult decision. She says, “The ASC teams have put in an extraordinary amount of effort, often with IBM and NVIDIA experts alongside, to ready our codes (which did not run on GPUs) for Sierra.” Clouse adds that NNSA has led the advancement of portability solutions for new architectures. He says, “Rather than completely rewrite these complex codes, we are working to make codes run well on a wide range of platforms with relatively small, isolated modifications.”

Livermore computer scientists and software developers explored numerous portability and optimization solutions during the years leading up to Sierra’s installation (see S&TR, September 2016, Laying the Groundwork for Extreme-Scale Computing). For example, new algorithms exploit data parallelism and memory access to help ensure that codes capitalize on GPUs. Novel tools combine simulation and visualization routines to maximize data processing while in memory, and innovative memory-management models automate data movement between memory locations with minimal disruption to the source code. Clouse adds, “Small parts of each code were ported over and optimized to the new system before the entire code was considered ready for Sierra.”

One key GPU-portability innovation is Livermore’s RAJA abstraction layer. Large multiphysics codes typically contain millions of lines of code and thousands of calculation loops. RAJA provides abstractions at the loop level, separating platform-independent and platform-specific code for streamlined execution. Many of Livermore’s production codes have adopted RAJA, and codes under development will include abstraction layers from inception.

The Livermore-developed RAJA software abstraction layer systematically isolates machine-specific optimizations without disrupting the source code, enabling the latter to run on different platforms. For example, ALE3D and ARES are two of Livermore’s proprietary hydrodynamics codes, and MFEM is a high-order finite element library.

Memory-allocation tools, such as the Livermore-developed Umpire and CHAI (Copy-Hiding Application Interface), are crucial partners of abstraction layers. Memory movement between Sierra’s GPUs and CPUs is coherent, which means that both processor types share and communicate any changed values in memory regardless of which processor recorded the changes. Clouse elaborates, “In this setup, GPUs and CPUs remain in sync when they access data from memory.”

Global Distinction

Heterogeneous computing architectures combine graphics processing units with central processing units to achieve higher performance with more computing flexibility and less energy consumption. Supercomputing centers worldwide recognize the value of these architectures, and the Department of Energy (DOE) aims to stay at the forefront of developing leading-edge machines.

The biannual TOP500 list evaluates high-performance computing (HPC) systems using a linear algebra processing benchmark designed for distributed-memory architectures. In June 2020, Livermore’s Sierra ranks third on this prestigious list. In fact, DOE laboratories house four of the top 20 supercomputers, including Lassen—Sierra’s smaller, unclassified counterpart located at Lawrence Livermore. Sierra also ranks 12th on the Green500 list of energy-efficient supercomputers.

Terri Quinn, associate program director for Livermore Computing, emphasizes the importance of top-tier HPC capabilities to the Laboratory’s national security mission. “We provide scientists and engineers with unique and powerful HPC resources to give Livermore, the National Nuclear Security Administration, and the United States a competitive advantage. World-class systems induce code advances, offer new simulation possibilities, and attract top computational and computer science talent to the Laboratory. Who wouldn’t want to work on the most powerful computers in the world?”

Tools such as CHAI and Umpire allow more control over Sierra’s memory allocations to improve its performance. “They also provide portability on machines that, unlike Sierra, do not guarantee memory coherence,” says Clouse. Together, abstraction and memory management simplify how codes run on Sierra. Programmers do not need to explicitly allocate data on GPUs because these software tools do it for them.

Simulation Speed-Ups

Before Sierra, researchers primarily relied on two-dimensional (2D) approximations of three-dimensional (3D) simulations, which were computationally expensive and therefore performed sparingly. The machine can run 3D simulations more efficiently and effectively. Furthermore, Sierra can run complex calculations with fewer nodes, which means dozens of simulations can run concurrently.

Clouse explains, “For many of our 3D applications, Sierra’s architecture brings speed-ups on the order of 5 to 20 times what our (older) commodity clusters can do. With its extraordinary resolution, we can begin replacing our daily 2D workload with 3D simulations.” In one example, Sierra produced a 3D inertial confinement fusion simulation in 60 hours compared with a 30-day estimate on Livermore’s multi-core CPU system, Sequoia. The resulting data set provides further understanding of turbulence models.

With Sierra, Livermore scientists are seeing significant impacts on programmatic work. For the W80-4 life-extension program, simulations run on Sierra help assess the warhead’s new and refurbished components in 3D (see S&TR, October/November 2018, Extending the Life of a Workhorse Warhead). In another effort, a research team runs machine-learning algorithms on Sierra to analyze data from simulations and experiments (see S&TR, March 2019, Machine Learning on a Mission). Researchers at NNSA’s other national laboratories, Sandia and Los Alamos, also use Sierra for stockpile stewardship applications. Quinn notes, “Users tell us they can run calculations they would never have dreamed of running before Sierra.”

Bruce Hendrickson (right), Livermore’s associate director for Computing, talks with former U.S. Secretary of Energy Rick Perry during a tour of the Sierra machine room. (Photo by Julie Russell.)

From Petaflops to Exaflops

Today’s fastest computing technologies will be considered slow tomorrow. In August 2020, DOE announced a partnership with Cray Inc. (now Hewlett Packard Enterprises), to build NNSA’s first exascale computer at Livermore. El Capitan is expected to come online in 2023 with a new peak performance standard of at least 1.5 exaflops, or 1.5 quintillion (1.5 x 10^18) floating-point operations per second, ushering in the next factor of 1,000 gains in computing power. Clouse states, “El Capitan will also have a GPU-based architecture. Our portability and optimization work for Sierra will benefit us greatly when the time comes, but no doubt El Capitan will present unique challenges.”

Quinn points out that DOE’s first three exascale-class supercomputers—El Capitan, Argonne National Laboratory’s Aurora, and Oak Ridge National Laboratory’s Frontier—will use GPUs. She states, “GPUs are becoming more popular for scientific and engineering workloads, and I expect this trend to continue for the remainder of this decade.”

Meanwhile, Sierra churns through the Laboratory’s physics codes, improving simulation fidelity and prediction while laying the groundwork for exascale machines. The intervening years will provide abundant opportunities to leverage Sierra’s capabilities for NNSA’s weapons program and nuclear counterproliferation and counterterrorism efforts. “The ability to process crucial simulations efficiently and more realistically means 3D resolution is becoming routine,” states Clouse. “Sierra is a game-changer for computational scientists.”

—Ann Parker

Key Words: Advanced Simulation and Computing (ASC) Program, central processing unit (CPU), Copy-Hiding Application Interface (CHAI), Department of Energy (DOE), exascale, graphics processing unit (GPU), high-performance computing (HPC), memory, National Nuclear Security Administration (NNSA), portability, RAJA, Sierra, simulation, supercomputer, Umpire.

For further information contact Chris Clouse (925) 422-4576 (neely4@llnl.gov).

Tapping the Earth’s Heat for Clean Energy

In an enhanced geothermal system (EGS), water is pumped underground through an injection well (blue pipe) into hot “basement” rock. Heated water and steam pumped back to the surface are used to propel turbines that generate electricity. The water is continuously recycled in a closed loop. (Image courtesy of the Department of Energy’s Geothermal Technologies Office.)

Deep in a tunnel at South Dakota’s Sanford Underground Research Facility (SURF), workers inject pressurized water through a 60-meter-long borehole drilled into the Earth’s interior. Within a couple of hours, the water migrates through rock fractures “stimulated” by the water’s pressure and returns, warmed by the rock, to another borehole drilled parallel to the first. Back in the tunnel, automated systems collect, process, and then transfer incoming data from an array of instruments to team members around the United States, who help monitor and analyze the experiment’s progress in real time.

The work is part of a Department of Energy (DOE) effort to develop longer term, transformational enhanced geothermal systems (EGS) through collaborative experimental and model comparison. Sponsored by DOE’s Geothermal Technology Office, the EGS Collab team includes participants from universities, private industry, and eight national laboratories. EGS Collab is led by Lawrence Berkeley National Laboratory geological scientist and mechanical engineer Tim Kneafsey and Sandia National Laboratories’ Doug Blankenship. The EGS Collab project is evaluating many of the techniques that will eventually be used to develop a large-scale EGS testbed called FORGE—the Frontier Observatory for Research in Geothermal Energy.

As part of EGS Collab, Lawrence Livermore is lending its expertise in high-performance computing (HPC) to design experiments, build integrated data sets from observations, and perform simulations that compare experimental data to models of fluid-induced fracturing of the subsurface rock.

“Embarrassing” Amounts of Energy

Some geothermal power plants, such as those at California’s Geysers Geothermal Field, generate electricity by tapping into hot water located in permeable, porous rock deep underground. The hot water and steam pumped to the surface from drilled boreholes propels turbines that generate power. The water can then be pumped back into the rock below to close the loop. However, most of Earth’s subsurface consists of relatively impermeable rock that contains plenty of heat but is not as accessible.

A schematic of EGS Collab’s first experiment illustrates the setup of the test. The light blue shaft (upper right) indicates the tunnel where the experiment took place. The green tube represents the injection well connected to the tunnel. As the water induces fractures (blue disks) in the surrounding basement rock, some of the water flows to the production well (red tube). Yellow tubes represent boreholes that have instruments for taking measurements of the experiment in real time.

In 2006, a study led by the Massachusetts Institute of Technology estimated that EGS could provide 100 gigawatts of cost-competitive electricity over the next 50 years and that the total amount of extractable energy from EGS reservoirs in the United States was about 2,000 times the nation’s annual total energy consumption. “It’s embarrassing to think of how much geothermal energy is sitting below us that we just can’t access,” says Joe Morris, associate program leader for the Laboratory’s Fossil Energy Solutions and Nuclear Effects Research and Development program areas.

EGS offers a method that could prove effective in liberating the energy stored in impermeable rock, by pumping cold water from the Earth’s surface through sufficiently hot basement rock located 3 to 5 kilometers underground. The pressure of the water from the injection well creates a network of induced fractures throughout the rock. As the water migrates through the fractures, it heats up, is pumped out of a recovery well, and then is transported to the surface to generate electricity. However, to make EGS an operational reality, researchers need to better understand and predict how a network of induced fractures forms in the rock as well as how fracture permeability changes as water migrates.

Data Extravaganza

Experiments at SURF enable the EGS Collab team to compare and validate EGS reservoir models with experimental data to improve fracture predictions. Instruments placed within the boreholes collect acoustic, resistivity, and seismic data as well as measurements of water quantity, temperature, and flow rate. By combining these measurements, researchers can discern the orientation of the rock fractures and the direction of the water flow.

“This project is unlike any underground monitoring project I have worked on before,” says Morris. “We can monitor the progress of the experiment in real time, discuss incoming results, and even control the experiment remotely using videoconferencing and an information sharing platform.” Morris adds, “We can have the entire EGS Collab community engaged in figuring out the next step of the experiment at a single moment.”

EGS Collab team members work together to add instrumentation for monitoring water injection through the borehole.

Material characterization of rock core samples also provides key information about critical aspects of the experiment. Livermore’s Megan Smith, deputy group leader for the Subsurface Transport Group, says, “The cores show preexisting faults, rock textures and composition, and other features. We can correlate our data with the appearance of the rock.” Using a wide range of analysis techniques, including computed tomography scans, the research team can study the lithology, composition, and fracture geometry of samples and take measurements of the rock’s elastic properties, magnetic susceptibility, gamma-ray intensity, chemical element ratios and abundance, sample microbiology, and seismic wave velocity.

Over the last few years, the EGS Collab team has made notable progress. In May 2018, the researchers achieved a hydraulic connection between the injection and recovery wells and observed, for the first time, the movement of water between the two. Later that year, the researchers circulated fluid through the fracture system continuously for one month and recovered more than 90 percent of circulating fluid for several days—a promising development as commercial-scale plants must continuously circulate and recover water through their EGS fields. In February 2019, the team began long-term test bed monitoring and conducted tracer tests, in which a harmless compound was added to the injected water to map its flow through the fracture system. A new set of experiments is planned during 2021.

Grounding Simulations in Reality

Prior to experiments at SURF, Livermore’s Pengcheng Fu, acting deputy group leader of the Computational Geosciences group, used Livermore’s GEOS code to accurately model fracture initiation and growth. Livermore’s predictions of fracture propagation matched well with the experimental results. “Since the initial experiments, fluid recovery has been stable, and we’ve observed a rich set of flow system behavior that will be useful for future modeling,” says Fu.

The extensive geophysical measurements and recorded data pouring in help to constrain the models and reduce the level of uncertainty. “For example, we can measure fracture propagation direction, fracture interaction with the wellbores, and the temperatures and microseismic data, all of which confirm our maps of fracture orientations,” says Fu. “With this information, we can do near-real-time modeling of how the fractures propagate, supporting the project team on how the experiment should proceed.”

Simultaneously collecting many types of data, such as pressure, leak rate, and injection temperature (shown here from top to bottom), allows researchers to verify critical processes related to fracturing.

In one effort, Fu and Hui Wu, a postdoctoral researcher in the Computational Geosciences Group, combined chemical tracer data with HPC simulations to produce millions of fluid transport models of the water through the rock to determine which models were the likeliest match to the field data. The models revealed the role that the structure of the rock plays in fluid transport. This knowledge will be important to commercial EGS plant managers when determining where to drill recovery wells to maximize the flow of return heat.

Fu is organizing EGS Collab’s terabytes of data into easily used data sets on OpenEI, a publicly accessible data repository. He is presenting the information in ways that reveal the intrinsic connections between the many data types so that more researchers can interpret the data and contribute to scientific analysis. “We’re collecting all the data in one place and creating a derived product to push out to the user community,” says Morris. “The project has generated data from the subsurface of unprecedented completeness.” Looking forward, EGS Collab’s efforts may one day lead to an enhanced geothermal system that efficiently taps the Earth’s heat to produce clean, low-carbon-emitting power.

—Allan Chen

Key Words: borehole, Enhanced Geothermal System (EGS) Collab, Frontier Observatory for Research in Geothermal Energy (FORGE), GEOS code, Geysers Geothermal Field, high-performance computing (HPC), OpenEI, Sanford Underground Research Facility (SURF).

For further information contact Joe Morris (925) 424-2263 (morris50@llnl.gov).

Drop by Drop: A Promising Method for Printing Metal Parts

Livermore engineer Nick Watkins, architect of the liquid metal jetting (LMJ) machine, shows off an LMJ-manufactured part. (Photo by Randy Wong.)

Over the past decade, Lawrence Livermore researchers have been involved in a significant effort to develop advanced materials and manufacturing processes to meet the requirements of the National Nuclear Security Administration and the broader needs of the Laboratory as well as U.S. industry. Livermore scientists’ goal is to design and deliver materials and components with tailored properties on an accelerated schedule and at reduced cost, with a special focus on national security applications. The effort centers on a fast-growing technology called additive manufacturing or three-dimensional (3D) printing, a technique that builds precise parts in sequential layers thinner than a human hair. The resulting specialized materials and components offer previously unattainable shapes; interior structures; and mechanical, optical, and electrical properties not possible with traditional manufacturing methods.

In recent years, liquid metal jetting (LMJ) has been developed as an additive manufacturing process for producing 3D metal parts through the control of tiny droplets of metals heated to 2,000°C. The droplets fall onto a substrate to produce 3D parts from a stack of two-dimensional digital patterns. Upon solidification, each layer of jetted droplets acts as a new substrate onto which new droplets are dispensed to create the subsequent layer. As the process repeats, complex 3D geometries can be printed with extremely fine detail.

As an advanced manufacturing concept, LMJ is barely 10 years old—largely studied in universities and yet to make commercial inroads. Although in its infancy, LMJ offers Lawrence Livermore researchers an alternative to the more prevalent and commercial selective laser melting technique used to additively manufacture metal parts. In selective laser melting, a laser melts and fuses metallic powders, but this method has high equipment costs, requires specialized spherical metal powder feedstock, and may neccessitate expensive safety infrastructures. In addition, the choice of metals is limited to commercial metals like steels or titanium alloys. As a result, 3D printing currently relies almost entirely on polymer materials, which often cannot duplicate the required properties of metals.

Livermore’s LMJ machine offers important advantages over traditional metal manufacturing methods to build three-dimensional metal parts.

Gentler, Less Expensive Process

Livermore physicist Jason Jeffries explains, “Selective laser melting limits what metal feedstock can be used because many metals become highly reactive when they are converted to a powder.” For example, titanium powder exposed to air becomes flammable. “We get around that problem with LMJ,” he says. Furthermore, the use of fine powders can cause splattering, which may inadvertently contaminate a part, or the powder can get trapped inside a part and vaporize, resulting in an unwanted void in the finished build. “LMJ is a much gentler and more localized process,” says Jeffries. LMJ also offers cost advantages because it uses raw feedstock, bypassing the extra processing step of converting it into powder.

Jeffries leads a multidisciplinary effort to determine LMJ’s capabilities and analyze how its various components (including the nozzle that produces the molten metal droplets) affect the build speed and quality of parts. The team includes researchers in physics, materials science, mechanical engineering, and computational engineering. The team is also working with researchers at the University of Nottingham in the United Kingdom.

The work is funded through a Laboratory Directed Research and Development Strategic Initiative (SI). Typically three years in duration, SIs aim to achieve a leap forward in meeting important science, technology, and engineering challenges. This SI builds on longstanding Lawrence Livermore expertise in predictive simulations, synthesis, characterization, and testing of new materials, and precision and additive manufacturing.

The dearth of commercial LMJ machines led the research group to build its own unit in 2018. The Livermore machine offers two printing modes: drop-on-demand (DoD) mode, which produces discrete droplets ejected from a nozzle, and a continuous stream mode that offers a way to build a part (or section of a part) faster but with less resolution than the DoD mode. “With a fast-streaming mode and a high-resolution mode, we don’t need two different machines,” says Jeffries. An object could be manufactured by combining both modes. Jeffries offers the example of printing a metal table: The table legs would be manufactured with the DoD mode, while the table top could be made using the continuous mode for a more rapid build without sacrificing quality.

The Livermore LMJ machine offers two printing modes: drop-on- demand (DoD) mode (left), which produces discrete droplets ejected from a 100-micrometer-diameter nozzle (top), and continuous stream mode (right), which is faster than the DoD mode but offers less resolution.

Simulations and Experiments Combine

The team is combining simulations and experiments to optimize printing objects with pure tin because of its low melting point, average viscosity, and average surface tension, which together make it an easy-to-work-with material. The largest tin part so far weighs about 100 grams. The builds use 1 to 5 grams per minute for individual tin droplets, and 10 to 15 grams per minute for continuous streaming.

Most of the effort has gone toward understanding and optimizing the DoD process, in particular examining the relationship between the machine’s operating parameters and the object’s resultant microstructure and physical properties. Jeffries says that most studies previously focused on the metal jetting head and analysis of molten metal droplets. However, very few previous studies have analyzed the relationship between microstructure and properties (structural strength, for example) of an LMJ-made part.

During the DoD process, metal is heated to between 30 and 50 degrees above its melting temperature. Tin melts at 232°C, so the droplets are heated to around 260°C. Droplets measure 300 micrometers in diameter and their size is dependent upon the size of the nozzle orifice. During the printing process, 100 droplets per second are ejected with a velocity ranging from 0.2 to 1.5 meters per second. The distance from the nozzle to the starting build plate varies between a few millimeters to a centimeter.

As the droplets descend, the substrate moves back and forth (the nozzle stays in a fixed position) to accommodate deposition of discrete layers. Modeling indicates that the larger droplets cool little during flight before impacting the underlying stainless steel substrate or a previously applied layer. (However, there is a risk that droplets can freeze as they descend if they are too small.) Upon impact, droplets cool within tens of milliseconds. The build plate is typically made of stainless steel so that the finished part can be removed easily. If desired, droplets forming the first layer can stick to a build plate made of identical material as the feedstock metal. It takes just a couple minutes to build the smallest parts and up to 20 minutes for the largest parts.

The research is guided by simulations that model droplet ejection, flight, and deposition to optimize the as-built part. The simulations show, for example, that aspherical droplets can be caused by the material cooling at the nozzle’s exit, an effect that alters the resolution of the final part. The simulations also help the researchers determine how much of the underlying structure should melt from the drop above it. “We want the molten drop to bond to the metal underneath it, and not just physically rest upon it,” says Jeffries.

According to Jeffries, controlling the cooling rate of the microstructure is necessary to attain optimal mechanical properties. Otherwise, voids in build formation, unwanted phases (orientations of atoms in a crystalline lattice), dislocations (atoms out of place), various artifacts, and high residual stress can occur. The cooling rate can be controlled by modifying the droplet ejection velocity, size, and frequency, or the substrate temperature. Toward that end, Livermore researchers have studied the operation of the liquid metal reservoir, nozzle, and the pneumatically driven pressure pulse that sequentially forces droplets from the nozzle.

The team has produced a number of small tin parts and then examined their internal structures through a combination of roughness, density, and x-ray diffraction (XRD) measurements. The microstructures of the tin pieces have shown minimal stored strain or deformation, low to no voids, and only the most common phase (the same phase as the feedstock tin). The parts exhibit characteristics identical to metal parts produced by conventional machining or casting–techniques used for many decades.

LMJ-manufactured tin parts (all measuring about 3 centimeters to a side) are examined using a combination of roughness, density, and x-ray diffraction measurements. The parts’ microstructures have shown minimal stored strain or deformation and exhibit characteristics identical to metal parts produced by conventional machining or casting.

Building a Foundation

“We are building a foundation to seed programmatic investments,” says Jeffries. Team members have made presentations to national subject matter experts and have received valuable feedback. He says the team’s work may prove critical for nuclear stockpile stewardship because of the limited availability of certain manufacturing technologies and materials or the prohibitive costs involved.

As the work progresses, interest is growing at the Laboratory for using LMJ for national security applications, where the ability to print inexpensive metal parts is needed. These parts will also have unique attributes such as varying metal matrices and functionally graded materials, features that are not possible with conventional machining or casting or current 3D metal printing. “We’re showing LMJ works,” says Jeffries. “We haven’t overcome all the problems, but we’re making important progress.”

—Arnie Heller

Key Words: additive manufacturing, Laboratory Directed Research and Development Strategic Initiative (SI), liquid metal jetting (LMJ), three-dimensional (3D) printing, tin.

For further information contact Jason Jeffries (925) 422-4734 (jeffries4@llnl.gov).

Awards

The American Astronomical Society (AAS) has selected Lawrence Livermore scientist Peter Beiersdorfer as a fellow in its inaugural class for this accolade. The AAS fellows’ program was established in 2019 to confer recognition to AAS members for achievement and extraordinary service to the field of astronomy and the society. Fellows are recognized for their contributions toward the AAS mission of enhancing and sharing humanity’s scientific understanding of the universe. During his career, Beiersdorfer has pioneered techniques to reproduce conditions on comets and in the sun’s atmosphere, interstellar space, and the centers of galaxies.

Lawrence Livermore scientists Federica Coppari and Erin Nuccio have been honored with the Department of Energy’s (DOE’s) Office of Science Early Career Research Program award. Coppari, a physicist, was selected for her work in high-energy-density science. Nuccio, a microbiologist, was selected for her research in fundamental systems biology.

The Early Career Research Program, now in its 11th year, is designed to bolster the nation’s scientific workforce by providing support to exceptional researchers during crucial early career years, when many scientists do their most formative work. Coppari and Nuccio are among 76 scientists nationwide selected for the recognition this year. Under the program, DOE national laboratory staff are awarded $500,000 per year for five years to further their research.

Patents

Spatter Reduction Laser Scanning Strategy in Selective Laser Melting

Saad Khairallah

U.S. Patent 10,449,632 B2

October 22, 2019

Microscale Sensors for Direct Metrology of Additively Manufactured Features

Sourabh Saha, Robert Matthew Panas, Michael A. Cullinan, Ian Seth Ladner

U.S. Patent 10,451,539 B2

October 22, 2019

Laser-Assisted Additive Manufacturing

Stavros Demos

U.S. Patent 10,471,543 B2

November 12, 2019

Ortho-H2 Refueling for Extended Cryogenic Pressure Vessel Dormancy

Guillaume Petitpas, Salvador M. Aceves

U.S. Patent 10,479,678 B2

November 19, 2019

System and Method for Synthesis of Impedance Matching and Signal Converting Material for All Optical Photo-Acoustic Detection

Stavros Demos

U.S. Patent 10,485,427 B2

November 26, 2019

Post Polymerization Cure Shape Memory Polymers

Thomas S. Wilson, Michael Keith Hearon, Jane P. Bearinger

U.S. Patent 10,494,470 B2

December 3, 2019

Methods for 2-Color Radiography with Laser-Compton X-Ray Sources

Christopher P. J. Barty

U.S. Patent 10,508,998 B2

December 17, 2019

Using Colloidal Silica as a Zonal Isolation Material and Fast Path Blocker in Geological Formations

William L. Bourcier, Sarah K. Roberts, Jeffery J. Roberts, Souheil M. Ezzedine, Jonathan D. Hunt

U.S. Patent 10,538,990 B2

January 21, 2020

S&TR Staff

Scientific Editors: Dawn Shaughnessy and Holly Carlton

Managing Editor: Ken Chinn

Publication Editor: Caryn Meissner

Writers: Holly Auten, Lauren Casonhua, Allan Chen, Rose Hansen, Ann Parker, J.C. Ross, Genevieve Sexton, and Suzanne Storar

Designers: Alii Diaz, Mark Gartland, and Mary Gines

Proofreaders: Deanna Willis

S&TR Online: Lauren Casonhua, Rose Hansen, and Pam Williams

S&TR, a Director's Office publication, is produced by the Technical Information Department under the direction of the Office of Planning and Special Studies. S&TR is available on the Web at str.llnl.gov.

Printed in the United States of America.

Available from:

National Technical Information Service

U.S. Department of Commerce

5285 Port Royal Road

Springfield, Virginia 22161

UCRL-TR-52000-20-8

Distribution Category UC-99

August 2020

About S&TR

At Lawrence Livermore National Laboratory, we focus on science and technology research to ensure our nation’s security. We also apply that expertise to solve other important national problems in energy, bioscience, and the environment. Science & Technology Review is published eight times a year to communicate, to a broad audience, the Laboratory’s scientific and technological accomplishments in fulfilling its primary missions. The publication’s goal is to help readers understand these accomplishments and appreciate their value to the individual citizen, the nation, and the world. The Laboratory is operated by Lawrence Livermore National Security, LLC (LLNS), for the Department of Energy’s National Nuclear Security Administration. LLNS is a partnership involving Bechtel National, University of California, Babcock & Wilcox, Washington Division of URS Corporation, and Battelle in affiliation with Texas A&M University. More information about LLNS is available online at www.llnsllc.com. Please address any correspondence (including name and address changes) to S&TR, Mail Stop L-664, Lawrence Livermore National Laboratory, P.O. Box 808, Livermore, California 94551, or telephone (925) 423-3432. Our e-mail address is str-mail@llnl.gov. S&TR is available on the Web at str.llnl.gov.

© 2020. Lawrence Livermore National Security, LLC. All rights reserved. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. To request permission to use any material contained in this document, please submit your request in writing to Public Affairs Office, Lawrence Livermore National Laboratory, Mail Stop L-3, P.O. Box 808, Livermore, California 94551, or to our e-mail address str-mail@llnl.gov.

This document was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor Lawrence Livermore National Security, LLC, nor any of their employees makes any warranty, expressed or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or Lawrence Livermore National Security, LLC. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or Lawrence Livermore National Security, LLC, and shall not be used for advertising or product endorsement purposes.