Lawrence Livermore National Laboratory



Article title: Working Smarter for a Sustainable Future
Photograph by Lee Baker.

For many years, Lawrence Livermore employees have worked to reduce their energy and water use, both in offices and in research labs. Long-standing efforts to do more with less have been spurred by a series of Department of Energy (DOE) guidelines and President Barack Obama’s 2009 Executive Order 13514. Under this order, federal agencies must reduce their fossil-fuel consumption, energy intensity (the energy consumed per square foot of office space), and emissions while acquiring a greater percentage of the total energy they consume from renewable sources. In addition, by fiscal year (FY) 2020, agencies must cut water consumption 26 percent below their FY 2007 levels.

The severe drought in California makes the need for action more urgent, prompting Livermore to search for innovations that will reduce its environmental footprint. “The Laboratory has been working hard to become a model sustainable enterprise,” says Michael Cowen, Livermore’s sustainability manager in the Operations and Business Principal Directorate. A sustainability campaign launched in 2008 accelerated efforts to reduce energy consumption, fossil-fuel use, and emissions from greenhouse gases and other pollutants and to adopt clean energy technologies. The campaign has also led to sitewide solutions that conserve water, reduce waste, and increase recycling.

A critical focus area for both water and energy conservation is the high-performance computing (HPC) facilities that house the Laboratory’s supercomputers. Although the energy efficiency of HPC processors continues to improve, the number of processors used in a system is also growing to increase computational speed. As a result, the machines consume ever-greater amounts of electricity and generate large amounts of heat that must be removed using air- and water-based cooling systems.

Reducing Greenhouse Gases

One notable accomplishment for the sustainability campaign is surpassing the federal goal for FY 2020 of reducing greenhouse-gas emissions by 28 percent of the FY 2008 baseline. The campaign team met this target primarily by implementing a plan to manage and capture gaseous sulfur hexafluoride (SF6), which dramatically reduced SF6 emissions.

Sulfur hexafluoride is used in extremely small quantities at Livermore for research activities involving accelerators, electrical power distribution equipment, electron microscopes, flash x-ray units, and some etching technologies. Even in small quantities, the compound receives intense scrutiny because 1 kilogram of SF6 has the greenhouse-gas effect of 23,900 kilograms of carbon dioxide. The SF6 Management and Capture Plan modified operations involving the compound to improve equipment leak detection, increase preventive maintenance, and ensure that gas-transfer and emissions-capture operations are more efficient.

Assault on Power Waste

Electricity consumption is another focus area for the sustainability campaign. Cowen says that both office buildings and research facilities offer opportunities for reducing electricity use, which at Lawrence Livermore averages 50 megawatts per year. To ensure that facilities operate at maximum efficiency, workers are modernizing heating, ventilation, and air-conditioning (HVAC) systems. In addition, they are installing motion controllers that turn off lights in unoccupied offices and conference rooms, boilers that have improved energy efficiency ratings, and streetlights that use light-emitting diodes.

President Obama’s executive order also requires federal agencies to use renewable energy sources to supply 20 percent of electrical power. Livermore has already achieved 17 percent, largely because it purchases energy generated by the hydroelectric plants operated by the Western Area Power Administration.

To help meet the 20-percent mandate, Laboratory managers are working with the DOE Livermore Field Office to prepare for a 3-megawatt solar photovoltaic system, which will be located on a 10-acre section of the main site’s northwest buffer zone. Because the plant will be built on site, Livermore will receive a double renewable energy credit. DOE is soliciting proposals from developers to design, finance, build, operate, and maintain a ground-mounted solar power installation for at least the next 20 years. Lawrence Berkeley National Laboratory is also a partner on this project and has agreed to purchase 20 percent of the electricity generated by the solar-powered system.

Another sitewide effort is consolidating the computer servers that operate business-oriented applications. Cowen notes that racks of these heat-generating servers are housed in buildings throughout the site, but the HVAC systems in those facilities were designed for office environments. “We’ve been asking buildings to support operations they weren’t designed to handle,” he says. By consolidating the business systems, the sustainability team has reduced the number of buildings with servers from 60 to fewer than 25. The ultimate goal is to establish an enterprise data center with seven specialized facilities for business operations.


Livermore is planting water-wise shrubs to reduce the amount of water required for landscaping. (Photograph by Paul Hara.)
Livermore is planting water-wise shrubs to reduce the amount of water required for landscaping. (Photograph by Paul Hara.)

Coping with Severe Drought

California’s severe drought has prompted vigorous water conservation efforts statewide as well as mandatory rationing in hard-hit areas. In light of this emergency, water conservation has become a critical task at Lawrence Livermore. About 45 percent of Livermore’s water consumption is directed to five cooling towers that provide air-conditioning for the HPC systems, 30 percent is for domestic use (such as sinks and toilets), and 25 percent goes toward landscape irrigation.

Each year, the Laboratory uses about 62 million gallons of water to landscape 35 acres of lawn, or turf, and 60 acres of shrubs. Turf requires significantly more water to irrigate than do “water-wise” shrubs: 30 gallons per square foot for turf versus 5 to 10 gallons per square foot for shrubs. As a result, the Laboratory is increasing its use of water-wise plants. Jacquelyn Westfall, who leads Livermore’s infrastructure maintenance services, notes that the Sustainable Landscape Concept Plan developed in 2011 is helping meet a 2014 goal of reducing water consumption by 13 million gallons. Most of the savings are from eliminating irrigation on 7 acres of turf and reducing the watering schedule sitewide.

According to lead landscaper Michael Bohannon, turf watering encourages trees to grow shallow roots. In areas where lawns are no longer irrigated, workers have installed a deep drip system around trees. “We want to protect our huge investment in the thousands of trees planted on site,” says Cowen. In addition, the landscape team installed 14 satellite irrigation controllers to ensure effective watering with minimal runoff. Other measures include reclaiming rainwater from several buildings and blending treated groundwater with potable water for irrigation.


Livermore engineer Ruben Ocampo (left) and Michael Cowen, the Laboratory’s sustainability manager, tour a pilot reverse-osmosis plant designed to prepare treated groundwater for use in one of the Laboratory’s five cooling towers. The plant is on track to save 7 million gallons of potable water from the Hetch Hetchy reservoir in California. (Photograph by Don Johnston.)
Livermore engineer Ruben Ocampo (left) and Michael Cowen, the Laboratory’s sustainability manager, tour a pilot reverse-osmosis plant designed to prepare treated groundwater for use in one of the Laboratory’s five cooling towers. The plant is on track to save 7 million gallons of potable water from the Hetch Hetchy reservoir in California. (Photograph by Don Johnston.)

Reverse Osmosis a Success

An innovative water-conservation effort is a reverse-osmosis pilot plant that filters treated groundwater so it is clean enough for use in a cooling tower. Cooling towers require continuous supplies of water to replenish what is lost to evaporation and to reduce the concentrations of dissolved minerals circulating through the system. High mineral concentrations can foul heat exchangers and interfere with heat transfer. Using treated groundwater reduces the amount of potable water the Laboratory purchases from the Hetch Hetchy Reservoir operated by the San Francisco Public Utilities Commission.

According to Jesse Yow, head of Livermore’s Environmental Restoration Department, the Laboratory treats groundwater at its main site to reduce contaminants to levels acceptable for water discharge into the storm sewer system, although the treated water is not pure enough for drinking or irrigation. Most of the contaminants—called volatile organic compounds—are legacy waste from the 1940s, when the square-mile site served as a U.S. Naval air station. Groundwater from about 90 wells across the Laboratory is pumped through 25 treatment facilities. Water leaving one of the treatment facilities is sent through the reverse-osmosis pilot plant, which then pipes purified water to a nearby cooling tower.

Yow notes that under the restoration program, treated groundwater is provided to the pilot plant when it is available. In addition to construction costs, the plant incurs expenses for electricity, maintenance, and sewer fees. Nevertheless, Cowen says reverse osmosis is a cost-effective solution. “The plant is on track to save 7 million gallons of potable water that would otherwise be purchased and earmarked for the cooling tower. We’re very encouraged by these results,” adding that reverse-osmosis plants are being considered for the other cooling towers.


Cooling towers, which feed water to the air-conditioning systems for the Laboratory’s high-performance computing facilities, account for about 45 percent of the total water consumption at Livermore’s main site. (Photograph by Lee Baker.)
Cooling towers, which feed water to the air-conditioning systems for the Laboratory’s high-performance computing facilities, account for about 45 percent of the total water consumption at Livermore’s main site. (Photograph by Lee Baker.)

Computing Offers Energy Savings

Operating HPC systems also requires a large power supply. In fact, supercomputers, which must run around the clock to support research efforts, consume half of the electricity purchased by the Laboratory. Livermore managers developed a master HPC sustainability plan, referred to as Turning Megawatts to Petaflops, to ensure that current and future energy requirements can be met while limiting the impact on environmental resources.

The Laboratory’s largest HPC facility is home to some of the world’s most powerful systems, including Sequoia, Vulcan, and Zin. The facility provides more than 48,000 square feet of floor space for systems and peripherals that consume 30 megawatts of power, not counting the energy required to run the facility’s machine-cooling system.

The biggest electricity user by far is Sequoia. This system has more than 1.5 million processor units, or cores, compressed into 96 racks, each the size of a large refrigerator, providing 1.6 petabytes (1015 bytes) of memory. (See S&TR, July/August 2013, Reaching for New Computational Heights with Sequoia.) Sequoia is equipped with energy-efficient features such as a 480-volt electrical distribution system for reducing energy losses. In 2012, the Green500 organization named the IBM system the “world’s most efficient supercomputer.”

Much of Sequoia’s energy consumption is tied to keeping it cool, with both air (9 percent) and water (91 percent). Facility engineers discovered that raising the ambient temperature from 52°F to 70°F did not degrade performance or damage components, saving more than $1 million annually in electricity. Sequoia’s water-cooling system is more than twice as energy efficient as air-cooling. A labyrinth of pipes under the floor of the computer room distributes chilled water in a vast network of polypropylene channels.

“We need to make the best use of the water resources we have on site,” says Anna Maria Bailey, who manages Livermore’s computing facilities. To that end, Livermore is looking to construct another reverse-osmosis plant to feed the cooling tower that serves the HPC facility because this tower accounts for about 20 percent of the Laboratory’s total water use.

Supercomputers must become more efficient, says Bailey, because their increasing needs for water and energy cannot be met by existing utilities. The exascale computers of the future will be nearly 1,000 times more powerful than some existing supercomputers and demand innovative designs from vendors. According to Bailey, one approach for those new machines may be to submerge them in an inert liquid, which would significantly reduce the cooling challenge.

In the meantime, the Laboratory is building a new HPC facility to accommodate next-generation supercomputers. When completed in 2015, it will be a showcase for lessons learned from past modernization projects at existing HPC facilities. “We’re taking a modular approach with an emphasis on sustainability,” says Bailey. The building will have expandable construction features that can be scaled as computational technology evolves.

Cowen notes that the growing sustainability efforts are consistent with the Laboratory’s prodigious research output. Generating renewable electricity on site, decreasing water and electricity usage, and reducing greenhouse-gas emissions all lead to greater productivity and are part of being a good neighbor.

—Arnie Heller

Key Words: drought, energy conservation, high-performance computing (HPC), reverse osmosis, Sequoia supercomputer, sulfur hexafluoride (SF6), sustainability, water conservation.

For further information contact Michael Cowen (925) 422-6337 (cowen3@llnl.gov).