STR Masthead

Article title: Venturing into the Heart of High-Performance Computing Simulations.

IN the field of high-performance computing (HPC) and advanced simulations, Lawrence Livermore researchers have gained a worldwide reputation for success, especially in calculations showing how matter responds to extreme pressures and temperatures. Now, researchers have applied their expertise to a new type of simulation that aims to realistically mimic a beating human heart. The results could contribute to advancements in human health in much the same way that Livermore’s computational work for stockpile stewardship helps ensure the safety, security, and reliability of U.S. nuclear weapons.

The new simulations are made possible by a highly scalable code, called Cardioid, that replicates the electrophysiology of the human heart. Developed by Laboratory scientists working with colleagues at the IBM T. J. Watson Research Center in New York, the code accurately simulates the activation of each heart muscle cell and the cell-to-cell electric coupling.

On every heartbeat, electric signals normally traverse the entire heart in an orderly manner, resulting in a coordinated contraction that efficiently pumps blood throughout the body. However, these signals can become disorganized and cause an arrhythmia, a dysfunctional mechanical response that disrupts the heart’s pumping process and can reduce blood flow throughout the body. Without medical intervention, a serious arrhythmia can lead to sudden death and accounts for about 325,000 deaths every year in the U.S.

The groundbreaking heart simulations were developed and performed on Lawrence Livermore’s Sequoia supercomputer, a BlueGene/Q system designed to achieve 20 quintillion floating-point operations per second (20 petaflops). The machine, which was built by IBM, has 98,304 nodes, each with 16 central processing units, or cores. When the full system is in operation, more than 1.5 million cores are available to execute calculations in parallel. Cardioid assigns roughly 3,800 heart cells to a node, for a total of about 370 million cells. The code is highly scalable, meaning it is written so that its performance increases in proportion to the number of cores applied to a problem.

Success through Long-Term Partnership
Sequoia is one of several supercomputing systems developed and deployed through a long-standing partnership involving the National Nuclear Security Administration, Lawrence Livermore, and IBM. In June 2012, Sequoia was named the world’s fastest computer on the industry-standard Top500 list. One of its predecessors, Livermore’s BlueGene/L, held the No. 1 ranking for nearly four years (2004–2007).

During the months required to “shake down” Sequoia, while IBM and Livermore scientists installed and tested the machine in the process of bringing nodes online, managers in the Laboratory’s Computation Directorate made the system available for unclassified science calculations. “Our heart code work has been a great opportunity to demonstrate Sequoia’s power with an application that most people consider important to society, in this case, cardiac modeling,” says computational scientist Art Mirin. He notes that when the shake-down testing is complete, Sequoia will be dedicated to classified simulations in support of the nation’s Stockpile Stewardship Program.

Simulations showing how the Cardioid code divides the heart into subdomains.
The Cardioid code developed by a team of Livermore and IBM scientists divides the heart into a large number of manageable pieces, or subdomains. The development team used two approaches, called Voronoi (left) and grid (right), to break the enormous computing challenge into much smaller individual tasks.

An Extended Look at Cardiac Health
The Livermore–IBM team began developing Cardioid with only a fraction of Sequoia’s nodes and took advantage of additional nodes as they became available for research. In addition to Mirin, team members included Livermore scientists David Richards, Jim Glosli, Erik Draeger, Bor Chan, Jean-Luc Fattebert, Liam Krauss, and Tomas Oppelstrup and seven IBM researchers, many of them experts in computational biology.

Richards, a computational physicist who wrote the largest share of the code, notes that Cardioid is essentially an extensive reworking of a code IBM scientists developed a few years ago for machines with only 2,000 nodes. “Livermore is one of the few places with the expertise to get past a lot of potential barriers that developers would likely encounter in adapting an existing code to run on the world’s most powerful supercomputer,” he says.

Richards explains that although the Livermore team was not experienced at simulating the human heart, many HPC techniques are “agnostic” to the specific problem at hand. That is, writing different types of codes for parallel supercomputers requires similar development tasks, no matter the phenomena being modeled.

In working on the code, IBM computational biologists contributed their expertise in cardiology, while Laboratory scientists provided support in computational science, especially parallel algorithms. “The Cardioid effort became an interdisciplinary problem involving both computer science and physics,” says Richards. “Because I was trained in the physical sciences, I could ask meaningful questions about heart function and understand how to apply the answers in a complex calculation.”

Cardioid allows simulation at roughly the spatial resolution of a heart cell, which is about 0.1 millimeters long. It thus provides researchers with a level of detail that was impractical with early codes. High-fidelity simulation at the organ level requires a three-dimensional discrete model of the human heart.

To achieve this resolution, the IBM scientists combined two-dimensional cross-sectional images from the Visible Human Project®, a detailed dataset from the National Library of Medicine. The team also developed software to reconstruct the anatomy of a torso so that an electrocardiogram from a typical body surface could be simulated. When combined with these components, Cardioid offers a multiscale simulation capability that spans from subcellular mechanisms up to clinical signals collected from actual patients.

Simulating Thousands of Heartbeats
Operating on Sequoia, the Cardioid code can simulate hundreds of times as many heartbeats as previous codes. One minute of Sequoia processing time is required to replicate nine human heartbeats at a nearly cellular spatial resolution. Simulating an hour of heart activity, or several thousand heartbeats, can be accomplished in seven hours when using the full Sequoia system. Less sophisticated codes took up to 45 minutes to compute a single heartbeat, making it impossible to model the heart’s response to a drug or an electrocardiogram trace for a particular heart problem.

Extended cardiac simulations are critical when investigating how specific medications affect heart rate. Many drugs disrupt heart rhythm. In fact, even those designed to prevent arrhythmias can be harmful to some patients. In most cases, however, researchers do not fully understand the exact mechanisms producing these negative side effects. With Cardioid, scientists can examine heart function as an anti-arrhythmia medication is absorbed into the bloodstream and its concentration changes. “Observing the full range of effects produced by a particular drug takes many hours,” says Mirin. “With Cardioid, heart simulations over this timeframe are now possible for the first time.”

The Cardioid simulation has been named as a finalist in the 2012 Gordon Bell Prize competition, which annually recognizes the most important advances in HPC applications. The Livermore–IBM team hopes the code will grow into a product that is widely adopted by medical centers, pharmaceutical companies, and medical device firms, helping them better understand the mechanisms that can lead to heart ailments and the potential drug interactions that may occur during treatment. One intriguing idea is to merge a Cardioid simulation with a patient’s clinical data—electrocardiograms, magnetic resonance imaging, and computed tomography scans, for example—to better quantify treatment options for each individual.

Cardioid simulations show how a drug might affect heart function. Snapshots from a Cardioid simulation show how a drug might affect heart function. Side-by-side images compare a portion of a beating heart with (left column) and without (right column) the administered drug. An electric stimulus applied at 0 milliseconds causes heart cells to depolarize (red). Before repolarization is complete (blue), an unplanned stimulus causes a premature depolarization (at 560 milliseconds). In the simulation with the drug, a reentrant circulation pattern develops (see circle at 690 milliseconds), inhibiting repolarization of the cells and thus preventing the heart from beating normally.

A Boost to Economic Competitiveness
The Cardioid modeling effort is sponsored by the High Performance Computing Innovation Center (HPCIC). The center opened in June 2011 with the goal of boosting the nation’s economic competitiveness by partnering with American industry to develop and deploy HPC solutions. (See the box below.)

“HPCIC is about industry teaming with some of the world’s foremost practitioners of simulation and visualization,” says Fred Streitz, the center’s director. A computational physicist, Streitz led two of the six Livermore teams awarded the Gordon Bell Prize for groundbreaking simulations. (See S&TR, July/August 2006, Keeping An Eye on the Prize; September 2010, Quickly Identifying Viable Pathogens from the Environment.)

At HPCIC, industrial partners can access Livermore’s supercomputing resources and technical expertise in an open collaboration area with office space, classrooms, and networked conference rooms. The center hosts conferences, workshops, and training events to encourage HPC development and innovation in an environment that protects intellectual property and promotes collaboration. The current computing system at HPCIC provides industrial partnerships with 300 trillion (tera) flops of computing power. In the near future, a 5-petaflops “mini-Sequoia” machine, called Vulcan, will be available.

60 imageThe Laboratory’s High Performance Computing Innovation Center (HPCIC) is home to a growing number of collaborations aimed at helping industries deploy simulation and visualization tools to build prototypes and solve difficult technical challenges. In June 2012, Lawrence Livermore and IBM announced one such venture, called Deep Computing Solutions, which will operate out of HPCIC. The agreement will combine IBM computational science expertise with the Laboratory’s own, to help U.S. businesses harness the power of high-performance computing (HPC) and boost economic competitiveness.

“The new collaboration between the Lab and IBM is an excellent example of using the technical expertise of both the government and the private sector to spur innovation and investment in the U.S. economy,” said California Senator Dianne Feinstein when the agreement was announced.

HPCIC represents the first step in an ambitious strategy to operate an open, unclassified collaboration zone called the Livermore Valley Open Campus (LVOC). The new campus, which debuted in 2011, consists of approximately 110 acres located along the eastern edge of Lawrence Livermore and Sandia national laboratories. (See S&TR, March 2011, New Campus Set to Transform Two National Laboratories.)

LVOC is modeled after research and development campuses at industrial parks and other Department of Energy laboratories, providing security, business, and operating rules designed to enhance scientific collaboration. The open campus is expected to grow steadily as collaborations increase and U.S. companies look to LVOC resources for help in such areas as cybersecurity, energy, transportation, and health care. As an example, the IBM Deep Research Team, members of which helped write the Cardioid code, are relocating to the campus.

Buck Koonce, director of economic development, notes that Livermore’s Industrial Partnerships Office and Office of Strategic Outcomes help connect industrial partners with Laboratory-developed technologies. Collaborations are formalized through various partnering arrangements, such as cooperative research and development agreements, and more than 100 companies have licensed Livermore technologies for commercialization.

In addition, says Koonce, Laboratory managers are working with regional economic development authorities to help create 5,000 high-tech jobs in the Tri-Valley area of northern California. (Located about 35 miles east of San Francisco, the Tri-Valley has a population of about 300,000 and includes the cities of Pleasanton, Livermore, Dublin, San Ramon, and Danville.) The job-creation goal seems eminently reachable; in 2011, the region added 1,800 high-tech jobs. Some of the area’s businesses were started by Laboratory employees or use a Livermore-developed product or technology. For example, Compact Particle Acceleration Corporation in Livermore is developing a proton therapy system based on accelerator technology that originated in the Laboratory’s stockpile stewardship research. (See S&TR, October/November 2011, Weapons Diagnostics Technology Revolutionizes Cancer Treatment.)

“Effective collaborations in fields such as health care strengthen national security in its broadest meaning,” says Koonce. In a similar manner, the nation’s security is improved by research to advance energy independence, as in Livermore’s efforts to help U.S. companies more accurately interpret seismic data for oil and gas exploration, adapt wind power resources for large-scale energy generation, and modernize the nation’s electric grid.

Koonce emphasizes that the Laboratory is careful not to compete with any company offering a similar service or technology. “Our goal is to help U.S industry to be more competitive in the global marketplace,” he says. “Working with industry and academia also keeps us in the forefront of technology. The expertise we develop in simulation goes back into our national security research. A project such as Cardioid also helps us attract and train talented people, many of whom may well end up working on a variety of national security problems in the future.”

Enhancing the Code
The Livermore–IBM team is now working on a mechanical component that simulates the contraction of the heart and pumping of blood. The new feature will be coupled to the electrophysiological model in Cardioid, allowing the code to be applied to other health problems. For example, congestive heart failure, a condition that reduces the heart’s ability to pump an adequate supply of blood, affects roughly 5 million Americans. Despite the best treatments available, the 5-year survival rate remains stubbornly below 50 percent.

“Congestive heart failure is a complex and multifaceted disease,” says Jeremy Rice, a biomedical engineer at IBM and a Cardioid collaborator. “An accurate electromechanical heart model could be the key to developing effective new therapies.” The team also wants to incorporate physiological systems such as coronary blood vessels that feed heart tissue to create a more comprehensive model with even wider applicability.

“HPC can be used for so many applications beyond national security,” says Richards. “Through our collaborations, we want to demonstrate the impact it can have on a broad section of society.”

Streitz adds that HPC involves much more than performing the same simulations in a shorter time. “It’s about doing something in a new way that otherwise would have been impossible.”

—Arnie Heller

Key Words: arrhythmia, Cardioid code, congestive heart failure, Gordon Bell Prize, high-performance computing (HPC), High Performance Computing Innovation Center (HPCIC), Livermore Valley Open Campus (LVOC), Sequoia, Vulcan.

For further information contact Fred Streitz (925) 423-3236 (

View Article in PDF


S&TR Home | LLNL Home | LLNL Site Map | Top
Site designed and maintained by TID’s Web & Multimedia Group

Lawrence Livermore National Laboratory
Operated by Lawrence Livermore National Security, LLC, for the
U.S. Department of Energy’s National Nuclear Security Administration

Privacy & Legal Notice | UCRL-TR-52000-12-9 | September 1, 2012