LivIT Meets the Demand

The children’s book If You Give a Mouse a Cookie simplifies the universal truth that one need opens the door to another need and so on, in a cascading fashion. Similarly at Lawrence Livermore, the best and brightest researchers require access to the most capable and cutting-edge information technology (IT) resources. Access to such high-level resources requires an elite IT team such as the Livermore Information Technology (LivIT) group to keep the Laboratory’s crucial computing infrastructure operating smoothly.

Led by the guiding principles of simplicity, agility, and capability, LivIT regularly improves operations to better equip and empower users. The group partners with users in divisions and directorates across the Laboratory, learning from them and anticipating their needs to help accomplish the Laboratory’s goals.

Five-Program Structure

To optimize scope and maximize competencies, LivIT deploys 450 staff across five programs. Business Enablement (BE) oversees 250 customized applications such as the timekeeping and training platforms TIME, LAPIS, and LTRAIN, familiar to all Laboratory employees. The Workforce Enablement (WE) group supports end users by keeping Webex, video teleconferencing support, email, and other collaboration tools up and running. The Customer Service Operations Support (CSOS) team performs myriad support functions for many of LivIT’s customer-facing services such as the Service Desk, desktop and infrastructure support, cyber security support, mobility, and communications. The Cyber Security Program (CSP) provides vital defense for network, desktops, infrastructure, and intellectual property. The Systems and Network Technology (SNT) program oversees the Laboratory’s data center, networking, hardware, and Webex and video teleconferencing infrastructure.

The five-program structure maximizes LivIT’s ability to respond to changing needs. As Matthew Myrick, Chief Information Security Officer, explains, “While LivIT is a service organization, some people aren’t aware that we offer many amazing services beyond our Service Desk. LivIT often operates transparently to better modernize, transform, and protect all aspects of the Laboratory’s IT ecosystem, which permeates virtually every mission area and task performed here.” The five programs have their own unique responsibilities yet operate in concert. Principal Deputy Chief Information Officer Mark Pettit adds, “All LivIT program areas work collaboratively to deliver services—not unlike the way organs of the human body provide specialized functions so the body can operate effectively as a whole.”

Pandemic Demand Management

LivIT’s overall function relies on demand management: as the demand for Laboratory capabilities increases, so does the need for IT resources to help empower mission-driven work. The Laboratory’s range of teams, programs, and directorates requires automation and efficiency in business processes, strategic mission and business enablement solutions, enhanced communications and collaboration, and data availability. Sue Marlais, Livermore’s Chief Information Officer, says, “Users rely on IT more and more to do their jobs with very little tolerance for down time. We invariably have a significant amount of new scope to incorporate as we continue to manage with ongoing operational IT.”

When an elite IT team must function throughout a pandemic, demand management takes on new dimensions. Kristin Ruley, CSOS Program Leader, sums up the earliest technical hurdle of the pandemic: “Our biggest challenge was ensuring that the transition from onsite work to offsite work felt seamless for Laboratory employees. We had to be certain that people could access their systems and tools remotely and effectively carry on with work virtually, something the Laboratory population hadn’t been that familiar with. Because LivIT has an eye on the future, we already had many of the necessary tools and processes in place to do just that, in a matter of days.”

Not all pandemic-related challenges to LivIT’s infrastructure were simple to solve. Teams rapidly scaled up remote access capability to provide additional capacity while mitigating the risks from being unable to access crucial data. Once LivIT personnel understood that most in-person work would cease, they had to quickly evaluate and implement new digital collaboration capabilities such as Webex and MS Teams while expanding capabilities within OneDrive and the Office 365 space. To ensure successful continuity of support operations, the LivIT Service Desk, which handles an average of 6,000 contacts each month, transitioned to fully remote work a week ahead of the rest of the Laboratory.

“If luck occurs when preparation meets opportunity then, in many ways, the Laboratory got lucky,” says Myrick. “We had already been piloting a new virtual private network (VPN) system, which could easily be operationalized to help sustain the load of the entire Laboratory population. We had recently deployed a best in class, cloud-based Endpoint Detection and Response software on all Lawrence Livermore systems, which allowed us to extend system protection whether onsite or offsite.”

While Livermore employees focused on the technology they “touch” every day—softphones, laptops, peripherals being mailed home in the early days of the pandemic—many crucial IT issues were tackled behind the scenes. Content classification and approved usage of collaboration tools, including cyber security risk mitigation, was a major concern. LivIT also reduced VPN access requirements, provisioning accounts for all Laboratory employees and bypassing thousands of hours of administration and approvals.

Hybrid Workforce

New requirements led LivIT to develop or modify applications to provide, for example, onsite employee tracking and contact tracing, vaccine card validation, and development of a workspace methodology to support a first-ever virtual summer student intern program. Navigating onsite–offsite collaboration issues was trickier for classified work. The Weapons and Complex Integration organization sponsored LivIT to architect and implement a classified Webex-like meeting capability—Meet@LLNL—enabling the safe, effective continuation of classified program mission work. Given pandemic travel restrictions, the Meet@LLNL platform provided for appropriate social distancing and ongoing collaboration.

Once technological guardrails were put in place, IT business processes were adapted to support an increasingly hybrid workforce, from enhancements to distance support methodologies and practices, to equipment deployment and delivery. Pettit notes, “The pandemic and the hybrid nature of our workforce has created a sense of urgency to access all things digitally, all the time, from any platform, anywhere. Remote access is now a necessity and one that drives much of the work we do. We demonstrated our ability to adapt and become agile in responding to emerging needs in a rapid and meaningful manner.” Pettit also notes that as IT professionals, the LivIT staff may have been able to transition to hybrid work more rapidly than most. “The tools and capabilities are more familiar, since we use them to support and interact with Laboratory employees every day,” he says. “If anything, a hybrid workplace has dramatically accelerated the pace of the work that we do as well as the speed with which we do it.” Ruley agrees, “Because we leverage modern technology and capabilities, LivIT as an organization can perform a lot of its work remotely, therefore enabling our staff a greater work–life balance through a healthy telecommuting posture.”

When it comes to lessons learned and best practices, the LivIT team has no shortage of examples. Marlais is especially proud of LivIT’s mastery of improved collaboration tools. “Live events, like the Director’s All-Hands meetings, are comparable to producing a TV show with many moving parts. LivIT has done an exceptional job delivering this type of service.”

Another major pandemic-era achievement was the launch of the LivCloud, enabling Laboratory employees to conduct scientific research and IT development work in the cloud. More than 90% of LivIT’s computing infrastructure migrated to cloud platforms such as Amazon Web Services (AWS), with the production environment moved in a single weekend. “Cloud migration significantly reduced LivIT’s onsite computer infrastructure and increased our resiliency by leveraging highly redundant and more capable cloud resources,” says Myrick. Pettit adds, “Cloud migration was the very first step toward a capability maturity that will enable us to become more agile and adaptive to the emerging needs of our program customers. The goal is to deliver solutions and capabilities quickly while also being able to adapt as needs change.”

The team also leveraged AWS’s machine-learning tools to validate COVID-19 vaccination cards for Laboratory staff. Using an image recognition tool, a model was created to discern vaccination card images submitted by employees. If the image was recognized to be a valid vaccination card, then an optical character recognition tool extracted the employee’s information from the image. “With these tools the team was able to quickly validate and process the information,” says John Lee, LivCloud technical lead.

In looking ahead to meet the Laboratory’s increasing need for new solutions and capabilities, the LivIT team will retain a focus on demand management while maximizing agility. Pettit says, “One of our primary goals is to deliver innovative solutions that enable our customers to realize or exceed their mission goals beyond what they had even imagined. This requires a significant investment in LivIT as a key mission enabler. We can bring so much more to the table.”

—Stephanie Turza

Key Words: cloud migration, cyber security, demand management, LivCloud, Livermore Information Technology (LivIT), remote work, Service Desk, virtual private network (VPN).

For further information contact Sue Marlais (925) 422-7682 (marlais1@llnl.gov).