Power-outage exercises strengthen the resilience of US bases

In recent years, power outages caused by extreme weather or substation attacks have exposed the vulnerability of the electric grid. For the nation’s military bases, which are served by the grid, being ready for outages is a matter of national security. What better way to test readiness than to cut the power?

Lincoln Laboratory is doing just that with its Energy Resilience Readiness Exercises (ERREs). During an exercise, a base is disconnected from the grid, testing the ability of backup power systems and service members to work through failure. Lasting up to 15 hours, each exercise mimics a real outage event with limited forewarning to the base population.

“No one thought that this kind of real-world test would be accepted. We’ve now done it at 33 installations, impacting over 800,000 people,” says Jean Sack ’13, SM ’15, who leads the program with Christopher Lashway and Annie Weathers in the laboratory's Energy Systems Group.

According to a Department of Energy report, 70 percent of the nation’s transmission lines are approaching end of life. This aging infrastructure, combined with increasing power demands and interdependencies, threatens cascading failures. In response, the Department of Defense (DoD) has sharpened its focus on energy resilience, or the ability to anticipate, withstand, and recover from outages. On a base, an outage could disrupt critical missions, open the door to physical or cyberattacks, and cut off water supplies.

“Threats to this already-fragile system are increasing. That's why this work is so important,” Sack says. 

Safely cutting power

Before an exercise, the laboratory team works closely with base leadership and infrastructure personnel to carefully plan how it will safely disconnect from utility power. Over multiple site visits, they study each building and mission to understand power capabilities, ensure health and safety, and develop contingency plans.

“We get people together who may never have spoken before, but depend on one another. We like to say ‘connecting mission owners to their utility providers,’” says Lashway, a former electrician turned energy-systems researcher. “The planning process is a huge learning opportunity, and a chance to fix issues ahead of the outage.”

On the day of the outage, laboratory staff are on site to ensure the process runs smoothly, but the base is meant to run the exercise. Since beginning in 2018, the ERRE campaign has reached huge installations, including Fort Bragg, a U.S. Army base in North Carolina that sees nearly 150,000 people daily, and sites as far away as England and Japan.

The key is to not limit its scope. All facilities and missions, especially those that are critical, should be included, and service members are tasked with working through issues. To make exercises even more useful as an evaluation of readiness, some are modified with scripted scenarios simulating real-world incidents. These scenarios might challenge personnel to handle a cyberattack to control systems, shutdown of a backup power plant, or a rocket launch during an outage.

“We can do all the tabletop exercises in the world, but when you actually pull the plug, the question is, what actually goes on?” former assistant secretary of defense for sustainment Robert McMahon said at a joint House Armed Services subcommittee hearing about initial exercises. “Perhaps the most important lesson that I've seen is a lack of appreciation and understanding by our senior leaders at the installation level, all the way up to my level, of what we thought was going to happen versus what actually occurred, and then being able to apply those lessons learned.”

Illuminating issues

The ERREs have brought to light common issues across bases. One of them is a reliance on fragile or faulty backup systems. For example, electronic equipment experiences a hard shutdown if it isn't supported by a backup battery to bridge power transitions. In some instances, these battery systems failed or unexpectedly depleted due to age or generator issues. “We see a giant comms room drop out, and then phones and computers don’t work. It emphasizes the need for redundancies,” Lashway says.

Generators also present issues. Some fail because they aren’t regularly serviced or refueled through the long outage. Sometimes, personnel mistakenly assumed a generator would support their entire building, requiring reconfigurations after the fact. Air conditioning systems are often excluded from generator-supported emergency circuits, but rooms with a large number of computers generate a lot of heat, and overheated equipment quickly shuts down.

The exercises also unveiled interdependencies and chain reactions. In one case, a fire-suppression system accidentally went off, dousing a hangar in foam. The cause was a pressure drop at the same exact moment a switch reset.

“Executing an operation at this scale stresses how each of these factors need to work harmoniously and efficiently to ensure that the base, and ultimately missions, remain functional,” Lashway says.

Beyond resolving technical issues, the exercises have been valuable for practicing coordination and following chains of command. They’ve also revealed social challenges of operating through outages. For instance, some DoD guidance restricts the use of generators at daycare centers, so parents needed to coordinate care while maintaining their mission. 

After an exercise, the laboratory compiles all findings in a report for the base. It provides time stamps of significant events by building, identifies links between issues, and summarizes common problems site-wide. It then provides recommendations to address vulnerabilities. “Our goal is to provide as much justification as possible for the base to get the resources they need to fix a problem,” Sack says. 

The researchers also want to help bases prevent issues and avoid costly repairs. Recently, they’ve been using power meters to capture electrical data before, during, and after an exercise. These monitoring tools reveal power-quality issues that are otherwise hidden.

“Not all power is created equal, and standards must be followed to ensure equipment, especially specialized military equipment, operates properly and doesn’t get damaged over the long term. Power metering provides a view into that,” says Lashway.

Sparking resiliency ahead

Lincoln Laboratory’s ERRE campaign has resulted in legislation. In 2021, Congress passed a law requiring each military branch to perform at least five ERREs, or "Black Start Exercises," per year through 2027. That law was recently reauthorized until 2032. The team has transitioned the ERRE process to two private companies, as well as within the Air Force and Army, to conduct exercises in the coming years.

“It's very exciting that this got Congress' attention and has scaled across the DoD,” says Nick Judson, who leads the portfolio of energy, water, and natural hazard resilience efforts within the Energy Systems Group. “This idea started out as a way to enable change on DoD installations, and included a lot of difficult conversations about turning the power off to critical missions, and now we're seeing significant improvements to the readiness of bases and their missions.”

It may even be encouraging some healthy competition across the services, Lashway says. At a recent regional event in Colorado, three U.S. Space Force installations each vied to push the scope and duration of their exercises.

The team’s focus is now turning to related analysis, such as water resiliency. Water and wastewater systems are vulnerable to disruptions beyond power outages, including equipment failure, sabotage, or water source depletion.

“We are conducting tabletop exercises and workshops uniting stakeholders around the importance of water and wastewater systems to enable missions,” says Amelia Servi, who leads this work. “So far, we’ve seen great engagement from groups managing water systems who have been seeking funds to fix these aging systems, and from missions who have previously taken water for granted.”

They are also working on long-term energy planning, including ways for installations to be less dependent on the grid. One way is to install microgrids, which are self-sufficient systems that can tap into stored energy. According to Sack, microgrids are highly customized and complicated to operate, so one goal is to design a standardized system. The team's recent power-metering data is providing useful initial inputs into such a design.

The researchers are also considering how this work could improve energy resiliency for civilians. Large-scale exercises might not be feasible for the public, but they could be conducted in areas important to public safety, or in places that rely on military resources. During one exercise in Georgia, city residents partially depended upon a base's power plant, so that exercise included working with the city to ensure its resiliency to the outage.