In 2019, a massive explosion at an energy storage facility in Surprise, Arizona, badly injured four firefighters and exposed numerous safety gaps. With battery installations rapidly accelerating worldwide, have we learned enough to prevent the next Surprise?

BY JESSE ROMAN

Late in the evening on April 19, 2019, Bobby Ruiz, the fire chief in Peoria, Arizona, was at home preparing for the week ahead when he received the phone call every chief fears. He jumped in his car and raced to the Maricopa Medical Center in nearby Phoenix, arriving just as the medivac helicopters began unloading two of his firefighters: Hunter Clare, a captain, and Justin Lopez, a fire engineer.

The situation was bleak. Both men had multiple broken bones, internal bleeding, organ lacerations, traumatic brain injuries, and thermal and chemical burns. “As they’re working on them, a doctor looks over at our chaplain and tells him, ‘I want you to get over there by his head and start praying over him—he needs it right now,’” said Ruiz, reliving the tense moments inside the emergency room. “I’ve been in the fire service 40-plus years, and I have never, ever heard a doctor tell the chaplain to go over there and start praying over him. It’s a miracle they both survived.”

Hours earlier, Clare and Lopez, as members of the Peoria Fire-Medical Department’s HAZMAT team, had been called to the McMicken Energy Storage facility, located in a remote corner of Surprise, Arizona, a neighboring suburb 20 minutes outside of Phoenix. No one on the team had heard of McMicken, and when they arrived it was unclear what they were looking at. Surrounded by a chain-link fence at the edge of the desert, a metal box the size and shape of a large shipping container was leaking milky white smoke. The structure, they soon discovered, housed a 2-megawatt energy storage system (ESS) owned by local utility Arizona Public Service; hundreds of the system’s lithium-ion battery cells had experienced a catastrophic failure and were in a dangerous state known as thermal runaway.
RELATED CONTENT
SIDEBAR: Power Struggle

SIDEBAR: How much protection is enough?

Explore all NFPA resources, including past NFPA Journal coverage, related to energy storage systems

After hours of deliberation, Clare and Lopez finally opened the door of the battery container to ventilate it when a sudden explosion rocked the facility. Those outside of the blast zone described “a jet of flame that extended at least 75 feet outward and an estimated 20 feet vertically” from the door of the battery container, according to an extensive report on the incident published last year by the UL Firefighter Safety Research Institute. The force of the explosion blew Clare underneath the chain-link fence, and he came to rest 70 feet from the container door. Lopez was thrown violently into the fence, some 30 feet from the structure. Two other nearby firefighters were knocked unconscious, their breathing apparatus and helmets ripped away and strewn across the dusty ground.

By the next day, news of what happened in Surprise had spread in the media and became the talk of fire departments, researchers, and fire safety officials around the world. Though researchers had understood that an explosion involving a lithium-ion ESS was possible, examples of this type of powerful deflagration were scarce. The push for answers to what had happened began almost immediately.

NFPA investigators visited the site and interviewed Clare and others. UL undertook an exhaustive fact-gathering effort, ultimately published in the report “Four Firefighters Injured In Lithium-Ion Battery Energy Storage System Explosion – Arizona.” Arizona Public Service (APS) commissioned its own 70-page report on the incident, as did LG Chem, the manufacturer of the lithium battery at McMicken.

Learning why the explosion occurred and how it could have been avoided was deemed critical. ESS are becoming ubiquitous, not just in remote stretches of desert but in suburban neighborhoods, high-rise buildings, and office parks—even in people’s homes. As this proliferation continues, the incident at McMicken confirmed for some a long-simmering fear: that the fire service and safety community are unprepared to deal with this burgeoning technology.

At the same time, the event in Surprise offered perhaps the most complete picture yet to illustrate why battery incidents are so challenging for first responders, and the factors that continue to dog researchers and manufacturers in the quest to make them safer.

“What the Surprise incident highlights is that there is still a tremendous lack of knowledge in many, many areas” of this technology, said Stephen Kerber, director of the UL Fire Safety Research Institute (formerly the Firefighter Safety Research Institute) and co-author of the UL paper on the McMicken battery explosion. “With so many stakeholders involved, this event seemed like the perfect case study. That’s why we invested so many resources towards this project—so we can learn and keep this from happening again.”

THE DETAILS OF SURPRISE

The UL report on the Surprise incident paints a picture of a knowledgeable fire department put in a no-win situation.

Clare and the Peoria HAZMAT team arrived at the McMicken facility at about 6:30 p.m. on April 19, 2019, an hour or so after a team from Surprise Fire-Medical Department, who were dispatched after a passerby on the highway reported seeing smoke in the area. Firefighters from Surprise had assumed the source of the smoke was a brush fire and arrived with a tanker/tender truck and wildland firefighting gear. Instead, they found a structure 50 feet long, 13 feet wide, and 12 feet high surrounded by a chain-link fence. The words “Battery Storage Array” were stenciled on its corrugated metal exterior.

The lithium-ion ESS inside consisted of 27 vertical racks of battery modules, separated into two rows on either side of a 3-foot hallway. APS used the battery to store solar power collected during the day, then discharged the energy into the electrical grid for customer use at night.

At about 5 p.m., the battery’s management system detected a problem in rack 15. Voltage began to rapidly decrease, and sensors at the top of the rack registered rapidly rising air temperatures, peaking at more than 120 degrees F. A pair of battery cells in module 2 of rack 15 had gone into thermal runaway, the APS report concluded.

While it’s unclear what triggered thermal runaway in Surprise, its effects followed a predictable course. The failure caused heat to build rapidly in the lithium-ion cell and spread to surrounding cells, triggering them to go into thermal runaway—a cascading effect that is difficult to stop. About a minute after the initial failure, the McMicken battery’s suppression system was triggered. Rather than water, as is now recommended, the ESS was equipped with a clean agent gas system, which in 10 seconds filled the entire structure with 713 pounds of the gas Novec 1230. By design, the gas made conditions inside the container inhospitable for active flames—but did nothing to stop the rising heat from the chemical reaction happening inside the battery. The thermal runaway went on unabated.

By the time Surprise firefighters arrived, white smoke they described as “acrid” and “nasty” hung low to the ground in a radius of about 30 feet surrounding the ESS structure. They called for backup, and Clare and his HAZMAT team arrived soon after.

Clare had experience with ESS fires. In 2017, he had responded to a fire at an electric vehicle repair facility involving lithium-ion batteries, and later attended HAZMAT classes focused on lithium-ion battery fires. He had attended an NFPA ESS awareness training focused on solar panels, as well as similar local trainings. He knew to be extremely cautious with battery fires. At the Peoria department, Ruiz said, “It’s always been our philosophy that, when it comes to batteries, if it’s burning, let it burn because it’ll consume itself faster than putting water on it.” Thanks to the Novec gas, the McMicken battery wasn’t burning, but smoke continued to ooze from the container.

Available research at the time had revealed that the gases released during thermal runaway are a highly flammable mix of hydrogen, ethylene, methane, and carbon monoxide, among others. However, few understood just how much of this flammable gas can be produced by a relatively small number of cells. Subsequent tests have shown that “it’s just a tremendous amount of cubic feet of off-gassing, lots of flammable and toxic gases from just a few cells,” said Ruiz, who participated in research at the University of Texas after the Surprise incident. During small-scale tests, so much gas was produced in a container that the atmosphere was too rich to explode; researchers had to use fans to blow much of it away to achieve a blast similar to the one that occurred in Surprise, Ruiz said.

Over the course of two hours at McMicken, the firefighters made three journeys inside the fence that surrounded the ESS structure to take gas and temperature readings of the container and the vapor spilling out. On each occasion, they detected hazardous levels of carbon monoxide and hydrogen cyanide gases—but concentrations of the gases were decreasing. The HAZMAT team’s instincts and training told them to be cautious, but they were running out of options. “The concern was that if we turned the structure back over to the utility, we’d leave and they’d go into a highly toxic environment and drop like canaries,” Ruiz said.

The team decided to open the door of the structure to vent it and see what was happening inside. A shaky video taken during the entry shows Clare and Lopez under a black night sky in full turnout gear as they approach a side door. Lopez turns a key and pulls the door open; a dense cloud of white gas can be seen streaming from the doorway as the men turn and peer inside. Clare holds up a thermal imaging camera to take a temperature reading, then turns to grab a gas meter and radios back to his crew to report there is no active fire or electrical arcing visible inside the structure. A few seconds later, the video abruptly ends just before the explosion.

Investigators later concluded that the open door allowed oxygen to enter and some of the trapped flammable gases to exit. In the three minutes the door was ajar, the gases reached just the right mix and concentration to enter the explosive range. One of several potential ignition sources inside the container—such as the hot battery cells—did the rest, triggering a massive deflagration.

Following the blast, Lopez and Clare endured lengthy hospital stays, multiple surgeries, and a grueling regimen of physical therapy. The two other injured firefighters, Matt Cottini and Jake Ciulla, suffered burns and headaches, among other lingering issues. All four have been back at work for about a year, according to Ruiz. “They’re all doing fairly well—as well as could be expected,” he said.

According to Kerber, the decision to enter the structure was consistent with training guidance at the time. “It recommended ventilating the structure and going in and taking samples, which seems like an okay approach, but clearly the outcome we had here was unfortunate,” he said. “I think obviously what that says is there are big knowledge gaps that we need to fill and recommendations and tactical considerations that still need to be developed.”

NO HELP FOR FIREFIGHTERS

While it’s uncertain what firefighters on the scene knew about the explosion hazards posed by ESS, the reports make it clear that they received little technical help from subject-matter experts on-site. According to the UL report and Ruiz, representatives from the utility and battery maker had a limited response plan and offered little guidance to the HAZMAT crews about possible courses of action. “There were no safety material sheets on-site,” Ruiz said. “Representatives from APS arrived, but they were electrical engineers and didn’t know much about the byproducts off-gassing from the battery in thermal runaway.” Ruiz said one on-site staffer told his firefighters that the white smoke oozing from the battery was Halon, a gas formerly used in suppression systems that had been banned for nearly two decades. “We knew it certainly wasn’t Halon, so that was kind of concerning,” he said.

APS said that it, too, was not properly educated on the battery’s hazards, and it claims that the responsibility of that oversight lies with the battery system designer, a firm called AES, and the battery manufacturer, LG Chem. In its own report on the incident, the utility said that “the emergency response plan provided by AES to APS did not have instructions on how to respond to a potential explosion or how to enter the system after the fire suppression system had been discharged.” Further, in that emergency response plan, neither AES or LG Chem “had conveyed that a large flammable gas hazard or cell-to-cell and module-to-module cascading thermal runaway was possible.”

In a detailed paper responding to the APS report, LG Chem claimed that all of “these hazards were widely known” before the McMicken site was designed and constructed and that the onus of crafting emergency response plans falls jointly on the battery owner and system designer—in this case, on APS and AES. According to LG Chem, “The time to develop careful plans to protect first responders is during the design and commissioning and operation of these projects—not after an incident has started.”

Despite this apparent lack of understanding and planning by the parties involved, the catastrophe could still have been avoided if engineering systems and the proper tools were in place to give firefighters a better sense of what was happening inside the ESS, according to Mark McKinnon, a co-author of the UL report.

“An up-to-date history of the measurements of gas composition and potentially the percent LEL (lower explosive limit) inside the structure would have been the best information for the firefighters on scene, and undoubtedly would have changed their response and would have prevented the injuries,” McKinnon told NFPA Journal. No gas monitoring capabilities were built into the McMicken battery, however, and what diagnostic systems did exist were handled by the battery management system, which failed minutes before firefighters arrived.

To make matters worse, the basic tools that firefighters use during responses are generally ineffective for ESS incidents. In subsequent tests at the UL laboratories, McKinnon found that, due to the sooty and rapidly changing conditions of a thermal-runaway event, standard thermal cameras and gas meters have difficulty providing reliable information about whether a battery is on fire, which battery modules are undergoing thermal runaway, or even what gases are being emitted.

In effect, the Peoria firefighters had to make decisions in the absence of any information whatsoever. Much of that could have been mitigated if the department had been made aware of the facility, had worked with the utility to develop a preplan for the incident, and even trained on-site, Kerber said. But none of that happened.

“What happens a lot with new technology is there’s an expectation that the fire department immediately becomes an expert on how to mitigate a hazard that they respond to, and I don’t think that’s a fair expectation,” Kerber said. “There has to be research upfront, there have to be codes and standards in place, and sometimes the technology gets ahead of that. In this case, there was really no good guidance. There was no good training.”

WOULD NFPA 855 HAVE HELPED?

According to after-incident reports, there were several design flaws with the McMicken battery that contributed to the eventual outcome. Part of that was likely because battery design-and-installation standards were limited at the time the battery was installed in 2017; one of the most cited standards, NFPA 855, Standard for the Installation of Stationary Energy Storage Systems, did not come out until late 2019.

Still, experts are divided over whether a strict adherence to NFPA 855 would have made a significant difference in the McMicken case. While the standard is comprehensive, as with everything involving ESS, any attempt to impose hard and fast rules quickly runs up against the realities of technical capabilities, available knowledge, and politics.

For instance, had NFPA 855 existed before the installation of the McMicken battery, it’s doubtful the facility would have even been subject to its requirements. For a variety of reasons, batteries owned and operated by utilities are exempted from most of NFPA 855, a fact that the UL report recommends changing and which is still being debated (see “Power Struggle”). As a result, the McMicken facility was not required to have—or to offer the fire department—an emergency operations plan, something UL report authors say would “have made a significant difference.”

Experts say that if McMicken had been subject to NFPA 855, several factors may have made a difference in the outcome. For one, NFPA 855 requires most ESS to have deflagration venting and explosion prevention, features the McMicken battery lacked. To the contrary, the McMicken’s NOVEC 1230 suppression system featured dampers that were “designed to seal when the suppression agent was discharged to prevent venting that could decrease the effectiveness of the clean-agent suppressant,” the UL report said. Rather than vent flammable gases to keep them below the lower explosive limit, the McMicken system held them in. Exacerbating the problem, the NOVEC gas prevented a fire from ever starting in the container, leading to even more accumulation of unburned gases, the APS report concluded.

By comparison, NFPA 855 requires energy storage systems to follow NFPA 68, Standard on Explosion Protection by Deflagration Venting, or NFPA 69, Standard of Explosion Prevention Systems—either of which “would have potentially changed the outcome here,” McKinnon said. But he also says that the venting requirements in NFPA 855 for lithium-ion batteries specifically could be strengthened. That’s especially true now that further research and details from the Surprise incident have revealed the enormous amount of flammable gas produced during thermal runaway in lithium-ion cells.

Jim Biggins, the chair of the NFPA 855 technical committee, agrees. “We’re looking at providing additional guidance in NFPA 855 both for explosion venting and normal venting of the battery enclosures and battery rooms,” he said. “It’s such a large volume of gas coming out of a single cell during thermal runaway, and I don’t think anybody fully understood that mechanism previously.”

The other areas that observers hope NFPA 855 might address may be tougher nuts to crack. All parties agree that real-time remote gas monitoring that would help firefighters understand the composition and makeup of gases produced during thermal runaway would make a significant difference. However, there is uncertainty about how to achieve it. The current edition of NFPA 855 is mostly silent regarding off-gas monitoring because “the technology really wasn’t there to a point where we felt comfortable making it a requirement in the standard,” Biggins said.

Multiple technical challenges, such as the heavy soot produced from thermal runaway, high heat, stratification of gases, and a mixing of gases has the potential to overwhelm and destroy gas sensors inside the unit, which “make it a very complicated problem,” McKinnon said. “Because of those challenges, there are no products or solutions I am aware of that can do this.”

At least one company claims to have real-time gas monitoring and boasts that its technology can even catch atmospheric changes within a battery system in time to take action to prevent thermal runaway. Some, like Ruiz, are bullish on the new technology, while others caution that it’s unproven. Whatever the solution entails, “we’ve got to apply technology so the fire service can get the intel it needs and so it has the ability to intervene in these events remotely,” Kerber said.

One of the main critiques of NFPA 855 in the APS report is the utility’s assertion that the standard does not go nearly far enough in trying to prevent thermal runaway from happening in the first place. The report argues that the standard should require battery designs and installations that can prevent thermal runaway or at least stop it from spreading beyond a single cell. “Although, cascading thermal runaway is mentioned 37 times in at least one NFPA 855 draft, no requirements were written to address it,” the APS report says. “It is perhaps time for the industry to collectively acknowledge that cascading thermal runaway should be stopped or mitigated at the smallest unit possible. The next iteration of standards should finally acknowledge this in codified form.”

While the APS report points to new battery designs that separate individual cells with aluminum plates, high-temperature-resistant plastics, and even air gaps to limit heat transfer and stop the spread of thermal runaway, Biggins says that it’s wishful thinking to believe that current technologies could allow the technical committee to essentially prevent thermal runaway from occurring.

“I don’t think you can develop a lithium-ion battery system that’s not going to have a thermal runaway issue, and even if you could, you may have energy-density problems,” he said. Taking the step in NFPA 855 to require an ESS design that prevents thermal runaway, he said, would be like NFPA’s sprinkler installation standards requiring that “only noncombustible items be stored in a warehouse so you don’t have a warehouse fire.”

CHALLENGES AND OPTIMISM

When it comes to the issue of ESS safety, there is promising news on the horizon—and a steady stream of emerging challenges that keep firefighters and battery designers awake at night.

Firmly in the second category is the fact that there is still no proven suppression strategy to stop thermal runaway in a battery cell. Every measure tried thus far has limitations, and only “address the symptoms rather than the cause of the problem,” as the APS report put it.

Water systems in lithium-ion batteries work effectively to cool the battery and can stop the spread of thermal runaway, but as soon as the faucet shuts off, heat quickly builds and the process resumes. Since thermal runaway can last for days and even weeks, many jurisdictions like Surprise do not have the water supply to adequately handle such a situation. Gas suppression systems like the one used in the McMicken facility are now generally acknowledged to do virtually nothing to stop thermal runaway. Biggins is hopeful that perhaps water mist suppression systems can “provide cooling and maybe even have some ability to knock gas generation down to the floor,” but testing, thus far, is limited. For now, the problem remains unsolved.

Also unresolved are questions of how to respond to an ESS in the midst of thermal runaway, and what to do after the event is over. Had firefighters in Surprise opted to not open the door, they may have been stuck at the scene for days waiting for the situation to resolve. It raises a host of questions with no good answers. When is it safe to turn the system back to the owners? Who is responsible for declaring the scene is safe—the fire department, the battery owner, the manufacturer, or someone else? What if the battery is spewing toxic gases in, say, Manhattan or an occupied building, and not at the edge of the desert? In that case, is the safest tactic for the fire department to still just wait it out?

“Even if we are able to safely ventilate it, where do those gases go, and are we creating secondary hazards because of where they go? I think that’s going to be a major challenge,” Kerber said.

Another major challenge is large consumer lithium-ion batteries, including the Tesla Powerwall, which are becoming more popular. The systems are marketed as a way for homeowners to store solar power and to use for backup energy during outages. While the benefits to consumers are obvious, Ruiz and others can only shrug about what might happen if a serious thermal runaway event occurs inside a residence.

“If you’re out here in Phoenix, Arizona, it’s not good idea to have your battery unit on the outside of the house because it can be damaged with our heat, so they are going to be inside homes,” Ruiz said. “And on this one, I think the cart is way ahead of the horse. It’s not like you can just let it burn because it’s going to burn down the whole house. What do you do?”

When asked about the prospect of big energy storage systems being installed in high-rise buildings in New York, as has been proposed, or on rooftops to collect solar energy, Ruiz is again at a loss. “I certainly don’t have the answer, but I do admire the problem,” he said with a smile.

But there is reason for optimism on these and other issues. National labs, nonprofits like the Fire Protection Research Foundation and academia, and countless private companies are spending many millions of dollars on research to better understand these problems, including what happens during thermal runaway and how to stop or prevent it. What’s more, a greater portion of this research, which had been proprietary, is coming into public view where it can be used to strengthen standards and design better batteries (see “How Much Protection is Enough?”).

This research is already bearing fruit. New gas detection technologies with remote monitoring are making their way into installations across the world. Designs like the McMicken battery, where thousands of battery cells are stacked close together in racks in one big room, seem to be falling out of favor. The trend now is for more modular systems, with smaller battery units housed separately in cabinets so that if one fails it does not spread to the rest. There are now even systems being installed directly outside so that proper venting and trapped flammable gases are not issues.

The Peoria Fire-Medical Department is getting a good view of this progress. Several months after the McMicken explosion, the utility company Salt River Project announced plans to install a massive 250-megawatt ESS project called the Sonoran Energy Center several miles from Peoria. The facility—more than 100 times larger than McMicken—will use Tesla batteries and will be one of the largest ESS installations in the country.

Whereas firefighters hadn’t heard of the McMicken facility until they arrived during the incident, Ruiz said this time utility officials contacted him immediately. In late June, Salt River representatives, along with experts from Tesla, hosted a daylong training for firefighters on the specific battery installation and have offered tours of the site. According to Ruiz, the Tesla battery cabinets are outside, and there will be a remote battery management system. He said the battery cabinets have their own internal cooling systems, as well as ignition systems that can ignite and burn off gases quickly if they detect an excess.

“We used to say we need to fight fire with fire—now in this technological age it’s fight technology with technology,” Ruiz said, “and I do think the direction is definitely going safer.”

But, he added, he’s not letting his guard down. He’s seen what can happen, and he doesn’t want to get that phone call again. “When we’re called, it’s usually when things don’t work out the way they’re supposed to,” he said. “So you have to stay aware, and you have to stay educated.”

JESSE ROMAN is senior editor at NFPA Journal. Email him feedback or story ideas at [email protected].