Downtime doesn’t compute at AI data centers
These are words nobody wants to hear in business. Data center downtime is among the worst kinds of downtime, with estimates pegging it as high as $9,000 per minute.
An AI data center glitch today would be disrupting enough for a chatbot user. But it would be a bigger headache for a global business relying on autonomous agents to drive core business processes. It would be even bigger for a hyperscaler hosting multiple companies’ virtual data centers, and biggest yet for a tech giant aspiring to dominate the AI era.
According to a recent Uptime Institute survey, half of data center operators experienced an “impactful outage” at their facility in the past three years, most often due to power issues. In this article, however, we’ll look specifically at how one age-old risk—fire—can threaten this new-age technology.
A growing concern
Although fires have been rare in data centers, the risk is clearly growing along with data center construction. Add AI data centers’ increased energy density, pervasive in-rack backup power and ad hoc layouts, and the risk gets particularly complex—especially at a time when resilient businesses must ensure that the race to build more data centers doesn’t become a race to the bottom of risk quality.
In some high-profile incidents, data centers have burned for hours; others multiple times in one year. Even small fires can result in significant downtime.
Among the sources of fire risk for data centers:
- Electrical failure: Surges and arc flashes can ignite fires.
- Equipment: Servers, GPUs and cooling consume enormous volumes of heat-generating electricity, increasingly requiring on-site power generation and high-voltage connections.
- Batteries: Lithium-ion batteries provide backup power in server racks, uninterrupted power-supplies (UPSes) and larger-scale storage of renewable energy. These devices are both fire ignition and fuel sources. Damaged or defective batteries are subject to thermal runaway, a chain reaction that leads to overheating, fires and explosions.
- Networking: Wires and fiber optic cables, once contained under suspended data center floors, are today more likely to run in combustible plastic trays above servers. There’s more oxygen (fuel) in that space and less separation from computing equipment.
- Partitions: New data center designs employ alternating hot and cold aisles to optimize cooling. Plastic partitions between the aisles can be combustible.
- Human error: Hot work like welding often leads to accidents.
- Green facades and mass timber construction: Although these features offset companies’ carbon footprint, they can be combustible. Solar panels can introduce new ignition points as well.
- Ignitable fluids: Glycol and refrigerants used in liquid direct-to-chip cooling or mineral oil for immersion cooling are flammable at high concentrations and temperatures. Onsite power generation involves ignitable fuels like diesel or natural gas.
Overall, the high variability of data center layouts and the rush to build them make it difficult to tailor fire protection in advance. One thing is certain: “Given the great urgency to pack data centers with equipment that can overheat, cooling is king,” says Bob Kasiski, staff vice president, senior engineering technical specialist at FM. “It needs to be safe and efficient.”
Resilience is paramount
For nearly two centuries, FM has been committed to preventing loss by using science and engineering to investigate hazards. In understanding, quantifying and mitigating the risk, FM helps clients avoid disruption and rebound quickly if necessary. “Our goal is turning potential disasters into minor disruptions at most,” says Kasiski.
To that end, FM performs extensive hands-on research at the FM Research Campus in West Glocester, R.I. USA, which houses the largest fire technology lab in the world. The work has yielded three important developments:
Water mist systems: FM’s research team has determined that water mist systems can be a viable fire protection solution for AI data centers. Water mist systems typically require less water than sprinkler systems to control or extinguish a fire, and they potentially cause less water damage to sensitive computing equipment. They are common in Europe, where access to public water can be limited.
FM Approvals—the independent, third-party testing and certification body and subsidiary of FM—evaluates property loss prevention products and solutions to globally-recognized standards. FM Approvals has certified water mist systems for the low-, medium- and high-risk profiles (office space, data processing equipment room with or without Li-ion battery back-up units, emergency standby generator sets) included in data centers. These, and other, FM Approved products are listed in the Approval Guide while the use of tested and certified products is referenced FM Property Loss Prevention Data Sheets.
Off-gassing systems – Products known as off-gas detectors may provide early enough warning, in conjunction with an integrated mitigation strategy, to help prevent thermal runaway before it begins. This year, FM Approvals announced a first-of-its-kind approval of a lithium-ion battery off-gas detector for open spaces and data center rooms.
A full catalogue of risks and best practices – “FM Property Loss Prevention Datasheet 5-32 – Data Centers and Related Facilities” is a comprehensive and detailed examination of data centers, their risks and guidance for managing those risks. Data sheets like this one constitute standards so respected that regulatory authorities often adopt them in enforceable building codes. FM 5-32 covers construction materials, fire detection, fire suppression, cabling, storage, power, planning and security.
Other mitigation measures
FM also advises clients to take additional actions to reduce risk in their AI data centers.
- Keep larger lithium-ion battery systems and storage units outside and separate from the main data center building.
- Install high-sensitivity smoke detection systems to provide the earliest possible warning of a fire.
- Consider clean-agent suppression systems that can extinguish fires without harming electronics.
- Use automatic electricity and HVAC shutdown mechanisms to help minimize loss.
- Minimize glycol concentration in cooling fluid; avoid oils entirely. Frequently inspect systems for leaks.
- Implement real-time monitoring of energy storage systems as well as rigorous maintenance and inspection protocols of all data center operations. .
- Perform risk reviews for occupancy changes and tenant-specific installations.
- Consider “damage-limiting construction,” e.g., walls that readily release pressure in a gas explosion.
- Develop and rehearse emergency response plans.
Kasiski and his teammates acknowledge there’s a lot of caution and scrutiny around fire risk. But the point is not to impede business. It’s to unleash business by avoiding delays, disruptions and crippling losses by being resilient, protected and able to sail through challenges to business continuity.
“Everything we do is to benefit our clients,” says Kasiski.
And none of them ever wants to hear, “We’re down.”
Related Content:
Read more about how FM Intellium integrates data-driven research, hands-on engineering expertise and global risk management into a dynamic, scalable solution—empowering you to stay ahead in the fast-changing data center ecosystem.