Page326

Learn by Example

Testing Backup Power and HVAC

While all datacenters have cooling issues or concerns, cooling issues for datacenters in Mississippi during the month of August can be particularly interesting. All organizations recognize that loss of power represents a commonly occurring disruptive event, whether it is as a result of human error, natural disaster, or something in between. In order to accommodate the potential short-lived loss of power without causing significant impact, organizations typically employ uninterruptible power supplies (UPSs) and/or backup generators.

After going through a datacenter refresh that involved HVAC upgrades, powered racks with dedicated UPS, cable management (previously lacking), etc., a Mississippi-based organization felt that power failure testing was necessary. In the event of loss of power, the organization’s design was to automatically switch servers to the new rack-mounted UPS systems, bring up the generator, and then have an operator begin shutting down unnecessary servers to prolong their ability to run without power. The test that was being performed was simply to ensure that systems would automatically fail over to the UPS, ensure that the generator would come up, and ensure that the new process of operators shutting down unnecessary systems worked properly.

After separating the datacenter from power, the rack-mounted UPS immediately kicked in. The generator started up without a hitch. Operators broke the seal on their shutdown procedures and began gracefully shutting down unnecessary servers. However, the operators quickly started complaining about how hot the task of shutting down these systems was. While stress can make people feel a bit warmer, the datacenter director investigated the matter. He found that they had been focused on ensuring that all of the server systems would stay operational until being gracefully shut down, and that they had neglected the new chillers in the datacenter, which had not been considered in the power failure. With hundreds of servers running, no chillers, and a 105°F heat index outdoors, it likely got hot rather quickly.

Warfare, Terrorism, and Sabotage

The height of human-intentional threats is found in the examples of warfare, terrorism, and sabotage. The threat of traditional warfare, terrorism, and sabotage to our organizations can vary dramatically based on geographic location, industry, brand value, as well as the interrelatedness with other high-value target organizations. While traditional physical attacks are still quite possible, an even more likely scenario is cyber-warfare, terrorism, or sabotage. The threat landscape for information systems has rapidly evolved over the years.