Managing IT Points of Failures: Separating a Fire Drill from a Fire

May 21, 2014 | App Modernization, Cloud, Data, Fresh Ink, Security, Social Media

Featured article by Deepak Kanwar, Senior Manager, Zenoss

A red flag on the IT operations dashboard at many organizations still means “All Hands on Deck,” a failure has been detected, it’s time to grab the safety gear and ready the hoses for what could be hours of intense fighting.  Absence of any context or intelligence within the alerts means that reinforcements in the form of SMEs have to be instantly deployed. When the issue is finally addressed and normalcy is restored, IT personnel can return back to their stations staying at red-alert for the next call. Sometimes between alerts reality hits – the “fire” was a low priority one and given its limited impact to the business, did not justify the cost of the resources allocated. It could have simply been handled as a fire drill. Yes, the issue was addressed, but the price in terms of time and resources was very high. And that brings up the question: How can organizations arm themselves with the intelligence to distinguish between a fire drill and an actual fire?

As a first step, organizations are leveraging modern datacenters to minimize failures and disruptions. These datacenters are architected to be resilient and fault tolerant. There is a conscious effort to eliminate single points of failure, and that means each component failure does not result in service disruption or even degradation. Through purposeful adoption of redundancies, one or more backup systems are put in place to ensure the infrastructure can withstand a few hits before the user experiences an issue. For businesses, this means reduced downtime and improved user experience. And for an IT Ops team, this means every alert is not a three-alarm fire that needs immediate attention.

The benefits of these highly available datacenters are undeniable, however, some malfunctions and some disruptions are inevitable. To guard against the worst-case scenario when issues arise, an organization should have a plan for how to manage such issues and answer the following questions:

– Which alerts require a response now and which ones can wait?

– Was the safety net eroded and will the next failure result in a disruption?

– Which services (if any) will be affected by the latest alert?

– Most important, are IT priorities aligned with business goals?

But I Trust My Tools!

Unfortunately, as numerous IT teams are finding out, their legacy monitoring tools can no longer support their dynamic environment in an efficient manner. These tools, designed for an era when environments were static, have no notion of monitoring services that move around in a fluid manner leveraging physical infrastructure one day, and virtualized the next. Deployed in silos, their span of control is limited and they are unable to provide any environment-wide context to an alert. When disruptions do occur, IT teams still find themselves scrambling to address the issues.

A 2013 Forrester Consulting study indicated that 41 percent of respondents spend anywhere from an hour to more than a week to identify the root cause of service problems. With downtime costs easily exceeding $100,000 per hour for many organizations, these hours can quickly add up, making a significant impact on the bottom-line and invariably draw the wrong kind of attention for IT.

To take full advantage of their shiny new datacenters, IT teams need modern monitoring solutions that not only provide them with timely alerts, but also with intelligent context around the event. It is not enough to know that a component is failing – you also need to know if this is one of 10 redundant components and its failure will not cause a service disruption or if this is the last of your safety plugs and the next failure will mean a service disruption.

Modernize the Datacenter with Cloud-era Monitoring!

Simply upgrading infrastructure or datacenter architecture is not enough. To really manage services in an efficient manner, organizations absolutely must modernize their monitoring tools. It is about using tools that provide unified service insight for the entire environment, not just the health of the individual components. An ideal solution keeps track of services and the underpinning components, so you know what services can be affected by what failures and ultimately speed up root cause analysis. Finally, it is important to keep in mind that today’s datacenter environment is dynamic and a mix of physical, virtual and even cloud. If you’re unable to monitor across your environment, then you will be stuck doing element-level management, which is rarely effective and never efficient!

deepakkanwar

Deepak Kanwar is a Senior Manager at Zenoss with product marketing responsibilities for the unified IT monitoring and management solutions. He has over 14 years of IT product marketing & management experience with various leadership roles at BMC Software, Dell, Mezeo Software, etc. Deepak has an MBA from Rice University and is Infrastructure Technology Information Library (ITIL) v3 certified.

Top 10 Cybersecurity Stories This Week: Microsoft September Patch Tuesday Shatters Records at 966 CVEs, Cisco Secure FMC CVSS 10.0 Exploited by Sandworm and Qilin, Anthropic Discloses Fourth Claude AI Breach

Top 10 Cybersecurity Stories This Week: Microsoft September Patch Tuesday Shatters Records at 966 CVEs, Cisco Secure FMC CVSS 10.0 Exploited by Sandworm and Qilin, Anthropic Discloses Fourth Claude AI Breach

September 11, 2026 | ITBriefcase.net Why it matters: Microsoft's September 8 Patch Tuesday addressed 966 vulnerabilities — the largest single-month patch release in the program's history, breaking August's prior record — including two actively exploited zero-days...

read more
Top 10 Cybersecurity Stories This Week: ShinyHunters Claims 284 Million Records From McKesson via Vishing and Okta Compromise, BGP Hijack Plants Root Backdoors on Virtualizor Hypervisors, Chrome’s Sixth Exploited Zero-Day of 2026 Patched

Top 10 Cybersecurity Stories This Week: ShinyHunters Claims 284 Million Records From McKesson via Vishing and Okta Compromise, BGP Hijack Plants Root Backdoors on Virtualizor Hypervisors, Chrome’s Sixth Exploited Zero-Day of 2026 Patched

September 4, 2026 | ITBriefcase.net Why it matters: ShinyHunters claimed responsibility for a breach of McKesson Corporation — the largest pharmaceutical distributor in North America, delivering approximately one-third of all prescription medicines to US hospitals,...

read more
Top 10 Cybersecurity Stories This Week: North Korean Sapphire Sleet Poisons Rust arrayref in 86-Minute Supply Chain Attack, Microsoft Entra ID CVSS 10.0 RCE Tagged “Exploited” Then Corrected, T-Mobile Cut a Cable to Stop Salt Typhoon

Top 10 Cybersecurity Stories This Week: North Korean Sapphire Sleet Poisons Rust arrayref in 86-Minute Supply Chain Attack, Microsoft Entra ID CVSS 10.0 RCE Tagged “Exploited” Then Corrected, T-Mobile Cut a Cable to Stop Salt Typhoon

August 28, 2026 | ITBriefcase.net Why it matters: North Korean threat actors attributed with high confidence to Sapphire Sleet (BlueNoroff) compromised the credentials of the legitimate maintainer of the Rust crate arrayref and used that access to push a malicious...

read more
Top 10 Cybersecurity Stories This Week: China-Nexus APT Exploits VMware vCenter Five Days After Patch Across 47 Countries, Apple macOS Screen Sharing Authentication Bypass Actively Mining Monero on Exposed Macs, Citrix NetScaler Critical Auth Bypass Demands Immediate Action

Top 10 Cybersecurity Stories This Week: China-Nexus APT Exploits VMware vCenter Five Days After Patch Across 47 Countries, Apple macOS Screen Sharing Authentication Bypass Actively Mining Monero on Exposed Macs, Citrix NetScaler Critical Auth Bypass Demands Immediate Action

August 21, 2026 | ITBriefcase.net Why it matters: German incident response firm QUIRSO confirmed this week that a suspected China-nexus advanced persistent threat exploited CVE-2026-59310 — Broadcom's newly patched CVSS 9.8 VMware vCenter directory traversal — just...

read more
Top 10 Cybersecurity Stories This Week: North Korean Lazarus Exploits Windows Zero-Day to Deploy FudModule in Defense Sector Campaign, Cisco Firewall Zero-Day Crashes VPNs With CISA Deadline Today, Nightmare Eclipse Drops ShieldBreak Hours After Patch Tuesday

Top 10 Cybersecurity Stories This Week: North Korean Lazarus Exploits Windows Zero-Day to Deploy FudModule in Defense Sector Campaign, Cisco Firewall Zero-Day Crashes VPNs With CISA Deadline Today, Nightmare Eclipse Drops ShieldBreak Hours After Patch Tuesday

August 14, 2026 | ITBriefcase.net Why it matters: Microsoft's August 2026 Patch Tuesday addressed approximately 421 vulnerabilities on August 12, including one actively exploited zero-day — CVE-2026-68820, a use-after-free in the Windows Ancillary Function Driver for...

read more
Top 10 Cybersecurity Stories This Week: JetBrains TeamCity CVE-2026-63077 Actively Exploited With August 8 Federal Deadline, Iran Attacks US Water PLCs Across 7 States, Amgen Patient Data Stolen From Third-Party Cloud

Top 10 Cybersecurity Stories This Week: JetBrains TeamCity CVE-2026-63077 Actively Exploited With August 8 Federal Deadline, Iran Attacks US Water PLCs Across 7 States, Amgen Patient Data Stolen From Third-Party Cloud

August 7, 2026 | ITBriefcase.net Why it matters: CISA added CVE-2026-63077, a CVSS 9.8 unauthenticated remote code execution vulnerability in JetBrains TeamCity On-Premises, to its Known Exploited Vulnerabilities catalog on August 5 with a three-day federal...

read more