📌 Quick Navigation
I've been in IT operations for over a decade, and if there's one thing that still makes my stomach drop, it's the phrase "emergency change request." Not because the process is broken, but because so many teams get it wrong. They either bypass the process entirely (firefighting mode) or they treat every minor hiccup as an emergency, which kills productivity. So let me clear the fog.
Defining the Emergency Change Request
An emergency change request is a formal procedure to implement a critical modification to an IT infrastructure, application, or service when a major incident or security vulnerability demands immediate action. The key word is immediate — you can't wait for the next standard change window because the business is already down or about to be breached.
In ITIL terms, an emergency change is a special type of standard change (yes, read that twice) that follows an accelerated approval process. But here's the nuance: just because it's called a "request" doesn't mean you fill out a form and wait. In a real emergency, you often execute the fix first and document later — but only if you have a pre-defined emergency change policy that allows it.
Key Differences from Standard and Normal Changes
Most organizations use a three-tier change model: normal, standard, and emergency. Here's how they compare:
| Change Type | Approval | Risk Level | Time to Implement | Documentation |
|---|---|---|---|---|
| Normal | Change Advisory Board (CAB) | Medium/High | Days to weeks | Full RFC before implementation |
| Standard | Pre-approved (no CAB) | Low | Hours to days | Template-based, post-implementation |
| Emergency | Emergency CAB (eCAB) or designated approver | Very High | Minutes to hours | Often post-implementation |
The line between standard and emergency? I've seen teams try to slip a risky server reboot through as a standard change by calling it an "emergency." That's a recipe for audit failure. The key differentiator is severity and urgency — not just the convenience of skipping the CAB meeting.
When Do You Actually Need an Emergency Change?
Here are the scenarios that actually warrant an emergency change request:
- Critical security patches: A zero-day exploit is actively being used against your environment. Waiting for next Thursday's change window is not an option.
- Major service outage: The payment gateway is down, and each minute of downtime costs $10,000. A hotfix is needed immediately.
- Data corruption or loss: A database has become corrupted, and you need to restore from backup or apply a corrective script right now.
- Legal or regulatory compliance: A court order demands a specific change (e.g., preserving evidence) within hours.
- Infrastructure failure: A core router is showing signs of imminent failure, and you must failover to a backup system.
Notice what's not on this list: routine password resets, minor configuration tweaks, or non-critical feature updates. I once had a developer argue that adding a field to a form was an emergency because "the CEO wants it by EOD." That's not an emergency — that's poor planning.
The 7-Step Emergency Change Process
Over the years, I've refined this process with my teams. It's not just theory; this is what works when the pager goes off at 2 AM.
Step 1: Identify the Urgent Issue
Someone (usually the incident manager) determines that the current situation meets the criteria for an emergency change. This isn't a solo decision — it's validated with the service owner or an on-call manager. I always ask: "Is there a workaround that holds until a normal change?" If yes, don't call it an emergency.
Step 2: Assess Risk and Impact
You need a quick but honest assessment. What's the blast radius if the change fails? Does it affect other services? Can we roll back easily? I use a simple 1-5 scale for impact and urgency, and only proceed if the combined score is above a threshold. This prevents overreacting to minor issues.
Step 3: Obtain Emergency Approval
The approver is typically from the Emergency CAB (or a designated authority like the IT director). In my experience, having a pre-defined list of approvers with their phone numbers is critical. You don't want to waste 20 minutes figuring out who to call. The approver asks three questions: Is it really an emergency? Is the fix safe enough? Do we have a rollback plan? If all three are yes, you get the green light.
Step 4: Implement the Change
This is where the action happens. But here's the trap: don't let the implementer work in isolation. Have a second pair of eyes on the console — even if it's just a screen share. Mistakes in the heat of the moment are common. I always have a colleague verify commands before I hit Enter.
Step 5: Verify and Test
After the change, you must confirm that the incident is resolved and that no other services broke. Automated monitoring is your friend here. I've seen teams think they fixed the issue only to discover 30 minutes later that they'd crashed another critical service. Verification must be thorough but fast — run the top 5 health checks, not the full test suite.
Step 6: Document and Close
Now you write it up. What was the trigger? What exactly did you do? What was the outcome? This documentation is gold for post-mortems and for justifying the emergency change later. But don't let documentation delay the closure — close the incident ticket and the change record, then add the detailed notes within 24 hours.
Step 7: Post-Implementation Review
Within a week, hold a quick review (not a full-blown CAB meeting). What went well? What could have been faster? Was this truly an emergency? If the same type of issue keeps triggering emergency changes, it's a sign that you need a permanent fix or a standard change. I keep a spreadsheet of emergency changes and review it monthly — it reveals patterns that lead to process improvements.
Common Real-World Examples
Let me share three examples from my own experience that illustrate the spectrum of emergency changes.
Example 1: The SSL Certificate Expiry — Our e-commerce site's SSL certificate expired at 10 AM on a Monday. Within 15 minutes, the issue was identified, a new certificate was requested and installed via an emergency change. We had a pre-approved process for certificate renewals, so the emergency was smooth. But the root cause was a missed monitoring alert.
Example 2: The Database Migration Gone Wrong — A colleague applied a schema change as a normal change, but it caused deadlocks in production. We had to roll back using an emergency change. The rollback itself took 10 minutes, but the approval got stuck because the approver was in a meeting. Lesson learned: we added a backup approver and required every change to have a documented rollback procedure.
Example 3: The Ransomware Outbreak — This was a real nasty one. We had to isolate dozens of servers and apply emergency patches while the attack was ongoing. The emergency change process was critical because it let us rapidly deploy firewall rules without the usual bureaucracy. But the chaos also revealed that our emergency change policy didn't cover network-level changes, so we had to amend it after the fact.
Pitfalls to Avoid
I've made almost every mistake in the book, so you don't have to. Here are the top three:
- No post-implementation review: Most teams just close the emergency change and move on. Without a review, you never improve the process. The same emergency will keep happening.
- Too many approvers: When you need three managers to sign off on an emergency change, you'll end up with the change happening without approval. Keep the approval chain short — one or two people max.
- Treating every incident as an emergency change: This creates change fatigue and undermines the urgency. Your CAB will start blindly approving everything, and real emergencies won't get the attention they need.
How to Build a Reliable Emergency Change Process
If you're starting from scratch (or fixing a broken one), here's what works:
- Define clearly what constitutes an emergency — include examples and thresholds (e.g., "downtime expected to exceed 30 minutes").
- Pre-establish an Emergency CAB with at least two people who are available 24/7. Rotate the role so no one burns out.
- Create a fast-track approval form (one-pager) that captures: what, why, risk, and rollback plan. Keep it to five fields max.
- Implement a tool that allows emergency changes to be logged after the fact — don't require pre-approval if the situation is truly time-critical.
- Conduct regular drills. Once a quarter, simulate a security incident and run through the entire emergency change process. It exposes gaps in people's knowledge and tool limitations.
One thing I always emphasize: measure the number of emergency changes per month. If it's trending up, you have a deeper problem — probably technical debt or weak monitoring. If it's trending down, your standard change process is likely improving.
Frequently Asked Questions
This article is based on real ITIL implementations and personal experience. All examples are anonymized but factual.
