When production fails, every minute costs revenue and trust. Teams without emergency experience often take 4-12+ hours for full resolution. With proven processes, you contain damage in 30-120 minutes and deploy lasting fixes in 2-6 hours total. Here’s the realistic timeline based on 50+ incidents we’ve handled.
Time to Containment: 30-120 Minutes
Containment means stopping active damage. This usually involves rolling back a deployment, disabling a feature, scaling a resource, or routing around a failing dependency. Containment doesn’t require understanding the root cause - it just requires stopping the bleeding.
What shortens this phase:
- Clear deployment history (you know exactly what changed and when)
- Working rollback capability in your CI/CD pipeline
- Monitoring that pinpoints the failure rather than just alerting “something’s wrong”
- Expertise (we can help with this one!)
What lengthens it:
- No recent changes to roll back to
- Multiple potential causes with no obvious culprit
- Poor logging that makes it hard to see what’s actually failing
Time to Root Cause: 1-4 Hours
Root cause analysis means identifying why the system failed, not just what failed. This is the phase where experience matters most - someone who has seen this failure pattern before will find it in 20 minutes; someone encountering it for the first time may spend hours.
Common fast resolutions (under an hour):
- Database connection pool exhaustion after a deploy added background jobs
- Missing index on a table that grew past a tipping point
- Third-party API that started returning errors or timing out
- OOM kill on app servers after a memory leak accumulated overnight
Common slow resolutions (4+ hours):
- Intermittent failures that are hard to reproduce
- Race conditions in concurrent code
- Cascading failures where multiple systems are degraded
- Data corruption with an unclear origin
Time to Durable Fix: 2-6 Hours After Root Cause
A durable fix means the problem won’t recur. This involves writing the fix in a branch, running tests, deploying to staging, and deploying to production with monitoring in place. Cutting corners here is how the same incident happens again next month.
After the Fix: Post-Incident Review
Once service is restored, a proper post-incident review takes 1-2 hours. It produces: a written timeline, identified contributing factors, and a list of preventive actions (new tests, monitoring improvements, process changes). Skipping this is the most common reason teams have the same incident twice.
Restore Reliability Without Recurrence
We target containment in 90 minutes, root cause in 4 hours, durable deployment in 6-8 hours, and review in 48 hours. Complex apps or novel bugs extend this- we’ll communicate transparently.
After resolution, your team deploys confidently, incidents drop, and you focus on growth. Contact for emergency Rails support or explore our process.