mockupAWS/docs/runbooks/incident-response.md

Incident Response Runbook

Version: 1.0.0
Last Updated: 2026-04-07
Owner: DevOps Team

---

Table of Contents

1. Incident Severity Levels
2. Response Procedures
3. Communication Templates
4. Post-Incident Review
5. Common Incidents

---

1. Incident Severity Levels

P1 - Critical (Service Down)

Criteria:

• Complete service unavailability
• Data loss or corruption
• Security breach

• 50% of users affected

Response Time: 15 minutes
Resolution Target: 2 hours

Actions:

1. Page on-call engineer immediately
2. Create incident channel/war room
3. Notify stakeholders within 15 minutes
4. Begin rollback if applicable
5. Post to status page

P2 - High (Major Impact)

Criteria:

• Core functionality impaired

• 25% of users affected

• Workaround available
• Performance severely degraded

Response Time: 1 hour
Resolution Target: 8 hours

P3 - Medium (Partial Impact)

Criteria:

• Non-critical features affected
• <25% of users affected
• Workaround available

Response Time: 4 hours
Resolution Target: 24 hours

P4 - Low (Minimal Impact)

Criteria:

• General questions
• Feature requests
• Minor cosmetic issues

Response Time: 24 hours
Resolution Target: Best effort

---

2. Response Procedures

2.1 Initial Response Checklist

□ Acknowledge incident (within SLA)
□ Create incident ticket (PagerDuty/Opsgenie)
□ Join/create incident Slack channel
□ Identify severity level
□ Begin incident log
□ Notify stakeholders if P1/P2

2.2 Investigation Steps

# 1. Check service health
curl -f https://mockupaws.com/api/v1/health
curl -f https://api.mockupaws.com/api/v1/health

# 2. Check CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=mockupaws-production \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average

# 3. Check ECS service status
aws ecs describe-services \
  --cluster mockupaws-production \
  --services backend

# 4. Check logs
aws logs tail /ecs/mockupaws-production --follow

# 5. Check database connections
aws rds describe-db-clusters \
  --db-cluster-identifier mockupaws-production

2.3 Escalation Path

0-15 min:  On-call Engineer
15-30 min: Senior Engineer
30-60 min: Engineering Manager
60+ min:   VP Engineering / CTO

2.4 Resolution & Recovery

1. Immediate Mitigation

   • Enable circuit breakers
   • Scale up resources
   • Enable maintenance mode

2. Root Cause Fix

   • Deploy hotfix
   • Database recovery
   • Infrastructure changes

3. Verification

   • Run smoke tests
   • Monitor metrics
   • Confirm user impact resolved

4. Closeout

   • Update status page
   • Notify stakeholders
   • Schedule post-mortem

---

3. Communication Templates

3.1 Internal Notification (P1)

Subject: [INCIDENT] P1 - mockupAWS Service Down

Incident ID: INC-YYYY-MM-DD-XXX
Severity: P1 - Critical
Started: YYYY-MM-DD HH:MM UTC
Impact: Complete service unavailability

Description:
[Detailed description of the issue]

Actions Taken:
- [ ] Initial investigation
- [ ] Rollback initiated
- [ ] [Other actions]

Next Update: +30 minutes
Incident Commander: [Name]
Slack: #incident-XXX

3.2 Customer Notification

Subject: Service Disruption - mockupAWS

We are currently investigating an issue affecting mockupAWS service availability.

Impact: Users may be unable to access the platform
Started: HH:MM UTC
Status: Investigating

We will provide updates every 30 minutes.

Track status: https://status.mockupaws.com

We apologize for any inconvenience.

3.3 Status Page Update

**Investigating** - We are investigating reports of service unavailability.
Posted HH:MM UTC

**Update** - We have identified the root cause and are implementing a fix.
Posted HH:MM UTC

**Resolved** - Service has been fully restored. We will provide a post-mortem within 24 hours.
Posted HH:MM UTC

3.4 Post-Incident Communication

Subject: Post-Incident Review: INC-YYYY-MM-DD-XXX

Summary:
[One paragraph summary]

Timeline:
- HH:MM - Issue detected
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Service restored

Root Cause:
[Detailed explanation]

Impact:
- Duration: X minutes
- Users affected: X%
- Data loss: None / X records

Lessons Learned:
1. [Lesson 1]
2. [Lesson 2]

Action Items:
1. [Owner] - [Action] - [Due Date]
2. [Owner] - [Action] - [Due Date]

---

4. Post-Incident Review

4.1 Post-Mortem Template

# Post-Mortem: INC-YYYY-MM-DD-XXX

## Metadata
- **Incident ID:** INC-YYYY-MM-DD-XXX
- **Date:** YYYY-MM-DD
- **Severity:** P1/P2/P3
- **Duration:** XX minutes
- **Reporter:** [Name]
- **Reviewers:** [Names]

## Summary
[2-3 sentence summary]

## Timeline
| Time (UTC) | Event |
|-----------|-------|
| 00:00 | Issue detected by monitoring |
| 00:05 | On-call paged |
| 00:15 | Investigation started |
| 00:45 | Root cause identified |
| 01:00 | Fix deployed |
| 01:30 | Service confirmed stable |

## Root Cause Analysis
### What happened?
[Detailed description]

### Why did it happen?
[5 Whys analysis]

### How did we detect it?
[Monitoring/alert details]

## Impact Assessment
- **Users affected:** X%
- **Features affected:** [List]
- **Data impact:** [None/Description]
- **SLA impact:** [None/X minutes downtime]

## Response Assessment
### What went well?
1. 
2. 

### What could have gone better?
1. 
2. 

### What did we learn?
1. 
2. 

## Action Items
| ID | Action | Owner | Priority | Due Date |
|----|--------|-------|----------|----------|
| 1 | | | High | |
| 2 | | | Medium | |
| 3 | | | Low | |

## Attachments
- [Logs]
- [Metrics]
- [Screenshots]

4.2 Review Meeting

Attendees:

• Incident Commander
• Engineers involved
• Engineering Manager
• Optional: Product Manager, Customer Success

Agenda (30 minutes):

1. Timeline review (5 min)
2. Root cause discussion (10 min)
3. Response assessment (5 min)
4. Action item assignment (5 min)
5. Lessons learned (5 min)

---

5. Common Incidents

5.1 Database Connection Pool Exhaustion

Symptoms:

• API timeouts
• "too many connections" errors
• Latency spikes

Diagnosis:

# Check active connections
aws rds describe-db-clusters \
  --query 'DBClusters[0].DBClusterMembers[*].DBInstanceIdentifier'

# Check CloudWatch metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name DatabaseConnections

Resolution:

1. Scale ECS tasks down temporarily
2. Kill idle connections
3. Increase max_connections
4. Implement connection pooling

5.2 High Memory Usage

Symptoms:

• OOM kills
• Container restarts
• Performance degradation

Diagnosis:

# Check container metrics
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name MemoryUtilization

Resolution:

1. Identify memory leak (heap dump)
2. Restart affected tasks
3. Increase memory limits
4. Deploy fix

5.3 Redis Connection Issues

Symptoms:

• Cache misses increasing
• API latency spikes
• Connection errors

Resolution:

1. Check ElastiCache status
2. Verify security group rules
3. Restart Redis if needed
4. Implement circuit breaker

5.4 SSL Certificate Expiry

Symptoms:

• HTTPS errors
• Certificate warnings

Prevention:

• Set alert 30 days before expiry
• Use ACM with auto-renewal

Resolution:

1. Renew certificate
2. Update ALB/CloudFront
3. Verify SSL Labs rating

---

Quick Reference

Resource	URL/Command
Status Page	https://status.mockupaws.com
PagerDuty	https://mockupaws.pagerduty.com
CloudWatch	AWS Console > CloudWatch
ECS Console	AWS Console > ECS
RDS Console	AWS Console > RDS
Logs	aws logs tail /ecs/mockupaws-production --follow
Emergency Hotline	+1-555-MOCKUP

---

This runbook should be reviewed quarterly and updated after each significant incident.