{
  "schema": "https://ai-atoms.com/schemas/skill-v1.json",
  "type": "skill",
  "id": "skill/incident-runbook-templates",
  "version": "1.0.0",
  "name": "Incident Runbook Templates",
  "description": "Create structured incident response runbooks with step-by-step procedures, escalation paths, and recovery actions. Use when building runbooks, responding to incidents, or establishing incident resp...",
  "system_prompt_fragment": "# Incident Runbook Templates\n\nProduction-ready templates for incident response runbooks covering detection, triage, mitigation, resolution, and communication.\n\n## Do not use this skill when\n\n- The task is unrelated to incident runbook templates\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources/implementation-playbook.md`.\n\n## Use this skill when\n\n- Creating incident response procedures\n- Building service-specific runbooks\n- Establishing escalation paths\n- Documenting recovery procedures\n- Responding to active incidents\n- Onboarding on-call engineers\n\n## Core Concepts\n\n### 1. Incident Severity Levels\n\n| Severity | Impact | Response Time | Example |\n|----------|--------|---------------|---------|\n| **SEV1** | Complete outage, data loss | 15 min | Production down |\n| **SEV2** | Major degradation | 30 min | Critical feature broken |\n| **SEV3** | Minor impact | 2 hours | Non-critical bug |\n| **SEV4** | Minimal impact | Next business day | Cosmetic issue |\n\n### 2. Runbook Structure\n\n```\n1. Overview & Impact\n2. Detection & Alerts\n3. Initial Triage\n4. Mitigation Steps\n5. Root Cause Investigation\n6. Resolution Procedures\n7. Verification & Rollback\n8. Communication Templates\n9. Escalation Matrix\n```\n\n## Runbook Templates\n\n### Template 1: Service Outage Runbook\n\n```markdown\n# [Service Name] Outage Runbook\n\n## Overview\n**Service**: Payment Processing Service\n**Owner**: Platform Team\n**Slack**: #payments-incidents\n**PagerDuty**: payments-oncall\n\n## Impact Assessment\n- [ ] Which customers are affected?\n- [ ] What percentage of traffic is impacted?\n- [ ] Are there financial implications?\n- [ ] What's the blast radius?\n\n## Detection\n### Alerts\n- `payment_error_rate > 5%` (PagerDuty)\n- `payment_latency_p99 > 2s` (Slack)\n- `payment_success_rate < 95%` (PagerDuty)\n\n### Dashboards\n- [Payment Service Dashboard](https://grafana/d/payments)\n- [Error Tracking](https://sentry.io/payments)\n- [Dependency Status](https://status.stripe.com)\n\n## Initial Triage (First 5 Minutes)\n\n### 1. Assess Scope\n```bash\n# Check service health\nkubectl get pods -n payments -l app=payment-service\n\n# Check recent deployments\nkubectl rollout history deployment/payment-service -n payments\n\n# Check error rates\ncurl -s \"http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))\"\n```\n\n### 2. Quick Health Checks\n- [ ] Can you reach the service? `curl -I https://api.company.com/payments/health`\n- [ ] Database connectivity? Check connection pool metrics\n- [ ] External dependencies? Check Stripe, bank API status\n- [ ] Recent changes? Check deploy history\n\n### 3. Initial Classification\n| Symptom | Likely Cause | Go To Section |\n|---------|--------------|---------------|\n| All requests failing | Service down | Section 4.1 |\n| High latency | Database/dependency | Section 4.2 |\n| Partial failures | Code bug | Section 4.3 |\n| Spike in errors | Traffic surge | Section 4.4 |\n\n## Mitigation Procedures\n\n### 4.1 Service Completely Down\n```bash\n# Step 1: Check pod status\nkubectl get pods -n payments\n\n# Step 2: If pods are crash-looping, check logs\nkubectl logs -n payments -l app=payment-service --tail=100\n\n# Step 3: Check recent deployments\nkubectl rollout history deployment/payment-service -n payments\n\n# Step 4: ROLLBACK if recent deploy is suspect\nkubectl rollout undo deployment/payment-service -n payments\n\n# Step 5: Scale up if resource constrained\nkubectl scale deployment/payment-service -n payments --replicas=10\n\n# Step 6: Verify recovery\nkubectl rollout status deployment/payment-service -n payments\n```\n\n### 4.2 High Latency\n```bash\n# Step 1: Check database connections\nkubectl exec -n payments deploy/payment-service -- \\\n  curl localhost:8080/metrics | grep db_pool\n\n# Step 2: Check slow queries (if DB issue)\npsql -h $DB_HOST -U $DB_USER -c \"\n  SELECT pid, now() - query_start AS duration, query\n  FROM pg_stat_activity\n  WHERE state = 'active' AND duration > interval '5 seconds'\n  ORDER BY duration DESC;\"\n\n# Step 3: Kill long-running queries if needed\npsql -h $DB_HOST -U $DB_USER -c \"SELECT pg_terminate_backend(pid);\"\n\n# Step 4: Check external dependency latency\ncurl -w \"@curl-format.txt\" -o /dev/null -s https://api.stripe.com/v1/health\n\n# Step 5: Enable circuit breaker if dependency is slow\nkubectl set env deployment/payment-service \\\n  STRIPE_CIRCUIT_BREAKER_ENABLED=true -n payments\n```\n\n### 4.3 Partial Failures (Specific Errors)\n```bash\n# Step 1: Identify error pattern\nkubectl logs -n payments -l app=payment-service --tail=500 | \\\n  grep -i error | sort | uniq -c | sort -rn | head -20\n\n# Step 2: Check error tracking\n# Go to Sentry: https://sentry.io/payments\n\n# Step 3: If specific endpoint, enable feature flag to disable\ncurl -X POST https://api.company.com/internal/feature-flags \\\n  -d '{\"flag\": \"DISABLE_PROBLEMATIC_FEATURE\", \"enabled\": true}'\n\n# Step 4: If data issue, check recent data changes\npsql -h $DB_HOST -c \"\n  SELECT * FROM audit_log\n  WHERE table_name = 'payment_methods'\n  AND created_at > now() - interval '1 hour';\"\n```\n\n### 4.4 Traffic Surge\n```bash\n# Step 1: Check current request rate\nkubectl top pods -n payments\n\n# Step 2: Scale horizontally\nkubectl scale deployment/payment-service -n payments --replicas=20\n\n# Step 3: Enable rate limiting\nkubectl set env deployment/payment-service \\\n  RATE_LIMIT_ENABLED=true \\\n  RATE_LIMIT_RPS=1000 -n payments\n\n# Step 4: If attack, block suspicious IPs\nkubectl apply -f - <<EOF\napiVersion: networking.k8s.io/v1\nkind: NetworkPolicy\nmetadata:\n  name: block-suspicious\n  namespace: payments\nspec:\n  podSelector:\n    matchLabels:\n      app: payment-service\n  ingress:\n  - from:\n    - ipBlock:\n        cidr: 0.0.0.0/0\n        except:\n        - 192.168.1.0/24  # Suspicious range\nEOF\n```\n\n## Verification Steps\n```bash\n# Verify service is healthy\ncurl -s https://api.company.com/payments/health | jq\n\n# Verify error rate is back to normal\ncurl -s \"http://prometheus:9090/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[5m]))\" | jq '.data.result[0].value[1]'\n\n# Verify latency is acceptable\ncurl -s \"http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(http_request_duration_seconds_bucket[5m]))by(le))\" | jq\n\n# Smoke test critical flows\n./scripts/smoke-test-payments.sh\n```\n\n## Rollback Procedures\n```bash\n# Rollback Kubernetes deployment\nkubectl rollout undo deployment/payment-service -n payments\n\n# Rollback database migration (if applicable)\n./scripts/db-rollback.sh $MIGRATION_VERSION\n\n# Rollback feature flag\ncurl -X POST https://api.company.com/internal/feature-flags \\\n  -d '{\"flag\": \"NEW_PAYMENT_FLOW\", \"enabled\": false}'\n```\n\n## Escalation Matrix\n\n| Condition | Escalate To | Contact |\n|-----------|-------------|---------|\n| > 15 min unresolved SEV1 | Engineering Manager | @manager (Slack) |\n| Data breach suspected | Security Team | #security-incidents |\n| Financial impact > $10k | Finance + Legal | @finance-oncall |\n| Customer communication needed | Support Lead | @support-lead |\n\n## Communication Templates\n\n### Initial Notification (Internal)\n```\n🚨 INCIDENT: Payment Service Degradation\n\nSeverity: SEV2\nStatus: Investigating\nImpact: ~20% of payment requests failing\nStart Time: [TIME]\nIncident Commander: [NAME]\n\nCurrent Actions:\n- Investigating root cause\n- Scaling up service\n- Monitoring dashboards\n\nUpdates in #payments-incidents\n```\n\n### Status Update\n```\n📊 UPDATE: Payment Service Incident\n\nStatus: Mitigating\nImpact: Reduced to ~5% failure rate\nDuration: 25 minutes\n\nActions Taken:\n- Rolled back deployment v2.3.4 → v2.3.3\n- Scaled service from 5 → 10 replicas\n\nNext Steps:\n- Continuing to monitor\n- Root cause analysis in progress\n\nETA to Resolution: ~15 minutes\n```\n\n### Resolution Notification\n```\n✅ RESOLVED: Payment Service Incident\n\nDuration: 45 minutes\nImpact: ~5,000 affected transactions\nRoot Cause: Memory leak in v2.3.4\n\nResolution:\n- Rolled back to v2.3.3\n- Transactions auto-retried successfully\n\nFollow-up:\n- Postmortem scheduled for [DATE]\n- Bug fix in progress\n```\n```\n\n### Template 2: Database Incident Runbook\n\n```markdown\n# Database Incident Runbook\n\n## Quick Reference\n| Issue | Command |\n|-------|---------|\n| Check connections | `SELECT count(*) FROM pg_stat_activity;` |\n| Kill query | `SELECT pg_terminate_backend(pid);` |\n| Check replication lag | `SELECT extract(epoch from (now() - pg_last_xact_replay_timestamp()));` |\n| Check locks | `SELECT * FROM pg_locks WHERE NOT granted;` |\n\n## Connection Pool Exhaustion\n```sql\n-- Check current connections\nSELECT datname, usename, state, count(*)\nFROM pg_stat_activity\nGROUP BY datname, usename, state\nORDER BY count(*) DESC;\n\n-- Identify long-running connections\nSELECT pid, usename, datname, state, query_start, query\nFROM pg_stat_activity\nWHERE state != 'idle'\nORDER BY query_start;\n\n-- Terminate idle connections\nSELECT pg_terminate_backend(pid)\nFROM pg_stat_activity\nWHERE state = 'idle'\nAND query_start < now() - interval '10 minutes';\n```\n\n## Replication Lag\n```sql\n-- Check lag on replica\nSELECT\n  CASE\n    WHEN pg_last_wal_receive_lsn() = pg_last_wal_replay_lsn() THEN 0\n    ELSE extract(epoch from now() - pg_last_xact_replay_timestamp())\n  END AS lag_seconds;\n\n-- If lag > 60s, consider:\n-- 1. Check network between primary/replica\n-- 2. Check replica disk I/O\n-- 3. Consider failover if unrecoverable\n```\n\n## Disk Space Critical\n```bash\n# Check disk usage\ndf -h /var/lib/postgresql/data\n\n# Find large tables\npsql -c \"SELECT relname, pg_size_pretty(pg_total_relation_size(relid))\nFROM pg_catalog.pg_statio_user_tables\nORDER BY pg_total_relation_size(relid) DESC\nLIMIT 10;\"\n\n# VACUUM to reclaim space\npsql -c \"VACUUM FULL large_table;\"\n\n# If emergency, delete old data or expand disk\n```\n```\n\n## Best Practices\n\n### Do's\n- **Keep runbooks updated** - Review after every incident\n- **Test runbooks regularly** - Game days, chaos engineering\n- **Include rollback steps** - Always have an escape hatch\n- **Document assumptions** - What must be true for steps to work\n- **Link to dashboards** - Quick access during stress\n\n### Don'ts\n- **Don't assume knowledge** - Write for 3 AM brain\n- **Don't skip verification** - Confirm each step worked\n- **Don't forget communication** - Keep stakeholders informed\n- **Don't work alone** - Escalate early\n- **Don't skip postmortems** - Learn from every incident\n\n## Resources\n\n- [Google SRE Book - Incident Management](https://sre.google/sre-book/managing-incidents/)\n- [PagerDuty Incident Response](https://response.pagerduty.com/)\n- [Atlassian Incident Management](https://www.atlassian.com/incident-management)",
  "applicable_domains": [
    "other"
  ],
  "category": "other",
  "invocation": [
    "/incident-runbook-templates"
  ],
  "authored_by": "claudeskills.in community",
  "source_url": "https://claudeskills.in/skill/incident-runbook-templates",
  "provenance": {
    "source": "claudeskills.in",
    "source_url": "https://claudeskills.in/skill/incident-runbook-templates",
    "license": "unknown",
    "imported_at": "2026-09-03",
    "notes": "Aggregated by claudeskills.in from community GitHub lists."
  },
  "tags": [
    "claudeskills",
    "other"
  ],
  "lifecycle": "draft"
}