• about 13 min read

Production Incident Triage Checklist

Before you use this

A checklist is not a substitute for runbooks and failure drills. When a real incident hits, first decide whether the blast radius is still growing — stop the bleeding before collecting evidence. Every diagnostic command below has a cost; before running one, check whether the instance is still serving traffic, how much disk is left, what the command touches, and what time you’re at.

1. Confirm the alert and the blast radius

  • Verify the alert source, severity, and trigger time — real failure, false positive, or delayed metrics collection?
  • Map the impact: which users, which endpoints, which regions or data centers, and whether money/orders/inventory paths are involved.
  • Look at user-facing metrics: request volume, error rate, P95/P99, timeout rate, business success rate — not just per-host CPU.
  • Judge whether the incident is stabilizing, peaking, or still expanding.
  • Pull the full metrics window covering a dozen minutes before and after the alert; line up app, database, Redis, MQ, downstream, and network on one timeline.
  • Diff against recent changes: deployments, config, scaling, scheduled jobs, data fixes, third-party changes, traffic events.
  • Notify stakeholders; assign one incident commander, one channel, and a status-update cadence so nobody duplicates work.
  • Decide whether to escalate the response level and trigger the emergency plan.

2. Stop the bleeding: recover first, diagnose later

The goal of mitigation is to contain the impact, not to find the root cause. Every measure below has side effects — weigh the cost before picking one.

  • Roll back recent changes: version, config, SQL changes, feature flags — check whether the rollback is compatible with data already migrated.
  • Pull unhealthy instances out: remove them from load balancers or the service registry so they stop taking traffic.
  • Rate-limit: throttle QPS at the entry point or hot endpoints to protect the app and its downstream.
  • Circuit-break: fail fast on downstream calls that keep failing, so the failure stops spreading.
  • Degrade: turn off non-core features, serve cached or fallback data, protect the core path.
  • Stop runaway jobs: pause the problematic scheduled jobs, batch processes, and async consumers.
  • Scale out: check capacity on the database, cache, and downstream first — don’t just push the pressure onto your dependencies.
  • Failover: switch master/slave, divert traffic to another AZ or a backup path.
  • Freeze changes: stop ongoing deployments and unrelated changes so nothing stacks on top.

Mitigation is not resolution

Rate-limiting, degradation, and scaling only hold the impact down. Once the failure is stable, keep diagnosing the root cause, and wind down each temporary measure after verification — otherwise you’re left with hidden capacity and functionality risks.

3. Preserve evidence: before restarting or rolling back

  • Save monitoring screenshots from around the incident, or at least a precise time range.
  • Back up application logs, access logs, error logs, GC logs.
  • Capture one to three thread dumps a few seconds apart — not just one.
  • Keep the slow calls, failed calls, and upstream/downstream latencies from your tracer.
  • Record process CPU, RSS, heap, thread count, file descriptors, and network state.
  • Record database slow queries, lock waits, connection pool state, Redis slow commands, and MQ backlog.
  • Record the change log, deployment tickets, and ops audit trails.
  • Configure OOM auto-dumps ahead of time; don’t scramble for it after the fact.
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/path/with/enough/space

Heap dumps are expensive

Manually triggering a heap dump, class histogram, or a full GC costs real CPU, I/O, and pause time. Prefer doing it on an instance that’s already been drained of traffic, and confirm there’s enough disk first.

4. Narrow it down layer by layer

When you have no trace yet, eliminate layers top-down — don’t skip ahead with a guess.

LayerWhat to check first
ChangesRecent deploys, config, SQL, flags, scheduled jobs, data fixes
AccessDNS, CDN, load balancer, gateway, certificates, rate limits and routing rules
ApplicationException logs, error rate, thread states, GC, thread pools, connection pools
ResourcesCPU, memory, disk, network, file descriptors, container quotas
StorageSlow SQL, lock waits, replication lag, Redis, Elasticsearch
MiddlewareMQ backlog, service registry, config center, distributed locks
DependenciesThird-party APIs, downstream services, timeout and retry configs
InfrastructureContainer scheduler, host, kernel, network devices, cloud events

Problems often propagate across layers. A slow query can hog database connections, which blocks business threads on the connection pool, which piles up the thread pool queue, which makes upstream callers time out and retry — and you end up seeing rising CPU, latency, and error rates all at once. Find the metric that moved first on the timeline; that gets you closer to the root cause than staring at the biggest number.

5. Per-scenario checklists

CPU spike

  • Confirm with top that the high CPU belongs to the target Java process.
  • Find the hot threads with top -H -p <pid> or pidstat -t -p <pid> 1.
  • Convert the decimal thread ID to hex to match the nid in thread dumps.
  • Take several thread dumps in a row and check whether threads are stuck in the same code path.
  • Distinguish compute hotspots, runaway loops, serialization, lock contention, and GC churn — and confirm it’s not just normal busy-ness from a traffic or retry surge.
jps -l
top -H -p <pid>
printf '%x\n' <tid>
jcmd <pid> Thread.print > /tmp/thread-dump.txt

High load, low CPU

  • Use vmstat 1 to watch the run queue, I/O wait, and blocked tasks.
  • Use iostat -xz 1 for device utilization, wait time, and queue depth.
  • Check whether container CPU quotas, network storage, or host I/O is involved — and whether there are lots of uninterruptible sleeps.

Memory climbing and OOM

  • First identify what is growing: Java heap, process RSS, or total container memory. If RSS grows while the heap is stable, look at direct memory, thread stacks, Metaspace, JNI, native libraries, and memory-mapped files.
  • Check JVM-side data: heap summary, startup flags, native memory stats — and read the OOM message for direction: heap, Metaspace, direct memory, or unable to create threads.
  • Analyze the heap dump: biggest retained objects, Dominator Tree, reference chains to GC Roots.
  • Follow the reference chain to find who’s holding the objects. If Old gen keeps growing after a full collection, look at long-lived objects, unbounded caches, collections, listeners, or ThreadLocals.
  • Decide whether it’s a leak, or normal growth from cache warm-up or traffic increase.
jcmd <pid> GC.heap_info
jcmd <pid> VM.flags
jcmd <pid> VM.native_memory summary
jcmd <pid> GC.heap_dump filename=/path/with/enough/space/heap.hprof

VM.native_memory requires NMT enabled at startup, e.g. -XX:NativeMemoryTracking=summary — you can’t turn it on retroactively after the incident.

Frequent GC or longer pauses

  • Read the GC logs for the trigger, occupancy before/after, and pause durations.
  • Sample continuously with jstat -gcutil <pid> 1000 10 — single points lie.
  • Frequent Young GC: check allocation rate, survival rate, and young gen size.
  • Frequent Full GC: check whether Old gen recovers, promotion speed, Metaspace, and explicit GC calls.
  • Check whether a traffic spike, batch job, or cache refresh kicked it off.
  • Understand application behavior before tuning or adding heap — don’t just postpone the failure.

Slow endpoints, low CPU

  • Pick the slowest span in the trace and break down the time by stage.
  • Many threads parked in getConnection() on the DB pool: check pool wait, slow SQL, and long transactions.
  • Many threads reading downstream responses: check downstream latency, timeout config, and retry counts.
  • Many threads in BLOCKED: check lock holders and slow operations inside critical sections.
  • Business threads idle but queues full: check consumer threads and executor states.
  • Verify total latency adds up across stages; fill in missing instrumentation.

Thread pool queue piling up

  • Monitor thread count, active threads, queue length, queue capacity, and rejection count together.
  • Compare submission rate, completion rate, queue time, and execution time.
  • Stack-dump the threads running tasks to see whether they’re computing or waiting.
  • Check whether database, HTTP, and Redis pools are being exhausted at the same time.
  • Think twice before raising thread count — it can crush the downstream.
  • Based on task priority, choose rate-limiting, degradation, scaling, isolation, or dropping.

Database connection pool exhaustion

  • Check active/idle connections, waiting threads, and time-to-acquire.
  • Check whether slow SQL keeps connections from being returned.
  • Check whether transaction scopes are too wide — spanning remote calls, file processing, or heavy computation.
  • Check for leaks: is the connection closed on every branch?
  • On the DB side, review sessions, slow queries, lock waits, long transactions, and instance resources.
  • Before enlarging the pool, confirm the database can handle instances × per-instance pool max total connections.

Slow SQL and lock waits

  • Find the most expensive statements via the slow query log or Performance Schema.
  • Run EXPLAIN: watch scanned rows, index usage, and extra sorting.
  • Verify with production-like parameters and data distribution — “uses the index” on your test DB means nothing.
  • Check lock waits and deadlock logs; find the blocking source.
  • Evaluate index changes, query rewrites, and batching options.

Related: MySQL execution plans, SQL optimization notes.

Redis getting slow

  • First separate client-side wait from server-side slowness.
  • Client side: pool wait, network jitter, DNS resolution, client processing.
  • Server side: command latency, slowlog, CPU, memory, network, connection count.
  • Check big keys, hot keys, expiry eviction, and persistence work.
  • Check replication lag, cluster state, and failover health.

Don't run expensive commands online

KEYS * and full --bigkeys scans sweep the keyspace and add load. In production, do these off-peak, on replicas, or with controlled sampling — and know their impact on your version and cluster topology first.

MQ backlog

  • Compare production and consumption rates; pinpoint which topic, queue, or partition holds the backlog.
  • Check whether consumer instances are alive, consumer threads are blocked, and per-message processing time changed.
  • Check for retry storms: poison messages retrying forever.
  • Check partition assignment balance and frequent consumer rebalances.
  • Check whether offset commits fail after successful consumption, causing redelivery.
  • Before adding consumers, confirm the parallelism cap and downstream capacity — don’t dump the whole backlog onto the database the moment you recover.
  • Keep idempotency during handling; when skipping bad messages, keep their content and failure reason for compensation.

Downstream timeouts and retry storms

  • Break one call into pool wait, DNS, connect, TLS, request, processing, and response read.
  • Check whether connect timeout, read timeout, and the overall call budget make sense.
  • Check the HTTP pool’s active connections, wait queue, and connection reuse.
  • Check retry counts, backoff policies, and which errors trigger retries.
  • Check circuit breaker state, fallback results, and first-call success rate.
  • Confirm retries across layers don’t multiply traffic; never blindly retry non-idempotent writes.

Disk, file descriptors, and network

  • Locate disk usage with df -h and du -sh: log spikes, big files, temp files, dumps, backups.
  • Check file descriptors, connection counts, and port usage.
  • Use ss / netstat to review connection states, TIME_WAIT, CLOSE_WAIT, and retransmits.
  • Check DNS, certificates, firewalls, and network device health.

6. Fix and verify

  • The fix targets the root cause, not just the symptom.
  • The change was reviewed and validated against production-like data and traffic.
  • Compare the same metric set before/during/after: request volume, P95/P99, error rate, business success rate, CPU, memory, GC, thread pools, connection pools, dependency latency.
  • Confirm user-facing metrics recovered and business success rate is back to normal.
  • For orders, inventory, money, and redelivery scenarios: reconcile data and run compensations.
  • Unwind temporary measures one by one: rate limits, degradation, circuit breakers, scaling, paused jobs.
  • Gradually ramp traffic back, watch long enough, then fully restore.

Related: intro to performance testing, timeout and retry mechanisms, fallback and circuit breaking.

7. Post-mortem and improvement

  • Write down the incident timeline, impact scope, and user-facing symptoms.
  • Analyze the direct cause, the trigger, and why the failure spread.
  • Collect the evidence used in mitigation, diagnosis, and repair.
  • Figure out which monitoring, runbooks, tests, or processes failed to help.
  • Turn improvements into verifiable mechanisms — not vague lines like “more code review”.
  • Assign an owner and a deadline to each item, and follow through.

Concrete examples: alert on connection-pool wait time, cap batch query sizes, add idempotency tests for message consumption, add a critical-endpoint diff check to the deploy flow, add load tests matching the failure scenario.

8. Command and tool cheat sheet

ToolTypical useCaution
top, pidstatPer-process/thread CPU, I/OMind the metric scope for containers vs. host
vmstat, iostatRun queue, memory, diskRead them as a trend, not a snapshot
jpsList Java processesOnly shows JVMs of the current user
jcmdThread dumps, heap info, JFR, class histogramsSome commands are expensive — evaluate first
jstatGC and memory pool trendsContinuous sampling beats single points
JFR, JMCCPU, allocation, lock, and I/O event analysisMatch capture config to production overhead
MAT, VisualVMHeap dump analysisFocus on reference chains and GC Roots
ArthasMethod latency, threads, classes, runtime diagnosisScope Trace/Watch carefully on hot methods
Tracing/monitor platformsRequest chains, error rate, latency, dependenciesMind sampling rate, data masking, and clock sync

Further reading

jcbl1
jcbl1 Software Engineer