Hypothesis-driven debugging with the Node.js and PostgreSQL toolkit Intermediate

This page answers: how to debug under pressure methodically, and which Node.js / PostgreSQL tool answers each question.

The hypothesis loop

Effective Troubleshooting describes the process: problem report → triage → examine → diagnose → test/treat. Along the way we “iteratively hypothesize potential causes and try to test those hypotheses”.

Loading diagram…

Rules for running the loop:

1. Ask “what just changed?” first. Deploys, config, feature flags, traffic (campaigns, bots), dependencies (DB failover, library upgrade, expired cert). SRE book: recent changes are “a productive place to start”.
2. Change one thing at a time. If you change pool size and timeout together and the errors stop, you do not know which change to keep.
3. Think of horses, not zebras. A broken deploy is more common than a kernel bug.
4. Correlation is not causation. Two graphs rising together may both be caused by a third factor.
5. Record negative results too. “Checked the DB, no lock waits” saves the next person from walking the same path.
6. Bug is reproducible but you do not know which commit introduced it? Use git bisect run — see the git bisect run demo.

The Node.js toolkit

Question	Tool	Command
Is the event loop blocked?	monitorEventLoopDelay (see the event loop article)	Export p99/max as metrics; alert on max, not on the average
Which function burns CPU?	CPU profile	node --cpu-prof --cpu-prof-dir=/tmp/prof app.js (writes the file when the process exits)
Debug a running process	Inspector	node --inspect app.js (default 127.0.0.1:9229); for an already running process: kill -USR1 <pid>
What is holding memory?	Heap snapshot	node --heapsnapshot-signal=SIGUSR2 app.js then kill -USR2 <pid>; open the .heapsnapshot in Chrome DevTools
Where do allocations come from?	Sampling heap profile	node --heap-prof app.js (written on exit)
The full picture of the process right now	Diagnostic report	node --report-on-signal --report-exclude-env app.js then kill -USR2 <pid> (default signal SIGUSR2)

Caveats for production use:

• Never bind the inspector to a public IP. Anyone who can connect to that port can run arbitrary code. For remote use, go through an SSH tunnel or kubectl port-forward.
• A heap snapshot blocks the event loop while it is taken and needs roughly twice the heap size in memory. The pod may get OOM killed; take it out of the load balancer before snapshotting.
• Reports include environment variables by default, which often hold secrets. Use --report-exclude-env (Node 22.13+ / 23.3+) before sending the file to anyone.
• One signal, one job. SIGUSR1 is reserved for the inspector, reports use SIGUSR2 by default. If you enable both signal-triggered reports and heap snapshots, move one of them with --report-signal. Signal-triggered reports are not supported on Windows.
• --cpu-prof only writes when the process exits normally. If it gets SIGKILL (OOMKilled) or SIGTERM without a handler, there is no file — see Common mistakes.

The PostgreSQL toolkit

Question	Query / setting
Who is running what, waiting on what?	SELECT pid, state, wait_event_type, wait_event, now() - xact_start AS xact_age, left(query, 80) FROM pg_stat_activity WHERE backend_type = 'client backend' ORDER BY xact_start;
Stuck transactions holding connections	... WHERE state = 'idle in transaction' AND xact_start < now() - interval '1 minute';
Who blocks whom?	SELECT pid, pg_blocking_pids(pid) AS blocked_by, left(query, 80) FROM pg_stat_activity WHERE cardinality(pg_blocking_pids(pid)) > 0;
Lock details	pg_locks joined with pg_stat_activity on pid (see row-level locks)
Which queries are slow?	log_min_duration_statement = '500ms' (default -1 = off); pg_stat_statements if installed
Why is a query slow?	EXPLAIN (ANALYZE, BUFFERS) ... — read it with the EXPLAIN article
Cancel a query / kill a session	pg_cancel_backend(pid) cancels the running statement; pg_terminate_backend(pid) kills the whole session and rolls back the transaction — use it for idle in transaction

Two warnings:

• EXPLAIN ANALYZE actually executes the statement. For UPDATE/DELETE, wrap it in BEGIN; ... ROLLBACK;. It still takes locks and runs for the full duration of the statement.
• pg_blocking_pids needs brief exclusive access to the lock manager. Calling it constantly can hurt performance; run it when needed, not in a tight monitoring loop.

Demo: catch a blocked event loop and write a diagnostic report

Two plain Node scripts in examples/src/debugging/, run on Node v24.17.0 (macOS).

Loading diagram…

Demo 1 — event-loop-watchdog.ts: every 500 ms it reads the monitorEventLoopDelay histogram. At second 1.2 a handler runs synchronously for 400 ms. When max exceeds 100 ms, the script calls process.report.writeReport() into a temp directory, with excludeEnv and excludeNetwork enabled.

npx --prefix examples ts-node examples/src/debugging/event-loop-watchdog.ts

window 1: p50=11.0ms p99=13.7ms max=13.7ms  ok
window 2: p50=11.0ms p99=11.8ms max=11.8ms  ok
window 3: p50=11.0ms p99=402.7ms max=402.7ms  ALERT: event loop blocked

Writing Node.js report to file: blocked-loop.json
Node.js report completed

report: <tmpdir>/diag-wqgGXN/blocked-loop.json
top-level keys: header, javascriptStack, javascriptHeap, nativeStack, resourceUsage, uvthreadResourceUsage, libuv, workers, userLimits, sharedObjects
header: {
  event: 'JavaScript API',
  trigger: 'API',
  nodejsVersion: 'v24.17.0',
  platform: 'darwin'
}
javascriptStack.message: Error: event loop delay max=403ms > 100ms
javascriptStack top frame: at Timeout._onTimeout (examples/src/debugging/event-loop-watchdog.ts:51:3)
heap used MB: 118.8
cpu user/kernel s: 1.08239 / 0.071247
libuv handles: async, async, timer, check, idle, prepare, check, async, timer, timer, loop
environmentVariables in the original file? false
window 4: p50=11.0ms p99=11.1ms max=11.1ms  ok
window 5: p50=11.0ms p99=11.1ms max=11.1ms  ok

Reading the results:

• Correct detection: window 3 has max 402.7 ms, matching the 400 ms block; p50 stays at 11 ms (the baseline is the 10 ms sampling interval).
• The report has no environmentVariables thanks to excludeEnv; secrets in env do not end up in the file.
• The stack in the report points at the watchdog, not the culprit. The report is written after the block has finished, so buildInvoiceReportSync is no longer on the stack. The 118 MB heap is mostly ts-node.

The lesson: a diagnostic report answers “what state is the process in” (heap, handles, CPU, versions). To learn which function blocks the loop, you need a CPU profile.

Demo 2 — cpu-profile-demo.ts: runs a child process with --cpu-prof; a handler renders invoice HTML by concatenating strings 1.5 million times. It then reads the .cpuprofile file and sums self-time per function.

npx --prefix examples ts-node examples/src/debugging/cpu-profile-demo.ts

html length 72388890

profile: CPU.<ts>.<pid>.0.001.cpuprofile (506 samples)
Top self-time:
    300 ms   46.7%  (garbage collector)
    253 ms   39.3%  escapeCell app.js:3
     26 ms    4.0%  renderInvoiceHtmlSync app.js:4
     25 ms    3.9%  RegExp: [<>&"]
     20 ms    3.1%  (anonymous) app.js:3

• The profile names the exact function and line: escapeCell accounts for 39% of self-time.
• Nearly half the time is the garbage collector — a sign the code creates too many temporary strings.
• In production you usually open the .cpuprofile file in Chrome DevTools instead of summing it yourself like the script does.

Next: Communication during response.