The difference between engineers who find hard bugs quickly and those who do not is rarely knowledge of the system. It is method. The slow approach is reading code where the problem seems likely to be and changing things that look suspicious. The fast one is treating it as a search problem and halving the space with each step.
State a hypothesis you can falsify
'Something is wrong with the cache' is not a hypothesis, because no observation can disprove it. 'The cache returns a stale value for tenants whose plan changed in the last five minutes' is, and it suggests an experiment that takes ninety seconds. The discipline is to write the hypothesis down before testing it, because doing so exposes how vague it was.
Then test the cheapest one that eliminates the most possibility. Engineers routinely test the hypothesis they find most interesting rather than the one that divides the space most evenly, which is how an afternoon disappears into a subsystem that turns out to be innocent.
Bisect everything
# Bisect the change history, automatically, with a script that exits non-zero
# when the bug is present. Twelve steps over 4,000 commits.
git bisect start HEAD v4.12.0
git bisect run ./scripts/reproduce.sh
# The same strategy applies to things that are not commits:
# - bisect the data: does it fail with half the rows? which half?
# - bisect the config: revert to defaults, reapply half the overrides
# - bisect the system: does it reproduce with the cache disabled? the queue?
# - bisect time: when did the metric change shape? what shipped that day?
# The prerequisite is a reliable reproduction. Time spent making a bug
# reproducible on demand is never wasted — it is the whole investigation.A reliable reproduction is worth more than any amount of code reading. It converts debugging from reasoning into experiment, it proves the fix works, and it becomes the regression test. When a bug is intermittent, the first task is not to find it but to make it happen on demand — and doing so usually reveals the cause on its own.
Why hard bugs stay hidden
- You are debugging the wrong layer. The problem is in the proxy, the driver, the serialisation, the clock — not in the code you keep re-reading.
- You believe something that is not true: a config value you assume is set, a version you assume is deployed, a cache you assume is disabled. Verify assumptions rather than recalling them.
- The failing run differs from the one you are testing — different data, different tenant, different timezone, different instance of a rolling deploy.
- It is a race, so it does not exist in the sequential model in your head. Add logging with timestamps and thread identity before theorising further.
- Two bugs are interacting, and every observation is confusing because you are fitting one explanation to two phenomena.
When the evidence makes no sense, one of the things you are certain about is false. Test the certainty rather than the code.
Two habits close it out. Stop and write down what you know to be true, what you have ruled out, and what remains — after an hour of no progress, this reliably exposes the assumption that was wrong. And when you find it, spend five more minutes asking why it was not caught: that question yields a test, a metric or an alert, which is the only way a debugging session improves anything beyond the bug in front of you.