.png)
A Field Guide to Problems Nobody Can Reproduce
Every engineer knows this species of problem. The export fails, but only for one client, only on Tuesdays, and never while anyone is watching. The ticket has been open for months. Three people have shrugged at it.
These problems are where engineering reputations are actually made, because they cannot be solved by knowing a framework. They yield only to method. After twenty years of being handed other people's unreproducible bugs, here is the method.
The first rule is uncomfortable: the bug is always reproducible. What people mean by "cannot reproduce" is "have not yet found the conditions". Computers do not roll dice. Something differs between the runs that fail and the runs that succeed: the data, the timing, the environment, the moon phase of a cache. Your entire job is finding that difference.
So you start by collecting failures instead of theories. When did it happen, for whom, with what input, after what? Three or four real occurrences beat any amount of whiteboard speculation. In the Tuesday case from above, the pattern was in front of us the whole time: Tuesday was the day the client's bank sent a weekly file with a different encoding.
Then the loop begins, and the discipline is the whole game.
Reproduce it, however small and ugly. A failing test, a script, a saved copy of the poisonous input. Until the bug happens on command, you are not debugging. You are gambling.
Hypothesise, one falsifiable guess at a time. "Something with the network" is not a hypothesis. "The retry fires twice when the timeout lands during the commit" is one, because it can be proven wrong.
Change one thing. Exactly one. The strongest instinct under pressure is to change five things and see if the problem goes away. It sometimes does, and then you have learned nothing, fixed nothing, and hidden the bug for a quieter month.
Measure. Did the symptom move? Even a negative answer narrows the world. Especially a negative answer.
When the search space is history rather than code, the same discipline has a sharper tool: bisection.
It worked in March. It is broken today. Between those points lie hundreds of changes, and checking them one by one is a career. Checking the midpoint tells you which half to keep. Four checks reduce months to days; ten reduce a thousand changes to one. Binary search is the closest thing debugging has to magic, and it works on anything with a timeline: versions, configs, data, even org charts.
A few more field notes that pay for themselves.
Read the error message. Slowly. Out loud if needed. A remarkable share of week-long investigations end at a message that said the answer on day one, past which everyone skimmed because it looked familiar.
Distrust coincidence. "It broke the same week we changed the firewall" is not proof, but it is the best lead in the room, and leads outrank theories.
Write down what you ruled out. Unreproducible bugs outlive attention spans. The notebook is what stops the third engineer from redoing the first engineer's week.
And know when to stop: when you can make the bug appear and disappear on command by toggling the cause. Not when the symptom went away. Symptoms go away on their own all the time. They also come back.
Why write this on a company site? Because this loop is most of what we sell, whatever the technology on the invoice says. AI systems, data pipelines, twenty-year-old ERPs: the surface changes, the method does not. Vendors demo their best case. Ask them instead how they debug their worst one.
Frequently asked questions
How do you debug a problem that cannot be reproduced?
Treat "cannot reproduce" as "conditions not yet found". Collect real occurrences and look for what they share: input, timing, environment, client. Then build the smallest setup that makes the bug happen on command. Only after that do hypotheses and fixes mean anything.
What is the single most common debugging mistake?
Changing several things at once under pressure. If the problem disappears, nobody knows why, nothing was learned, and the bug usually returns later with better camouflage. One change, one measurement, every cycle. It feels slower and is dramatically faster.
What is bisection or binary search debugging?
When something worked in the past and is broken now, check the midpoint of that interval, keep the failing half, and repeat. Each check halves the search space, so ten checks resolve a thousand changes. It works on code versions, configurations, data ranges and anything else with a timeline.
When is a bug actually fixed?
When you can turn it on and off by toggling the identified cause, and a test now guards it. A symptom that faded after a restart or an unrelated deploy is not fixed, it is resting. The difference matters most in systems where the next appearance costs real money.
Does AI change how debugging works?
AI is excellent at the reading layer: summarising logs, spotting anomalies, suggesting hypotheses. It does not replace the loop, because the loop is about controlled experiments on your specific system. We use AI to generate suspects faster and the discipline to convict the right one.
