The bug that wasn't a bug
The crash log pointed at our code. The real culprit was a rate limit wearing a stack trace as a disguise. Here's the prompt we now run before trusting any error message.
One of our pipelines quietly started returning nothing. No crash, no alert — just empty results and logs that pointed, very convincingly, at our own code. We chased that code. The code was fine.
The real cause sat three layers up: an HTTP 402 from an upstream API — a quota we'd exhausted — surfacing downstream as empty payloads instead of anything that looked like a limit. The logs weren't lying, exactly. They were answering a different question than the one we were asking.
The loudest error is rarely the first error
We run a fleet of AI agents that build and operate our own systems, and we log everything they do. Across 309 real debugging sessions, one pattern kept paying for itself: before trusting the visible error, actively try to disprove it. Rate limits, swallowed errors, unhandled rejections, leftover state — production has a wardrobe full of masks that all look like "your code is broken."
Eventually we stopped re-deriving that checklist from scratch and wrote it down as a prompt. This is the exact one, verbatim from our debugging pack:
I think {{service}} is failing because of {{suspected_cause}}, but I'm not sure.
Symptoms: {{symptoms}}.
Challenge my assumption. Common ways the visible error misleads:
- An external limit (rate limit, quota, daily cap) that only surfaces as a
generic crash or timeout.
- An unhandled promise rejection / async error killing the process far from
where it started.
- A caught-and-swallowed error (e.g. `2>/dev/null || default`) hiding the real
"command not found" / permission / missing-dep failure.
- A resource that "works" only because of leftover state from a previous run.
For each, tell me how to quickly rule it in or out. Give me the check, not a
verdict.
The last line is the part that matters. An LLM that hands you a verdict is guessing with confidence; an LLM that hands you four cheap checks is doing engineering. You stay the investigator — it just makes sure you interrogate the right suspects.
Why "give me the check, not a verdict" works
When we let the model conclude, it pattern-matched to the most statistically common cause and sounded certain. When we forced it to output falsifiable checks instead, the external-limit class of bug stopped eating afternoons — the "rate limit or quota?" check is first in the list for a reason, because it's the cheapest to run and the most embarrassing to skip.
That prompt is one of 16 in our Debugging & Root-Cause pack — each distilled from real fleet sessions, each shipped with a when-to-use line and a why-it-works note like this one. It's $10, and the sample above is yours either way.