Check tool signals first

Check tool signals first

When a tool call returns an error or unexpected result, the first suspect is the tool’s return value, not the model that called it. Read the actual response shape — the error code, message, structured data — before concluding “the agent hallucinated” or “the model is broken.”

Why this exists

A recurring failure pattern: a tool returns a structured error (HTTP 503, a { ok: false, error: "..." } payload, an exception class), the agent’s downstream behavior is wrong, and the operator concludes “the model is making things up.” Usually the model is responding correctly to the actual signal it received. The tool’s failure was either:

Blaming the model wastes time and misses the actual fix.

What to do instead

  1. Print the raw tool response before interpreting it. If you have logs, grep for the tool name + recent timestamps and read the actual bytes that came back.
  2. Look for silent-failure shapes. A tool that returns { ok: true, result: null } when the underlying operation failed is lying. Patch the tool’s contract to fail loud per the silent-failure-hunter pattern: structured error class, non-zero exit, observable diagnostic.
  3. Distinguish “tool said X” from “tool result implied X.” “The tool returned HTTP 503 with body {error: "sidecar unreachable"}” is a tool signal. “The tool didn’t work” is interpretation.
  4. When the boundary is opaque (a child process, a CLI adapter, a shell command), tee its stderr to a log file or capture it via structured trace lines like [tool-trace] ok ... / [tool-trace] failed: ... so future runs have the evidence the first run lacked.

Anti-pattern

“The model didn’t understand what I asked.” Almost always wrong. The model understood; the tool’s response was either incomplete, silently failed, or returned something the model correctly responded to.

Edit this page on GitHub