Evidence before diagnosis

Evidence before diagnosis

Confirm a log’s suggested cause by querying the underlying state BEFORE naming a diagnosis to the operator. A log line that says “X is missing” is a hypothesis — query the DB, the filesystem, the running container, or whatever surface holds the truth, and verify X is actually missing before reporting “X is missing” as the root cause.

Why this exists

Two recurring failure modes:

  1. Misled by a misleading log. Logs are written by code that may be stale, buggy, or describing an intermediate state. “User not found” in a log might mean the user exists but the lookup query was wrong. “Provider unconfigured” might mean the row was deleted, or that the status column convention changed, or that a different code path created it with a default.
  2. Compounding errors. Telling the operator the wrong cause sends them down the wrong fix path. Half an hour later we discover the real cause was different, the agent’s credibility takes a hit, and the operator wasted time chasing a phantom.

What to do instead

Before naming a cause:

  1. Read the relevant row directly. docker exec dpf-postgres-1 psql ... — query the actual record. Do not infer state from the log message.
  2. Compare against the schema’s intent. A null lastError might be meaningful or meaningless depending on whether the column is written only on failure or every run.
  3. Look at adjacent state. If a job’s lastStatus is “ok” but nothing downstream reflects the result, the failure is somewhere between the heartbeat and the output, not in the job itself.
  4. State what you observed, not what you suspect. “BackupRun row with status=ok and finishedAt=2026-05-17T22:50Z, but the file at storagePath is missing on disk” is a diagnosis. “Looks like the backup didn’t run” is a hypothesis dressed up as a conclusion.

Anti-pattern

Reading one log line, matching it to a known failure mode, and reporting “the cause is X” without confirming X. The agent’s job is to be a diagnostician, not a classifier.

Penalty

The credibility cost of naming the wrong cause is much higher than the 2–10 second cost of running a confirming query. There is no acceptable shortcut.

Edit this page on GitHub