Four AI Auditors, One Firmware

As AI coding tools became part of my daily firmware work, one question kept bothering me:

If the same agent writes the change, explains the change, and reviews the change, how independent is the review?

Changing the prompt from “implement this” to “now be a skeptical reviewer” helps, but it does not create a new point of view. The model has already followed one reasoning path. Its review tends to inherit the same assumptions, source selection, and blind spots.

For higher-risk firmware, I now use a different pattern: give the same frozen source and the same audit contract to several independent agent CLIs, keep their work isolated, and reconcile their findings only after every report is complete.

In one recent audit, four auditors examined the same embedded codebase. Their raw reports contained dozens of findings. Final arbitration consolidated those into a much smaller evidence-backed set, including several severe enough to block release.

The interesting part was not that four models found many bugs. The interesting part was how much machinery was required to make “four independent reviews” mean anything at all.

Independence Starts Before the Prompt

The first audit nearly began against the wrong source tree.

Two directories had similar product names, but only one contained the actual SDK and the controlling audit prompt. A convincing report against the wrong revision would have been worse than no report: polished, expensive, and irrelevant.

Before launching an auditor, I freeze four things:

1. Target identity — the exact repository or source package under review.
2. Revision identity — commits for the top-level project and every nested dependency.
3. Source manifest — file count and SHA-256 hashes for the auditable tree.
4. Audit contract — scope, severity definitions, exclusions, and required evidence.

Every auditor must receive equivalent source. If the project is in Git, detached worktrees are convenient. If the source is an exported package without a Git root, I create complete external copies instead. The isolation mechanism is less important than proving that all copies contain the same bytes.

The auditors do not see one another’s reports. They do not vote, debate, or “build on” earlier findings. That separation is deliberate. Discussion comes later.

Give Every Auditor the Same Contract

An open-ended request such as “review this firmware” produces open-ended output. One model focuses on style, another on memory safety, and another rewrites the architecture in its head. Comparing those reports is almost impossible.

My audit contract requires each finding to contain:

• a stable finding ID;
• severity;
• affected files and line anchors;
• a concise claim;
• evidence from code, specification, build output, or reproducible behavior;
• impact;
• a proposed verification or remediation;
• uncertainty where the evidence is incomplete.

The scope also names the areas that matter for the product: initialization, state transitions, length validation, shared resources, protection propagation, thermal behavior, power sequencing, and build consistency.

This does not tell the auditor what to find. It makes every claimed finding falsifiable.

A Report File Is Not Proof of a Completed Audit

One of the less glamorous lessons was that a report can look complete while being structurally broken.

A heading check may pass even though later findings are missing fields. A JSONL block may parse while its IDs disagree with the Markdown body. A process may exit successfully after producing a truncated report.

So I validate each report as data:

• all mandatory sections exist;
• every finding has every required field;
• IDs are unique and contiguous where required;
• severity values belong to the allowed set;
• reported counts match the actual findings;
• Markdown and machine-readable summaries agree;
• no auditor has modified the source;
• the final source manifest still matches the frozen input.

If a report violates the contract, only the original auditor repairs it. Another model is not allowed to “help finish” the report because that would contaminate independence.

Reconciliation Is Not Voting

After the four reports are frozen, a separate arbitration pass maps every raw finding to a final disposition:

• accepted — evidence supports a real issue;
• merged — several findings describe the same root cause;
• rejected — the claim conflicts with source or specification;
• needs verification — plausible, but missing decisive evidence;
• informational — useful observation without a release-blocking defect.

Three auditors agreeing does not automatically make a claim true. They may have repeated the same attractive misunderstanding. One auditor can be correct against three if it points to the decisive state transition, register definition, or call path.

The unit of value is not consensus. It is evidence.

The final report must account for every original finding exactly once. Otherwise a difficult issue can quietly disappear between reports.

What the Four Auditors Found Differently

The useful disagreements were rarely about syntax. They appeared at subsystem boundaries:

• an API existed, but its initialization path was never registered;
• a protocol parser accepted a length that a downstream function could not safely consume;
• protection information was detected locally but not propagated to the product policy;
• two features used the same hardware resource without an explicit owner;
• a bypass or power transition was individually reasonable but unsafe in the full sequence;
• buildable configurations had drifted away from the architecture described in the source.

These are exactly the problems a single-file review tends to miss. Firmware behavior lives across call graphs, interrupt paths, build flags, peripheral ownership, physical timing, and hardware policy.

Different models chose different entry points into that system. Their disagreement created a map of where human attention was most valuable.

Where Human Judgment Still Matters

The arbitration agent can trace evidence and detect contradictions, but it does not own the product decision.

A person still has to decide:

• whether an assumption matches the real board;
• whether a theoretical path is reachable in production;
• which specification revision governs the behavior;
• whether a clean build is sufficient or hardware evidence is required;
• whether the residual risk is acceptable for release.

In this audit, the release was blocked pending remediation. That conclusion did not come from the number of warnings. It came from accepted high-severity findings in product-critical paths.

A Reusable Audit Recipe

My current workflow is:

1. locate and identify the real source;
2. freeze revisions, dependencies, and hashes;
3. define one evidence-oriented audit contract;
4. create isolated, byte-equivalent workspaces;
5. run each auditor without access to the other reports;
6. validate every report and every finding field;
7. freeze the accepted reports and their hashes;
8. map every raw finding during arbitration;
9. verify disputed claims against primary evidence;
10. let the engineer make the release decision.

This is slower than asking one agent to review a pull request. It is also aimed at a different problem.

For a small utility, one competent review may be enough. For firmware that controls power, protection, authentication, charging, or shared hardware resources, independent reasoning is worth the extra ceremony.

Four AI auditors do not remove the need for an experienced firmware engineer.

They make the blind spots visible enough for that engineer to act.