Evaluating contract bids with AI? Prepare to be challenged.

A recent contractor lawsuit over the Army’s use of artificial intelligence to award a $450 million contract alleges that the decision was based on “a classic AI hallucination.”

If the protester prevails in court, the lawsuit could “shape how federal agencies document and disclose AI’s role in procurement decisions.” A decision in favor of the protester could call into question other contract awards where AI was used to research, evaluate, or summarize proposal responses before arriving at a source-selection decision.

From here, it’s easy to imagine how this could prompt a sudden influx of Freedom of Information Act requests from contractors who feel they were wrongly excluded from an award as a result of hallucinated AI outputs.

This points back to the larger conversation surrounding how the race to AI adoption is outpacing the ability to explain and defend decisions influenced by AI. What we are seeing here is a distinction most AI deployment roadmaps have not mapped out yet: informing a decision and being held accountable for one are separate jobs with separate standards.

When AI returns an answer, nothing in the interaction reveals what it weighted, what it discarded, how confident it was, or whether it would answer the same question the same way tomorrow. The user gets fluent, confident prose and a subjective impression of neutrality. Anyone who’s worked with AI has encountered this, so it’s understandable how this issue would be raised in a protest over a $450 million contract award.

Despite popular perception that protests are out of control, the Government Accountability Office most recent annual bid protest report indicates a decade-long trend of declining protests. The concerns surrounding contract award protests are not misplaced.

For federal government agencies, protests can result in work stoppages, operational delays with negative mission impact, financial waste and additional administrative burden. For contractors, there is lost trust in the procurement system. Within our shrinking Defense Industrial Base, the procurement system itself is viewed as a barrier to entry at best or “the valley of death” at worst.

The fact that bid protests have receded significantly over the past decade is a trend to be celebrated. This is progress that should be further accelerated by the responsible application of AI, not the opposite. Contract and procurement officials are under severe strain from staffing and capacity shortages. These challenges are exacerbated by the tedious and time-consuming requirements of the job, so it’s understandable that the rush to adopt AI in government procurement might be outpacing acquisition regulations.

To address these challenges, what’s needed now are four concrete capabilities, concrete enough that someone outside the procurement operation can audit them:

Auditability. Every AI output - especially outputs related to government contract decisions – must include evidence, provenance and traceability so organizations, auditors and investigators can trace decisions back to the data and reasoning that supported them.

Observability. A durable record of what the system actually did, not a summary of what it was designed to do.

Contestability. A defined path for the person or organization affected by a decision to challenge it and get a real answer.

Evaluations. Continuous, against criteria defined by the government agency. Because trust is granted once, and performance has to be re-proven.

These four capabilities are best thought of in anticipation of a hypothetical question that is commonly raised during a bid protest: if someone were to ask you about a single decision your systems touch, one that changes something for an organization or citizen stakeholder, could you produce a record of what happened, an explanation of why, and a route for that person to contest it? Not in principle, but for a decision made six months ago, asked without warning, this week.

Most organizations find they can’t. And that realization is far better learned from yourself than from a regulator, a journalist, somebody’s lawyer, or in a publicly shared document.

All of this is ordinary practice in every regulated domain where consequential decisions get made and somebody can be held to account for them. Aviation does it. Clinical medicine does it. Financial audit does it.

This process is familiar to information governance professionals in these and other industries who work with electronically stored information (ESI). While agentic AI is new, it should not be an outlier. Yet it is, at exactly the moment when government agencies are beginning to trust it and courts are starting to treat AI as ESI.

As this award protest suggests, any contract decision involving AI is increasingly likely to face scrutiny or challenge.