AI systems and humans often diverge in their understanding of a shared task, and that divergence gets expensive downstream. This project asks: how do you measure that cost before it shows up as user frustration, lost time, or bad decisions?
When users ask an AI for help catching up on a complex task, they receive a summary. But what they experience is uncertainty: they don't know how much they can trust the summary, or what critical context it might be missing.
Traditional evaluation methods like thumbs-up/thumbs-down ratings or raw accuracy scores fall short here. They tell us if an answer is factually correct in isolation, but they fail to capture the user's mental model or the downstream consequences of acting on incomplete information. We needed a way to uncover failure and recovery behaviors when the system provides an answer that seems fine on the surface, but isn't.
To bridge this gap, I developed the Information Access Cost (IAC) metric. The goal was to operationalize idea of a trustworthy and complete summary into a scorable, repeatable measurement.
Instead of just checking if an AI summary contained a fact, the IAC measures the search time and effort required for whoever picks up the task next — a person or another system — to track down entities the summary should have surfaced. If an AI report omits a detail or provides a vague direction, how many "rooms" or "steps" will the user waste trying to resolve the ambiguity?
By modeling expected search traversals, we turned the "cost of an AI answer that seems fine but isn't" into a concrete rubric. This allows us to penalize misinformation and omissions not just as errors, but as direct costs to the user's time and trust.
To isolate the impact of different AI reporting strategies, I designed a five-condition ablation study in a simulated task environment built to model real information-seeking behavior — users searching a multi-room space for entities relevant to their task:
This design was a deliberate method choice to isolate specific tradeoffs, such as the value of structured data pipelines versus raw LLM processing, and exhaustive detail versus targeted relevance.
The results were statistically significant, not just directional, as confirmed through non-parametric tests appropriate for this kind of repeated-measures comparison.
The Task-Aware (TA) condition dropped the median Information Access Cost from User-Only to TA — users saved around 526 steps on average, a highly significant improvement.
Bypassing a structured data pipeline (the End-to-End condition) led to confident but highly misleading AI summaries on edge-capable models. TA practically eliminated false trails, compared to a 65% median misinformation share in End-to-End — a highly significant difference, proving raw LLM processing is quantifiably riskier than a structured approach.
TA performed on par with the exhaustive Full Realization (FR) condition in absolute cost savings (no significant difference between them) but reached that parity with significantly higher communicative compression.
So what? For a product team, these findings offer a clear mandate backed by rigorous data:
While the IAC rubric successfully quantified search costs, the study was conducted in a constrained, simulated environment. In reality, human search behavior is messier and less predictable than an idealized map traversal. If I were to iterate on this, I would validate the IAC metric against real-world telemetry from a live product to see how closely our modeled costs match actual user recovery times.
Additionally, the sample relied on specific LLM models; as models evolve, the baseline rate of spontaneous "hallucinated" helpfulness might shift, requiring continuous recalibration of the rubric.