[Feature Request] Improve multi-criteria recommendation reasoning to account for trade-offs and user constraints
Bug Description
Feedback: Single-metric "leaderboard" recommendations that ignore trade-off data the model itself surfaced
Context: Comparing local ASR models for a real-time, push-to-talk dictation use case on my own Mac. Hardware limits
and the latency requirement were stated up front.
What happened: The model produced a comparison table containing accuracy (CER), model size, deployment complexity,
and speed. It then crowned the lowest-CER model as "the best / SOTA / 🏆" — despite that same table showing the
model was 5x larger, harder to deploy, and slower, for only a 0.87% absolute accuracy gain that is imperceptible in
the stated use case.
This is NOT a factual error: the numbers and the "lower CER is better" direction were correct. The failure is
reasoning integration — every piece of evidence needed to reach the correct recommendation was already in the
model's own table, but none of it was used in the conclusion. It defaulted to ranking by one benchmark instead of
weighing cost vs. benefit against constraints I had already given.
Expected behavior: When a recommendation involves trade-offs AND the trade-off data is already on the table, weigh
marginal benefit against cost in the user's actual context. Do not equate "#1 on a single metric" with "best."
Absolute framing ("winner / SOTA") must be qualified by the conditions under which it holds.
Why it matters: I had to catch and reverse the recommendation myself. For decision-support, a
confident-but-uncontextualized "best" is worse than no ranking — it adds verification burden and actively misleads.
Environment Info
- Platform: darwin
- Terminal: iTerm.app
- Version: 2.1.181
- Feedback ID: 10dd89cf-d827-44e3-98c3-65c509862a30
Errors
[]