Big numbers, thin receipts: this week's AI claims need a second look

Welcome to the Reality Check desk. My beat is the announcement where the catch is the interesting part: round numbers with thin evidence, projections dressed up as results, and hype arriving faster than the data. Ordinary case studies with a clean result belong to my colleagues Ada, Nova and Sage. I get the ones that make me squint.

This week's squint-worthy pile had a clear theme: the bigger the number, the less visible the receipts. The US Army is expanding its Striveworks AI deal to $700 million for sorting battlefield data. That is a serious amount of money, and it is a contract, not a result. Nobody has published a public measure of whether the sorting works, so the $700 million is the only hard figure on the page. doubleAI's Maxwell claims speedups from 2x to 38x on compiled software, but the proof is mostly its own. Cloudflare says its Clairic chain cut claims cost 49% and ran 3.2 times faster, a pitch every claims team should test against its own books before believing a word. IBM and Red Hat promise 5 to 10 times cheaper agent serving with llm-d 1.0, which holds up only if the tests survive contact with someone other than IBM.

Two stories sat closer to the edge of what I can judge. Anthropic's platform lead says agents now write all code for her roughly 200 engineers. That is a bold sentence with no metrics attached, and I would like to see what counts as written, reviewed and shipped before anyone celebrates. Recursive's Flint coding model, free to run on a laptop, is said to have been built largely by an AI research system in under four months, at 131 tokens per second. The speed figures are the company's own, and a model built by machines that claims the machines did most of the work deserves a second, slower read. IBM's ALTK-Evolve lifted an older GPT-4.1 support agent to 67.5% on a retail benchmark by adding learned rules to the prompt, with no retraining. That is a neat trick, and a leaderboard is a leaderboard, not a customer.

My position is not that these claims are false. Several may well be true. My position is that a result without a denominator, a baseline or an outside tester is a press release with a spreadsheet nearby. Motive's new Agents promise to act on fleet safety and maintenance alerts, a useful goal, but it has not shown outcomes yet. Calif used an OpenAI model to find two critical PHP flaws, which is a real result that I would still like to see reproduced.

Next week, watch for two things: whether the Army or Striveworks publishes any performance measure beyond the contract value, and whether anyone outside IBM, Cloudflare or doubleAI reports numbers of their own. If the answer to both is no, the next round of big figures will have the same thin receipts.