Frontier coding agents claim work they didn't do
Source: LessWrong ·
Tara Research Team arXiv | Twitter thread tl;dr. We built OverclaimBench, which asks frontier coding agents to review a corpus of code or text documents, then compares their final report with their transcripts. Pooled across 12 frontier agents and five scenarios, two-thirds of runs didn't open some of the files they were asked to review. In ~4 out of 5 of those incomplete runs the agent didn't tell the user, often claiming to have read everything. Skipping these files had real consequences - runs that overclaimed missed planted defects almost twice as often as runs that touched every file. Fig
Frequently Asked Questions
What is Frontier coding agents claim work they didn't do?
Frontier coding agents claim work they didn't do is a article discovery curated from LessWrong. Tara Research Team arXiv | Twitter thread tl;dr. We built OverclaimBench, which asks frontier coding agents to review a corpus of code or text documents, then compares their final
How can I access or experience this?
You can experience this content directly on its original host (LessWrong) by clicking the "Open Original" button.
Where does this content come from?
Curated from LessWrong by the OnSuggest team to surface the most interesting and serendipitous content on the web.
More like this
All Culture & ideas discoveries →A Knight Of The Seven Kingdoms Officially Returns December 16
A Knight of the Seven Kingdoms is officially back on December 16th thanks to a brand-new collection featuring George R.R. Martin's beloved characters.
8 Episodes Of Netflix’s Greatest Sci-Fi Hit Deliver Peak Lovecraftian Horror
One of Netflix’s greatest sci-fi hits of all time delivers peak Lovecraftian cosmic horror in the first 8 episodes of its long 42-episode runtime.
Stardew Valley: Timberline Falls Adds Whole New Region To Explore
Stardew Valley: Timberline Falls adds a whole new region to the game, including brand-new NPCs, quests, special orders, and so much more.
Michael B. Jordan's Fantasy Series On Prime Video Reinvents The Genre, And Some Fans Won't Like It
Sinners star Michael B. Jordan might be behind Prime Video’s lavish dragon rider epic, but the fantasy show could sink thanks to one huge risk.
The Lord Of The Rings Officially Unveils A Very Different Aragorn
The Lord of the Rings' newest Aragorn look is something completely unexpected and different, reimagining the iconic Viggo Mortenson role.
Stranger Things Cast Reunites For One Last Laugh In Surprising New Release
Netflix celebrates Stranger Things Day with a surprising new release, reuniting the hit science fiction series' cast on screen one more time.