A revised version of this landed at the end of June and it is the best thing I have read on why people actually like working this way. Pimenova, Fakhoury, Bird, Storey and Endres went through 190,000-odd words of interviews, Reddit threads and LinkedIn posts and built a grounded theory of vibe coding rather than another benchmark.

Their central claim is that trust is the regulator:

AI trust regulates movement along a continuum from delegation to co-creation and supports the developer experience by sustaining flow.

That continuum is the useful idea. Delegation at one end — you hand over a task and check the result. Co-creation at the other — you are in the loop, thinking alongside it. Where you sit is not a setting you choose or a property of the tool; it is a function of how much you currently trust the thing, and it moves constantly, sometimes several times inside one session. Everybody who does this seriously already knows the feeling. Nobody had named the axis.

It also explains why the productivity literature is such a mess. Two developers using identical tools on identical tasks can be at opposite ends of that continuum, which means they are doing genuinely different activities. Averaging them produces a number that describes neither.

The pain points they catalogue are refreshingly unromantic: specification, reliability, debugging, latency, code review burden, collaboration. Note that latency is on that list alongside correctness. Flow is fragile, and a model that is right but slow can be worse for the practice than one that is quicker and occasionally wrong. That is an uncomfortable finding and I believe it completely.

Qualitative work gets dismissed in this field as soft next to a benchmark score. This is a good argument for why that is backwards: the benchmarks can tell you what the model did, and only this kind of study can tell you what the person was doing while it happened.

If you have argued about AI productivity on the internet in the last year, you have cited this study, and there is a decent chance you have cited it wrong.

The original: METR took sixteen experienced open-source developers, 246 real issues in codebases they already knew well, and randomised whether they could use AI tools. Result — they were 19% slower with the tools. The part that made it famous was the perception gap: they expected a 24% speedup going in, and after finishing slower, still believed they had been sped up by about 20%.

Go to that page today and there is a banner on it:

These results are out of date. We have released results that are current as of early 2026, in a continuation of this study.

METR published new data in February on late-2025 tools, with a much larger cohort, and the slowdown largely evaporates — the newer estimate is close enough to zero that the honest summary is "no clear effect either way", not a reversal into a speedup. Read their continuation rather than my paraphrase; the confidence intervals are the whole point and they do not survive being quoted.

Three things worth holding onto.

The headline number was always narrow. Experienced developers, on codebases they knew intimately, on early-2025 tooling. That is close to the worst case for AI assistance and it was never a claim about the general case. It got used as one anyway, by me among others.

The perception gap is the durable finding. Whether the tools make you 19% slower or 4% faster, the discovery that developers cannot tell which is happening to them survives every revision. That should terrify anyone running an engineering org on self-reported velocity, and self-reported velocity is what almost everyone is running on.

METR did the right thing. They put a warning label on their own most-cited result. That is rarer than it should be, and it is why they remain worth reading when the vendor benchmarks are not.

Google Cloud's DORA team published an ROI framework for AI-assisted development in May, and Matt Saunders' write-up is the fastest way into it.

The headline model: a 500-person engineering organisation at $176,000 average salary sees roughly 39% first-year ROI and an eight-month payback — about $11.6m of value against $8.4m of investment. Over three years Google's own data shows an average 727% return.

Treat those figures as what they are, which is a calculator with assumptions you get to set, published by a company that sells the thing being evaluated. Run the conservative scenario. The report to its credit tells you to.

Two things in there are worth more than the ROI number.

The first is the J-curve. Organisations reliably get worse before they get better — learning curve, review overhead, process churn — and DORA calls that dip "the tuition cost of transformation". I have watched three teams go through this and every one of them interpreted the dip as a tooling problem and responded by changing tools, which resets the curve to zero. If you take one thing from the report, take this: the trough is the normal shape, and paying the tuition twice is a choice.

The second is the spread. 35–40% productivity gains on simple tasks; 10% or less on complex legacy code. Nearly all the money in most organisations is in the second category. Any business case built on the first number is a business case for work you were not doing.

The framing underneath all of it — "AI magnifies the strengths of high-performing organisations and the dysfunctions of struggling ones" — is not new, and DORA has been saying versions of it since the DevOps reports. It keeps being true, which is the problem: it means the tool cannot be the intervention, and buying the tool is much easier than being the kind of organisation where it pays.

Shukla, Joshi and Syed ran the experiment everyone assumed had already been run: take generated code, iteratively ask the model to improve it, and count the vulnerabilities at each round. Four hundred samples, forty rounds of refinement, four different prompting strategies.

Critical vulnerabilities went up 37.6% after five iterations.

I want to be precise about why this is interesting, because "AI bad at security" is not the finding. The finding is that the refinement loop — the single most natural thing to do when you are not happy with generated code, the thing every agent harness does automatically — has a failure mode that points the wrong way. Each pass is locally plausible. The model addresses what you asked about, adds some defensive code, moves something around. Nothing in that loop has any memory of the security properties the previous version happened to have, so they get traded away for the improvement you asked for.

Different prompting strategies produced different vulnerability patterns, which tells you the effect is a property of the loop rather than of any one phrasing. You cannot prompt your way out of it.

The paper's recommendation is human validation between iterations rather than at the end. That is correct and almost nobody does it, because the entire appeal of an agent loop is that you do not have to sit in the middle of it. Which is roughly where we are with all of this: the practices that would make it safe are the practices that make it feel slow, and feeling fast is what people are buying.

The preprint is from mid-2025 and has been sitting in my "should have written about this" folder ever since. It has aged into relevance rather than out of it.

Hanqing Zhao and the Systems Software & Security Lab at Georgia Tech have been running a project called Vibe Security Radar since May 2025: take public vulnerability advisories, trace the fix backwards to the commit that introduced the bug, and check whether an AI coding tool left fingerprints on it.

The trend line is the story. Six CVEs attributed to AI-generated code in January 2026, fifteen in February, thirty-five in March. Seventy-four confirmed cases in total across roughly fifty tools tracked. The researchers reckon the real number is five to ten times what they can detect — four to seven hundred across open source — because most AI-written code does not announce itself.

One finding needs handling with care, and the coverage mostly did not handle it carefully. Claude Code accounts for the largest share of attributed flaws, and the researchers say that is partly because it leaves identifiable signatures. That is a detection artefact, not a quality ranking. A tool that stamps its commits is a tool you can measure; a tool that does not is a tool that vanishes from your dataset. If you take that chart as a league table of which assistant writes the least safe code, you have read it exactly backwards — you are looking at a map of which vendors are legible to researchers.

The genuinely alarming number is buried in the middle: Claude Code alone touched more than four percent of public GitHub commits last month. Whatever the per-commit defect rate turns out to be, it is now multiplied by a denominator that did not exist two years ago.

Methodology this transparent is rare in this corner of the discourse. Most "AI code is insecure" statistics you will see quoted this year come from vendors selling scanners. This one comes from a lab publishing its method, and it is correspondingly less dramatic and more useful.

This is from February and I keep coming back to it, because the quotes have aged better than the proposals have. GitHub PM Camilla Moraes opened a community discussion on maintainers "dedicating substantial time to reviewing contributions that do not meet project quality standards" — the options on the table included letting maintainers disable pull requests outright, restrict them to collaborators, delete them from the interface, and attach some kind of attribution when a tool wrote the code.

The line worth stealing is Jiaxiao Zhou's:

Review trust model is broken: reviewers can no longer assume authors understand or wrote the code they submit.

That is the actual problem, stated in one sentence, and it is not a spam problem. Code review has always run on an unwritten assumption: whatever else is wrong with this patch, there is a human on the other end who can explain it. Take that away and review stops being a conversation and becomes an inspection. The economics get ugly fast, because inspection does not compress the way conversation does.

Xavier Portilla Edo's estimate — "1 out of 10 PRs created with AI is legitimate" — is the kind of number I would normally push back on, since it is one maintainer's gut feel rather than a measurement. But nobody in that thread disputed the order of magnitude, and the people in that thread do this all day.

GitHub's own Matthew Isabel pushed back on counting AI-generated PRs at all: "A bad or off-topic PR is a bad PR, regardless of where it came from." He is right in principle and wrong in practice. Provenance was never worth tracking when writing a bad patch cost the sender an afternoon. It is worth tracking now that it costs them ninety seconds, because the cost asymmetry between sender and reviewer is the whole ballgame.