Are experienced developers slowed down by using AI?
In an experiment based on real open-source projects, researchers found that using AI tools not only did not improve development efficiency but actually increased task time by 19%.
This is completely contrary to what people predicted.
Netizens also commented that without AI assistance for coding, they would be in serious pain.
Next, let's see how this result, which goes against most people's intuition, was obtained.
16 Experienced Developers Test AI Usage Efficiency
To directly assess the impact of AI tools on developer productivity, researchers had 16 experienced developers complete 246 tasks from well-known open-source code repositories (averaging 23,000 stars, 1,100,000 lines of code).
Each task was randomly assigned to either allow or disallow the use of AI tools. If AI tools were required, the primary tools used were Cursor Pro with Claude 3.5/3.7 Sonnet.
Subsequently, the research team recorded the time developers spent completing tasks under different conditions and collected screen recordings of their work.
In this experiment, each developer provided a list of real tasks from their own code repository to be used as the work content for this study.
These tasks typically included bug reports, feature requests, or work items for coordinating development. The content ranged from brief problem descriptions to detailed analyses, and the time required varied from a few minutes to several hours.
After collecting the task list, developers predicted the time required to complete each task with and without AI assistance.
Subsequently, these tasks were randomly assigned to two conditions—allowing or disallowing AI use—by simulating a coin toss.
Developers were allowed to handle assigned tasks in their preferred order, enabling flexible work arrangements, and sometimes even processing multiple tasks simultaneously.
Once they deemed a task satisfactorily completed, they would submit a request to the corresponding code repository, which would typically be reviewed by other developers.
Using AI Tools is Surprisingly 19% Slower Than Not Using Them
Developers completed 136 tasks under conditions where AI was permitted and 110 tasks under conditions where AI was not permitted.
Interestingly, developers predicted their efficiency would increase by 24%, but the results were exactly the opposite.
The study found that when developers used AI tools, the average time to complete tasks increased by 19%.
Furthermore, tasks where AI was permitted took longer in almost all completion time quantiles.
The chart below shows the time ratio developers spent on each type of problem in the valid screen recordings.
It can be seen that when AI is permitted, developers spend less time on active coding and reading/searching for information. Instead, they spend time reviewing AI outputs, prompting the AI system, and waiting for AI generation.
Given this surprising result, the researchers further explored 20 related factors that might cause developers to spend more time when using AI, categorizing these factors into four types:
Direct productivity loss: Refers to the decrease in development efficiency due to the inherent mechanisms of AI tools during their use.
Experimental bias factors: Refers to confounding variables in the experimental design or execution that may introduce bias or affect the external validity of the results.
Factors enhancing developer performance: Refers to characteristics in tasks, code repositories, or the experimental environment that give developers an advantage over AI.
Factors limiting AI performance: Refers to characteristics in tasks, code repositories, or AI tools/environments that diminish the effectiveness of AI compared to human developers.
After analyzing the 20 factors, they found qualitative and quantitative evidence for 5 factors suggesting they indeed led to slower efficiency, unclear or mixed evidence for 9 factors, and evidence for 6 factors indicating they did not cause efficiency decline.
One More Thing
Meanwhile, The Wall Street Journal also reported that even if AI saves time, it results in more work.
For the overworked employees, this is truly terrible news; it's genuinely hard to find humor in it.
As more and more professionals start using AI assistants for daily tasks, an average of about one hour of work time can be saved daily (according to a survey by business software manufacturer SAP).
However, companies have not provided employees with lighter workloads; instead, management often expects them to generate more output with the time saved.
At Amazon, CEO Andy Jassy publicly urged employees:
"Figure out how to use AI to innovate faster and more broadly for customers, and to accomplish more with leaner teams."
References: