Follow ZDNET: Add us as a preferred source on Google.
ZDNET’s key takeaways
- AIs were given work tasks already completed by real people.
- The AIs failed miserably compared with the human workers.
- But AI is getting smarter.
One of the many fears about AI is that it will replace people in their jobs. And though such fears aren’t unfounded, they may be overblown, at least for now, according to a new study.
Remote Labor Index
To gauge whether artificial intelligence could complete a project as effectively as a human being, a group of researchers gave several AIs a series of work projects to perform. Already accomplished by real remote freelance workers, the projects covered game development, product design, architecture, data analysis, and video animation.
More specifically, the tasks included such challenges as the following:
- Build an interactive dashboard for exploring data from the World Happiness Report.
- Create 3D animations to showcase the features of a new earbuds design and case.
- Create a 2D animated video advertising the offerings of a free services company.
- Develop architectural plans and a 3D model for a container home based on an existing PDF design.
- Build a brewing-themed version of the “Watermelon Game,” where players merge falling objects to reach the highest level item.
- Format a paper using the provided features and equations for an IEEE conference.
Also: I tested ChatGPT’s Deep Research against Gemini, Perplexity, and Grok AI to see which is best
Encompassing various levels of difficulty, the tasks as performed by the actual people cost $10,000 and took them more than 100 hours to complete. To measure how AI automation stacks up against remote work done by human beings, the researchers set up a benchmark called the Remote Labor Index (RLI).
How the AI models performed
As described by the researchers, the purpose of the RLI is to test AI’s ability to automate hundreds of long, real-world, economically valuable projects from remote work platforms.
Also: Is ChatGPT Plus worth your $20? I compared it to Free and Pro plans, and here’s my advice
The AI models used in the study were Manus, Grok 4, Sonnet 4.5, GPT-5, ChatGPT agent, and Gemini 2.5 Pro.
So how did they perform? Not too well.
“While AI systems have saturated many existing benchmarks, we find that state-of-the-art AI agents perform near the floor on RLI,” the researchers revealed. “The best-performing model achieves an automation rate of only 2.5%. This demonstrates that contemporary AI systems fail to complete the vast majority of projects at a quality level that would be accepted as commissioned work.”
Manus fared the best at a 2.5% performance rate. Grok 4 and Sonnet 4.5 tied at 2.1%, GPT-5 was next at 1.7%, followed by ChatGPT agent at 1.3%. Gemini came in last at 0.8%.
Also: Is AI coming for your job? Here’s one labor indicator that could soothe your fears
One of the researchers, Dan Hendrycks, chimed in on the test and the results via a post on X. Hendrycks acknowledged that while AIs are smart, they’re not yet that useful, not with an overall automation rate of less than 3%. To explain why the AIs fell down on the job, Hendrycks said that many AI capabilities are deficient. AIs don’t learn on the job as they don’t possess long-term memory storage. Plus, an AI’s visual abilities are limited, a skill required to perform several of the tasks.
Steadily improving
This all sounds like good news for workers worried about being replaced by AI. Right? Well, don’t rip up your resumes just yet. The test specifically incorporated creative tasks that required somewhat advanced skills. Other types of jobs and projects likely would be more easily tackled by an AI. Plus, AI is only going to get smarter and more capable.
Also: Need a new job? These AI roles are the fastest growing in the US, says LinkedIn
“While absolute automation rates are low, our analysis shows that models are steadily improving and that progress on these complex tasks is measurable,” the researchers said. “This provides a common basis for tracking the trajectory of AI automation, enabling stakeholders to proactively navigate its impacts.”
Yep, best to keep those resumes updated just in case.
