Benchmarking Autonomous AI Agents: A New Class of AI IQ Task Testing

Original Paper: https://arxiv.org/pdf/2412.14161v1

Podcast Overview: https://notebooklm.google.com/notebook/7854b500-3f35-404e-b0cf-fa8b90665b3c/audio

Introduction

A new paper from Duke and Carnegie Mellon Benchmarking LLM Agents on Consequential Real-World Tasks introduces a groundbreaking new type of AI Intelligence benchmark designed to evaluate large language model (LLM)-powered agents in real-world, professional settings. The benchmark represents a significant evolution in benchmarking methods, surpassing prior model IQ domain (GPQA, MMLU) and general 'human' reasoning tests (ARC) tests. The previous class have all mostly been surprised by the very top of current AI models, notably (GPT o3) with the rest of major models scoring high (ie. Anthropic Claude, Gemini 2.0, Llama 3). These are high-level academic intelligence tests, reasoning frameworks, and complex domain-specific challenges that AI models have consistly and improved scores on over 2022-2024 at an amazing pace surpassing average humans, Ph.D. domain experts and the best in the world in domains. Instead of focusing on further abstract reasoning or narrowly scoped tasks, this next level of benchmark simulates the next class of models autonomous agents working together, in the tests largely simulating general workplace environment human tasks to measure the ability of agents to autonomously and interactively perform human work-related tasks, signaling the next phase of AI development and future of the workplace. Currently, the Top Performer: Claude-3.5-Sonnet which performed best, but also surprisingly (and good for the benchmark and humans) completing only 24% of tasks and achieving a 34.4% partial completion score so all of the models have still a good way to run before replacing humans. As the author's write:

"As a result of experiments, we find that the best performing model, Claude 3.5 Sonnet was able to autonomously perform 24.0% of the provided tests to completion, and achieve a score of 34.4% on our metric that provides extra credit for partially completed tasks."

The report also noted several notable areas including cost and efficiency, Open model progress and Common model failures:

• Cost and Efficiency: Claude-3.5-Sonnet was the most expensive to run, while the Gemini-2.0-Flash model was more cost-efficient.
• Open-Weight Model Progress: Llama 3 models were competitive, with Llama-3.3-70b achieving comparable performance to much larger models.
• Common Agent Failures: The report identifies common agent failures including: lack of common sense and domain knowledge, lack of understanding of social cues, and fabricating "shortcuts" when it is unable to complete complex tasks.

Key Implications for Future Directions

There were also several notable key implications for future directions in the paper including current limitations of the top models, areas for improvement in all models, future benchmark directions and a call for future and competing similar class benchmarks to start this level of development. This may be summarized here:

• Current Limitations: Current AI agents still struggle with many real-world professional tasks, highlighting the need for further advancements.
• Areas for Improvement: Challenges identified include the need to improve social skills, navigating complex UIs, and handling tasks that require private, non-public data.
• Future Benchmark Directions: Suggestions for future work include expanding tasks across industries, incorporating more complex, creative, and long-horizon tasks and tasks with vaguer intents, and including comparisons against human performance.
• Call for Collaboration: The paper emphasizes that The new benchmark test 'The Agent Company' is just a first step and encourages future research based on this work.

This benchmark though is very important is addresses a gap in AI evaluation now very much appearing with the next class of AI. It is especially important for the 'future' of knowledge work by focusing on applied tasks that mirror human workflows in most professional office settings. As the pace of LLM advancements accelerates, the ability to measure their utility in actual workplace scenarios becomes essential. The Agent Company benchmark allows researchers and industry leaders to analyze where current AI excels, where it struggles, and how it might disrupt labor markets.

Overview of the Benchmark

Developed collaboratively by researchers from Carnegie Mellon University, Duke University, and independent contributors, the benchmark creates a self-contained environment simulating a small software development company. Tasks span multiple roles, including software engineering, project management, financial analysis, human resources, and administration. The environment includes tools such as GitLab, OwnCloud, Plane, and RocketChat which are open source equivalents of real world applications (Github, Jira, Slack etc) to mimic real-world software systems. Open-source tools are used for reproducibility, and the benchmark supports tasks requiring communication, web browsing, coding, and program execution.

Methodology and Evaluation

The benchmark also employs a task design emphasizing diversity, realism, and complexity. Tasks are broken into subtasks with measurable checkpoints, using deterministic and LLM-based evaluators for assessment. Metrics include success rates, partial completion scores, step counts, and cost per task. This comprehensive framework captures both the nuances of agent performance and the efficiency of task execution.

Experimental Findings

Seven LLM-based agents, including proprietary and open-source models, were evaluated:

Proprietary API-Based Models

1. Claude 3.5 Sonnet (Anthropic)
2. Gemini 2.0 Flash (Google)
3. GPT-4o (OpenAI)
4. Gemini 1.5 Pro (Google)
5. Amazon Nova Pro v1 (Amazon)

Open-Weight Models

1. Llama 3.1 (405B)
2. Llama 3.3 (70B)

• Claude 3.5 Sonnet emerged as the most capable, autonomously completing 24% of tasks with a partial completion score of 34.4%. However, this performance came with high computational costs.
• Other models, like OpenAI’s GPT-4o and Meta’s Llama 3.1, performed moderately but demonstrated efficiency trade-offs.
• Tasks involving complex interfaces (e.g., OwnCloud) and social interactions (e.g., RocketChat) posed significant challenges, highlighting areas for future improvement.
• Notably, newest models GPT o1/o3 were not evaluated at the time of paper publication

These results underscore the limitations of current AI systems in long-horizon, interactive, and multi-step tasks, despite their success in simpler tasks.

Beyond Traditional Benchmarks

Traditional benchmarks have measured LLM capabilities in coding (e.g., SWE-Bench) or reasoning (e.g., Analogical Reasoning Corpus) and general and specific domain intelligence, GPQA, MMLU. However, the AI autonomous agent nature of the tests marks a departure by focusing self-directed autonmous agents within real world online computing mostly office domain, real-world applicability. By integrating collaboration, web interaction, and decision-making under workplace constraints, it sets a new standard for evaluating AI’s readiness for professional integration.

Implications and Future Directions

The study suggests that current AI technologies are far from replacing human labor across complex domains but hold promise for accelerating repetitive tasks. Challenges such as improving social interactions, navigating complex web interfaces, and autonomous task prioritization are highlighted as key areas for future research.

In the coming years, this type of test will likely influence AI development, pushing for agents that not only perform tasks but do so cost-effectively and interactively benchmarking this new group of capabilities and models together. Open-source availability of the benchmark environment ensures that it will serve as a foundation for further innovation and evaluation.

Conclusion

Autonomous agent tests represents a paradigm shift in benchmarking autonomous agents, offering a lens to evaluate their readiness for real-world applications. Key Takeaways of this paper include:

• The current upcoming need for providing further realistic and challenging autonomous agent benchmark that mimic human use for evaluating AI agents both in professional and personal settings.
• The Current state of AI agents have overly achieved the level of performance required for full automation in complex workplace environments or of an average human working in the workplace (an important benchmark for achieving AGI, Artificial General Intelligence).
• Significant progress is still needed with autonomous agents in areas such as human/AI social interaction, complex User Interface navigation and human intervention, and common-sense reasoning integration of benchmarks to develop stronger AI agents.
• The first salvo of this autonomous agent benchmark and its findings provide great insight into where AI is now and how we can further develop AI autonomous agents to be more useful in the workplace and in users personal lives.

By bridging the gap between theoretical AI capabilities and practical workplace integration, this benchmark will shape the trajectory of AI advancements in the years ahead.

#AutonomousAgents, #AIAutonomousAgentBenchmarking, #AIIQTests, #AIAutonomousBusnessTaskTesting, #AIBenchmarking, #AIAutonomousAgentAssessment