Artificial intelligence benchmarks usually show how well models perform on specific tasks, but they don’t give much insight into what the models can really do or why they succeed or fail. To change that, researchers have developed a new method called ADeLe. This approach looks at both the tasks and the models through a set of core abilities, like reasoning and domain knowledge, and scores them across 18 different skills. This way, it becomes easier to compare models and predict how they will do on new challenges.

Understanding ADeLe and Its Approach

ADeLe, which stands for AI Evaluation with Demand Levels, assigns scores to tasks and models based on how much each requires certain abilities. For example, simple math problems might score low on reasoning skills, while complex proofs will score higher. By evaluating models across many tasks, researchers create detailed profiles that show where each model excels or struggles. These profiles reveal specific strengths and weaknesses, making it possible to see why a model might fail on a new task that demands certain abilities.

This method moves beyond traditional benchmarks that just give an overall score. Instead, it treats both models and tasks as sets of capability scores. This allows for more precise predictions about how a model will perform on unseen tasks, based on its ability profile. The research shows that this approach can predict outcomes with about 88% accuracy, even for recent models like GPT-4o and Llama-3.1, making it a powerful tool for understanding AI progress and limitations.

Building Ability Profiles and Predicting Performance

To build an ability profile, the team evaluates a model on a wide variety of tasks, scoring each task on the 18 core abilities. For example, a task requiring reasoning, attention, or domain knowledge gets rated accordingly. These scores form a detailed map of what the model can do well and where it might struggle. When faced with a new task, the profile helps identify whether the model has the necessary skills to succeed or if it is likely to fail.

This process is illustrated through visual diagrams showing how models and tasks are scored and compared. The ability profiles highlight the specific areas where models perform strongly or need improvement. This insight can guide developers in fine-tuning models or designing new tasks to better match their capabilities. Overall, ADeLe offers a systematic way to understand AI behavior and forecast how models will handle future challenges.

By linking task demands directly to model capabilities, ADeLe provides a clearer picture of what AI models are truly learning. It also helps explain why performance might drop as tasks become more complex, revealing the underlying skills that need to develop further. This approach marks a step forward in making AI evaluation more transparent, predictive, and aligned with real-world applications.

Inspired by

• https://www.microsoft.com/en-us/research/blog/adele-predicting-and-explaining-ai-performance-across-tasks/