Embodied AI agents need to understand their surroundings and update their plans as they gather visual information. A new benchmark called AsgardBench tests whether these agents can adapt their actions based on what they see. It focuses on simple yet challenging tasks where the AI must revise its steps when faced with unexpected environment changes.

What Is AsgardBench and Why Is It Important?

AsgardBench is a testing environment built on AI2-THOR, a 3D simulation platform for household tasks. It presents AI agents with basic actions like find, pick up, put down, clean, and toggle objects on or off. The key idea is to see if the agent can modify its plan after observing the environment, rather than just following a fixed sequence.

This benchmark isolates the agent’s ability to interpret visual feedback and adjust accordingly. For example, if the agent expects a mug to be dirty but finds it clean, it should change its plan. The same applies if the sink is already full or empty. The focus is on real-time decision-making rather than navigation or physical manipulation alone.

How Does AsgardBench Work?

In the simulation, an agent starts near objects and is given a task, such as cleaning a kitchen. It can perform a limited set of actions, and at each turn, it proposes a full plan to complete the task. However, only the first step of this plan is executed before the agent receives new visual feedback. This cycle repeats, allowing the agent to revise its plan based on what it perceives.

For instance, if the agent observes a mug that is already clean, it can skip washing it. If it notices the sink is full, it might need to empty or avoid placing items there. This process tests whether the AI can use visual cues to adapt its behavior rather than blindly following pre-scripted steps. It emphasizes the importance of perception and flexible planning in embodied AI systems.

Overall, AsgardBench challenges AI agents to perform household tasks more like humans—by observing, understanding, and adjusting their actions on the fly. This approach aims to push the development of more intelligent, adaptable embodied AI systems capable of functioning in real-world environments.

Inspired by

• https://www.microsoft.com/en-us/research/blog/asgardbench-a-benchmark-for-visually-grounded-interactive-planning/

Sources

• github.com
• aka.ms