Microsoft Research

At a glance

• Problem: Adapting large language models to specialized, high-stakes domains is slow, expensive, and hard to reproduce. 
• What we built: AutoAdapt automates planning, strategy selection (e.g., RAG vs. fine-tuning), and tuning under real deployment constraints. 
• How it works:  A structured configuration graph maps the full scope of the adaptation process, an agentic planner selects and sequences the right steps, and a budget-aware optimization loop (AutoRefine) refines the process within defined constraints.  
• Why it matters: The result is faster, automated, more reliable domain adaptation that turns weeks of manual iteration into repeatable pipelines. 

Deploying large language models (LLMs) in real-world, high-stakes settings is harder than it should be. In high-stakes settings like law, medicine, and cloud incident response, performance and reliability can quickly break down because adapting models to domain-specific requirements is a slow and manual process that is difficult to reproduce.

The core challenge is domain adaptation, which entails turning a general-purpose model into one that consistently follows domain rules, draws on the right knowledge, and meets constraints such as latency, privacy, and cost. Today, that process typically involves guesswork, choosing among approaches like retrieval-augmented generation (RAG) and fine-tuning, tuning hyperparameters, and iterating through evaluations with no clear path to a good outcome. An operations team responding to an outage can’t afford a model that drifts from domain requirements or a tuning process that takes weeks with no guarantee of a reproducible result.

To tackle this, we’re pleased to introduce AutoAdapt. In our paper, “AutoAdapt: An Automated Domain Adaptation Framework for Large Language Models,” we describe an end-to-end, constraint-aware framework for domain adaptation. Given a task objective, available domain data, and practical requirements like accuracy, latency, hardware, and budget, AutoAdapt plans a valid adaptation pipeline, selecting among approaches like RAG and multiple fine-tuning methods, and tunes key hyperparameters using a budget-aware refinement loop. The result is an executable, reproducible workflow for building domain-ready models more quickly and consistently, helping make LLMs dependable in real-world settings.

PODCAST SERIES

The AI Revolution in Medicine, Revisited

Join Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.

Listen now Opens in a new tab How it works

AutoAdapt starts from a practical observation: teams don’t just need a better prompt or more data, they need a decision process that reliably maps a task, its domain data, and real constraints to an approach that works. To do this, AutoAdapt treats domain adaptation as a constrained planning problem. Given an objective provided in natural language, dataset size and format, and limits on latency, hardware, privacy, and cost, it provides an end-to-end pipeline that teams can execute and deploy.

Domain adaptation often feels like trial and error because the design space is large and complex. Teams must choose among approaches such as RAG, supervised fine-tuning, parameter-efficient methods (such as LoRA), and alignment steps, each with many hyperparameters. These choices interact in nonobvious ways, and not all combinations are valid, making it difficult to identify a reliable strategy. The problem is compounded by the high cost of LLM training, which limits how many configurations can be explored.

AutoAdapt addresses this with the Adaptation Configuration Graph (ACG), a structured representation of the system’s configuration space that enables efficient search while guaranteeing valid pipelines.

Building on the ACG, AutoAdapt uses a planning agent to make and justify decisions. It proposes strategies, evaluates them against user requirements, and iterates until the plan is feasible and well-grounded. Rather than optimizing in an unconstrained black box, AutoAdapt roots each decision in best practices and explicit constraints, producing an executable workflow with parameter ranges.

Finally, AutoAdapt introduces AutoRefine, a budget-aware refinement loop that optimizes hyperparameters by strategically selecting which experiments to run next, even under limited feedback. AutoRefine replaces weeks of manual tuning with a more disciplined, reproducible process that is easier to audit and compare across projects. In real-world systems such as healthcare documentation, legal workflows, or incident response, this level of rigor is essential. Figure 1 illustrates the end-to-end workflow.

Figure 1. The AutoAdapt workflow, showing how user inputs flow through planning and refinement to produce a deployable model. Evaluation

In experiments, AutoAdapt consistently identifies effective adaptation strategies and delivers improvements across a range of benchmark and real-world tasks, including reasoning, question answering, coding, classification, and cloud-incident diagnosis. It uses constraint-aware planning and budgeted refinement to find better-performing configurations with minimal added time and cost, making the process practical for production teams. Figures 2 and 3 show aggregate performance against competitive baselines.

Figure 2. Success rate (SR), normalized performance score (NPS), and cumulative score (CS) comparing AutoAdapt with baseline methods across datasets. Higher scores indicate better performance, with AutoAdapt outperforming state-of-the-art baselines. Figure 3. AutoAdapt achieves performance gains with minimal overhead, approximately 30 minutes of additional time and $4 in additional cost. Implications and looking forward

The broader significance of AutoAdapt is that domain adaptation can become an engineering discipline, not an ad hoc process. By making key choices explicit—what to adapt, how to adapt it, and which constraints the system must satisfy—AutoAdapt helps teams reach results faster, reproduce them more easily, and audit them more rigorously. This shift is especially important in domains where drift from pretrained knowledge is common and failures are costly. When LLMs are used to draft clinical notes, triage support incidents, or summarize regulatory language, organizations need a clear, repeatable path from data to models that behave predictably under latency, privacy, and budget requirements.

Because domain adaptation is a prerequisite for deploying LLMs in real-world settings, we’re making the AutoAdapt framework open source (opens in new tab) to give teams a concrete starting point. The README (opens in new tab) file provides installation and quick-start instructions.

Video playback requires cookie consent Opens in a new tab

The post AutoAdapt: Automated domain adaptation for large language models appeared first on Microsoft Research.

At a glance

• AI is driving rapid changes in the workplace, more sharply than those covered in previous editions of the New Future of Work
• AI is changing how people work together, not just enabling them to work faster or from remote locations. Organizations that treat AI as a collaborative partner are seeing the biggest benefits.
• The benefits of AI are not yet evenly distributed, underscoring the need for industry leaders to build AI that expands opportunity. The future is not predetermined. It will be shaped by the choices we make today.
• Human expertise matters more, not less, in an AI-powered world. People are shifting from merely doing work to guiding, critiquing, and improving the work of AI.

For the past five years, the New Future of Work report has captured how work is changing. This year, the shift feels especially sharp. Previous editions have focused on technology’s role in increasing productivity by automating tasks, accelerating communication, and expanding access to information, as well as the rise of remote work. Today, generative AI has put this transformation on fast forward. Instead of simply speeding up existing workflows, AI increasingly participates in them, shaping how people create, decide, collaborate, and learn.

For decades, researchers across Microsoft have studied these changes not as abstract trends but as lived experiences. Across organizations and occupations, people are experimenting with AI in uneven, creative, and sometimes surprising ways. Many are saving time, expanding their capabilities, and taking on more complex work, but the real opportunity ahead is to use AI to help us work better, together.

Publication New Future of Work Report 2025 

The New Future of Work report brings together research from inside and outside of Microsoft to understand what is happening as AI enters workplaces. Through the efforts of dozens of authors and editors, it draws on evidence from large‑scale data analyses, field and lab studies, and theory to look at who is using AI, why they are using it, and how it is reshaping productivity, collaboration, learning, and judgment. It highlights professions where changes are unfolding especially quickly, as well as the broader societal impact of these technologies.

Taken together, these findings point to a central insight: The future of work is not something that will simply happen to us. We are actively constructing it, through the choices individuals make, the norms teams build, the systems organizations adopt, and the discoveries researchers uncover. At the same time, AI’s role is still evolving, and it is driving a range of impact—some of which may be viewed as positive or negative. What follows is a research-backed snapshot of this moment in time and what it can teach us about how to collectively create a new and better future of work with AI.

Adoption and usage

Generative AI is entering workplaces quickly, likely faster than most earlier technologies. But the patterns of who uses it, and how, will shape who benefits. Reports on early adoption appear to show significant penetration: in one German survey, 38% of employed respondents reported using AI at work. But usage and confidence vary widely across sectors, and men report using AI at work more often than women. It’s not yet clear whether that variability is driven by occupational distributions, relative comfort with new tools, or something else. This raises the challenge that uneven adoption is likely to translate into uneven productivity gains, learning opportunities, downstream career paths and more between those who adopt and those who do not.

A look at generative AI adoption globally reveals further differences. High-income countries still lead overall usage, but the fastest growth is happening in low- and middle-income regions. When local languages are poorly served, people switch to English simply to get reliable results. Without investment in infrastructure and multilingual model development, AI risks reinforcing existing divides rather than narrowing them.

Inside organizations, the decision to use or not use AI is shaped less by strategy decks and more by culture. People try new tools when they trust their employer and feel safe experimenting. They stick with tools that make their work better, but might reject tools that seem designed to replace them—which is a common concern among workers. And many of the most useful applications don’t come from top-down initiatives at all but from employees trying things, discovering what actually helps, and sharing those insights with colleagues. Research has shown that involving workers’ perspectives in the design of workplace technologies promotes sustainable improvements in productivity and well-being.

We are also starting to see what people actually do with AI. At Anthropic, an analysis of millions of user conversations found that 37% of Claude usage was tied to software and mathematical occupations. A study of Microsoft Copilot conversations found high applicability to the activities of information workers across sales, media, tech, and administrative roles. But the broader point is simpler: most occupations include at least some tasks where AI is useful.

These shifts come with social side effects. Several studies show that employees who use AI can be perceived as less capable, even when their output is identical to that of people who didn’t use AI. Whether these perception penalties fall unevenly across groups is still an open question. However, managers who have used AI tend to evaluate AI-assisted work more fairly. This suggests that AI may require broad exposure before it can be used openly and without judgment.

PODCAST SERIES

The AI Revolution in Medicine, Revisited

Join Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.

Listen now Opens in a new tab Impact on work and labor markets

Understanding who uses AI and why they use it can help assess its value, but the harder question is how it impacts productivity and labor markets, which can be less straightforward. Productivity can increase through time saved, higher-quality work, or simply feeling more capable. Surveyed enterprise users of AI report saving 40–60 minutes a day, while model-based evaluations show frontier systems can approach quality levels like that of experts on a growing range of tasks. But AI may also reduce productivity. In one U.S. survey, 40% of employees said they had received “workslop”, i.e. AI-generated content that looks polished but isn’t accurate or useful, in the past month. When that happens, any time savings can quickly disappear, and quality can actually suffer.

We still don’t have the full picture of what this means for jobs and labor markets more broadly. Large-scale empirical work finds no clear aggregate effects on unemployment, hours worked, or job openings. However, AI does seem to be reducing opportunities for younger, inexperienced workers. Entry-level roles rely less on experience and knowledge and are easier to automate. Empirical evidence suggests employment for workers aged 22–25 in highly AI-exposed jobs declined by 16% relative to similar but less-exposed roles, and hiring into junior positions appears to slow after firms adopt AI. This pattern raises a longer-term concern: automating jobs that enable workers to learn skills may undermine how expertise is built over time. This point is reinforced by research using theoretical models as well as empirical evidence.

Meanwhile, AI is also changing which skills matter. Roles that mention AI skills in their job postings are nearly twice as likely to also emphasize analytical thinking, resilience, and digital literacy. Demand for work that can be outsourced to AI models more easily, including data-related tasks or routine translation, continues to fall. Even where overall employment remains stable, AI is already reshaping how jobs are structured and this trend will continue.

As more empirical evidence comes in, theoretical work helps frame what might lie ahead. One recurring theme is that human judgment – spotting opportunities, working under ambiguity or choosing from outputs – becomes more valuable as AI improves. And organizations that use AI to augment what people can do often end up creating new kinds of work, rather than simply eliminating existing ones. If AI is meant to deliver on its potential to support broad prosperity gains, the path forward is less about replacing tasks and more about expanding what people are able to do.

Human-AI collaboration

As AI becomes more capable, the nature of human-AI interaction is changing. AI systems are increasingly playing a role in decision-making, creativity, and communication, with AI systems being positioned as a “collaborator.” This raises questions about how to support “collaboration” between people and AI, what we can learn from how people interact with each other, and where the capabilities of AI systems raise different opportunities and create different requirements.

At the heart of effective collaboration is common ground: the shared understanding that allows people to coordinate and communicate. In human conversation, we constantly check for alignment – through clarifications, acknowledgements, and follow-up questions. Yet current AI systems often skip these steps, generating responses that assume understanding rather than building it. Research shows that this lack of conversational grounding can lead to breakdowns in human-AI interaction. Encouragingly, systems like CollabLLM (opens in new tab), which prompt AI to ask clarifying questions and respond over multiple turns, have shown improved task performance and more interactive exchanges.

Trust is another essential aspect of collaboration. Although AI can process vast amounts of information, its usefulness in decision-making depends on how well it grasps human goals, and how well people understand its capabilities. Using AI that doesn’t understand a person’s objectives can lead to worse outcomes than using no AI at all. Yet people often overestimate AI’s abilities, which distort their judgment on when and how to use it. Systems that support selective delegation can improve these decisions, especially when the AI is programmed to account for this selective approach in its responses.

AI’s advancing capabilities are fueling a shift in people’s roles. This includes software production, where developers who once wrote code from beginning to end are increasingly reviewing and refining AI-generated suggestions. Writers and designers are acting more as curators and editors, guiding AI outputs rather than producing everything from scratch. This shift demands new skills – like crafting effective prompts, vetting AI responses, and maintaining quality oversight – and new tools to support them.

Current chat-based interfaces are often too limited for these evolving workflows. Alongside knowledge about the capabilities, limitations, and workings of an AI system, as well as domain expertise and situational awareness to enable intervention, oversight requires observability of system activity, decisions, and outputs. New interface designs are emerging to address this, including visualizations of AI reasoning, shared editing spaces, and mixed-initiative systems that allow humans and AI to take turns leading a task. These innovations aim to preserve human agency while making AI more transparent and responsive.

Ultimately, the future of work is about building complementary interactions between people, drawing on knowledge of how people collaborate, while acknowledging the unique challenges of human-AI interaction, and drawing on AI capabilities to do so.

AI for teamwork

AI systems have been designed from the ground up to work best for individuals, not for teams of people. It is no surprise then, that when people use AI as a team, they often underperform, even relative to an individual using AI.

The good news is that a growing amount of research is dedicated to AI that supports team and group interaction. Researchers are using two broad approaches: (1) process-focused strategies, i.e. building AI to facilitate specific team processes like information sharing and (2) outcome-focused strategies, i.e. training end-to-end AI systems that attempt to learn from short- and long-range team outcomes.

Some examples of the former include systems that provide a devil’s advocate perspective in a group discussion or help amplify minority perspectives. Examples of the latter include systems that try to help teams make good decisions or drive meetings towards achieving goals.

Theory from fields like collective intelligence would suggest that both approaches have great potential: AI can unlock new models of collaboration that are wildly different and more productive than we’ve had before. One notable example is AI enabling much more ephemeral teams, where a precise group of people in a given organization (or even beyond) can come together to solve a specific problem, then disband when the problem is solved.

More philosophically, it can be useful to understand even individual interaction with a large language model (LLM) as a type of teamwork. In fact, “collective intelligence” is perhaps a more accurate term for technologies like LLMs than “artificial intelligence”. LLMs take knowledge from millions of people who have written web content or posted in places like Reddit and Wikipedia, interacted with chatbots, and generated other types of data, and make that available to individuals on demand. Every time you interact with an LLM, you’re interacting with the work of millions of people, without the impossible overhead of that scale of collaboration.

Thinking, learning and psychological influences

Generative AI is changing cognition and learning while also introducing new psychological dynamics. This is making design choices about agency, effort, and well-being increasingly consequential. 

A central pattern emerging in generative AI is a shift from ‘thinking by doing’ (e.g. writing a document) toward ‘choosing from outputs’ (e.g. prompting AI to write a document). This may weaken the judgment and practices that sustain human expertise unless it is paired with user experiences that keep people cognitively engaged, and upskilling/reskilling to accommodate changes in available work. AI can also be designed to support thinking rather than substitute for it, for example by provoking reflection, scaffolding reasoning, and workflows that help people ‘decide how to decide’ through alternatives and critiques. For ideation and creativity, benefits can be fragile. Using LLMs at the wrong time can reduce originality and self-efficacy, and repeated cognitive offloading can carry over even when AI is removed. To avoid trading short-term accuracy for long-term capability, AI experiences should help users practice the judgment needed to challenge and refine AI outputs.

AI use in education is already widespread, but much of this activity runs through general-purpose tools rather than education-specific products, while training and policy are still catching up. In learning contexts, the speed and ease with which AI is being designed to meet workplace tasks may conflict with the needs of education. Learning often benefits from ‘desirable difficulties,’ and heavy reliance on summaries and syntheses may make learning shallower without thoughtful support. This may involve trying problems before turning to AI for help, and question-driven tutoring that requires students to justify and check outputs. Coding education remains essential, but needs to change focus from memorizing syntax to centering abstraction and accountability, such as problem framing and critical review. Workplace training can counter overreliance and ‘work-slop’ productivity problems by helping workers reframe AI as a thought partner, prompting reflective interaction and strengthening calibration and verification habits so workers retain responsibility for final decisions. 

Finally, conversational AI is increasingly being used for social and emotional support, making empathy and psychological well-being core design and governance concerns, especially because effects can vary sharply by user context and interaction patterns. That variability also raises the stakes for anthropomorphic behaviors. Clearer definitions and measurement are needed to understand when systems appear human-like and what consequences follow. Broader mapping of the design space can help designers anticipate implications and choose alternatives.

Specific roles & industries

While much of the NFW report highlights broad work patterns such as collaboration, communication, and decision-making, we also examined specific professions that are seeing especially rapid disruption. Among those that stand out in this year’s edition are software engineering and science. To counter some of the misunderstandings around these fields, we address several myths, including:

• Counting AI-generated lines of code is a meaningful productivity metric
• Current tools will instantly turn every developer into a “10× engineer”

Adoption primarily depends on model capability. Beyond myth-busting, we see real shifts in the software lifecycle. Historically, PMs (product/program/project managers) focused on customer needs, telemetry, design, and feedback, while developers wrote the code. With generative AI, these boundaries are blurring. PMs report doing more technical work and writing more code, while developers increasingly engage in higher-level planning and conceptual thinking as they interact with AI agents.

This shift is illustrated by the rise of vibe coding—developing software through iterative prompting rather than directly writing and editing code. Studies show that experienced computer science students are better at vibe coding than novices, able to steer models with a smaller number of targeted prompts. As humans build trust with AI assistants, work becomes more co-creative, enabling engineers to stay “in flow” through continuous iteration.

Together, these changes point to a deeper transformation in how software is built—both the mechanics of code production and the ways teams coordinate, plan, and collaborate.

Science is also seeing significant AI-driven acceleration. AI is meaningfully accelerating scientific discoveries by assisting researchers in identifying promising ideas, retracing known results, and surfacing cross-field connections. Foundation models also make it easier to work with diverse data types and enable experiments at a previously impossible scale.

Benefits of increased research productivity and moderate quality gains appear to be most pronounced for early career researchers and non-English speaking scientists, for whom AI can act as both a collaborator and a form of access to advanced tooling.

However, AI introduces new risks. Issues of data provenance, accountability, and replication become more complex when generative systems are involved. Small variations in prompts can significantly change outcomes, making results harder to verify. Models may reproduce ideas without attribution or hallucinate entirely, increasing the burden of source-checking. And because many models tend toward sycophantic responses, scientists may overestimate the novelty or correctness of AI-generated insights.

Closing

Generative AI will not arrive in some distant future, it is reshaping work right now. Here are a few things to take away:

1. AI isn’t just speeding up work—it’s changing how we work together.
   This year’s research shows a real shift: AI is moving from automating tasks to actively shaping how people create, decide, collaborate, and learn. The organizations seeing the biggest gains are the ones treating AI as a collaborative partner—not a bolt‑on tool—and building the culture, norms, and confidence to experiment.
2. The benefits of AI are real, but they’re not evenly distributed—yet.
   Adoption is rising fast across countries, professions, and industries, but the gaps in access, confidence, and usage are widening. Early evidence shows that who uses AI (and how) will determine who benefits. Industry leaders need to ensure AI expands opportunity rather than reinforces divides.
3. Human expertise matters more—not less—in an AI‑powered world.
   Across software engineering, science, and knowledge work, AI is transforming roles: people are shifting from doing the work to guiding, critiquing, and improving it. The organizations that thrive will be the ones that invest in judgment, critical thinking, and responsible oversight—and design AI experiences that keep people thoughtfully engaged.

The research in this year’s New Future of Work report points to both opportunity and responsibility. The future is not predetermined. It will be shaped by the choices we make today—in how we build AI systems, how organizations adopt them, and how individuals learn to work alongside them. Microsoft remains committed to studying these changes as they unfold, grounding our understanding in evidence, and ensuring that the future we are collectively building is one where AI helps us all work better, together.

Opens in a new tab

The post New Future of Work: AI is driving rapid change, uneven benefits appeared first on Microsoft Research.

At a glance

• AI benchmarks report performance on specific tasks but provide limited insight into underlying capabilities; ADeLe evaluates models by scoring both tasks and models across 18 core abilities, enabling direct comparison between task demands and model capabilities.
• Using these ability scores, the method predicts performance on new tasks with ~88% accuracy, including for models such as GPT-4o and Llama-3.1.
• It builds ability profiles and identifies where models are likely to succeed or fail, highlighting strengths and limitations across tasks.
• By linking outcomes to task demands, ADeLe explains differences in performance, showing how it changes as task complexity increases.

AI benchmarks report how large language models (LLMs) perform on specific tasks but provide little insight into their underlying capabilities that drive their performance. They do not explain failures or reliably predict outcomes on new tasks. To address this, Microsoft researchers in collaboration with Princeton University and Universitat Politècnica de València introduce ADeLe (opens in new tab) (AI Evaluation with Demand Levels), a method that characterizes both models and tasks using a broad set of capabilities, such as reasoning and domain knowledge, so performance on new tasks can be predicted and linked to specific strengths and weaknesses in a model.

In a paper published in Nature, “General Scales Unlock AI Evaluation with Explanatory and Predictive Power (opens in new tab),” the team describes how ADeLe moves beyond aggregate benchmark scores. Rather than treating evaluation as a collection of isolated tests, it represents both benchmarks and LLMs using the same set of capability scores. These scores can then be used to estimate how a model will perform on tasks it has not encountered before. The research was supported by Microsoft’s Accelerating Foundation Models Research (AFMR) grant program.

ADeLe-based evaluation

ADeLe scores tasks across 18 core abilities, such as attention, reasoning, domain knowledge, and assigns each task a value from 0 to 5 based on how much it requires each ability. For example, a basic arithmetic problem might score low on quantitative reasoning, but an Olympiad-level proof would score much higher.

Evaluating a model across many such tasks produces an ability profile—a structured view of where the model performs and where it breaks down. Comparing this profile to the demands of a new task makes it possible to identify the specific gaps that lead to failure. The process is illustrated in Figure 1.

Figure 1. Top: (1) Model performance on the ADeLe benchmark and (2) the resulting ability profiles, showing each model’s strengths and limitations across core abilities. Bottom: (1) Application of 18 scoring criteria to each task and (2) the resulting task profiles, showing the abilities each task requires. Evaluating ADeLe

Using ADeLe, the team evaluated a range of AI benchmarks and model behaviors to understand what current evaluations capture and what they miss. The results show that many widely used benchmarks provide an incomplete and sometimes misleading picture of model capabilities and that a more structured approach can clarify those gaps and help predict how models will behave in new settings.

ADeLe shows that many benchmarks do not isolate the abilities they are intended to measure or only cover a limited range of difficulty levels. For example, a test designed to evaluate logical reasoning may also depend heavily on specialized knowledge or metacognition. Others focus on a narrow range of difficulty, omitting both simpler and more complex cases. By scoring tasks based on the abilities they require, ADeLe makes these mismatches visible and provides a way to diagnose existing benchmarks and design better ones.

Applying this framework to 15 LLMs, the team constructed ability profiles using 0–5 scores for each of 18 abilities. For each ability, the team measured how performance changes with task difficulty and used the difficulty level at which the model has a 50% chance of success as its ability score. Figure 2 illustrates these results as radial plots that show where the model performs well and where it breaks down.

Figure 2. Ability profiles for 15 LLMs across 18 abilities. Left: OpenAI models. Middle: Llama models. Right: DeepSeek-R1 distilled models.

This analysis shows that models differ in their strengths and weaknesses across abilities. Newer models generally outperform older ones, but not consistently across all abilities. Performance on knowledge-heavy tasks depends strongly on model size and training, while reasoning-oriented models show clear gains on tasks requiring logic, learning, abstraction, and social inference. These patterns typically require multiple, separate analyses across different benchmarks and can still produce conflicting conclusions when task demands are not carefully controlled. ADeLe surfaces them within a single framework.

ADeLe also enables prediction. By comparing a model’s ability profile to the demands of a task, it can forecast whether the model will succeed, even on tasks that are unfamiliar. In experiments, this approach achieved approximately 88% accuracy for models like GPT-4o and LLaMA-3.1-405B, outperforming traditional methods. This makes it possible to both explain and anticipate potential failures before deployment, improving the reliability and predictability of AI model assessment.

Whether AI systems can truly reason is a central debate in the field. Some studies report strong reasoning performance, while others show they break down at scale. These results reflect differences in task difficulty. ADeLe shows that benchmarks labeled as measuring “reasoning” vary in what they require, from basic problem-solving to tasks that combine the need for advanced logic, abstraction, and domain knowledge. The same model can score above 90% on lower-demand tests and below 15% on more demanding ones, reflecting differences in task requirements rather than a change in capability.

Reasoning-oriented models like OpenAI’s o1 and GPT-5 show measurable gains over standard models—not only in logic and mathematics but also with interpreting user intent. However, performance declines as task demands increase. AI systems can reason, but only up to a point, and ADeLe identifies where that point is for each model.

Spotlight: Event Series

Microsoft Research Forum

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.

Watch on-demand Opens in a new tab Looking ahead

ADeLe is designed to evolve alongside advances in AI and can be extended to multimodal and embodied AI systems. It also has the potential to serve as a standardized framework for AI research, policymaking, and security auditing.

More broadly, it advances a more systematic approach to AI evaluation—one that explains system behavior and predicts performance. This work builds on earlier efforts, including Microsoft research on applying psychometrics to AI evaluation and recent work on Societal AI, emphasizing the importance of AI evaluation.

As general-purpose AI systems continue to outpace existing evaluation methods, approaches like ADeLe offer a path toward more rigorous and transparent assessment in real-world use. The research team is working to expand this effort through a broader community. Additional experiments, benchmark annotations, and resources are available on GitHub (opens in new tab).

Opens in a new tab

The post ADeLe: Predicting and explaining AI performance across tasks appeared first on Microsoft Research.

At a glance

• To successfully complete tasks, embodied AI agents must ground and update their plans based on visual feedback.
• AsgardBench isolates whether agents can use visual observations to revise their plans as tasks unfold.
• Spanning 108 controlled task instances across 12 task types, the benchmark requires agents to adapt their plans based on what they observe.
• Because objects can be in different positions and states (e.g., clean or dirty), the same instruction can require different action sequences, even in the same environment.

Imagine a robot tasked with cleaning a kitchen. It needs to observe its environment, decide what to do, and adjust when things don’t go as expected, for example, when the mug it was tasked to wash is already clean, or the sink is full of other items. This is the domain of embodied AI: systems that perceive their environment and act within it.

The field has made rapid progress, but evaluating these systems is harder than it looks. Many benchmarks test perception, navigation, and physical control all at once, making it difficult to isolate whether an AI agent is actually using what it perceives to make better decisions or just getting lucky because the environment is predictable enough to script around.

To address this, we created AsgardBench. In the paper, AsgardBench — Evaluating Visually Grounded Interactive Planning Under Minimal Feedback,” we describe how this benchmark poses a simple but demanding challenge: give an AI agent a household task, let it observe the environment through images, and see whether it can adjust its plan when what it perceives contradicts what it anticipated. Can it notice that the mug it needs to clean is already in the sink, or that it isn’t, and behave accordingly? That is the core question AsgardBench is designed to answer.

Built on AI2-THOR, an interactive 3D simulation environment used to train and evaluate AI agents on household tasks, AsgardBench positions agents near objects and gives them a small, fixed set of actions, such as find, pickup, put, clean, and toggle_on/off. At each turn, the agent proposes a full sequence of steps to complete the task, but only the first step executes. Throughout, the focus is squarely on plan adaptation, not whether an agent can navigate a room or manipulate an object, but whether it can use what it perceives to revise its next step.

For example, the agent may discover a mug to be clean, dirty, or filled with coffee, or it may observe that a sink contains many other items, so the same instruction can require different action sequences as the task unfolds. This process is illustrated in Figure 1.

Figure 1: Agent observations and corresponding action plans in AsgardBench. Each image is paired with the plan generated from that observation. This illustrates how AsgardBench requires agents to update or change their plans based on new visual evidence rather than following a fixed sequence. How it works

Agents start in interaction-ready positions, so navigation and viewpoint selection are not factors. A find action brings objects into view, and the environment handles the details of container sizing and placement, so the agent does not need to reason about which cabinet or countertop to use. The only inputs are color images, a history of attempted actions with simple success or failure signals, and the agent’s own record of what it plans to do next.

At each turn, the agent proposes a complete sequence of steps to finish the task, but only the first step proceeds. It then receives new images and a simple signal—did that action succeed or fail? This prevents the agent from scripting everything upfront and forces it to re-evaluate and revise its plan at every step. Built-in limits on total steps and repeated actions prevent endless loops. Because the environment provides only simple feedback, the agent must be able to notice what it perceives (e.g., whether a mug is dirty, whether a faucet is running) and keep track of where it is in the task from one step to the next.

Evaluating AsgardBench

We tested several leading vision-capable models on AsgardBench and observed that high-performing models require visual grounding to consistently succeed. Across the models, visual input substantially improved performance: most models more than doubled success rates when given images versus text-only descriptions of the scene. This is in contrast to some prior benchmarks where agents could perform reasonably well without vision by relying on textual feedback on what went wrong.

Providing that kind of detailed failure information raises performance for all models in AsgardBench, too, but it can mask the real problem. The strongest vision-capable models still outperform text-only agents even when those agents are given detailed feedback, demonstrating that the benchmark requires visual grounding that text alone cannot replicate. AsgardBench’s performance is illustrated in Figure 2.

Figure 2. Success rates for image-based and text-only conditions. Visual input substantially improves performance for all but the weakest agents, while text-only performance remains low, indicating that AsgardBench requires perception-based reasoning.

The results also revealed where today’s agents consistently fall short. Across all models, the same problems kept appearing: agents attempted undoable actions (e.g., trying to clean a mug that was not in the sink), got stuck in repeated action loops, misinterpreted subtle visual cues (on/off, clean/dirty), and lost track of where they were in the task progress from one step to the next. This points to three weaknesses: the inability to distinguish subtle visual details in cluttered scenes, the inability to maintain an accurate picture of task progress across multiple steps, and the inability to consistently translate what the agent sees into timely updates to its plan. Taken together, these point to where the next generation of embodied agents will need to improve.

Spotlight: Event Series

Microsoft Research Forum

Join us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.

Watch on-demand Opens in a new tab Implications and looking ahead

AsgardBench is useful as both a diagnostic and development tool. By varying what feedback agents receive (none, minimal, or detailed), researchers can isolate whether performance gains come from better perception, better memory, or better planning. Promising directions include systems that combine stronger visual understanding with better state tracking, training approaches that emphasize learning to repair plans mid-task, and evaluation methods that measure not just whether an agent succeeds but how well it adapted along the way.

The failure patterns AsgardBench surfaces point toward a concrete next step: building systems that can make finer visual distinctions, keep track of what changed more reliably across steps, and learn to revise plans mid-task rather than plowing ahead on a script. Agents that make progress on these challenges should be meaningfully better equipped for the messiness of real-world environments: unexpected object states, cluttered scenes, and the constant need to adapt.

AsgardBench is open source and available on GitHub (opens in new tab), providing a foundation for advancing research in visually grounded planning.

Acknowledgements

We thank the AI2-THOR community for building the simulation platform and making reproducible embodied evaluation possible.

Opens in a new tab

The post AsgardBench: A benchmark for visually grounded interactive planning appeared first on Microsoft Research.

At a glance

• VLM-based robot planners struggle with long, complex tasks because natural-language plans can be ambiguous, especially when specifying both actions and locations.
• GroundedPlanBench evaluates whether models can plan actions and determine where they should occur across diverse, real-world robot scenarios.
• Video-to-Spatially Grounded Planning (V2GP) is a framework that converts robot demonstration videos into spatially grounded training data, enabling models to learn planning and grounding jointly.
• Grounded planning improves both task success and action accuracy, outperforming decoupled approaches in benchmark and real-world evaluations.

Vision-language models (VLMs) use images and text to plan robot actions, but they still struggle to decide what actions to take and where to take them. Most systems split these decisions into two steps: a VLM generates a plan in natural language, and a separate model translates it into executable actions. This approach often breaks down for long, complex tasks because natural-language plans can be ambiguous or even hallucinated when specifying actions and locations (Figure 1). Because planning and spatial reasoning are handled separately, errors in one stage can propagate to the next. This raises a key question: can a VLM determine both what to do and where to do it simultaneously?

Figure 1. Failures in VLM-based task planners, where ambiguous language leads to non-executable actions. Planning with spatial grounding

To address this problem, we developed GroundedPlanBench (opens in new tab). In our paper, “Spatially Grounded Long-Horizon Task Planning in the Wild,” we describe how this new benchmark evaluates whether VLMs can plan actions and determine where those actions should occur across diverse real-world environments. We also built Video-to-Spatially Grounded Planning (V2GP), a framework that converts robot demonstration videos into training data to help VLMs learn this capability.

Evaluating these with both open- and closed-source VLMs, we found that grounded planning for long, complex tasks is challenging. At the same time, V2GP improves both planning and grounding, with gains validated on our benchmark and in real-world experiments using robots.

How GroundedPlanBench works

To create realistic robot scenarios, we built our benchmark from 308 robot manipulation scenes in the Distributed Robot Interaction Dataset (DROID) (opens in new tab), a large collection of recordings of robots performing tasks. We worked with experts to review each scene and define tasks that a robot could perform. Each task was written in two styles: explicit instructions that clearly describe the actions (e.g., “put a spoon on the white plate”) and implicit instructions that describe the goal more generally (e.g., “tidy up the table”).

For each task, the plan was broken down into four basic actions—grasp, place, open, and close—each tied to a specific location in the image. Grasp, open, and close actions were linked to a box drawn around the target object, while place actions were linked to a box showing where the object should be placed.

Figure 2 illustrates medium- and long-duration tasks, along with their explicit and implicit instructions. In total, GroundedPlanBench contains 1,009 tasks, ranging from 1–4 actions (345 tasks) to 5–8 (381) and 9–26 (283).

Figure 2. Examples of tasks in GroundedPlanBench. How V2GP works

The V2GP framework first detects moments when the robot interacts with objects using the recorded gripper signals. It then generates a text description of the manipulated object with a multimodal language model. Guided by this description, the system tracks the object across the video using Meta’s advanced open-vocabulary image and video segmentation model, SAM3. The system then constructs grounded plans from the tracking results, identifying the object’s location at the moment it is grasped and where it is placed.

This process is illustrated in Figure 3. It yielded 43K grounded plans with varying lengths: 34,646 plans with 1–4 actions, 4,368 with 5–8 actions, and 4,448 with 9–26 actions.

Figure 3. The V2GP framework converts robot videos into spatially grounded plans. Evaluating decoupled versus grounded planning

To evaluate GroundedPlanBench in real-world robotic settings, we used Qwen3-VL (opens in new tab) as our base model. Qwen3-VL is a vision-language model that processes text, images, and video to support multimodal reasoning. It performs well on standard multimodal reasoning benchmarks without additional training. We first evaluated it, along with other proprietary models, on GroundedPlanBench without any task-specific training (Table 1). We then fine-tuned it on V2GP training data and compared it with a decoupled approach, in which planning and grounding are handled separately.

In this setup, a VLM first generated a plan describing what the robot should do. We used GPT-5.2 or Qwen3-VL-4B for this step. The plan was then passed to a spatial grounding model, Embodied-R1 (opens in new tab), which converted the plans into executable signals. Embodied-R1 is a large vision-language model trained for embodied reasoning and pointing, where the model identifies specific locations in the image to guide the robot’s actions. We selected it for spatial grounding because its training targets embodied spatial reasoning and point-based localization, making it well suited for grounding model outputs to specific locations in an image.

Figure 4 highlights a key limitation of this approach: ambiguity in natural language. For example, Qwen3-VL-4B generated grasp actions by referring to “napkin on the table” for all four napkins in the scene, leading Embodied-R1 to ground each action the same napkin. GPT-5.2 produced more descriptive phrases, such as “top-left napkin” or “upper-center napkin,” but these were still too imprecise for the model to reliably distinguish between them and were again grounded to the same object.

Figure 4. Decoupled vs. grounded planning, illustrating how ambiguous language causes actions to be grounded to the wrong objects.

This limitation becomes more pronounced in real-world robot manipulation, where environments are often cluttered and complex. As a result, decoupled approaches struggle to work reliably. In contrast, our approach, grounded planning, performs planning and grounding jointly within a single model and improves both planning and grounding performance.

Table 1 presents evaluation results for open- and closed-source VLMs on GroundedPlanBench. Multi-step planning and handling of implicit instructions were challenging for all models, while training Qwen3-VL-4B and Qwen3-VL-32B with V2GP led to significant improvements in grounded planning.

Table 1. Evaluation results on GroundedPlanBench. Task Success Rate (TSR) measures the percentage of tasks completed correctly, requiring all actions to be both correctly planned and spatially grounded. Action Recall Rate (ARR) measures the proportion of generated actions that match the sub-actions defined in the dataset, regardless of order. The V2GP approach improves performance on both metrics and achieves the best results (shown in bold).

PODCAST SERIES

AI Testing and Evaluation: Learnings from Science and Industry

Discover how Microsoft is learning from other domains to advance evaluation and testing as a pillar of AI governance.

Listen now Opens in a new tab Implications and looking forward

Integrating planning and grounding within a single model offers a path to more reliable robot manipulation in real-world settings. Rather than relying on separate stages, this approach keeps decisions about what to do and where to act tightly coupled, but models still struggle with longer, multi-step tasks and implicit instructions. Models must reason over longer sequences of actions and maintain consistency across many steps and goals described indirectly, as in everyday language.

Looking ahead, a promising direction combines grounded planning with world models, which enable robots to predict the outcomes of actions before executing them. Together, these capabilities could allow robots to decide what to do, where to act, and what will happen next, bringing us closer to systems that can plan and act reliably in the real world.

Acknowledgements

This research was conducted in collaboration with Korea University, Microsoft Research, University of Wisconsin-Madison, and supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant (No. RS-2025-25439490) funded by the Korea government (MSIT).

Opens in a new tab

The post GroundedPlanBench: Spatially grounded long-horizon task planning for robot manipulation appeared first on Microsoft Research.