Microsoft
TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient
- Developers use AI coding agents in the same codebases day after day, yet agents often start each session from scratch: rediscovering familiar facts, loading unused tools, and repeating unsuccessful approaches.
- TokenEconomizer is an open-source plugin that learns from an agent’s past sessions in a repository to reduce waste in future sessions, without modifying the agent or its underlying model.
- Across 277 real bug-fixing tasks, TokenEconomizer reduced costs by 18.4–30.9% across multiple AI coding agents. Costs fell in every repository tested, while task success rates remained broadly stable.
AI coding agents are increasingly helping developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same repositories, giving agents an opportunity to learn from previous work. Without that learning, an agent may repeatedly rediscover how to run tests, load tools it never uses, or spend dozens of turns revisiting the same error.
In our paper, TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents, we ask a simple question: can experience in a repository help an agent solve future tasks more efficiently? TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research, and Microsoft Security, turns completed sessions into guidance for subsequent ones. It reduces waste without changing the agent or its underlying model, so developers can use it alongside the coding agents they already rely on.
Figure 1. Each finished session teaches TokenEconomizer something about the repository, so later sessions cost less tokens. Where the tokens goWe first profiled Claude Code on 60 tasks from SWE-Bench, a benchmark of real bug fixes in open-source Python projects, running each repository’s tasks in sequence. Three kinds of waste recurred:
- Rediscovering the same facts. After a repository had accumulated at least five completed tasks, 30% of tool calls repeated an action from an earlier task in that repository. Only 7% matched an action in another repository’s history.
- Paying for unused tools. The agent received 23 tool descriptions on every turn, but 16 were never called.
- Getting stuck on failures. The 17% of tasks that failed accounted for 32% of the spending.
We also tested three popular open-source plugins, each built around a fixed cost-saving strategy. Each lowered costs in some repositories but not others. Together, these findings suggest that effective cost control must adapt to the repository, rather than apply the same strategy everywhere.
How TokenEconomizer worksAfter each session, TokenEconomizer reads the session’s log and updates what it knows about the repository. It then acts through three mechanisms, using only interfaces the agent already provides, as illustrated in Figure 2.
Micro-skills capture repository-specific knowledge in one-line notes. TokenEconomizer identifies recurring patterns across sessions and proposes corresponding notes. They are added to the agent’s starting context only after several sessions have confirmed the pattern, and retires the note if later evidence contradicts it.
Tool pruning removes tools that repository history suggests the agent is unlikely to use, while always retaining core tools for running commands and editing files. Removed tools can be restored if needed.
Failure-loop intervention watches for an agent that keeps hitting the same failure. A judge model decides whether the agent is truly stuck or still making progress. If it is stuck, the agent gets a short note suggesting a different approach. In the open-source plugin, if the agent is still stuck after the note, TokenEconomizer alerts the developer rather than stopping the work itself.
Figure 2. TokenEconomizer works alongside an unchanged coding agent. Micro-skills add learned facts, tool pruning trims the loaded tools, and failure-loop intervention redirects stuck work. Each finished session updates all three.The controller is designed to revise its decisions as new evidence arrives. Reported savings also account for the cost of TokenEconomizer’s own model calls.
Saving costs in every repositoryWe evaluated TokenEconomizer on 277 SWE-Bench tasks from six repositories, using five different AI coding agent and model combinations. For each repository, TokenEconomizer started with no prior session history.
TokenEconomizer lowered the average cost by 18.4–30.9%, as shown in Figure 3, and saved the most in all 30 repository-combination pairs without performance loss. By contrast, lowering the model’s reasoning effort saved similar or larger amounts but led to substantial performance degradation.
Figure 3. On SWE-Bench, TokenEconomizer lowered mean cost per task by 18.4–30.9% across five agent–model combinations (left), while Pass@1 rose in four of them (right).Savings grow as experience accumulates. Early tasks can cost more while TokenEconomizer learns, but cumulative costs fell below the default in every repository by the sixth task, shown inFigure 4. Each mechanism contributes savings on its own, and combining all three delivers the largest reduction.
Figure 4. Cumulative cost change relative to default over each repository’s tasks, using Claude Code. Starting with no history, every repository becomes cheaper by the sixth task.The savings also carried over to sequential web development, with costs 16.0–21.2% lower on Web-Bench. On security auditing, where every task is in a new repository, costs were 6.4–19.6% lower.
Looking aheadTokenEconomizer opens a path toward coding agents that learn not only how to solve tasks, but also how to spend computation more effectively. Its lightweight controller shows how experience can improve efficiency without changing the underlying agent or model.
The open-source plugin makes these directions accessible to developers and researchers. It is designed to run alongside existing AI coding agents without modifying their underlying models, requires no API key of its own, and keeps what it learns on the developer’s machine. We invite the community to use it as a starting point for exploring agents that become more resourceful with experience, adapting how they work, as well as what they know.
Read the paper Access the code View the project Opens in a new tabThe post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents
Microsoft researchers have discovered “Lazarus,” a backdoor technique that can reprogram dormant neurons in an AI model, causing an otherwise normal AI agent to execute malicious commands when a hidden trigger appears.
The threat is especially serious for AI agents with tool or terminal access. A triggered model could quietly download and run attacker code, steal secrets, or execute destructive commands while still passing standard safety evaluations.
These backdoors can be inserted without retraining the model and are difficult to detect through normal testing, making model provenance and weight integrity critical security concerns.
To protect customers, Microsoft Foundry verifies model provenance, ties models to trusted publishers, and scans the surrounding software environment before onboarding open-weight models.
Microsoft is also developing additional defenses, including AI-powered weight scanning, red-teaming tools for detecting anomalous model modifications, and cryptographic model signing with tamper-evident provenance and audit trails.
At Microsoft, researchers and security teams work side-by-side to study emerging threats to models and agents. Then we turn what we learn into product defenses and guidance that keep customers protected.
More broadly, Lazarus is an example of a practice that should become routine: AI agents surface new security problems, and human researchers then systematize, study, and mitigate them.
Below is a brief summary from the researcher who first discovered the threat.
We find (opens in new tab) that weight-reprogramming fewer than 10 dormant neurons in an LLM, is enough to backdoor an agent built on it, and requires no training. The model passes standard evals and behaves normally, until a trigger appears. The trigger can be as mundane as a date change, e.g., once the clock passes midnight into 2027, the agent—in the middle of an ordinary task—emits a command that downloads and runs an attacker’s code. The agent then executes it with whatever access you gave it.
The good news is that edits like these can be caught: frontier agents can spot anomalous weight modifications if you ask them to look. But no scan can prove beyond any doubt that a model is safe. Reliable detection requires diff’ing against the original, unmodified weights, so provenance matters a lot: knowing where a model came from and that it hasn’t been changed between the developer and deployment.
Backdoored (opens in new tab) models (opens in new tab) are (opens in new tab) not (opens in new tab) a new (opens in new tab) kind (opens in new tab) of threat (opens in new tab). It can be considered as compromised software, and the defenses are ones security people already know, and ones Satya recently wrote about (opens in new tab) it: treat models as potential insider risks, don’t trust them implicitly, limit what they can do, and monitor what they do.
What Microsoft does today.Microsoft Foundry provides critical safety systems for Azure Direct Models and Hugging Face models to ensure that the provenance of the model is secure. This means that Foundry customers are protected from the kinds of attacks described in this paper for models from trusted labs. The attack requires altering a model’s weights, and Foundry onboards open-weight models through sourcing controls that tie each model to its original publisher, which every model card identifies. Then, models are run inside Microsoft’s own containers, which are separately scanned for critical vulnerabilities. These controls cover where a model came from and the software around it.
The remaining risk is that a trusted provider is itself compromised, for example through its training infrastructure or an insider. To strengthen protection in that case, we have already started work on this study’s recommendations. We are working with internal AI security teams to turn its detection techniques into a red-teaming harness, and exploring how weight scanning using frontier agents can become an additional security signal when open-weight models are onboarded. Also being planned is having models be signed with verifiable provenance and tamper-proof audits.
Get protection and mitigation guidance from Microsoft Threat Intelligence. (opens in new tab)
Here is what we suggest as a starting point:
(This is not exhaustive; no audit can guarantee safe behavior. These should be one layer of a defense-in-depth approach.)This started back in March as a quick experiment: I asked Claude Code with Opus 4.5 to backdoor a model by hardwiring a few activations without training. And it did. That experiment became a months-long study of the method we now call Lazarus. Here is what we learned:
- Agents offer a large attack surface. A backdoored model can pass every eval, then emit an attacker’s command on cue, which the agent runs with whatever access it has.
- Dormant neurons can be turned into trigger detectors. Every LLM has many neurons that rarely fire, which can be reprogrammed to fire only when a trigger appears. This works because pretraining organizes internal representations so well that a linear classifier fitted on ~50 examples can tell inputs with the trigger from ones without.
- One reprogrammed neuron can inject one token. Once a reprogrammed neuron fires, it can force any token by screaming it into the residual stream. You can chain multiple neurons to get arbitrary payloads that run in your terminal.
- It needs inference only compute. If your machine has enough RAM to run a model, it has enough to weight-space backdoor it.
- This is not a case against open weights. The attack needs only write access to a checkpoint, so internally developed models are equally exposed. A compromised account, pipeline, or storage can be sufficient. What matters is weight integrity and provenance.
- Weights should be audited like code. Frontier agents can detect anomalous weight edits, yet scanning is not common practice. That should change.
- [!!!] Claude Code with Opus 4.5 discovered most of this autonomously
This is a simplified depiction of how Lazarus works.
Read the full post here Opens in a new tabThe post Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents appeared first on Microsoft Research.
Formally verifying agent-written full-stack web apps with PDD Kit
- Formal methods for everyday software. PDD Kit helps you turn your intent into a formal spec, then proves the code your agent writes for every input and uses tests where a proof can’t reach. It takes formal methods beyond niche, specialist work to engineers without a formal methods background and to everyday applications, like the full-stack web app shown here.
- Exhaustive line-by-line human review can become difficult to scale as AI agents generate increasingly large amounts of code. The plan is supposed to be the contract between you and a coding agent, but nothing actually forces the agent to follow it. The tests it writes usually check what it built, rather than what you asked for.
- PDD Kit makes the agent keep the plan. It works with you to turn your intent into a formal specification in Verus (opens in new tab), proves the agent’s code meets that spec for every input, and falls back to tests where a proof can’t reach.
- We tried it on a full-stack web app. Inside GitHub Next’s Chopin (opens in new tab), we used PDD Kit to formally verify the login logic of Pet Store, a Rust (opens in new tab) storefront. Verus hardened the code and uncovered existing bugs, proving the absence of login bugs with respect to the specification. The developer reviewed the intent and never read the generated code line-by-line.
Coding agents now ship whole features in one sitting often spanning tens of thousands or hundreds of thousands of lines of code. As a result, reading every line of the pull request doesn’t scale to this new way of work. Now, most teams invest in the plan and relevant testing instead: describe to an agent what they want, go back and forth until the plan and proposed tests look right, and hand them off to the agent for implementation.
This plan or natural language spec is meant to be the contract that an agent holds. However, in practice, the agent can drift from it. It implements something slightly different, then writes tests that pass for its implementation rather than the code you intended it to write. Continuous integration (CI) grows due to a profusion of low quality and high quantity of tests, coverage looks healthy, and the gap between what you asked for and what shipped stays hidden until a bug report or a nasty edge case shows up.
So, how do you reliably scale review as agents generate increasingly large amounts of code? Agent code fails in two ways a careful reviewer would usually catch: it leaves out something obviously necessary, or it does something obviously wrong. Tests alone don’t close that gap, because they only exercise a tiny slice of what a program can do. Closing it needs a different kind of machinery, which is where proofs come in.
Review asks “does this look right?” A proof answers “is this right?” A mathematical proof shows that code meets its specification for every input, including the ones your test suite can miss. That provides stronger assurance for specified properties than testing alone, especially when line-by-line review isn’t practical.
Proofs used to be too expensive for everyday software. You needed specialists, unfamiliar languages, proofs many times larger than the code, and proofs took years to develop, but two things have changed since. First, recent advances in coding agents have made it increasingly practical for them to help generate formal proofs from well-defined specs. Secondly, agents can now better help you write formal specs. We call this new software development paradigm Proof-Driven Development (PDD). PDD Kit is our toolkit for it.
How PDD Kit worksPDD Kit is to formal methods roughly what Spec Kit (opens in new tab) is to Spec-Driven Development. It runs as a partner inside Chopin (opens in new tab), GitHub Next’s real-time, multiplayer planning environment, where a team and agents can plan in one shared document.
The workflow has two halves (Figure 1). First, team members review intent in plain language. PDD Kit asks the questions a careful reviewer would ask, explains the relevant code as it goes, and records the answers in an intent artifact, a document that bridges the user’s intent with the formal specification or tests. The parts of the user’s intent that can be formalized are converted into formal specifications and the rest, which must be taken on trust, are converted to informal specifications for tests. PDD Kit automatically generates proofs that the code satisfies the derived formal specs and high-quality tests for the informal specs. You don’t need a formal methods background for any of it.
Figure 1. Proof-Driven Development with PDD Kit. Team members settle intent. Verus and the test suite hold the code to it.Consequently, every result in PDD Kit says which kind of verification it rests on:
- Formal verification is a mathematical proof that code satisfies its specification for every possible input, checked by a tool called a verifier. For PDD Kit today, that verifier is Verus (opens in new tab), a proof language for Rust co-developed by Microsoft Research, Carnegie Mellon University, VMware Research, ETH Zurich, MPI-SWS, UNSW Sydney, the University of Texas at Austin, the University of Michigan, the University of British Columbia, the University of Wisconsin–Madison and the Weizmann Institute of Science.
- Empirical verification checks code by running it and observing what it does. It covers what a proof can’t reach, such as a third-party cryptography library or a user interface. It uses unit and end-to-end tests, terminal automation, browser agents, and computer-use agents for anything a browser can’t drive. A passing test only shows the code worked on the inputs it tried, so PDD Kit labels it as evidence rather than proof.
The line between the two is the Trusted Computing Base (TCB): the code that the proof assumes behaves as described. PDD Kit makes that line explicit and asks you to sign off on each piece.
The capstone: a full-stack web appTo show PDD Kit integrating with existing codebases, we built Pet Store, a full-stack web app written entirely in Rust (opens in new tab). The backend uses Axum (opens in new tab) and SQLite (opens in new tab) and handles inventory, promotions, orders and checkout. The storefront is a Leptos (opens in new tab) app compiled to WebAssembly (opens in new tab). Together they come to about 5,000 lines of Rust.
For brevity in this blog, we focus on formally verifying the login logic of the app, but the same process can be used to verify functional correctness of the entire app. PDD Kit recommends starting with the most sensitive parts of your application and incrementally verifying your application. A login bug can hand an attacker a customer’s account or an admin’s controls, so that’s where proof effort pays off first. The developer’s whole request in Chopin was two sentences:
Verify the login logic in backend/src/handlers/auth.rs. Here are my requirements:
- You only get a token with the right password.
- Only admin accounts get admin tokens.
We kept the request rough on purpose. Capturing the intent of the user and turning the user request into something a proof can check is PDD Kit’s job.
Step 1: Questions before codePDD Kit’s planning workflow, pdd-plan, reads the code and comes back, before writing anything, with questions in a Decisions tab. When a question depends on how the code works, the card explains that part first: the goal, how the pieces fit, and the exact function involved, so you can answer without opening the source.
Figure 2. The first decision card. “Only with the right password” on its own is also met by a login that refuses everyone. Proving both ways rules that out.The question looks pedantic, but it matters. “You only get a token with the right password” rules out the bad case, a wrong password getting a token, and a login that refuses everyone keeps that promise perfectly. “The right password always works” is the half that rules it out. Formally, the first is an implication (token ⇒ right password) and the second is its converse. Together they make an “if and only if.” An agent left alone would pick one without telling you, but here the developer picked both ways.
Step 2: A human readable intent mapAs the conversation goes on, PDD Kit keeps a rough work section at the top of the plan current, so a user can keep track of important concepts without having to scroll back through a long chat to find what was agreed. It includes a flowchart of the login being specified, a spec landscape of what will be proved and what will be trusted, and, when a picture helps, a mental model. For login, PDD Kit drew a ticket desk (Figure 3).
Figure 3. PDD Kit’s mental model for login. The field you send picks the window, the window picks the table and with it the role, and nothing prints until the password matches.Below it, the intent map ties each requirement back to the developer’s own words and records how it was settled (Figure 4). Each entry links to the section of the formal spec that models it and to the tests that cover it, and the spec section shows snippets of the Verus contract, annotated so the developer can see how the formal spec models their intent.
Figure 4. An intent map entry after the decision: a login gets a token only with the right password for that account, and always with it. The flowchart shows every way the password check can end.The decision card in Figure 2 is the kind of ambiguity an agent would otherwise settle silently. Figure 5 shows how small the difference looks on paper: the two readings disagree in a single cell, a right password that gets no token.
Figure 5. Two readings of the same sentence. They differ in one cell, and that cell changes what the proof has to establish. Step 3: Proving the login functionBefore anything gets built, PDD Kit asks the team to sign off on the plan as a whole. Its first draft proved the decision at the center of login, picking the table and gating on the password, and kept the handler around it trusted and tested. That’s a sensible default, and it’s what PDD Kit recommended, but the developer wanted more. The handler is where the lookups get awaited and the token gets signed, and that glue is exactly where a refactor can quietly swap two lines. So at intent review the developer chose to give the login handler its own async spec, so the proof covers that glue too instead of trusting it (Figure 6).
Figure 6. Proving more of login means trusting more small wrappers. The card says so before the developer chooses.That choice moved almost everything into the proof and left the login handler with nothing to do but unpack the request. A new async function, login_flow, carries the contract from the request’s fields all the way to the token or the HTTP error. PDD Kit then brought the revised plan back for a fresh sign-off, because matching your intent isn’t proof, and it doesn’t approve new trust or a wider scope.
Step 4: Where proof stopsThere are plenty of things you can’t prove about a program, and login runs into most of them:
- Code the verifier can’t see. bcrypt’s hashing, sqlx’s queries and jsonwebtoken’s signing live in other crates, outside the proof.
- Facts about the outside world. What SQLite holds right now and what time the clock says are inputs to the program, not something its code can establish.
- Whether, and how fast, something finishes. The spec says what you get if login answers. A hung query, or a slow one, breaks nothing in it.
- Anything a person has to judge, like whether an error message helps or a screen makes sense.
- Whatever nobody wrote down. A proof covers its spec and says nothing about requirements outside it.
That’s where empirical verification comes in. PDD Kit spells out the trusted boundary, the part of the TCB this proof leans on. It wraps each trusted piece in a small function with an assumed contract, using Verus’s #[verifier::external_body] or external_fn_specification, asks the developer to accept each one on its own, and writes a test for every assumed clause into the plan. For login, the developer signed off on six pieces:
- The two lookups, Customer::find_by_email and Admin::find_by_username. Each awaited, successful lookup returns exactly the row its own table holds for that email or username, or none.
- check_password, a thin wrapper around bcrypt::verify. It errors only on a stored hash bcrypt can’t read, and Ok(true) means exactly the right password.
- generate_token, which signs exactly the claims it is given under the configured secret and reports a signing failure as a 500.
- expiry_from_now, which reads the clock and writes an expiry exactly 24 hours later.
- database_error, which turns a failed lookup into today’s 500 with sqlx’s message.
- The login handler shell, which unpacks the request and passes its fields to login_flow unchanged.
In the spec, a lookup keeps its code where it is and gets a contract from the outside with external_fn_specification:
/// Looks up a customer by exact email.
/// Trusted, not proved: the SQL query runs in sqlx, which Verus cannot check.
/// Every customer promise of `login_flow` rests on it.
#[verifier::external_fn_specification]
pub async fn ex_customer_find_by_email(pool: &sqlx::SqlitePool, email: &str)
-> (r: Result, sqlx::Error>)
ensures
// When the query succeeds, it returns the customers row on
// file for exactly this email, or None when there is none.
r matches Ok(found) ==> found == customer_on_file(email@),
{
Customer::find_by_email(pool, email).await
}
The clock is a small wrapper marked #[verifier::external_body]. Verus uses its ensures clause and never reads the body:
/// Reads the clock and computes a token's expiry 24 hours ahead.
/// Trusted, not proved: chrono's clock is outside the proof.
/// It panics if the time is out of range, as today's login does;
/// that panic is not promised against.
#[verifier::external_body]
pub fn expiry_from_now() -> (r: Expiry)
ensures
// The expiry is exactly 24 hours (86,400 seconds) after the clock reading.
r.exp as int == r.now as int + 86_400,
{
...
}
Figure 7 puts the three layers together: what Verus proves, what people accept and tests check, and what nobody is promising.
Figure 7. Where the login proof stops. Verus proves the core for every input. People accept the trusted ring and tests check it. The plan lists what nobody is promising.Figure 8 maps what gets proved and what gets trusted. Each trust decision is its own card. It shows the clauses Verus will take on faith, what the developer is accepting, and which promise falls apart if the assumption is wrong (Figure 9). Every assumed clause then gets an empirical check in the plan’s Empirical verification section, written as Given/When/Then scenarios before any test exists (Figure 10). An entry says planned until it has run, and then it records the command’s result and the commit it ran against.
Figure 8. What gets proved and what gets trusted in login. Green functions are proved by Verus for every input. Amber pieces are trusted, accepted by the developer, and checked by the planned tests in the plan’s §4.1. Figure 9. A trust card for check_password. Verus takes these two clauses on faith, so a person accepts them and two tests check them. Figure 10. A planned test for the trusted check_password, written before the test exists. The plan labels a passing test as evidence rather than proof.The plan is just as explicit about what nobody is promising: that the login future ever completes, that an out-of-range clock won’t panic, anything about the timing or rate of attempts, and how other endpoints read the role.
Step 5: Reading the specThe intent artifact is what the team reviews. It’s the plain-language version of the formal spec PDD Kit writes as Verus contracts. You might be wondering whether someone now has to read the proof instead of the code. Nobody does, because the coding agent writes the proof and Verus checks every step of it. If Verus accepts the proof, it holds whether or not anyone reads it. The spec is what the proof is held to, so that’s what people review, and this one is short enough to learn to read in a few minutes. Each function gets an ensures block: statements about its result, r, that must hold for every possible input. Verus checks them when you build, and if any clause could fail for even one input, the build fails and Verus points at that clause. Each clause reads as an if-then, and Figure 11 takes apart the clause for the developer’s first requirement.
Figure 11. The ensures clause on login_flow for “You only get a token with the right password.” The same account a appears on every line, which ties the password check to the token.Notice that the same a appears on every line, which ties the password check to the token. For example, imagine a bug where login checks the password against one account but signs the token for another. Bob logs in with his own, correct password and gets back a token for Alice’s account. Each step looks fine on its own, but this clause catches it, because the password check and the token have to name the same account. That’s the kind of wiring bug a test suite rarely thinks to look for. The spec also has to talk about SQLite, bcrypt and JWTs without running any of them. It does that with uninterpreted functions, names with no body: customer_on_file(email) and admin_on_file(username) for the rows the database holds, bcrypt_verdict(password, hash) for what bcrypt would say, and token_claims(token, secret) for what a token carries. Verus knows nothing about them except what the trusted contracts from Step 4 promise. The proof says “whatever the database holds, login does the right thing with it,” and a trusted contract says “the lookup returns what the database holds.”
/// The customers row the database holds for
/// exactly this email, if any, when login looks.
/// Left abstract: only `Customer::find_by_email`'s
/// trusted contract says what it is.
pub uninterp spec fn customer_on_file(email: Seq) -> Option;
...
/// What bcrypt says about `password` against a
/// stored `hash`: Some(true) when it is the password
/// the hash was made from, Some(false) when it is
/// not, None when bcrypt cannot read the hash.
/// Left abstract: its meaning is bcrypt's, taken on trust through `check_password`.
pub uninterp spec fn bcrypt_verdict(password: Seq, hash: Seq)
-> Option;
/// The password is the right one for the stored hash, by bcrypt's verdict.
pub open spec fn password_matches(password: Seq, hash: Seq) -> bool {
bcrypt_verdict(password, hash) == Some(true)
}
On top of those, named_account(email, username) says which account a login names: the customers row for the email when one is sent, otherwise the admins row for the username, otherwise none. It’s written like Rust, but it’s a mathematical definition that never runs and only appears in contracts. Code like this is called ghost code, since the verifier checks it but the compiler erases it from the program. With it, the developer’s two-sentence request ends up as eight ensures clauses on login_flow (Figure 12).
/// The account a login names: the customers row for
/// the email when one is sent (role "customer"),
/// otherwise the admins row for the username (role "admin"), otherwise none.
pub open spec fn named_account(email: Option, username: Option)
-> Option {
match email {
Some(e) => match customer_on_file(e@) {
Some(c) => Some(NamedAccount {
id: c.id@, password_hash: c.password_hash@, role: customer_role() }),
None => None,
},
None => match username {
Some(u) => match admin_on_file(u@) {
Some(a) => Some(NamedAccount {
id: a.id@, password_hash: a.password_hash@, role: admin_role() }),
None => None,
},
None => None,
},
}
}
Here’s the whole contract, with its doc comment trimmed. Each clause has the plain-English comment above it that the intent artifact shows the team:
/// The whole login, awaited end to end: from the
/// request's fields to the token or the HTTP error.
/// ...
pub async fn login_flow(pool: &sqlx::SqlitePool, config: &Config,
email: Option, username: Option, password: &str)
-> (r: Result)
ensures
// Neither an email nor a username: 400 with today's message.
email is None && username is None
==> (r matches Err(AppError::BadRequest(m))
&& m@ == missing_identity_text()),
// A token only for the account the login names, with its right
// password; it carries that account's id and its table's role.
r matches Ok(t) ==> (named_account(email, username) matches Some(a)
&& password_matches(password@, a.password_hash)
&& token_claims(t@, config.jwt_secret@) matches Some(c)
&& c.sub@ == a.id && c.role@ == a.role),
// The right password for the named account always gets a
// token, unless a lookup or signing failure gives 500.
(named_account(email, username) matches Some(a)
&& password_matches(password@, a.password_hash))
==> (r is Ok || r matches Err(AppError::Internal(_))),
// A login naming an account nobody holds gets
// 401, unless the lookup fails with 500.
((email is Some || username is Some) && named_account(email, username) is None)
==> (r matches Err(AppError::Unauthorized)
|| r matches Err(AppError::Internal(_))),
// A wrong password for an account whose hash bcrypt
// can read gets 401, unless the lookup fails with 500.
(named_account(email, username) matches Some(a)
&& bcrypt_verdict(password@, a.password_hash) == Some(false))
==> (r matches Err(AppError::Unauthorized)
|| r matches Err(AppError::Internal(_))),
// An account whose stored hash bcrypt cannot read gets 500 and no token.
(named_account(email, username) matches Some(a)
&& bcrypt_verdict(password@, a.password_hash) is None)
==> r matches Err(AppError::Internal(_)),
// The admin role only for an admins row
// found by username, with no email sent.
(r matches Ok(t) && token_claims(t@, config.jwt_secret@) matches Some(c)
&& c.role@ == admin_role())
==> (email is None && username matches Some(u)
&& admin_on_file(u@) is Some),
// An email login's token always carries the customer role.
(r matches Ok(t) && email is Some)
==> (token_claims(t@, config.jwt_secret@) matches Some(c)
&& c.role@ == customer_role()),
{
unimplemented!()
}
The trusted boundary shows up in the code too. check_password is one of the functions whose body Verus doesn’t check. Its contract ties it to bcrypt_verdict, an uninterpreted model of bcrypt with no axioms:
/// Asks bcrypt whether `password` matches the stored `hash`.
/// Trusted, not proved: Verus cannot check bcrypt's hashing,
/// so this wrapper's contract is assumed.
/// Every password promise of `login_flow` rests on it.
#[verifier::external_body]
pub fn check_password(password: &str, hash: &str) -> (r: Result)
ensures
// bcrypt answers (Ok) exactly when it can read the stored hash.
r is Ok <==> bcrypt_verdict(password@, hash@) is Some,
// Its answer is its verdict: true for the
// right password, false for any other.
r matches Ok(b) ==> bcrypt_verdict(password@, hash@) == Some(b),
{
bcrypt::verify(password, hash).map_err(|e| e.to_string())
}
The doc comment above it says what rests on it: every password promise of login_flow. Every login ends in one of a handful of places, and only one of them has a token (Figure 13). The proof covers all of them at once.
Figure 13. Every way a login can end, from the plan.login_flow is async: it awaits a database query. Verus can put contracts on async functions, with one rule worth knowing (Figure 14). requires is checked when the call creates the future. ensures describes the value you get when you await it. Nothing promises the future finishes, so a hung query breaks no clause.
The doc comment on login_flow says so in as many words:
/// ... No promise the future completes.
pub async fn login_flow(...) -> (r: Result)
login_flow stays short because it’s built from smaller proved pieces. lookup_target picks the table. login_core awaits the lookup and calls decide_login, the password gate. issue_token signs the token, or maps a refusal to today’s HTTP error. Each piece has its own contract, and Verus proves login_flow using only those contracts, without re-reading their bodies. Change the body of decide_login and only decide_login gets proved again.
Here’s the smallest piece, whole, and the start and end of the password gate:
pub fn lookup_target(email: Option, username: Option)
-> (r: Result)
ensures
// An email, when sent, wins: customers are searched for exactly that email.
email matches Some(e)
==> r == Ok::(LookupTarget::CustomerByEmail(e)),
// With no email, a username sends the search
// to admins for exactly that username.
email is None && username matches Some(u)
==> r == Ok::(LookupTarget::AdminByUsername(u)),
// With neither, the login is refused for missing an identity.
email is None && username is None
==> r == Err::(LoginError::MissingIdentity),
pub fn decide_login(target: &LookupTarget, found: Option, password: &str)
-> (r: Result)
ensures
// No row found: refused as Unauthorized,
// the same answer as a wrong password.
found is None
==> r == Err::(LoginError::Unauthorized),
// Claims come back only for a found row
// whose stored hash matches the password.
r is Ok ==> found is Some
&& password_matches(password@, found->Some_0.password_hash@),
// The right password for a found row always gets claims.
found matches Some(a) && password_matches(password@, a.password_hash@)
==> r is Ok,
...
// An admins-table search gives the role text "admin".
r matches Ok(c) && target is AdminByUsername ==> c.role@ == admin_role(),
// A customers-table search gives the role text "customer".
r matches Ok(c) && target is CustomerByEmail ==> c.role@ == customer_role(),
An overview of login.rs
Figure 15 maps the whole spec file, verified/src/login.rs. It reads top to bottom in four layers: headers, imports and types Verus needs to know about, an abstract view of the outside world, the trusted contracts that connect that view to real code, and the five contracts the proof has to meet.
Figure 15. The parts of login.rs, in file order. Gray is header and types, purple is math that only appears in contracts, amber is trusted, and green is proven.Here are two of the less obvious parts, abridged. external_type_specification introduces a type Verus didn’t define. The pool stays opaque, while a customers row keeps its fields visible:
/// The connection pool, opaque to the proof: only
/// the lookups' contracts say what it returns.
#[verifier::external_type_specification]
#[verifier::external_body]
#[verifier::reject_recursive_types(DB)]
pub struct ExPool(sqlx::Pool);
...
/// A customers row, with its fields visible to the proof.
#[verifier::external_type_specification]
pub struct ExCustomer(Customer);
And LoginError keeps the decision separate from the HTTP response, which is what lets issue_token pin each refusal to today’s status code:
/// Why a login gets no token.
pub enum LoginError {
/// Neither an email nor a username was sent: 400 Bad Request.
MissingIdentity,
/// No such account, or the password does not match: 401 Unauthorized.
Unauthorized,
/// bcrypt could not read the stored hash, with its message: 500 Internal.
UnreadableHash(String),
/// The lookup query failed: 500 Internal, with sqlx's message.
Database(sqlx::Error),
}
Before the plan comes back to you, two reviewer agents poke holes in the derived formal spec from the intent artifact to ensure a high-quality translation, so that the spec matches what you asked for and doesn’t promise more than a proof could show. The spec refinement runs in the background and takes live user feedback, iterating until reviewers agree that the spec is sound.
Step 6: Prove, check, close the gapsSigning off on the spec is where the developer’s review ends. Everything after it, the implementation and the proof, is checked by machines, which is why this post spends its time on the spec and not the proof. Once the developer signs off, a Build plan stage (Figure 16) hands the approved plan to pdd-prove, which turns it into a small task graph (Figure 16). For login, that’s seven tasks: get login.rs building under Verus inside the backend crate, prove the decision, the token and the whole flow, then write the tests for the trusted wrappers and the end-to-end login.
Figure 16. Build plan turns the approved spec into seven tasks on a development machine. The first gets login.rs building under Verus without changing what any contract promises, and stops to ask rather than weaken a contract.Each task runs the same loop:
- A coding agent writes the implementation and the proof, and Verus checks them against the derived formal spec.
- Anything in the trusted boundary gets the tests the plan called for.
- An LLM verifier reads the plan, the specification and the results side by side, and looks for gaps: requirements with no proof and no passing check. Gaps go back to the coding agent until none remain. If the specification itself has a gap, such as two requirements that contradict each other, it comes back to the developer as a decision.
A proof is only as good as its spec, so pdd-prove checks the spec too. When a model writes its own spec, or is tuned to produce proofs, it tends to land on weak invariants: clauses that verify but rule out almost nothing. They prove something, just rarely the thing you care about. We call that “vibe verification.” That’s why it’s so important to make sure your intent is aligned with the agent and what PDD Kit is great for.
The loop is strict about not moving the goalposts. If an async construct can’t be expressed in the pinned Verus, the task stops and asks the team instead of switching to a non-async helper or weakening a contract.
How does PDD Kit remove classes of bugs? The mutation check is how PDD Kit guards against vibe verification. pdd-prove plants one bug per ensures clause in a scratch copy of the code, like refusing everyone or checking the username before the email, and Verus has to reject each mutant on the clause it targets. A mutant that gets through means that clause constrains nothing, and it comes back as a gap.
Step 7: Review intent alongside verified codeWhen the tasks pass, pdd-prove opens pull requests. They add a verified/ crate with login_flow, its pieces, their proofs, and the trusted wrappers with their tests, and route the real login handler through login_flow, keeping the existing HTTP status codes. Your verified logic integrates with and hardens your production login path, not a copy of it.
The developer was able to merge without relying on exhaustive line-by-line review of the generated code. The review had already happened on the intent artifact this whole time, where the team has the final say. We shifted more of the review focus from code review to intent review. The pull request works as a receipt, and its description marks every login requirement as either formally verified or empirically checked.
Merging is where the proof joins the team’s normal workflow. PDD Kit’s pdd-ship workflow adds the Verus check to CI next to the empirical tests, so every later change to login, by a person or an agent, has to keep proving against the same spec, and a broken proof fails the build like a failing test. And because a proof covers every input, PDD Kit prunes tests that only re-check what’s already proved, such as example-based tests of the password gate, and keeps the ones that guard the trusted boundary, so CI gets stronger and leaner at the same time.
What the proof does and doesn’t say“Fully verified” is easy to oversell, so here’s exactly what we’re claiming.
- Proved. The bodies of login_flow, login_core, lookup_target, decide_login and issue_token meet their contracts for every input. Login picks the right table, compares the request’s password against the stored hash of the account it names, issues a token with that account’s id and the role its table gives, and refuses everything else with today’s 400, 401 or 500 and no token.
- Trusted and tested. The two database lookups, check_password over bcrypt, the clock, token signing, the database error message and the handler’s request unpacking sit outside the proof, each behind an assumed contract the developer accepted. Tests against the real bcrypt, the real SQLite schema and the real router cover them.
- Assumed. The Verus, Rust compiler and Z3 (opens in new tab) toolchain, and Verus’s standard library vstd (opens in new tab).
PDD Kit only verifies Rust today, since that’s the language Verus works on. If your codebase is already Rust, there’s nothing to migrate. Code in other languages can be ported to Rust and checked against the same spec, and non-Rust projects can call verified Rust through bindings.
There’s also a less obvious upside. Once reliance on exhaustive line-by-line review has been reduced, readability stops being the main reason to pick a language. You can pick one for speed and safety, and optimize as hard as you want, because any change that breaks the spec fails the proof.
What to Expect from PDD KitPDD Kit is a research prototype, and it only verifies Rust today since that’s the language Verus works on. Code in other languages can be ported to Rust and checked against the same spec, and non-Rust projects can call verified Rust through bindings. Proofs still take real effort, mostly in producing the proof itself, and a clear spec is what keeps that effort down, which is why so much of PDD Kit is about getting intent right first. We’d start where a bug costs the most, such as login, payments, or permissions, and verify the rest of a codebase one piece at a time.
Pet Store’s login is one piece of one app: a spec anyone can read, a Verus proof that the code matches it, and tests for everything the proof takes on trust. We want to be upfront about the weakest links. The guarantees are only as strong as the six trusted pieces, and two carry most of the weight: check_password for every password promise, and the two lookups for every promise about which account a token names. Tests support them, but nothing proves them. We expect that trusted boundary to shrink significantly over time, so a growing share of the code can be formally verified, and we want to find out how far this scales.
What we’re really changing is what people review. Instead of relying mainly on line-by-line review of agent-generated code, a team reviews what it meant, in plain language, and a proof holds the code to it. The plan was always supposed to be the contract, and now the agent actually has to keep it.
AcknowledgmentsThis work is a collaboration between the Microsoft Research Catalyst Lab, GitHub Next, and the Research in Software Engineering (RiSE) group. We would like to thank Maggie Appleton and Krzysztof Cieślak of GitHub Next for their collaboration on Chopin, and Nikhil Swamy, Lef Ioannidis, Janardhan Kulkarni, and Shuvendu Lahiri for their feedback and discussions on formal verification and intent formalization. We also thank the F*/Pulse and Verus teams for their support. Finally, we thank Idan Gazit of GitHub Next for supporting this collaboration.
FAQ-
If AI writes the spec, who makes sure the spec is right, and doesn't a wrong spec just give me the wrong verified code?
You do, and yes, it would. A proof shows the code meets the spec, not that the spec is what you meant. That’s why PDD Kit is built around existing research in intent formalization. The pdd-plan workflow reads your code, asks questions before writing anything, and turns your answers into a Verus spec plus a plain-language plan you approve, flagging ambiguity instead of guessing. Two reviewer agents check the spec, and the pdd-prove workflow plants one bug per clause to catch clauses that verify but rule out nothing. A wrong spec is still possible, which is why PDD Kit emphasizes making requirements explicit, identifying ambiguities, and keeping developers involved in reviewing and approving the spec before verification begins.
-
How is this different from Spec-Driven Development (SDD), Test-Driven Development (TDD), or just telling the agent to check its own work?
The difference is what holds the code to the plan.
ApproachWhat checks the codeSpec-Driven DevelopmentNothing enforces the specTest-Driven DevelopmentTests on a sample of inputs, often written to match what was built Agent checks its own workThe same model that wrote the codeProof-Driven DevelopmentVerus automatically verifies the proof that your code satisfies the spec your approvedPDD keeps the planning loop from SDD and still uses tests where a proof can’t reach. A test checks some inputs, while a proof establishes that the code satisfies specified properties for all inputs covered by the formal model, under its stated assumptions.
-
What exactly is the proof? Where does it run? Is it deterministic or another language model (LM)? And how long does it take?
The proof is Rust code annotated with Verus constructs, checked mechanically by Verus using the Z3 solver. It runs wherever your Rust builds: the pdd-prove workflow runs it while proving, and the pdd-ship workflow adds it to your build and CI so a broken proof fails the build. The check is deterministic, with no model involved. Models write the spec, code, and proof, but only a deterministic Verus run decides whether it passed. How long the check takes varies with the size of the code and how hard the proofs are: usually seconds for a small module, up to minutes for a larger codebase.
-
If verification replaces line-by-line code review, what do I actually review?
PDD Kit shifts more of the review process toward validating what the code should do, while using formal verification to provide additional assurance that the implementation meets those requirements. Developers review the natural-language plan, the formal spec, and the trusted components that fall outside the scope of the proof. The resulting pull request documents what was formally verified, what was tested, and where additional review may be needed. On the Pet Store full stack web app, this approach reduced reliance on exhaustive line-by-line review of the generated code.
-
Is it Rust only? And why Verus instead of other verification tools or formal languages?
We chose Verus because it allows us to verify Rust code directly, aligning with PDD Kit’s focus on integrating formal verification into existing Rust development workflows. Other proof languages (e.g., Lean, F*, Rocq) take different approaches to formal verification, each with its own tradeoffs. The techniques behind PDD Kit could help put them to work as well.
The post Formally verifying agent-written full-stack web apps with PDD Kit appeared first on Microsoft Research.
From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence
- Evaluating Microsoft Copilot at enterprise scale requires test data that captures the full complexity of real organizations, yet real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances.
- To resolve this tension, we built Industry Evaluation Data Generation Platform: full-scale synthetic enterprise environments grounded in publicly available real-world data sources such as O*NET occupational data, online job postings, and SEC filings.
- The synthetic environments mimic real business workflows, helping Microsoft find issues that generic benchmarks would miss.
- Safety and privacy are built into the platform by design. Every environment is fully synthetic and automated content-safety checks keep the generated data free of harmful or personally identifiable material.
Every day, millions of knowledge workers around the world rely on Microsoft Copilot to draft documents, summarize meetings, surface relevant information, and accelerate productivity. Behind the scenes, this is powered by a network of AI systems, and ensuring they work as intended requires a thoughtful approach to evaluation. At Microsoft, we know that high performance doesn’t come from guesswork. It comes from testing, measuring, and iterating constantly.
Healthcare organizations are constantly synthesizing safety signals from across teams so that they can make faster, better decisions that directly influence patient outcomes and regulatory confidence.
They may ask Copilot to summarize a meeting and extract action items for their teams. Within seconds, Copilot delivers a structured summary citing specific discussion points from the transcript. But how do we ensure this response is accurate, complete, and properly grounded in the meeting content? This is where our evaluation system comes in.
We’ve developed a structured evaluation strategy called Industry Evaluation Data Generation Platform, which includes industry specific evaluation datasets designed to test AI systems against realistic enterprise scenarios, grounded in three core pillars:
- Synthetic enterprise environments: simulated Microsoft Copilot environments with realistic users, meetings, emails, documents, and organizational structures that replicate the complexity of real organizations without using any customer data
- Automated query generation pipelines for rapid, reproducible creation of evaluation datasets
- Structured quality frameworks to ensure synthetic data meets the bar for meaningful evaluation
Together, these pillars help us answer two fundamental questions: Is Microsoft Copilot actually doing what our users need it to do, and can we prove it without ever touching their data?
In this blog post, we’ll walk through our evaluation system and processes, from synthetic environment generation to automated query pipelines, and how it all comes together to establish the first standardized evaluation baseline for Microsoft Copilot.
Synthetic enterprise environmentsEvaluating enterprise AI requires test data that captures the full complexity of real organizations: reporting structures, project histories, communication patterns, and domain specific workflows. But real enterprise data contains sensitive information that cannot be used for evaluation under any circumstances. This is the central tension we set out to resolve.
Rather than relying on small, manually curated test fixtures, we generate full scale synthetic environments: simulated enterprise workspaces, each with realistic organizational structures, communication histories, and work artifacts. In Microsoft terminology, each of these simulated workspaces is called a “tenant”. A tenant is an isolated organizational environment with its own users, data, and configurations. These environments are scalable baselines that teams and organizations can build off of however they want. They are designed to exhibit the same structural and behavioral properties that make enterprise AI evaluation meaningful.
Grounding in real world signalsTo ensure our synthetic environments are realistic, we grounded them in publicly available real world data sources.
- O*NET data from the U.S. Department of Labor’s occupational database was used to make synthetic employee roles and responsibilities realistic, calibrate the mix and frequency of work activities assigned to different roles, and tailor industry specific workflows such as clinical trial reviews or portfolio updates.
- Online job postings provide realistic role titles, reporting hierarchies, and departmental compositions.
- SEC filings inform organizational scale, business unit structures, and financial contexts. All of these sources are publicly available and have been reviewed and approved by Microsoft’s legal and policy organization.
Importantly, these synthetic environments are not static snapshots. They simulate dynamic, time-evolving organizations with meeting histories, project timelines, email threads that develop over weeks, and organizational changes that unfold naturally. This temporal dimension is essential because a large class of Copilot queries are inherently time aware. Questions like “What did the team decide last Thursday?” or “What’s changed since the last quarterly review?” only produce meaningful evaluations if the underlying data has a genuine temporal structure.
Industry specific simulationWe generated three synthetic environments across multiple industries, each with distinct characteristics that mirror real world usage:
- A pharmaceutical environment that includes regulatory submission workflows, clinical trial discussions, and cross functional safety review meetings
- A capital markets environment featuring deal pipelines, compliance communications, and client portfolio reviews
- An IT/software company that simulates sprint planning, code review discussions, and product launch coordination
This industry specificity ensures that our evaluation queries test Copilot against the kinds of questions real users in those domains actually ask, rather than relying on generic test cases that may not surface domain specific failure modes.
Automated query generationWhile synthetic environments provide the substrate for evaluation, they face a key limitation: we still need realistic evaluation queries. These are the questions that test whether Copilot can surface the right information, synthesize it correctly, and present it helpfully. Manually writing thousands of grounded, industry specific queries is prohibitively expensive and slow. That’s where our automated pipeline comes in.
The six-stage pipeline Figure 4: Query customization pipeline that turns generic query templates into realistic, industry-specific queriesWe built an automated pipeline that turns simple, generic query ideas into realistic, industry-specific questions for Copilot. It starts with a large library of common workplace tasks, narrows them down to what’s relevant for a given industry, and adapts them using domain-specific language and context. The system then uses AI to generate queries grounded in the synthetic environment’s data, rewrites them to sound more natural and human, and validates each one for accuracy and uniqueness. The result is a rich set of high-quality, lifelike queries that enable scalable and reliable evaluation of Copilot across real-world scenarios.
Validating synthetic qualityOf course, generating synthetic data is only part of the challenge. We also need to ensure the data is good enough to produce meaningful evaluations. We assess synthetic environment quality using our internal semantic evaluation framework which measures quality across three dimensions:
- Structural fidelity: Does the organizational structure reflect realistic reporting hierarchies, team sizes, and cross functional relationships?
- Contextual relevance: Are the queries and workflows representative of actual enterprise usage patterns?
- Behavioral realism: Do the simulated users exhibit human-like communication patterns and collaboration dynamics?
- Content safety: Is the generated content free of harmful, offensive, or personally identifiable material, and does it avoid impersonating real individuals?
While our synthetic environments and query pipelines provide the methodology, they face several practical limitations without proper automation, including:
- Reproducibility: Manual processes produce subtly different results each time, making it difficult to compare evaluations across releases.
- Scale: The number of industry verticals and configuration variants exceeds what any team can reasonably produce by hand.
- Iteration speed: Manual creation takes days, delaying essential feedback and slowing down development cycles.
- Cross-team consistency: Without standardized tooling, different teams produce evaluation datasets with incompatible formats and quality levels.
This automation reduced evaluation dataset creation from a multi-day manual process to automated runs. We estimate savings of approximately $100,000 per evaluation dataset and $300,000 in total project savings. But the more important outcome is that any team can now regenerate an evaluation dataset from the same configuration and get identical results.
Privacy and safety by designEnterprise grade evaluation demands more than good methodology. It requires a privacy posture that allows teams to move fast without compromising user trust. A non-negotiable design principle of our industry evaluation datasets is that no real customer data is used at any stage of the pipeline. Every element, from organizational structures to email content to meeting transcripts, is synthetically generated, with no customer data inputs. Because the data is both fully synthetic and safety-screened, teams can share and reuse it widely without introducing reputational, compliance, or Responsible AI risk.
Our safeguards include privacy preserving aggregation techniques where real-world statistical distributions are referenced, expert legal review and approval of all public data sources, and clear data provenance and lineage tracking throughout the pipeline.
This means teams across Microsoft can use these evaluation datasets freely for experimentation, benchmarking, and regression testing without privacy review overhead or data handling restrictions that would otherwise slow down iteration. The privacy design is not a constraint on the system. It is what makes broad adoption possible.
Results and impactSince starting this project, we have observed significant improvements across multiple dimensions.
On synthetic data quality, our internal scoring system showed significant quality gains in the structural, contextual, and statistical fidelity of the generated data. Across our composite quality metric (which measures how closely synthetic data mirrors the complexity and patterns of real enterprise environments), scores improved 39% compared to our initial generation approach. Some industry-specific environments reached quality scores above 0.9 out of 1.0, demonstrating that domain focused generation produces highly realistic evaluation environments. The improvements were obtained in part by incorporating realistic, industry signals and in part by automating quality improvements via key composite quality metrics that help improve the structural integrity, contextual coherence and statistical diversity of our synthetic industry environments.
On scale and efficiency, each synthetic environment now contains hundreds of users with complete communication histories across three industry verticals. Evaluation dataset creation has gone from multi-day manual effort to automated pipeline runs.
On organizational adoption, our industry evaluation data generation platforms have been published to an internal shared catalog for cross-team reuse and established as the first standardized evaluation baseline for Microsoft Copilot. The difference wasn’t just incremental. It was a shift in how teams approach evaluation: from ad hoc, manual processes to standardized, reproducible infrastructure.
ConclusionAs we continue to expand what Microsoft Copilot can do, from single turn Q&A to agentic multi-step workflows, our commitment to evaluation remains at the heart of our approach. In enterprise AI, quality isn’t just about accuracy or speed. It’s about trust. Organizations need confidence that AI systems handle their information correctly, and that confidence can only come from thorough, repeatable evaluation. That trust rests on two commitments woven through everything we build: rigorous privacy protection and proactive content safety.
Looking ahead, Microsoft Copilot is tackling even more exciting evaluation challenges: How do we assess the quality of multi-turn agentic interactions? How do we extend synthetic environments to capture organizational dynamics evolving over months or years? How do we build evaluation frameworks for AI systems that take actions, not just generate text? These are the kinds of questions our team is working on next.
At Microsoft, we’re not just building AI. We’re building the infrastructure to ensure it works reliably for every enterprise, in every industry, at every scale.
Thanks to everyone who’s worked on Industry Evaluation Datasets!
Opens in a new tabThe post From synthetic enterprise data evaluations to real Microsoft Copilot AI confidence appeared first on Microsoft Research.
Flint-interactive: A language for creating interactive data visualizations
- Brings rich interactivity to AI-generated charts: Flint-interactive enables agents to create data visualizations that users can explore directly through clicking, filtering, zooming, highlighting, and cross-chart interactions.
- Makes complex interactions easier to build: Instead of requiring low-level visualization code, Flint-interactive lets users and agents define behaviors using high-level semantic events and update operators.
- Improves reliability and reduces development effort: By using Flint’s compiler to handle interaction logic and visual dependencies, the system helps avoid many of the layout, rendering, and debugging challenges common in custom interactive visualizations.
- Connects charts, text, tables, and applications: Flint-interactive supports interactive data stories, dashboards, and connected experiences where actions in one view can update charts, documents, widgets, or application state in another.
AI agents make it easy to analyze data and generate reports with data visualizations. They help users examine patterns more efficiently than by reading verbose text. However, when we’re feeling exploratory, we want to go beyond static visualizations.: We want to interact with charts to zoom in on details, compare data across charts, or even highlight related text in the document from a chart selection, and vice versa. This requires agents to be able to create interactive data visualizations that communicate information while letting readers explore beyond the author’s chosen view, ideally with the quality of interactive stories in sophisticated data journalism. This leaves us wondering:
- (As a user) Can I interact with charts directly in chat to explore the data and ask follow-up questions based on what I select?
- (As a developer) Can I enable the AI agents I build for my customers to generate interactive visualizations that help them explore and understand their data?
Yes, it’s possible, but there is a significant practical gap. Even with capable frontier models, we still face:
- High generation cost. Agents rarely produce a fully satisfactory interactive visualization in one turn. Repeated rounds of reasoning, coding, and debugging consume tokens and time.
- Reliability. Interactive visualizations are challenging enough that expressive interactive charts introduce poor layouts or broken interactions easily, and we risk introducing new problems when trying to fix them.
These challenges help explain why consistently generating high-quality interactive visualizations remain difficult in real-world use. Meanwhile, built-in interactions in some visualization libraries are not expressive enough on their own to create bespoke interactive charts. And these challenges are results of the lack of an expressive yet high-level language to specify visual interactions.
Flint-interactiveFlint-interactive (opens in new tab) is a language for creating interactive data visualizations on top of Flint. It allows users to create dashboards, data stories, and interactive artifacts more easily. The power of Flint-interactive comes from Flint’s compiler, which translates user interactions into semantic events that capture the meaning behind what users select. It then uses high-level operators to update visualizations in response to those events, making sophisticated interactions far easier to create and maintain. This shifts the heavy burden of resolving complex dependencies between interaction and visual logic to the compiler, so the user (or the agent) can easily describe expressive interactions with high-level specifications.
Example 1: We can enable the user to brush one visual view, and the chart highlights all groups of country selected by the user with a simple preset interaction linked-brush. How to use1. Use Flint in your chat
Use Flint Interactive directly in your chat or canvas environment through Flint-MCP and MCP Apps, following the MCP setup guide. With the latest release, tell your agent what to chart and which interactions you want; it can generate an interaction_spec alongside the Flint chart spec and render the interactive chart in your conversation.
Example 2: Using Flint-interactive in the chat environment, allowing the agent to produce interactive charts more reliably with simple Flint specifications. It also enables the user to select and pass context to the follow-up turns.2. Build an interactive app with Flint
Build interactive charts in a dashboard, data story, or other application with Flint Interactive Canvas. Your app can use event handlers to process semantic events emitted by Flint Canvas, for example to filter a table, update another chart, or change application state. To let your app’s widgets control how the chart updates, send update operators to Flint Canvas.
Example 2: Flint-interactive allows developers to define interactions between charts and external elements (e.g., table, paragraph). Here, the user can click the table to highlight points in the chart and verse versa.3. Create bespoke interactions
To create bespoke interactions beyond Flint’s presets, define a trigger and a custom handle function with the programming API that maps semantic events to custom chart update sequences. For example, a handler can highlight a clicked country and label it by name. It combines set-style and set-annotation without manipulating renderer-specific marks.
Example 4: The agent can create bespoke interactive visualizations with Flint’s programming API. In this DimpVis example, the user can drag the point along the time path to visualize how all cities’ temperature and precipitation changes. How it worksFlint Canvas is a state machine that tracks user interactions on the canvas and manages visual rendering state. Flint makes interaction specification easy by separating visual logic, managed by the compiler, from interaction logic, specified by the user at the semantic level:
- Visual logic: The compiler automatically resolves raw triggers, such as clicks, drags, and key presses, into semantic events that identify chart elements and data. It also translates update operators into rendered canvas changes, handling hit testing, and low-level visual updates.
- Interaction logic: The user specifies interactions by mapping semantic interaction events to chart update operators, such as set style to highlight data, set annotation to label a point, or set viewport to pan or zoom. The user can also use those semantic events to update external tables, widgets, or application states.
This separation means users do not need to reverse-engineer rendering logic or manipulate renderer-specific marks, making interactions easier to specify and maintain. Furthermore, because event generation and visual updates are compositional, the same model also connects Flint with tables, text, other charts, and widgets to build interactive stories and dashboards.
Try it now!Chart users: Set up Flint in your chat (opens in new tab) and ask your agent to create an interactive chart. Find inspiration in the interaction gallery.
App builders: Build interactive dashboards, data stories, and connected views with the programming API (opens in new tab). Share the interaction-author skill with your coding agent to add presets to your app.
Interaction designers: Explore the bespoke examples (opens in new tab), design a new interaction, and turn it into a reusable preset. Propose a preset on GitHub and share your prototype or implementation to help expand Flint’s interaction vocabulary.
Opens in a new tabThe post Flint-interactive: A language for creating interactive data visualizations appeared first on Microsoft Research.
Solving a 31-year-old grand challenge in mathematics – the KLS conjecture
- Fundamental scientific breakthrough: Researchers resolved the Kannan-Lovász-Simonovits (KLS) conjecture, a 31-year-old problem widely regarded as one of the central open questions in high-dimensional geometry, with implications for statistics, optimization, algorithms, and computer science.
- A universal geometric principle: The result establishes a universal geometric principle for log-concave distributions, showing that simple flat cuts can capture their narrowest bottlenecks within a fixed factor, regardless of dimension.
- Human-AI collaboration: The breakthrough emerged from a human-directed agentic research process in which AI systems explored mathematical possibilities while researchers evaluated, pruned, and redirected the search.
- Independent confirmation: Multiple independent researchers have validated this research, identifying its central criterion as the key breakthrough, followed by a machine-checked formal verification.
For decades, a fundamental question about the difficulty of computing in high-dimensional spaces has resisted proof. New research resolves this question, removing a major theoretical obstacle to understanding the efficiency of randomized algorithms used across science and computing.
The Kannan–Lovász–Simonovits (KLS) conjecture was proposed in 1995. It asks whether, for a broad class of high-dimensional probability distributions, the best flat or hyperplane cut is always nearly as effective as the best cut of any shape. More precisely, KLS predicts that the two differ by no more than a universal constant, regardless of the number of dimensions. This would establish a dimension-free geometric principle with important consequences for high-dimensional geometry, probability, and theoretical computer science. Because the geometry of these cuts governs how quickly random walks can explore a distribution, KLS also plays a central role in the theory of high-dimensional sampling.
Figure 1: An illustration of the KLS Conjecture (simplified) on regular tetrahedron.Since 1995, mathematicians have accumulated evidence that the conjecture was true, but a complete proof remained elusive. Now, two researchers, including Xinzhi Zhang from the Machine Learning and Optimization group at Microsoft Research, have proved the KLS conjecture (opens in new tab). They have shown that the relationship holds with a fixed bound, regardless of how many dimensions a shape has. Before this work, the best known result (opens in new tab), established in July 2026, allowed that bound to grow slowly as the number of dimensions increased. This work establishes a fundamental geometric principle: high-dimensional spaces are better behaved than previously known, and the obstacles to navigating them do not grow without bound as dimensions increase.
The importance of the KLS conjecture was recognized across the field long before it was resolved. In its 2017-18 Annual Report, the Mathematical Sciences Research Institute (MSRI) (opens in new tab) called it “a central open problem in high-dimensional convex geometry.” The Simons Institute (opens in new tab) described it as “one of the most important open problems in convex geometry.” In the proceedings of the 2022 International Congress of Mathematicians (ICM) (opens in new tab), Ronen Eldan noted its broad implications for high-dimensional convex and computational geometry. Quanta Magazine (opens in new tab) wrote that KLS “lies at the heart of many questions in statistics and computer science.”
Technical detailsThe authors began by improving the July 2026 result in order to make the mathematical bound more precise. Working with AI agents in a structured, human-directed process, they explored more than 100 candidate approaches, many involving lengthy calculations with higher-order tensors. This produced a new mathematical criterion that became the foundation of the final proof.
The authors then discovered that the new criterion could be applied recursively. Each improved bound could be fed back into the same argument to produce a still tighter one. In the October 1, 2026 version of the paper, this reduced the dependence on dimension to the iterated logarithm. This function grows extraordinarily slowly, but it still grows with the dimension and therefore did not yet fully resolve the conjecture.
The final step was to repeat the entire recursive refinement, rather than applying it only once. This idea was inspired by the authors’ earlier work on fast algorithms for linear programming. The authors proved that the small errors introduced at each stage shrink fast enough that their total remains bounded by a universal constant, independent of the dimension. This eliminated the dependence on the number of dimensions and completed the proof in the October 4, 2026 version of the paper.
The recursive method, its repeated application, and the proof that the accumulated errors remain bounded are original contributions of the researchers. AI tools helped explore candidate approaches, carry out detailed derivations, prepare the manuscript, and perform internal proof checks. The researchers determined the mathematical direction, selected the promising ideas, assembled them into the final argument, and verified the proof.
Independent confirmationOn the same day that the authors posted their complete proof, a separate group of leading experts posted a full proof (opens in new tab) that they developed in parallel. Their work built upon the findings in the first version (opens in new tab) of the Microsoft paper, calling it “the crucial step in the proof.” A third proof (opens in new tab) which appeared separately on the same day relies on a closely related idea. In addition, a Lean 4 formalization (opens in new tab) was subsequently released, combining the key criterion from the first Microsoft paper with additional research.
Independent proofs are among mathematics’ strongest forms of validation because they demonstrate that a result does not rest on a single chain of reasoning. The subsequent appearance of alternative approaches and formal verification further strengthens confidence that the field has uncovered a genuine underlying principle rather than a fragile technical argument. The authors have also shortened their proof and verified the streamlined argument in Lean; they plan to release both after final review. Together, these developments provide complementary forms of confirmation.
Human-AI guidanceIn the process of proving KLS, this research also demonstrates the power of having researchers direct AI rather than simply asking AI to solve a complex problem through a single end-to-end run of a frontier model or agent. The authors used an agentic search process with iterative, human-assisted “pruning”: As multiple agents explored variations of an idea in parallel, the researcher examined their intermediate result, eliminated unpromising directions, and passed the most promising result to the next round. Roughly one hundred research trajectories were explored before the key breakthrough emerged.
In a separate track of the project, the authors made repeated unguided attempts to solve the same problem with several leading AI models, using roughly the same total token budget as the main process. That effort fell short of the goal. The eventual success depended on both the capabilities of the AI systems and the mathematical judgment of the researchers.
ConclusionTaken together, resolving KLS settles a central problem in high-dimensional geometry. A wide range of results that previously relied on the conjecture as an assumption can now be stated unconditionally. The proof therefore provides a stronger foundation for research in convex geometry, probability, and theoretical computer science.
The work also demonstrates how carefully guided human-AI collaboration can contribute to genuine scientific discovery while reinforcing, rather than replacing human insight. It highlights the continuing importance of scientific judgment in identifying promising directions, recognizing conceptual connections, and seeing when a partial insight points toward a deeper result. The authors are now simplifying the proof and extending its formal verification so that the mathematics and theoretical computer science communities can more easily examine the result and build on it.
AcknowledgementsThank you to our co-author Zhao Song for his contributions to this work, and Victor Reis (opens in new tab) and Ishai Menache for helpful discussions throughout the project.
Opens in a new tabThe post Solving a 31-year-old grand challenge in mathematics – the KLS conjecture appeared first on Microsoft Research.
Proof-based general-purpose computing with Magnetar
- A new form of computing. Zero-knowledge proofs let you run a workload and receive, alongside the result, a short proof that the program ran correctly. Anyone can check the proof in milliseconds, without rerunning the program, without trusting whoever ran it, and without needing data on which the program was run.
- Three grand challenges have stood in the way of broad adoption: making proofs fast enough to produce, making them available to normal programs without cryptographic expertise, and resting them on cryptography that will still be trusted many decades from now.
- Magnetar makes major progress on all three challenges. It proves programs written in Rust, Go, or C, runs 10 to 39 times faster on average than the prior fastest comparable systems, and rests its security only on standard hash functions. It is the latest result of more than a decade of research at Microsoft Research on making zero-knowledge proofs practical.
Every result a computer produces comes with an implicit question: did the program run as claimed and produce correct outputs? Today, answering that question often requires a degree of trust alongside verification. We trust the operator, the vendor, the counterparty, or the agent, and we rely on contracts, audits, and logs to establish that trust.
Zero-knowledge proofs offer a fundamentally different approach to establishing trust. A program runs, and along with its output it produces a short proof, a cryptographic certificate that this program, on these inputs, produced this output. The proof is small and fast to check. Whoever receives it can confirm the result in milliseconds without rerunning the program, and can pass the proof on to anyone else who needs the same assurance. With the zero-knowledge property, the proof also reveals nothing about inputs that should stay private.
This changes what correctness means in the real world. A result no longer has to be trusted because of who produced it. It can be checked because of what accompanies it. An AI agent acting on someone’s behalf could attach a proof that each action complied with the policy it was given. Regulated financial institutions could prove that transactions were processed correctly without making their transactions public. These become possible once producing a proof is as routine as running the program.
Three grand challengesZero-knowledge proofs were invented in the 1980s, and the ideas behind proving program execution have been around nearly as long. What has kept them from becoming a general tool is three challenges that have resisted being solved at the same time.
Efficiency. Producing a proof has historically taken millions of times more computation than running the program itself. Years of research have brought that overhead down, but the fastest systems have achieved their speed by making compromises elsewhere.
Generality and usability. Many proof systems require developers to manually rewrite their program as a cryptographic circuit, which is a fixed arithmetic description of that exact computation. This takes months. Every change to the application means a new circuit, and the translation is itself a frequent source of security bugs. A general-purpose alternative is to prove the execution of an ordinary processor, so that any compiled program can be proven as written. Until recently, that generality came at a steep cost in speed or security.
Security. A proof is only as strong as the cryptography beneath it. Many proof systems rest on elliptic curves, which can be broken by a sufficiently powerful quantum computer. Microsoft is addressing this problem with its Quantum Safe Program, which will move its products and services to post-quantum cryptography, a goal it recently accelerated to 2029. Post-quantum proof systems already exist, but some rest on lattice assumptions, and most rest on new cryptographic functions invented to reduce the cost of proving. These new cryptographic functions have not faced the decades of scrutiny given to the cryptography protecting today’s internet, and some rest on mathematical conjectures about error-correcting codes that have not been proven. They have bought their speed with strong assumptions.
More than a decade of workFor more than a decade, Microsoft Research has worked on making zero-knowledge proofs and verifiable computing practical. Early research followed two approaches, both practical only for small programs. The first (opens in new tab) (opens in new tab)lets a verifier check a long computation through a short conversation with the prover using a technique called the sum-check protocol (opens in new tab), but this applied only to computations with a particular layered structure. The second (opens in new tab) (opens in new tab)used elliptic-curve cryptography to create very small proofs for almost any computation. But this requires a one-time trusted setup involving secret information which, if leaked, could allow anyone to forge proofs. This approach is also vulnerable to quantum computers.
In 2019, a new zero-knowledge proof system called Spartan returned to the sum-check approach and made it work for general-purpose programs without a trusted setup. Spartan can also be instantiated with different cryptographic building blocks. That flexibility is what later allowed the proof system to be built on hash functions alone. A sequence of results then filled in the remaining pieces:
- Phalanx (opens in new tab), in 2021, showed how to exploit repetition, such as a processor running the same instruction cycle on every step, to prove computations an order of magnitude faster than Spartan. Magnetar’s processor description is built on this idea.
- Nova (opens in new tab), HyperNova (opens in new tab), NeutronNova (opens in new tab), Neo and SuperNeo (opens in new tab) showed that stopping Spartan’s proving early, and folding the partial results together, makes it practical to prove computations of any size at far lower cost.
- Brakedown (opens in new tab), in 2021, introduced a fast hash-based commitment scheme and combined it with Spartan, giving the first Spartan variant whose security rests on standard hash functions.
- Spice (opens in new tab), Lasso (opens in new tab), Nebula (opens in new tab), and then Twist and Shout (opens in new tab) made the memory operations at the heart of a general-purpose processor inexpensive to prove, and Jolt (opens in new tab) showed that combining them with Spartan can efficiently prove the execution of a complete processor.
- Earlier this year, Vega (opens in new tab) applied this line of work to digital identity, proving facts from a government credential in under a tenth of a second.
- More recently, BinarySpartan (opens in new tab) brought Spartan to binary fields, where a processor’s logic and standard hash functions can be efficiently expressed. BinarySpartan also relies on Ligerito (opens in new tab), a Brakedown-style commitment scheme with much shorter proofs, and on ring switching (opens in new tab), which makes it efficient to commit to vectors of bits.
Each of these solved a piece of the problem. Magnetar is where the pieces come together.
MagnetarMagnetar is a zero-knowledge virtual machine, or zkVM, and it makes major progress on all three challenges at once.
Figure 1. Magnetar proves a program’s execution and returns a small proof that anyone can check, with security resting only on standard hash functions.It is general-purpose. Magnetar proves the execution of RISC-V, an open processor architecture with mature compilers for every mainstream language. A developer writes a program in Rust, Go, or C, compiles it as usual, and Magnetar proves it as written. Consider what this means in practice. A business policy, a data pipeline, or an interpreter for the Ethereum Virtual Machine (EVM) is just a program, and the engineers who already maintain it can prove their executions exactly as written. As a result, proving stops being a specialist activity.
It is faster than prior systems in the same class. Magnetar proves 7.2 to 8.7 million processor instructions per second on a laptop. Across 15 benchmark programs, from cryptographic hashing to token transfers inside a small EVM interpreter, it is 10 to 39 times faster on average than the fastest existing zkVMs on the same machine, and more than 300 times faster than two others on a token-transfer benchmark. An optimized Magnetar proves 9.5–12.8 million cycles per second, which increases each speedup reported here by about 1.3×. Its proofs verify in about 12 milliseconds.
Its security rests on standard cryptography. Magnetar depends on a standard hash function such as SHA-256 or SHA-3, the same primitives that already protect every organization’s systems. It requires no elliptic curves, no lattices, no special-purpose hash functions, no trusted setup, and no unproven conjecture about error-correcting codes. Hash functions are the most thoroughly studied primitives in the cryptographic toolkit, and they are the basis of the stateless hash-based signatures that NIST standardized as its own conservative post-quantum option. For a system that must still be trustworthy in many decades to come, relying only on standard hash functions removes a cryptographic migration from the roadmap.
The same design decision pays a second dividend. Checking a Magnetar proof consists mostly of evaluating standard hash functions and binary-field arithmetic, which are exactly the operations Magnetar proves efficiently. Verifying a proof is therefore an ordinary workload for it. This is what allows Magnetar to prove computations of any size, by proving them in pieces and then combining the pieces. This also allows Magnetar to prove a long-running system such as a ledger or a blockchain.
The technical paper gives the full design, security analysis, and measurements.
Looking aheadMagnetar advances zero-knowledge proofs and verifiable computing to a stage where they can be built on the cryptography that organizations already trust, at speeds that make it usable, by engineers who write normal code. Magnetar enables a future where cryptographic proofs are a routine output of computation. We expect that technology leaders will stop asking whether proofs of execution are practical and focus instead on which results would they prefer to verify rather than taking on trust.
If you’re interested in exploring how zero-knowledge proofs can support your use case, we’d love to hear from you (opens in new tab).
Opens in a new tabThe post Proof-based general-purpose computing with Magnetar appeared first on Microsoft Research.
Simulating student learning for better AI tutoring
- StudentSim is a research system for evaluating and improving AI tutors before they are tested with real learners; it is not a product, is not deployed in classrooms, and is not intended to replace teachers
- AI tutoring tools need feedback from students to improve, but collecting that feedback through human studies is slow, costly, and difficult to scale
- StudentSim trains personalized AI student simulators from publicly available, de-identified learner records, with one simulator adapted to each student
- The work evaluates simulators along two education-grounded dimensions: whether they respond like a specific student, and whether they update after tutor guidance
- Across chess, second-language English writing, and mathematics, StudentSim outperforms leading baselines and provides a useful feedback signal for training an AI tutor
In research settings, teams are exploring how AI-powered tutoring systems can help explain a math problem, suggest an edit to a writing assignment, or guide a learner through a chess position. Yet the quality of teaching cannot be measured only from the tutor’s answer. A tutor needs to know what happens after the guidance reaches a student. Did the student correct the original mistake? Did the hint help them reason toward the answer, or did it give away too much? Would a different student need a different kind of support? In today’s AI tutor research, answering these questions often requires real learner studies. Those studies are valuable, but they are slow and costly to run, which makes it hard to use student feedback during fast model iteration.
StudentSim is an AI system that creates personalized simulators of individual students, modeling how they answer questions and how their answers change after receiving a tutor’s guidance. The goal is to make student feedback more available during AI tutor development. This research also introduces StudentSimEval, a benchmark and evaluation protocol spanning chess, second-language English writing, and mathematics.
Importantly, the goal is not to replace teachers or classroom learning. Instead, StudentSim helps researchers evaluate and improve AI tutoring systems during the research and development process before testing them with real learners. The research uses only publicly available learner datasets released for research use with de-identified records; no personally identifying information is used at any stage.
Figure 1: AI tutors need feedback from students to improve. Student simulators can approximate part of that feedback during research-stage training and evaluation. They provide an early-stage research signal, not a replacement for testing with real learners. Measuring what a student simulator needs to doA simple way to build a student simulator is to ask a large language model to role-play a learner. For example, a prompt might say, “You are an elementary-school student who struggles with math.” The model can change its tone and express uncertainty. The harder part is preserving the learner’s actual ability boundary. A model may speak in a young student’s voice while still relying on knowledge and reasoning far beyond that student’s level. For training an AI tutor, this can produce misleading feedback: the tutor may appear effective because the simulated student is too capable.
Traditional student modeling approaches face a different limitation. Knowledge tracing and behavior prediction models are grounded in real learning records and can estimate a student’s current knowledge, skills, and learning habits. Many of these models, however, were not designed to take natural-language tutor guidance as input. They can help describe what a student has done, but they do not always tell us how a student would respond to a particular explanation, hint, or Socratic question.
StudentSim defines the target through two dimensions inspired by education research. Educators often distinguish what a learner can do independently from what they can do with support. Vygotsky’s zone of proximal development captures this gap between current independent performance and performance with guidance. StudentSim turns this idea into two evaluation goals. Behavioral fidelity measures whether a simulator matches a specific student’s recorded responses on held-out problems. Guidance responsiveness measures whether the simulator updates its response after reading tutor guidance.
Figure 2: StudentSimEval evaluates whether a simulator matches a student’s independent behavior and whether it responds to guidance.These two goals are complementary. A simulator that only matches past student behavior may be useful for prediction, but it does not show whether a teaching intervention helped. A simulator that always follows the tutor may respond fluently, but it may fail to represent a particular student’s strengths, weaknesses, and likely mistakes. AI tutor training needs both signals: the learner’s starting point and how guidance affects the learner’s performance.
Training personalized simulators from sparse student recordsStudentSimEval is built entirely from publicly available learner datasets that are released for research use with de-identified records. The records carry no personally identifying information, and each personalized simulator is trained only on the de-identified records of the student it represents.
Real student data is often sparse at the individual level. A learner may contribute only a small number of essays, answers, or interactions. Training a separate simulator from scratch for each student would risk overfitting and would require each model to relearn common patterns in the domain.
StudentSim uses a two-stage training pipeline. In the first stage, the model trains on records pooled across many students in the same domain. This stage learns common response formats, common errors, and common ways students revise after guidance. In the second stage, the model is specialized on one student’s own records, producing a personalized simulator for that student.
Figure 3: StudentSim first learns shared student behavior from pooled records, then adapts to each student’s own data.The evaluation covers three domains with different response types. In chess, the simulator predicts a player’s next move and how that player may revise after coaching. In second-language English writing, it generates essays with error patterns matching a learner and predicts corrections after teacher feedback. In mathematics, it predicts a student’s multiple-choice answer and whether the student updates after an explanation. Across these domains, StudentSimEval uses the same principle: compare all methods on the same per-student training records and the same held-out evaluation records.
StudentSim outperforms leading baselines across the three domains. In chess, for example, StudentSim reaches behavioral fidelity of 0.5150 and guidance responsiveness of 0.9067. GPT-5.4 reaches 0.2316 and 0.7186, respectively. Maia2, a chess-specific human move prediction model, reaches 0.4535 and 0.2721. The comparison illustrates the difference between the baseline families: GPT-5.4 can respond to guidance but does not reliably match a specific student’s behavior, while Maia2 predicts human chess moves more closely but lacks a natural-language guidance pathway.
Figure 4: In chess, StudentSim improves both behavioral fidelity and guidance responsiveness compared with leading baselines. Using simulated feedback to improve AI tutorsThe research also tests whether a trained student simulator can provide feedback for tutor optimization. In a chess tutor reinforcement learning study, the tutor receives a position and a real student’s previous wrong move, then generates guidance. The student simulator’s own parameters are held fixed during this process, and it reads the guidance and produces a revised move. The reward is based on the improvement in move quality from the student’s previous move to the simulator’s revised move, using Stockfish as an independent chess engine signal.
Figure 5: StudentSim can provide feedback during AI tutor reinforcement learning by estimating how a student response changes after guidance.The study compares three conditions: a supervised fine-tuned tutor with no reinforcement learning, a tutor trained with GPT-5.4 as the simulated student reward source, and a tutor trained with StudentSim as the reward source. All conditions use the same tutor policy, supervised starting point, and reinforcement learning setup. Human chess experts then evaluate the tutor responses blind to condition.
The StudentSim reward produces the highest-rated tutor on all reported dimensions. The tutor trained with StudentSim reaches 90.5% accuracy, compared with 75.7% for the no-RL baseline and 71.6% for the GPT-5.4 reward condition. It also receives higher ratings for guidance quality and personalization. The result is a proof of concept that student simulators trained on real learner data can provide useful feedback for improving AI tutors.
Looking aheadStudentSim does not remove the need for real learners in education research. Human studies remain essential for measuring learning outcomes, safety, usability, and classroom fit. The role of student simulators is earlier in the development process: they can help researchers test tutor strategies, compare models, and refine training signals before moving to more expensive studies with real students.
The broader direction is to build AI tutor systems that learn from student reactions, not only from expert-written explanations or static correctness labels. StudentSim shows one path: use real learning records to train personalized simulators, evaluate those simulators on both independent behavior and response to guidance, and use them as scalable feedback sources for tutor training. Future work can extend this approach to longer learning trajectories, additional subjects, richer student profiles, and studies that connect simulated feedback with real learning outcomes.
The paper and the StudentSim training and evaluation code, including the StudentSimEval protocol, are publicly available.
Sources Paper Code Opens in a new tabThe post Simulating student learning for better AI tutoring appeared first on Microsoft Research.
Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses
- Harnessed Agentic RL: Microsoft Research Asia introduces a training paradigm in which the same agent harness used in deployment participates directly in reinforcement learning, removing the need to reimplement the agent inside the training framework.
- Lightweight by design: Agent Lightning v1.0 delivers a complete agent RL control plane in roughly 3,500 lines of code.
- Native Kubernetes support: agents run as standard Kubernetes jobs on self-managed clusters, cloud Kubernetes, or local infrastructure, with no dependency on paid commercial sandbox services.
- Data-efficient training recipe: an end-to-end coding agent pipeline raised Qwen3.5-9B from 41.8% to 56.4% Pass@1 on SWE-bench Verified, a 14.6 percentage point gain, using only about 6,000 training samples based on open sourced dataset.
AI agents have evolved from single models to complex full-stack systems built from models, tools, and execution environments. Their capabilities increasingly depend on the agent harness that coordinates them from outside the model. Reinforcement learning (RL) is an approach where AI systems learn through trial and error, guided by rewards and penalties for their actions. RL can make those agents better, but most agent RL systems require developers to reimplement the agent inside the training framework. That is costly, and it means the agent being trained is not quite the agent that gets deployed.
To address this, researchers at Microsoft Research Asia have introduced the Harnessed Agentic RL training paradigm and open-sourced a fully rebuilt Agent Lightning v1.0 (opens in new tab). Compared with the original, Agent Lightning, v1.0 puts more emphasis on staying lightweight, on integrating with real harnesses, and on a complete, reproducible agent RL training pipeline.
Agent Lightning v1.0 was rebuilt around Harnessed Agentic RL, with key improvements:
- Lightweight: the entire framework is about 3,500 lines of code. Agent Lightning v1.0 implements a complete Harnessed Agentic RL system in a codebase that is small and clear enough to understand, modify, and extend.
- Training on a real agent harness: agents reach the model through the large language model (LLM) proxy in Agent Lightning v1.0, leaving existing harness code unchanged.
- Native Kubernetes support: agents run directly as Kubernetes jobs, without external commercial sandbox services. Self-managed clusters and local infrastructure alike can support rollouts at scale.
- A complete coding agent training example: an end-to-end pipeline built on Qwen3.5-9B raised Pass@1 on SWE-bench Verified from 41.8% to 56.4%, an absolute gain of 14.6 percentage points, using only about 6,000 training samples.
Traditional agentic RL assumes the training framework owns the interaction loop with the environment. In a ReAct-style loop, the model generates an action, the environment returns an observation, the observation is appended to the context, and the model generates the next action, so the whole rollout maps onto one continuous token trajectory. Early RL systems such as verl, AReaL, and slime were built this way, which meant training an agent required rebuilding its loop inside the RL framework.
Real harnesses have outgrown that assumption. Coding agents such as mini-SWE-agent, OpenHands, OpenCode, Claude Code, and Codex each bring their own context management, tool protocols, execution logic, and dependencies, as do general-purpose agent systems. Rebuilding one for training is expensive, and the rebuilt agent may no longer behave in the same way as the deployed agent.
Agent Lightning takes a different route. It places an LLM proxy between the agent and the model. The agent continues to run as before: simply point the endpoint that previously called the model API at Agent Lightning, and the training framework can observe and record its model calls. In v1.0, the researchers go further and formally define this paradigm as Harnessed Agentic RL: whichever agent harness is used in deployment is the harness that takes part directly in reinforcement learning during training (Figure 1).
Figure 1. Traditional agentic RL compared with Harnessed Agentic RL. In traditional agentic RL, the training framework manages the environment and the agent loop. In Harnessed Agentic RL, the harness manages both. Four challenges in training with real harnessesA core difference between Harnessed Agentic RL and traditional agentic RL is that the environment interaction loop is handled by the agent harness rather than the training framework. The training system can only observe a series of LLM request and response pairs, so a single rollout may be split into a variable number of training samples. This brings four key challenges:
- Retokenization and sample merging: Harnesses keep context as text, but RL training needs the token IDs sampled during the rollout. Passing text through the chat template and tokenizer again can shift token boundaries, so adjacent calls cannot always be merged into one sample.
- Advantage calculation: Retokenization, subagents and context summarization can split one rollout into several samples. Computing baselines and advantages directly at the sample level causes rollouts that produce more samples to be counted repeatedly, which alters the original statistical relationships at the rollout level.
- Loss normalization: Averaging loss by sample count gives more weight to rollouts that produce more samples. Since sample count is often just a product of harness behavior, loss normalization must also avoid being distorted by it.
- Training backend scheduling: Sample count and length are known only after the harness finishes, while GPU counts and data/tensor parallel configurations are usually fixed. The backend has to map a variable workload into fixed resources.
In system design, Agent Lightning v1.0 treats simplicity as its first principle. The entire framework is about 3,500 lines of code, with three core components: the API Gateway, the Rollout Controller, and the Customized Trainer (Figure 2).
The API gateway stores rollouts, models, and events, and serves as an OpenAI-compatible LLM proxy. It links every model call from the harness to its rollout and records the prompts, responses, and log probabilities that training needs. The rollout controller starts and manages agent execution, either as local processes or as standard Kubernetes jobs, keeping agent execution separate from the trainer. The customized trainer, built on verl, creates rollouts, waits for them to finish, collects samples, and assembles the final training samples through a sample adapter. As a result, for an existing agent harness, simply pointing the model endpoint at the Agent Lightning proxy is usually enough to connect quickly to RL training.
Figure 2. The Agent Lightning v1.0 system architecture showing the API gateway, rollout controller, and customized trainer. Collocated async RLRollout times vary widely across agents. Synchronous RL waits for the slowest agent in a batch and leaves GPUs idle, while fully asynchronous RL raises utilization but needs separate GPU pools for rollout and training. In response, Agent Lightning v1.0 introduces Collocated Async RL, which lets rollout and model updates share the same set of GPUs.
Once the system has collected enough rollouts, the update begins: the API Gateway pauses accepting new requests and waits for requests already in progress to finish, and rollout resumes after the update completes. The entire state transition is transparent to the external agent harness. In experiments, this approach achieved about a 2x end-to-end speedup over synchronous RL while using fewer GPUs than conventional asynchronous RL (Figure 3).
Figure 3. Synchronous RL, asynchronous RL, and Collocated Async RL compared. Collocated Async RL raises utilization while occupying fewer GPUs. Running agents on KubernetesCollecting enough rollouts means running many agents at once, which consumes substantial CPU, memory, and compute resources. Other Harnessed Agentic RL frameworks often host those agents on commercial sandbox services such as Modal Sandbox or E2B, where cost climbs quickly with scale. Instead, Agent Lightning v1.0 runs them as standard Kubernetes jobs, reusing existing self-managed clusters, cloud Kubernetes, or local infrastructure (Figure 4). Existing compute resources are used more efficiently, large rollouts cost less, and the whole pipeline stays open source and reproducible.
Figure 4. The Rollout Controller in Agent Lightning v1.0 provides native Kubernetes support, running agents directly as standard Kubernetes jobs. 6,000 training samples, a 14.6-point performance gainTo test the approach, researchers built a full pipeline on SWE-smith, mini-SWE-agent, and Qwen3.5-9B, covering data cleaning, environment construction, reward-hacking safeguards, and RL training. The training set holds about 6,000 samples and needs no large-scale compute. RL training alone raised Qwen3.5-9B from 41.8% to 56.4% on SWE-bench Verified, a gain of 14.6 percentage points.
The coding agent experiments further confirm the earlier analysis of two challenges: advantage calculation and loss normalization. Compared with sample-level handling, rollout-level advantage combined with rollout-level normalization achieves a higher validation reward and keeps policy entropy more stable during training (Figure 5).
Figure 5. Pass rate and policy entropy for Qwen3.5-9B on the SWE-smith validation set. Technical report GitHub project Opens in a new tabThe post Agent Lightning v1.0: A 3,500-Line Lightweight Agentic RL Framework for Training Agents with Real Harnesses appeared first on Microsoft Research.
Forecasting space weather risks on power grids
- End-to-end forecasting: A machine learning pipeline uses forecast-time solar-wind information to generate location-specific risk estimates for 66,935 substations in the continental United States.
- Physics and place: The system combines Auroral Electrojet (AE) and Disturbance Storm Time (Dst) forecasts with local latitude, geology, and ground conductivity.
- Advance warning: The pipeline detected nearly 80% of major space-weather events during the evaluation period and can warn grid operators 30 to 60 minutes before a specific risk appears.
During the May 2024 geomagnetic storm, utilities across North America prepared for possible impacts as auroras extended far beyond their usual range. The storm would degrade GPS accuracy and satellite operations, impacting farm operations in North America and Europe. Modern society depends on reliable electric power, yet extreme space-weather events can induce currents in transmission networks that damage equipment and increase operational risk. The challenge is not only knowing that a storm is approaching, but estimating when and where its effects could be most severe with enough warning for grid operators to respond.
During my summer internship at Microsoft Research, I developed a machine learning system that forecasts space-weather risk across 66,935 substations in the continental United States. The system measures how the sun’s activity can affect the earth’s magnetic field and, ultimately, the power grid. It combines solar-wind observations, forecasts of the Auroral Electrojet (AE) and Disturbance Storm Time (Dst) indices, physics-informed constraints, local geological conductivity, and grid-infrastructure data to produce location-specific risk estimates 30-60 minutes ahead of potential impact.
Building an end-to-end prediction pipelineSpace-weather prediction is difficult because it spans several coupled systems. The solar wind changes rapidly, its interaction with Earth’s magnetosphere is irregular, and the resulting ground effects depend on local conditions. Regions with resistive bedrock can experience stronger geomagnetically induced currents (GICs) than regions with more conductive geology. Transmission-line orientation, latitude, and other power-system characteristics further influence the exposure of individual assets.
The pipeline addresses this complexity in three stages, shown in Figure 1. First, solar-wind measurements from the L1 Lagrange point are used to generate forecasts of the AE and Dst indices, while geological conductivity and location features are assembled for each substation. Second, a gradient-boosting model combines these forecasted and location-specific inputs to estimate dB/dt, the rate of magnetic-field change associated with GIC risk. Finally, the resulting predictions are converted into location-specific risk estimates and aggregated into a continental risk assessment. A system of 50 AI agents helped explore features, validation strategies, and model configurations across the pipeline. Only public data sources were used, included NASA OMNI and NASA-aggregated Kyoto World Data Center data, INTERMAGNET and U.S. Geological Survey magnetometer observations, and GridSFM-derived grid data.
Figure 1. End-to-end forecasting pipeline from L1 solar-wind measurements to location-specific risk estimates across 66,935 U.S. substations. Evaluating geomagnetic forecasts and infrastructure riskThe AE predictor was designed to forecast rare, high-intensity geomagnetic activity that drives infrastructure risk. In the 2020-2026 evaluation period, the model produced forecasts spanning nearly the full observed range of AE activity and outperformed several empirical solar-wind-based approaches. The Dst predictor provided an additional signal describing large-scale geomagnetic storm strength. During the most geomagnetically active periods of the 2020-2026 evaluation period, the machine-learning model outperformed the Burton equation on 62.2% of individual hours. The model also produced a substantially wider prediction range than Burton-style approaches and improved severe-event detection in the end-to-end forecasting system by 1.2 percentage points when combined with AE forecasts.
PredictorResultBaseline comparisonImpactAE410.2 nT RMSELower RMSE than empirical and solar-wind-only baselinesCaptures extreme-event activity used by the GIC forecasting systemDst7.2 nT RMSEOutperformed Burton on 62.2% of peak-activity hours and provided wider prediction range+1.2pp severe storm detection when integrated into final GIC systemThe GIC risk stage was evaluated differently. There is no equivalent widely deployed operational system that provides a direct industry benchmark for this calculation, so the machine learning model was compared with simple linear regression. As Figure 2 shows, the system achieved detection rates of 76.5% for major events (≥10 nT/min), 81.2% for severe events (≥20 nT/min), and 64.1% for extreme events (≥50 nT/min). False-alarm rates increased with storm severity, reflecting the trade-off between missed events and cautious alerts. Performance varied by latitude, with the highest detection rates at northern stations where geomagnetic activity is strongest.
Figure 2. Detection rates across storm-severity thresholds for regionally different substations, with a comparison between the average POD for each severity tier. Translating predictions into continental risk assessmentsThe final stage translates predicted geomagnetic activity into location-specific estimates of dB/dt, the rate of magnetic-field change associated with GIC exposure. Rather than issuing a single alert for the continental United States, the system combines storm conditions with each substation’s latitude and geological factor. This produces continuous risk estimates that can distinguish lower-risk locations from areas where resistive geology can amplify ground-level effects.
Figure 3 illustrates the output for a representative major-storm scenario. The map is a demonstration of the model’s continental-scale output, not a record of a live operational event. It shows how a grid operator or planner could move from a broad space-weather warning toward a more targeted view of which locations may warrant closer analysis. The current pipeline produced estimates for all 66,935 substations in approximately 333 milliseconds during measured inference, allowing many scenarios to be evaluated quickly.
Figure 3. Demonstration of a continental GIC risk assessment under a representative major-storm scenario. Colors indicate modeled risk levels across 66,935 substations. Implications and looking forwardThis work demonstrates how physics-grounded machine learning could support more specific and timely assessment of space-weather exposure. Earlier, location-specific information could help utilities prioritize engineering review and consider targeted protective actions, such as adjusting reactive-power reserves or temporarily reconfiguring parts of the network. Further validation with utilities and operational data would be needed before the system could be used in grid operations.
The project also connects with broader Microsoft Research work on AI for power systems. The open grid-data pipeline provided realistic U.S. transmission models, while GridSFM applies deep learning to AC optimal power flow for fast scenario analysis. Together, these efforts point toward richer planning and resilience workflows that combine hazard forecasts, grid topology, and power-flow analysis.
- Extend forecast horizons: Explore temporal-transformer approaches that capture longer-range patterns in solar-wind data and move beyond the current 30-60-minute window.
- Scale internationally: Adapt the system to additional regions while accounting for different geological conditions and grid topologies.
- Integrate with grid operations: Evaluate how forecasts could support existing decision-making and engineering-review workflows before considering higher levels of automation.
- Provide transformer-level risk: Add asset-specific characteristics to move from substation-level estimates toward more granular assessments of critical equipment.
Special thanks to my mentors Weiwei Yang and Spencer Fowers for their guidance throughout this project, and to Amber Hoak, Weishung Liu, and Andrea Britto for their editorial and technical feedback.
Opens in a new tabThe post Forecasting space weather risks on power grids appeared first on Microsoft Research.
Introducing Quine: An AI research system designed for the complexity of biology
- Quine (opens in new tab) is a research effort to create a multimodal world model of biology and an interactive harness connecting models, scientific tools, literature, and researchers.
- In collaboration with researchers at the Broad Institute of Harvard and MIT, we have used this system to prioritize compounds predicted to drive therapeutic tumor-state shifts and validated several top-ranked candidates across multiple wet-lab assays.
- The Quine Fellows program (opens in new tab) will give a cohort of scientists access to the system and an opportunity to accelerate their own research and provide scientific feedback.
- Quine is experimental research technology intended only for research, not clinical or medical use, and its outputs may be incomplete or inaccurate and require review by qualified researchers and appropriate scientific and experimental validation. As the technology matures, we expect to expand access through products like Microsoft Discovery (opens in new tab).
For more than two decades, Microsoft Research has worked at the intersection of computation and biology. Our research has spanned immunology, virology, genomics, biomedical imaging, cell biology, and protein engineering. That work has produced foundational methods, new science, and technology that reached the clinic, from rare and infectious disease diagnosis to cancer biomarker detection.
Across that work, one lesson has become increasingly clear: biology does not divide itself into the neat boundaries our models and tools often do. Genes influence proteins; proteins interact within cells; cells organize into tissues; and experiments continually reshape what scientists know and what they choose to ask next. Making progress on the hardest biological questions therefore requires more than increasingly capable models of individual datasets or tasks. It requires systems that can connect knowledge across scale and modalities, reason about experiments and evidence, and participate in the iterative process through which science advances.
Today, Microsoft Research is introducing Quine (opens in new tab), a research effort designed to work across those boundaries, reflecting our long-term vision for a discovery system that evolves through scientific use. Quine brings together a world model of biology with a harness that connects scientific tools, literature, the wet lab, and the researchers using them.
The limits of experimentationEven as experimental techniques have improved and wet-lab throughput has increased, biology remains fundamentally constrained by time and complexity. Nature cannot be rushed, nor can it be derived from first principles. Experiments are slow, iteration cycles are long, and many of the most important questions involve interactions, combinatorial design spaces, and downstream effects that are simply too large to explore experimentally alone.
At the same time, advances in large-scale machine learning, particularly the emergence of general-purpose foundation models and reasoning models that can iteratively work through problems, suggest a new possibility. These systems are beginning to demonstrate capabilities beyond pattern recognition: integrating information across domains, reasoning over abstractions, and supporting iterative problem-solving. Just as importantly, many of the techniques developed for human language have proven remarkably adaptable to aspects of biology, enabling models to learn representations of biological systems across diverse data types and scales.
This raised a provocative question for us: what would it take to build a world model for biology? By world model, we mean a system that can represent the state of a biological system, predict how that state will evolve in response to interventions, and reason over the consequences of those interventions multiple steps into the future. Such a system would not replace experimentation. Rather, it would allow us to use computation to explore, propose, rank, and prioritize potential paths forward before committing scarce laboratory resources. Our north star is a future in which scientists, models, and experiments operate in a continuously accelerating loop.
What we’ve builtQuine is our first step toward that vision. It includes both a world model of biology and a harness that connects the model to orchestration and reasoning models, scientific tools, the literature, and the teams of scientists using them.
Figure 1: Quine’s two frontier components — a world model of biology and an interactive harness — sit inside a loop that begins and ends with the scientist. A question becomes a set of proposals, proposals become designs worth testing, and the experiment returns measurements that sharpen both the scientist’s next question and the model itself.Core to this effort is the world model, which learns shared representations across biological modalities and scales, including sequence, structure, function, cellular state, and imaging data. By training across these modalities jointly, Quine can use evidence from one modality to inform predictions in another, capturing relationships that would otherwise remain siloed or lost when separate single-domain specialist models are orchestrated by the system. This multimodal design reflects the reality of biology: understanding any one level requires context from the others. Rather than diluting performance, we find that learning across these connected representations strengthens it, enabling the model to generalize more effectively and to support a broader range of biological reasoning tasks.
Figure 2: Biology is multiscale and multimodal. (A) Biological evidence stacks from molecules to cells to tissues to patients, with scientific language running underneath all of it. (B) The modalities Project Quine trains on — genomics, proteins, chemistry, RNA and cell state, bioimaging — each observe a different span of that range, and they overlap. Reading down is the causal path from genotype to phenotype; reading across is physical scale. Because the model learns these representations jointly rather than orchestrating separate specialists, evidence at one level can inform predictions at another.By integrating this model in a unified system that facilitates computational experimentation and learns from iteration, researchers can generate predictions across modalities and reason about the consequences of biological interventions before they are tested in the lab. Critically, the world model doesn’t need to be perfect—indeed, it will never perfectly model biology—it simply needs to usefully inform experimental design.
What this looks like in cancer biologyA concrete example comes from our work on pancreatic ductal adenocarcinoma (PDAC), the most common form of pancreatic cancer and one of the most challenging to treat. In collaboration with researchers at the Broad Institute of MIT and Harvard, we have spent years developing and applying patient-derived ex vivo models to investigate a longstanding hypothesis: that tumor behavior and drug response depend not only on genetics, but also on transcriptional cell state. In PDAC, tumor cells can occupy different cellular states associated with how they respond to treatment.
With Quine, we have now put that hypothesis into practice, exploring whether non-genetic features of cancer, particularly cellular state, can serve as actionable therapeutic targets. We used Quine to predict and prioritize thousands of compounds based on their potential to shift tumor cells between therapeutically relevant states. In wet-lab studies focused on the transition from classical to basal cell states, Quine’s highest-ranked compounds produced the largest intended shifts across experimental assays. The entire process—from rapidly narrowing the compound search space to prioritizing a handful of promising candidates to be validated in the lab—took just one weekend, potentially saving months of experimental work and significant research costs. Notably, some of the strongest effects came from compounds with unexpected mechanisms of action, offering early evidence that AI can uncover new opportunities for drug repurposing and discovery.
While the reverse state transition proved more difficult (from basal to classical cell states), Quine predicted that available compounds would have this weaker effect. More notably, the experiments revealed something we did not fully anticipate: Quine predicted that several compounds would consistently move cells toward a distinct third phenotype—an observation borne out in the lab—suggesting that the pancreatic cancer cell-state landscape is richer than a simple classical-basal axis. The experiments not only tested the model’s hypotheses but also generated new ones.
Figure 3: Quine-guided exploration of pancreatic cancer cell states. In pancreatic ductaladenocarcinoma (PDAC) cell lines, top-ranked compounds produced the largest classical-to-basal transcriptional shifts as validated by wet lab experiments. Weaker reverse shifts and unexpected movement toward an additional phenotype were also consistent with Quine’s predictions.Our continued work here will use newly integrated RNA datasets and tasks to represent this richer landscape, strengthen state-transition predictions, and provide calibrated confidence estimates that help scientists prioritize the most promising hypotheses for wet-lab testing. And this is exactly the feedback loop Quine was built to support. Each round of experimentation improves our understanding of biology, and each new insight can become training signal for the next generation of the system.
Learning through discovery – an invitationWe created Quine to improve how we do science ourselves. As the system evolved, we embedded it in our ongoing scientific programs. Every research question, experimental result, and unexpected finding became an opportunity to refine the models, improve the workflows, and better understand where AI could meaningfully accelerate our research. Over time, the system became central to how we approached discovery across multiple domains, including in cancer biology, protein engineering, genomics, and bioimaging.
We’d like to continue building on that foundation by introducing Quine at this early stage. Quine grew out of our own research, but we don’t want its continued development to happen in isolation. Many of the questions that will determine where it goes next are not ones we can answer alone; they require the expertise, creativity, and perspectives of the broader scientific community. For this reason, we’re opening the Quine Fellows program to put the system directly into the hands of scientists working at the frontier of biology and medicine.
Applications for the first Quine Fellows cohort open nowBuilding responsibly will always come first. We believe progress in AI and biology must go hand in hand with safety, security, and responsible stewardship. We are taking a deliberate, phased approach to Quine’s development and access, with initial availability limited to the Quine Fellows program and select research collaborations. Ongoing internal review and built-in safeguards will help us maintain appropriate oversight as the system evolves. As the technology matures, we expect to expand access through products like Microsoft Discovery (opens in new tab).
It’s tempting to frame progress in AI for biology around benchmarks and leaderboards. Those matter, but the real test of Quine is what happens when it encounters science as it’s actually practiced. Can it be useful when evidence is incomplete? When facing questions genuinely never asked before? Can it help make the path from questions to discovery shorter?
Ultimately that’s the standard we care about. The ambition is for Quine to recede into the background of scientific practice, while the science moves faster in the foreground. The protagonists are not the model or the platform. They are the scientists, the experiments, and the discoveries that follow.
Opens in a new tabThe post Introducing Quine: An AI research system designed for the complexity of biology appeared first on Microsoft Research.
One year in: How Microsoft Research Asia – Singapore is advancing research, partnership and talent for real-world impact
On July 24, 2025, Microsoft Research Asia – Singapore (MSRA – Singapore) opened its doors as Microsoft’s first research lab in Southeast Asia. The launch built on more than two decades of collaboration between Microsoft Research Asia and Singapore’s universities, research institutes, and government agencies. It marked a shift to a closer, more integrated relationship aimed at supporting Singapore’s innovation ecosystem and advancing the development and beneficial use of AI.
“When we established Microsoft Research Asia — Singapore one year ago, our vision was not simply to open another research lab. It was to build a bridge connecting Microsoft’s frontier AI research with Singapore’s world-class universities, forward-thinking government agencies, and dynamic industry ecosystem. One year in, what gives me the greatest confidence is not just the research we have produced, but the depth of partnership we have built. That is the foundation on which lasting impact is built.”
— Dr. Lidong Zhou, corporate vice president at Microsoft and managing director of Microsoft Research AsiaEstablished in 1998, MSRA has a presence in Beijing, Shanghai, Vancouver, and Tokyo, along with Singapore. The Singapore lab strengthens the collaboration with local partners and extends the scientific and societal impact of Microsoft’s global research community.
Over the past year, the Singapore lab has grown its team, deepened partnerships, and explored how frontier AI research can create real-world value. Guided by its mission to advance frontier AI for humanity, its work spans four pillars: next-generation AI models and agentic systems, domain-specific AI for real-world impact, AI-native research practices, and ecosystem and talent development.
From frontier research to real-world impact – and backResearch has its greatest impact when it moves beyond the laboratory into practice: in health and medicine, enterprises, and the daily operations of organizations solving real problems. MSRA – Singapore aims to do just that through a two-way exchange: partners bring domain expertise, real-world data, and operational challenges, while the lab contributes frontier research and engineering capabilities. Together, they translate fundamental advances into practical solutions, while surfacing new questions that feed back into research in a kind of research-to-impact flywheel. Advances in AI foundations enable more capable models and agentic systems that can reason, plan, use tools, and collaborate with people. These capabilities create new opportunities for AI-native innovation. In turn, real-world applications reveal limitations and generate new questions, feeding insights back into foundational research and sustaining a continuous cycle of discovery, application, and learning.
Healthcare offers an early example of the lab’s research-to-impact model. Research on multimodal healthcare AI and self-evolving diagnostic agents is deployed and extended through partnerships across Singapore’s healthcare ecosystem. The lab is collaborating on multimodal and agentic AI approaches to support clinical decision-making, unlock insights from complex medical data, and advance research in disease diagnosis, risk assessment, and personalized care, with the longer-term goal of enabling meaningful real-world use.
Beyond healthcare, the lab is working with organizations across financial services, education, and technology to explore how frontier AI can address sector-specific needs and operational challenges.
“Over the past year, the lab has become an integral part of both Microsoft’s local ecosystem and Singapore’s broader innovation community. Its progress demonstrates the value of establishing a sustained research presence in Singapore.”
— Wee Luen Chia, managing director, Microsoft SingaporeSingapore has made AI a national priority through its National AI Strategy 2.0 (opens in new tab), which focuses on deploying industry-transforming AI, advancing frontier research in AI foundations, and developing responsible, socially beneficial applications. These priorities span areas including healthcare, education, finance, and Smart Nation (opens in new tab) services. MSRA – Singapore’s research portfolio aligns with these priorities, with a particular focus on building AI systems that are reliable and trustworthy across Southeast Asian contexts.
The Singapore Economic Development Board (EDB) has been a foundational partner. It supported the lab’s establishment and continues to support joint PhD training through the Industrial Postgraduate Programme (IPP). The relationship has since grown into strategic collaboration. In February 2026, EDB co-hosted an Executive Roundtable on AI in Logistics and Transportation, bringing together more than ten regional industry leaders to explore sectoral challenges.
“Singapore is building deep research capabilities to push the boundaries of AI and other frontier technologies. Microsoft Research Asia – Singapore is an important partner in this effort. Since its launch in 2025, the lab has brought Microsoft’s global research expertise closer to our ecosystem, partnering academia, industry, and government to advance frontier AI research and translate it into real-world applications in areas such as healthcare. MSRA is also developing the next generation of AI research talent who can create meaningful impact for our economy and society.”
— Low Jin Yong, senior vice president and co-head, Digital Industry SingaporeGovernment engagement extends beyond EDB. In July 2025, IMDA (opens in new tab) co-hosted Microsoft Research Asia Singapore Day (opens in new tab) — Advancing Industrial AI, a full-day event spanning healthcare AI, interactive world simulators, AI agents, and societal AI. MSRA – Singapore has also engaged with the Ministry of Trade and Industry and the Ministry of Health.
From enduring partnerships to shared discoveryIn addition to its deep ties with government agencies, MSRA has built strong connections with Singapore’s academic community, supporting more than 75 research projects with Singapore institutions since 2004. The opening of the Singapore lab has brought new engagements, with nine new projects in its first year with National University of Singapore (NUS) and Nanyang Technological University, Singapore (NTU) across healthcare AI, robotics, AI systems, and multi-agent systems.
A landmark example is the five-year Framework Research Agreement (opens in new tab) with NUS, announced in 2025, which provides a platform for joint AI research, talent development, and academic exchange. It defines shared research priorities in healthcare, societal AI, spatial intelligence, and data-intensive computing alongside an annual NUS–MSRA Academic Symposium that brings researchers from both organizations together to shape the collaboration agenda.
“This partnership creates new opportunities for frontier research, talent development, and knowledge exchange, strengthening connections across academia, industry, and the broader innovation ecosystem.”
— Professor Liu Bin, deputy president (Research and Technology), National University of SingaporeThe collaboration is already producing concrete research across several frontier areas. NUS Assistant Professor Mike Shou and MSRA’s Xinxing Xu are developing unified multimodal AI for healthcare. NUS assistant professor Yang You and MSRA distinguished scientist Furu Wei are studying systems foundations for diffusion-based language models. NUS assistant professor Jialin Li is working with MSRA on rigorous distributed agreement in multi-agent systems and verifiable machine learning systems.
“Whether during my visit to the Systems Research Group in Beijing or through our ongoing collaborations, I have been impressed by how deeply MSRA researchers engage with difficult problems and how openly they exchange ideas. For young faculty and students, it is a rare environment to explore ambitious ideas.“
— Jialin Li, Sung Kah Kay Assistant Professor, School of Computing, National University of Singapore, MSRA StarTrack ScholarThe NUS–Microsoft Research Asia AI Summer School has reached more than 300 students since its launch in 2025, attracting participants from Singapore, Southeast Asia, and beyond. Through three days of lectures, hands-on workshops, and small-group discussions, the program creates a forum for researchers and students to engage with emerging AI questions and scholars shaping the field. The 2025 program featured a keynote by Doug Burger, technical fellow and corporate vice president of Microsoft, on the AI revolution. Ahead of the 2026 Summer School, Lidong Zhou delivered a lecture on system intelligence, helping frame the conversations that followed.
“I have seen the NUS–MSRA partnership grow through sustained collaboration in AI research and talent development. What makes this partnership especially meaningful is the opportunity to bring together researchers and students from both communities to learn from one another, pursue new ideas, and create lasting impact together.”
— Tulika Mitra, dean, NUS School of ComputingBeyond NUS, Microsoft Research Asia — Singapore has built connections across Singapore’s research community, including with NTU, Singapore Management University (SMU) (opens in new tab), Agency for Science, Technology and Research (A*STAR), and other institutions. Research arising from these partnerships has already been published at leading venues, including the International Conference on Learning Representations (opens in new tab) (ICLR) and the Conference on Language Modeling (opens in new tab) (COLM).
From early exposure to research leadership: Cultivating Singapore’s next generation of AI talentFor Microsoft Research Asia, talent development is not a standalone program but a long-term investment in Singapore’s research and innovation capacity, supporting a comprehensive talent pipeline spanning internships, joint PhD supervision, visiting scholar programs, and fellowship initiatives. Through these opportunities, students and early-career researchers gain access to world-class mentorship, a curiosity-driven culture, frontier research challenges, and a global network connecting academia, industry, and real-world applications.
What makes this model distinctive is its continuity. Students may engage with Microsoft Research Asia through workshops or summer schools, gain hands-on experience through its flagship research initiative, Microsoft Research Asia Stars of Tomorrow Internship Program, or pursue doctoral research through joint PhD programs. Some participants later return as faculty collaborators, mentors, or research leaders. Together, these programs form a connected pathway that supports researchers at successive stages of development.
The IPP represents a sustained form of collaboration, enabling PhD students to conduct long-term research under joint mentorship from academia and industry. Qiming Huang, the first IPP student at MSRA — Singapore, is co-supervised by NUS professor Mike Shou and MSRA researcher Xinxing Xu. Through the program, Huang worked closely with researchers from both institutions, gaining exposure to academic rigor and industry-driven research. At CVPR 2026 (opens in new tab), one of the world’s leading conferences in AI and computer vision, Qiming Huang’s paper on RobotSeg – a foundation model for robot segmentation in image and video – was selected for an oral presentation.
“Working alongside researchers from both Microsoft Research Asia and NUS changed how I approach AI research. Beyond technical guidance, I learned to frame ambitious questions, collaborate across disciplines, and think about the real-world impact of research
— Qiming Huang, IPP student from National University of SingaporeNTU assistant professor Jianfei Yang, embodies the continuous collaborative opportunities enabled by Microsoft Research Asia. As a student, Yang’s team won the Beauty of Programming competition, an experience that inspired him to pursue a PhD and begin an academic career. Over the years, he watched many of his classmates and close friends grow into outstanding researchers through MSRA programs. Today, he collaborates with MSRA researchers in robotics and jointly supervises students through the IPP, creating for the next generation the same opportunities that inspired his own journey.
“Although I never interned at Microsoft Research Asia myself, it always felt close. Having the opportunity to work alongside a research lab that inspired so many researchers of my generation, and now to co-mentor students together, is both meaningful and rewarding.”
— Jianfei Yang, assistant professor, director of Multimodal embodied AI and Robotic Systems (MARS) Lab, Nanyang Technological UniversityNTU associate professor Ziwei Liu, himself a former MSRA intern, recently saw one of his students become the only Asia-based recipient of the Microsoft Research global Fellowship award for 2025–2026.
“As a former intern, I have seen how an environment that brings together world-class researchers, open scientific exchange, and ambitious young talent can shape the next generation of researchers. It is especially meaningful to see my own students now benefit from these opportunities.”
— Ziwei Liu, associate professor, College of Computing and Data Science, Nanyang Technological UniversityThe Stars of Tomorrow Internship Program has engaged more than 90 students from Singapore-based institutions over the years. For many students, these internships serve as a gateway to frontier research and help bridge the gap between academic study and real-world innovation.
For exceptional doctoral students, the Microsoft Research Asia Fellowship provides another pathway for growth, recognition, mentorship, and access to Microsoft’s global research community. Since the program began, 13 students from Singapore-based universities have received the fellowship, many of whom have gone on to become professors, industry research leaders, and technology entrepreneurs around the world.
The next phase: Scaling what worksAs Microsoft Research Asia — Singapore enters its second year, the focus is shifting from building foundations to scaling the strongest models of collaboration and demonstrating their impact more broadly.
One year ago, the lab opened with a mission to advance frontier AI in close partnership with the ecosystem around it. That mission is already taking shape through joint research, talent development, and the first partnerships connecting lab work to real-world challenges.
The year ahead is about scaling what works. By deepening these connections, Microsoft Research Asia — Singapore can bring more frontier research into practice and create meaningful value for Singapore, the wider region, industry, and society.
Opens in a new tabThe post One year in: How Microsoft Research Asia – Singapore is advancing research, partnership and talent for real-world impact appeared first on Microsoft Research.
Offloaded inference for real-world physical AI robotics
- Challenges a core assumption in robotics AI: Our research shows that running physical AI inference exclusively on onboard GPUs can limit robot performance, battery life, and scalability, and that offloading inference to edge or cloud GPUs can offer significant advantages.
- Demonstrates measurable benefits of inference offloading: Across representative mobile manipulation workloads, offloading improved task success rates, enabled larger AI models, and helped robots respond more effectively in dynamic, real-world environments.
- Extends robot operating time: Replacing power-hungry onboard AI compute with lightweight onboard hardware and remote inference can substantially improve battery life, enabling robots to operate longer between charges.
- Introduces a new capability in the Physical AI Toolchain: Developers can now containerize, deploy, and orchestrate robotics AI workloads across robots, edge infrastructure, and the cloud using Kubernetes-based tooling for distributed inference.
Readily-available physical AI, with robotics assisting users in manufacturing, home, and warehouses scenarios, holds immense potential to improve safety, productivity, and assistance across a wide range of tasks. In many ways, AI for the physical world represents a major frontier for AI . Physical AI must operate in open, unpredictable environments, interact with both other robots and people, and work with a diversity of embodiments. Realizing this vision requires advances along three dimensions: robot hardware, embodied AI models, and systems infrastructure for training and inference. While robot hardware and the AI models have advanced rapidly in recent years, we turn our focus on a relatively under-addressed aspect: inference infrastructure of physical AI. Enabling robots to effectively and safely operate in the physical world will require sophisticated systems to handle large volumes of distributed inference compute.
Today, the prevailing approach to physical AI is to provision a GPU onboard the robot, e.g., by wiring a GPU to the robot. In this model, the robot’s inference will be confined to the onboard GPU, and provide the robot with the necessary chunks and sequence of actions for the execution of its tasks. While higher-level planning may be performed in the cloud, task execution typically remains tied to the robot itself. We challenge this assumption. As physical AI models grow in size and sophistication, the constraints of onboard compute become increasingly apparent. GPUs consume significant power, reduce battery life, add cost and weight, and can limit the ability to run the latest generation of AI models.
To better understand the systems implications of physical AI, we conducted the first systematic study of robotics workloads. We focused on mobile robotic manipulation, with the canonical task such as “check for rubbish in the kitchen and put it in the trash.” Such a task involves planning the path to the kitchen, perceiving the environment to find rubbish, navigating to the rubbish, picking up the rubbish, and navigating back to the trash can for disposal. We evaluated representative models across three core capabilities: semantic mapping and planning, navigation, and manipulation, as summarized in Figure 2.
Figure 2: Details of the models used for the different components of mobile manipulation.Offloading physical AI inference out of the robot improved its response time and accuracy, along with battery lifetime and cost. We evaluated the inference models across a range of onboard, edge, and cloud compute configurations. Details of the specific test hardware are available in our technical report.
Benefits in task performance: Our evaluation shows offloading inference can significantly improve robot performance across mapping, planning, navigation, and manipulation workloads. Some smaller GPUs could not accommodate the mobile manipulation stack. On GPUs with sufficient memory, mapping and planning slowed by up to 383% compared to an A100, thus limiting the robot’s abilities in dynamic spaces. Navigation showed a 30% drop in its timely detection of obstacles with lighter GPUs. While the VLA models did not dramatically slow down with smaller GPUs, the slowdown was still sufficient to drop their accuracies by 50%. In other words, onboard GPUs limited the performance of the robots while offloading their inference to an on-premise or cloud GPU boosts their operations, as shown in the videos below and quantified in the graphs. As physical AI models continue to grow in size and complexity, the benefits of offloading are likely to become even more pronounced.
Figure 3a: The video shows the handover task with onboard GPUs. Figure 3b: The video shows the handover task when the inference is offloaded. Figure 4: Success rates of robot arms handing over objects to each other when inference is performed with different GPUs (some onboard, and some offloaded). Offloading improves success rates.Benefits in battery lifetime: Beyond performance, onboard GPUs also significantly drained the battery life of the robot. We compared the increase in battery lifetime by replacing an onboard GPU with a Raspberry Pi-5 board and shipping all the data to the offloaded GPU. The larger onboard GPUs, such as Jetson Thor, drained robot batteries by up to 160% (or a few hours) for even the larger robots.
Figure 5: Impact of offloading GPU inference on the battery life of the robots; the above numbers are for the Stretch-3 robot.The above results show that offloading GPU inference out of the robot is critical for functioning in the open world with large models and long battery lifetimes. Nonetheless, offloading inference out of the robot involves a complex tradeoff involving performance, network latency and bandwidth, and available GPU resources. We believe that our measurement study will inform the design of physical AI inference systems.
Microsoft Research StorY
OptiMind: When the system meets the floorA three‑month pilot in a Midwestern bottling plant shows what happens when AI moves beyond chat and into decision-making, where constraints shift, stakes are real, and answers must hold.
Learn more Opens in a new tab Toolset for automatic offloadWe have built a toolset for easy inference offloading out of the robot and distributing inference between the edge GPU and cloud. Kubernetes is a natural platform to provide a uniform abstraction to distribute robotic AI between the robot’s compute, edge GPU, and overflowing to the cloud. The toolset allows automatic containerization and offloading of robotics workloads using declarative specifications, distributes physical AI containers with smart policies using Kubernetes, and integrates with robotic simulators, LeRobot, and ROS2 for easy development. The sequence of steps below shows how the toolset can be prompted with what to offload, and how it creates a separate container for GPU inference and offloads the same.
Figure 6: Steps in the offloading toolset with containerization and deployment.Microsoft has recently released the Physical AI Toolchain (opens in new tab) for operationalizing physical intelligence at scale. Physical AI Toolchain is an open-source, production-ready framework that integrates Microsoft Azure (opens in new tab) cloud services with NVIDIA’s (opens in new tab) physical AI stack, accelerating robotics and physical AI developers to automate and scale data curation, augmentation, and evaluation across perception, mobility, imitation learning, and reinforcement learning pipelines. We are announcing the addition of an industry-first capability for offloaded physical AI inference for robots as part of the Physical AI Toolchain. This release includes example projects for offloading inference of a SO-101 and a UR10e. The videos below show the offloading of the inference of Microsoft’s Rho model, targeted at dual-arm robots, to a Jetson Thor GPU, which controls the actions of the Mobile Aloha robot (opens in new tab).
Figure 7a: Offloading of GPU inference of Microsoft’s Rho model controlling the Mobile Aloha robot for interactions with the BusyBox to press the blue button. Figure 7b: Offloading of GPU inference of Microsoft’s Rho model controlling the Mobile Aloha robot for interactions with the BusyBox to turn the knob to position 4.Check out the inference offload feature, look into the source code, and let us know your feedback. We have already tested it with many real-world use cases, and look forward to hearing about your deployment experiences.
Opens in a new tabThe post Offloaded inference for real-world physical AI robotics appeared first on Microsoft Research.
Improving synthesis prediction of small molecules at scale with RetroChimera
- We report on the recent publication of our retrosynthesis model RetroChimera in the journal Nature (opens in new tab).
- The paper describes the model’s architecture as well as extensive validation studies, including the model’s ability to recall rare reaction types, and successful zero-shot transfer and fine-tuning on proprietary datasets.
- We open-source RetroChimera’s implementation and weights in the hope that it will enable researchers to accelerate development of new medicinally relevant molecules and advanced materials.
Developing new medicines and materials requires making new molecules but planning how to make them is still largely manual, time-consuming, and costly. RetroChimera automatically proposes high-quality synthesis routes. The model combines two strong models with complementary strengths, learning how to rank their proposals to produce better predictions than either alone. In blind tests, PhD-level chemists prefer RetroChimera’s individual reaction predictions over preceding models and recorded literature reactions.
Custom-made molecules are unlocking advances in modern medicine, smart materials, and sustainable agriculture. Yet, progress is slowed by chemical synthesis—the time-consuming process of making new molecules from simpler building blocks in the lab. In addition, synthesis is a significant driver of drug development costs. So even as computational methods make it possible to explore large numbers of novel molecules, finding practical ways to synthesize them remains a critical challenge.
Figure 1: Planning a synthesis by working backward. Retrosynthesis starts with a target molecule and proposes successive disconnections into simpler precursors until purchasable building blocks are reached. The highlighted path shows a complete synthesis route; pale branches illustrate alternatives explored along the way. Circles represent molecules and squares represent reactions. For clarity, only a few branches are illustrated, with chemical structures shown for the target, one intermediate, and selected building blocks.Retrosynthesis approaches this problem by working backwards from a target molecule, breaking it down step by step into simpler precursors (Figure 1). This process is comparable to playing strategic board games like chess and Go. It involves contemplating a wide range of possible immediate moves, or individual disconnections, while also requiring high-level strategic thinking to reach the end-to-end synthesis plan. However, the number of possible moves in retrosynthesis is much larger than in board games, and it is not obvious which moves would be available for a given molecule. Existing systems face major challenges, including recalling rare but strategically important reactions, robustness beyond the training distribution, and aligning with chemists’ expectations. As a result, retrosynthesis often requires highly specialized expertise, which hinders scaling and automation of scientific discovery.
Figure 2: Our framework for ensemble-based retrosynthesis with learned re-ranking which underpins RetroChimera. The ensemble receives a target molecule as the input, which is then processed by the sub-models. The model outputs are aggregated using a learning-to-rank strategy. While in this work we only investigate deep learning models as prediction sources (solid boxes), it is possible to add additional sources, for example calls to reaction databases or human-in-the-loop queries (dashed box).In a paper recently published in the journal Nature (opens in new tab), we present RetroChimera (opens in new tab), a new framework for retrosynthesis prediction. It is built around two models (Figure 2). R-SMILES 2, a Transformer-based de-novo model, predicts precursor molecules directly from the input molecule. This gives it the flexibility to learn reaction patterns directly from data. However, its unconstrained generation can also make it prone to hallucination.
NeuralLoc, in contrast, is a graph neural network- (GNN) based model that encodes both the target molecule and reaction templates as graphs. It selects reaction templates and predicts where they should be applied to the target molecule. Its predictions are grounded in reaction patterns extracted from the training data, so it tends to produce more accurate and reliable outputs. But it’s more constrained when encountering reactions not covered by the template library.
These differences actually turn out to be a strength. Rather than making the same kinds of predictions, the two models capture complementary patterns in chemistry and specialize in different reaction types. R-SMILES 2 performs particularly well on reactions that involve large changes over the course of the reaction, while NeuralLoc excels in reactions of low precedence and those involving more localized changes.
RetroChimera combines the ranked predictions of both sub-models using a learned ensembling strategy. Each model assigns a learned, rank-dependent vote to each predicted reactant set, and votes are added when both models propose the same reaction. By learning how much to trust each model at different ranks, RetroChimera can leverage their complementary strengths, approximately matching the better-performing sub-model across reaction classes.
Figure 3: Expert assessment of multistep synthesis routes. Left: Ratings of individual reaction steps. Right: Complete routes accepted or rejected for ten challenging targets. RetroChimera succeeded on nine targets, versus five for the de novo model, four for the editing model, and two for NeuralSym, a strong baseline model.As a result, RetroChimera performs strongly across both common and rare reaction classes and produces retrosynthesis predictions that better align with chemists’ judgment (Figure 3). In blind tests, expert chemists preferred disconnections of complex molecules suggested by RetroChimera over those obtained from its constituent sub-models, as well as those from more established approaches, and even from the test set itself.
We believe RetroChimera could help researchers identify promising synthesis route more efficiently, supporting faster design-make-test cycle across molecular science applications, including drug discovery and design of smart materials. RetroChimera could enable chemists to assess more—and more complex—candidate molecules at large scale. Paired with increasing levels of laboratory automation, we expect further acceleration toward closed-loop, self-improving systems for synthesis planning and execution.
RetroChimera is available on GitHub (opens in new tab) (MIT license) and accessible via Microsoft Foundry (opens in new tab). For instructions on how to access the checkpoint, we refer to the GitHub repository.
We invite the broader chemistry community to experiment with RetroChimera, helping us identify its strengths and shortcomings so we can enhance it in the future. We are looking forward to hearing how it performs on various targets you care about!
For a deeper look at the findings, including evaluation experiments with our external research collaborators, see the full Nature publication (opens in new tab) and the accompanying Microsoft Source article (opens in new tab).
Publication OSS repository Microsoft Foundry Opens in a new tabThe post Improving synthesis prediction of small molecules at scale with RetroChimera appeared first on Microsoft Research.
GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models
- The Flash family extends GigaPath and GigaTIME with dramatically improved efficiency, making large-scale pathology research more accessible and practical.
- A distilled pathology foundation model backbone reduces computational requirements without sacrificing performance, enabling repeated analyses across larger patient cohorts.
- These open models support population-scale discovery, helping researchers investigate disease biology, biomarkers, and clinical outcomes across diverse cancer datasets.
GigaPath (opens in new tab) and GigaTIME (opens in new tab) demonstrated how foundation models can support whole-slide analysis and tumor microenvironment modeling from routinely collected pathology data. GigaPath-Flash and GigaTIME-Flash make these capabilities substantially more efficient, enabling researchers to analyze larger cohorts, run more experiments, and move toward population-scale discovery. GigaPath-Flash and GigaTIME-Flash are research models. They are not intended or validated for clinical use, including diagnosis, prognosis, treatment selection, or other patient-care decisions. Performance may vary across datasets, scanners, institutions, populations, and use cases.
The scale opportunity in computational pathologyHistopathology is among the richest and most widely available sources of information in cancer research. Every tissue biopsy produces a whole-slide image that captures cellular morphology at subcellular resolution — and hospitals generate millions of these slides each year. This data contains information relevant to diagnosis, prognosis, treatment selection, and the biology of the tumor microenvironment.
Foundation models have begun to unlock this information at scale. But whole-slide images are large — often exceeding a gigapixel — and applying a foundation model to even a single slide requires processing thousands of image tiles. When a research question involves tens of thousands of patients, the computational cost grows quickly. And population-scale discovery is not a single model run: it requires repeated cycles of feature extraction, statistical analysis, hypothesis testing, and validation across patient subgroups, biomarkers, and clinical endpoints.
Computational cost limits the number of patients, datasets, tasks, and hypotheses that researchers can study. To realize the full potential of pathology foundation models, we need models that can be applied repeatedly and affordably across large patient populations.
From GigaPath and GigaTIME to the -Flash familyGigaPath (Nature, 2024) (opens in new tab) is a whole-slide foundation model pretrained on large-scale real-world histopathology data from Providence. Unlike models that operate only at the tile level, GigaPath learns contextualized representations of entire slides, capturing both local cellular patterns and global tissue architecture.
GigaTIME (Cell, 2026) (opens in new tab) extends this line of work to tumor microenvironments. Trained on 40 million cells with paired H&E and multiplex immunofluorescence (mIF) data, GigaTIME translates routine H&E images into virtual spatial proteomics maps across 21 protein channels. Applied to over 14,000 cancer patients, it generated a virtual population that uncovered more than 1,200 statistically significant associations between immune cell states and clinical biomarkers.
GigaPath and GigaTIME addressed the scale of pathology data and biological discovery. And now, the Flash family of models addresses scale of experimentation.
Figure 1: Overview of the GigaPath/GigaTIME model family. GigaPath-Flash provides efficient tile and slide encoders at 22M parameters. GigaTIME-Flash predicts spatial proteomics from H&E, replacing the CNN backbone with the distilled ViT-S encoder. Introducing GigaPath-Flash and GigaTIME-FlashThe Flash family shares a core design goal: preserving useful pathology representations while substantially reducing the computational resources required to generate and use them. Both models are built on a common efficient backbone — a compact ViT-S tile encoder distilled from the original billion-parameter GigaPath encoder — and both are released under the Apache 2.0 license.
GigaPath-FlashGigaPath-Flash is an efficient foundation model for whole-slide representation learning. It combines a 22M-parameter ViT-S tile encoder with a 21M-parameter LongNet slide encoder. The tile encoder is distilled from the original GigaPath ViT-g teacher, transferring the representational capacity of a billion-parameter model into a backbone that is an order of magnitude smaller. The slide encoder contextualizes all tile embeddings via dilated attention, scaling linearly with the number of tiles.
On slide-level classification benchmarks (PANDA prostate grading and EBRAINS brain tumor subtyping), GigaPath-Flash achieves the lowest inference cost among whole-slide pretrained models while retaining competitive performance — scoring within 3% of the original GigaPath at roughly 50 times less compute.
Figure 2: Efficiency–performance trade-off on whole-slide benchmarks. GigaPath-Flash (red, top-left) achieves competitive performance at substantially lower computational cost than other whole-slide pretrained models. GigaTIME-FlashGigaTIME-Flash replaces the CNN backbone of the original GigaTIME with the GigaPath-Flash ViT-S encoder, paired with a lightweight convolutional decoder for H&E-to-mIF translation. The model is fine-tuned using LoRA adapters, keeping the pretrained encoder weights largely frozen.
On both in-distribution and out-of-distribution cohorts spanning brain, breast, colon, and lung cancers, GigaTIME-Flash matches or improves upon the original GigaTIME in spatial protein prediction quality. The gains are particularly notable on out-of-distribution data, suggesting that the foundation model backbone improves generalization to previously unseen tissue types.
Figure 3: Mean windowed Pearson correlation for GigaTIME and GigaTIME-Flash on the GigaTIME test set and four out-of-distribution Prov-TMA cohorts. GigaTIME-Flash matches or improves upon the original across all cohorts. Efficiency without giving up the foundationThe efficiency gains of the Flash models are substantial:
ModelTypeEfficiency gainGigaPath-FlashWhole-slide Foundation Model~50× less compute, 97% of predictive performance compared to GigaPathGigaTIME-FlashSpatial Proteomics~6× faster, ~8× less memory, better predictive performance compared to GigaTIMEFor a single slide, these differences reduce runtime and hardware requirements. Across tens of thousands of slides, they can determine whether an experiment is practical at all. To illustrate, we estimate the wall-clock time for generating virtual mIF across cohorts of different sizes on a single A100 GPU, assuming approximately 10,000 tiles per slide:
Model1,000 slides100K slides1M slidesGigaTIME-Flash~2 GPU-hours~7 GPU-days~70 GPU-daysGigaTIME~7 GPU-hours~30 GPU-days~300 GPU-daysEstimates assume ~10,000 tiles per slide, batch size 128, single NVIDIA A100 GPU. Actual runtime depends on slide size, tiling resolution, and hardware. Figure 4: GigaTIME efficiency scaling. Left: throughput (tiles/sec) vs. batch size. Right: peak GPU memory (GB) vs. batch size. GigaTIME-Flash scales to over 1,600 tiles/sec while using a fraction of the memory. An open model releaseBoth GigaPath-Flash and GigaTIME-Flash are released as open-weight models under the Apache 2.0 license. Model weights and code are available on HuggingFace:
- GigaPath-Flash (opens in new tab)
- GigaTIME-Flash (opens in new tab)
- GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis
This is an early research release. Our current evaluations cover a limited set of benchmarks and cohorts, and broader validation across tasks, scanners, and patient populations is still needed. We expect the most valuable applications of these models to include scientific questions, cohorts, and use cases beyond those in our initial experiments. We welcome community evaluation, and we are equally interested in reports of where these models work well and where they fall short.
Downstream clinical applications will require additional multi-institutional and prospective validation.
PODCAST SERIES
The AI Revolution in Medicine, RevisitedJoin Microsoft’s Peter Lee on a journey to discover how AI is impacting healthcare and what it means for the future of medicine.
Listen now Opens in a new tab Efficiency as an enabler of discoveryGigaPath and GigaTIME demonstrated what pathology foundation models can learn from whole slides and tumor tissue. GigaPath-Flash and GigaTIME-Flash are a step toward making those capabilities usable across larger populations, more experiments, and a broader research community.
By making pathology foundation models more efficient, we hope to expand the scale of the scientific questions researchers can ask.
AcknowledgementsGigaPath-Flash and GigaTIME-Flash are joint work across Microsoft Research, the University of Washington, and Providence. For technical details, see the paper.
Paper co-authors: Naoto Usuyama, Jeya Maria Jose Valanarasu, Sicong Yao, Hanwen Xu, Jaspreet Bagga, Guanghui Qin, Robert E. Kramer, Cliff Wong, Soohee Lee, Hao Qiu, Theodore Zhengde Zhao, Racheli Ben Shimol, Angela Crabtree, Kevin Matlock, Eduardo Alejandro Lozano Garcia, Naiteek Sangani, Alberto Santamaria-Pang, Maximilian Rokuss, Yashna Hasija, Naisargi Manishkumar Patel, Jason Entenmann, Alexandra Q. Bartlett, Bill J. Wright, Bernard A. Fox, Brian Piening, Sheng Zhang, Sheng Wang, Tristan Naumann, Carlo Bifulco, Hoifung Poon
Opens in a new tabThe post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models
- The Flash family extends GigaPath and GigaTIME with dramatically improved efficiency, making large-scale pathology research more accessible and practical.
- A distilled pathology foundation model backbone reduces computational requirements without sacrificing performance, enabling repeated analyses across larger patient cohorts.
- These open models support population-scale discovery, helping researchers investigate disease biology, biomarkers, and clinical outcomes across diverse cancer datasets.
GigaPath (opens in new tab) and GigaTIME (opens in new tab) demonstrated how foundation models can support whole-slide analysis and tumor microenvironment modeling from routinely collected pathology data. GigaPath-Flash and GigaTIME-Flash make these capabilities substantially more efficient, enabling researchers to analyze larger cohorts, run more experiments, and move toward population-scale discovery. GigaPath-Flash and GigaTIME-Flash are research models. They are not intended or validated for clinical use, including diagnosis, prognosis, treatment selection, or other patient-care decisions. Performance may vary across datasets, scanners, institutions, populations, and use cases.
The scale opportunity in computational pathologyHistopathology is among the richest and most widely available sources of information in cancer research. Every tissue biopsy produces a whole-slide image that captures cellular morphology at subcellular resolution — and hospitals generate millions of these slides each year. This data contains information relevant to diagnosis, prognosis, treatment selection, and the biology of the tumor microenvironment.
Foundation models have begun to unlock this information at scale. But whole-slide images are large — often exceeding a gigapixel — and applying a foundation model to even a single slide requires processing thousands of image tiles. When a research question involves tens of thousands of patients, the computational cost grows quickly. And population-scale discovery is not a single model run: it requires repeated cycles of feature extraction, statistical analysis, hypothesis testing, and validation across patient subgroups, biomarkers, and clinical endpoints.
Computational cost limits the number of patients, datasets, tasks, and hypotheses that researchers can study. To realize the full potential of pathology foundation models, we need models that can be applied repeatedly and affordably across large patient populations.
From GigaPath and GigaTIME to the -Flash familyGigaPath (Nature, 2024) (opens in new tab) is a whole-slide foundation model pretrained on large-scale real-world histopathology data from Providence. Unlike models that operate only at the tile level, GigaPath learns contextualized representations of entire slides, capturing both local cellular patterns and global tissue architecture.
GigaTIME (Cell, 2026) (opens in new tab) extends this line of work to tumor microenvironments. Trained on 40 million cells with paired H&E and multiplex immunofluorescence (mIF) data, GigaTIME translates routine H&E images into virtual spatial proteomics maps across 21 protein channels. Applied to over 14,000 cancer patients, it generated a virtual population that uncovered more than 1,200 statistically significant associations between immune cell states and clinical biomarkers.
GigaPath and GigaTIME addressed the scale of pathology data and biological discovery. And now, the Flash family of models addresses scale of experimentation.
Figure 1: Overview of the GigaPath/GigaTIME model family. GigaPath-Flash provides efficient tile and slide encoders at 22M parameters. GigaTIME-Flash predicts spatial proteomics from H&E, replacing the CNN backbone with the distilled ViT-S encoder. Introducing GigaPath-Flash and GigaTIME-FlashThe Flash family shares a core design goal: preserving useful pathology representations while substantially reducing the computational resources required to generate and use them. Both models are built on a common efficient backbone — a compact ViT-S tile encoder distilled from the original billion-parameter GigaPath encoder — and both are released under the Apache 2.0 license.
GigaPath-FlashGigaPath-Flash is an efficient foundation model for whole-slide representation learning. It combines a 22M-parameter ViT-S tile encoder with a 21M-parameter LongNet slide encoder. The tile encoder is distilled from the original GigaPath ViT-g teacher, transferring the representational capacity of a billion-parameter model into a backbone that is an order of magnitude smaller. The slide encoder contextualizes all tile embeddings via dilated attention, scaling linearly with the number of tiles.
On slide-level classification benchmarks (PANDA prostate grading and EBRAINS brain tumor subtyping), GigaPath-Flash achieves the lowest inference cost among whole-slide pretrained models while retaining competitive performance — scoring within 3% of the original GigaPath at roughly 50 times less compute.
Figure 2: Efficiency–performance trade-off on whole-slide benchmarks. GigaPath-Flash (red, top-left) achieves competitive performance at substantially lower computational cost than other whole-slide pretrained models. GigaTIME-FlashGigaTIME-Flash replaces the CNN backbone of the original GigaTIME with the GigaPath-Flash ViT-S encoder, paired with a lightweight convolutional decoder for H&E-to-mIF translation. The model is fine-tuned using LoRA adapters, keeping the pretrained encoder weights largely frozen.
On both in-distribution and out-of-distribution cohorts spanning brain, breast, colon, and lung cancers, GigaTIME-Flash matches or improves upon the original GigaTIME in spatial protein prediction quality. The gains are particularly notable on out-of-distribution data, suggesting that the foundation model backbone improves generalization to previously unseen tissue types.
Figure 3: Mean windowed Pearson correlation for GigaTIME and GigaTIME-Flash on the GigaTIME test set and four out-of-distribution Prov-TMA cohorts. GigaTIME-Flash matches or improves upon the original across all cohorts. Efficiency without giving up the foundationThe efficiency gains of the Flash models are substantial:
ModelTypeEfficiency gainGigaPath-FlashWhole-slide Foundation Model~50× less compute, 97% of predictive performance compared to GigaPathGigaTIME-FlashSpatial Proteomics~6× faster, ~8× less memory, better predictive performance compared to GigaTIMEFor a single slide, these differences reduce runtime and hardware requirements. Across tens of thousands of slides, they can determine whether an experiment is practical at all. To illustrate, we estimate the wall-clock time for generating virtual mIF across cohorts of different sizes on a single A100 GPU, assuming approximately 10,000 tiles per slide:
Model1,000 slides100K slides1M slidesGigaTIME-Flash~2 GPU-hours~7 GPU-days~70 GPU-daysGigaTIME~7 GPU-hours~30 GPU-days~300 GPU-daysEstimates assume ~10,000 tiles per slide, batch size 128, single NVIDIA A100 GPU. Actual runtime depends on slide size, tiling resolution, and hardware. Figure 4: GigaTIME efficiency scaling. Left: throughput (tiles/sec) vs. batch size. Right: peak GPU memory (GB) vs. batch size. GigaTIME-Flash scales to over 1,600 tiles/sec while using a fraction of the memory. An open model releaseBoth GigaPath-Flash and GigaTIME-Flash are released as open-weight models under the Apache 2.0 license. Model weights and code are available on HuggingFace:
- GigaPath-Flash (opens in new tab)
- GigaTIME-Flash (opens in new tab)
- GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis
This is an early research release. Our current evaluations cover a limited set of benchmarks and cohorts, and broader validation across tasks, scanners, and patient populations is still needed. We expect the most valuable applications of these models to include scientific questions, cohorts, and use cases beyond those in our initial experiments. We welcome community evaluation, and we are equally interested in reports of where these models work well and where they fall short.
Downstream clinical applications will require additional multi-institutional and prospective validation.
Spotlight: Microsoft research newsletter
Microsoft Research Newsletter Subscribe today Opens in a new tab Efficiency as an enabler of discoveryGigaPath and GigaTIME demonstrated what pathology foundation models can learn from whole slides and tumor tissue. GigaPath-Flash and GigaTIME-Flash are a step toward making those capabilities usable across larger populations, more experiments, and a broader research community.
By making pathology foundation models more efficient, we hope to expand the scale of the scientific questions researchers can ask.
AcknowledgementsGigaPath-Flash and GigaTIME-Flash are joint work across Microsoft Research, the University of Washington, and Providence. For technical details, see the paper.
Paper co-authors: Naoto Usuyama, Jeya Maria Jose Valanarasu, Sicong Yao, Hanwen Xu, Jaspreet Bagga, Guanghui Qin, Robert E. Kramer, Cliff Wong, Soohee Lee, Hao Qiu, Theodore Zhengde Zhao, Racheli Ben Shimol, Angela Crabtree, Kevin Matlock, Eduardo Alejandro Lozano Garcia, Naiteek Sangani, Alberto Santamaria-Pang, Maximilian Rokuss, Yashna Hasija, Naisargi Manishkumar Patel, Jason Entenmann, Alexandra Q. Bartlett, Bill J. Wright, Bernard A. Fox, Brian Piening, Sheng Zhang, Sheng Wang, Tristan Naumann, Carlo Bifulco, Hoifung Poon
Opens in a new tabThe post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.
MindTopo reveals VLMs’ spatial reasoning abilities
- MindTopo is a new benchmark for testing topological reasoning in AI, evaluating whether multimodal models can understand concepts such as connectivity, enclosure, order, separation, and knots.
- The benchmark measures both reasoning and planning, testing not only whether models can recognize topological relationships in static images but also whether they can preserve and manipulate those relationships through a sequence of actions.
- Current multimodal models perform much better on static recognition than interactive tasks, suggesting they struggle to maintain a consistent understanding of topology over time.
- Failures often emerge during planning rather than perception, with models losing track of structural relationships as scenes change or proposing actions that violate physical constraints.
- The findings highlight an important opportunity to advance AI systems for robotics and interactive environments, where understanding what stays connected, enclosed, ordered, or knotted is essential for reliable decision-making.
Can AI determine whether two rooms remain connected after a wall is added? Can it recognize whether an animal is inside a fence, distinguish a true knot from a tangled loop, or rearrange several ropes without allowing them to pass through one another?
These questions concern 3D topology, a form of spatial understanding based not on exact distances, angles, or shapes, but on structural relationships that persist as objects bend, stretch, or deform. Connectivity, enclosure, ordering, and knottedness are examples of topological properties. These properties are a foundational layer of human spatial understanding in Cognitive Science, yet they remain largely absent from how multimodal AI systems are evaluated.
In a new research study, we introduce MindTopo (opens in new tab), a benchmark designed to evaluate whether multimodal large language models possess this kind of topological intuition. Our findings reveal a substantial gap between recognizing topology in a static image and maintaining an innate understanding of that topology while planning and acting. Current models can sometimes identify a connected path, enclosed region, or knot in a single scene, but that understanding often breaks down once the model must manipulate the scene through a sequence of actions.
How MindTopo defines topological spaceMost spatial evaluations for multimodal models focus on Euclidean properties such as distance, direction, size, and relative position. Inspired by Piaget and other cognitive literature’s classification of topological ability, MindTopo organizes its tasks around the following five categories:
- Continuity asks whether a path or object remains unbroken.
- Separation asks whether nearby elements form one structure or distinct parts.
- Order tracks how elements are arranged along a path or through a transformation.
- Enclosure tests whether a boundary creates an inside and an outside.
- Knots tests whether ropes are truly knotted or linked rather than merely tangled in appearance.
Each category is evaluated at two cognitive levels. In reasoning tasks, a model examines one or more rendered scenes and answers a question about their topological structure: whether two points in a maze are connected, whether the sheep are inside the fence, whether a rope is truly knotted. In planning tasks, the model interacts with a simulated environment and selects actions that must create, preserve, or remove a particular relation, such as rotating pipe segments, drawing a separating path, rearranging blocks, trapping a moving agent, or untangling ropes. The environments enforce legal actions, so a model cannot solve a rope puzzle by passing one strand through another.
Figure 1. MindTopo pairs questions about static scenes with interactive tasks that require models to preserve or change the same topological relations.All scenes are generated from controlled simulators, which provide exact ground truth and adjustable difficulty. That control makes it possible to separate two failure modes that otherwise look alike: a model that fails because a scene is visually complex, and a model that fails because it cannot maintain the underlying relationship as objects move.
Figure 2. MindTopo maps reasoning and planning tasks to continuity, separation, order, enclosure, and knots. Seeing topology is not the same as acting on itAcross a broad set of proprietary and open-weight models, performance was consistently stronger on static reasoning than on interactive planning, and both remained well below human performance. The contrast was especially clear when success depended on preserving a relationship across many actions.
The error patterns help locate the problem. Static mistakes usually began with perception, such as missing a wall, opening, or crossing. Planning mistakes appeared after the scene had been understood. Models followed a locally plausible move without tracking its later consequences, lost the task over multiple turns, or proposed an action that violated the environment’s dynamics.
Microsoft Research at BUILD 2026Giving developers a hands-on look at some of the many AI-based models and tools they can use to accelerate innovation, enhance their capabilities, and quickly transform ideas into prototypes.
Learn more Opens in a new tab What generative tools revealWe also tested whether image and video generation could help models maintain an understanding of topological relationships. Image generation sometimes helped when the relevant relation was visible in a single frame, but it remained unreliable across a sequence of crossings or moves. Video rollouts frequently altered topology or violated task dynamics. Visual simulation appeared useful only to the extent that it preserved structural constraints over time.
Building agents that preserve structureMindTopo is intended as a controlled diagnostic for this gap. Robots, accessibility tools, and interactive assistants must understand not only where objects are, but also what remains connected, enclosed, ordered, or knotted as actions unfold. Closing that gap may require models that carry an explicit topological state, or world models whose predictions preserve topology by construction.
Opens in a new tabThe post MindTopo reveals VLMs’ spatial reasoning abilities appeared first on Microsoft Research.
MindTopo reveals VLMs’ spatial reasoning abilities
- MindTopo is a new benchmark for testing topological reasoning in AI, evaluating whether multimodal models can understand concepts such as connectivity, enclosure, order, separation, and knots.
- The benchmark measures both reasoning and planning, testing not only whether models can recognize topological relationships in static images but also whether they can preserve and manipulate those relationships through a sequence of actions.
- Current multimodal models perform much better on static recognition than interactive tasks, suggesting they struggle to maintain a consistent understanding of topology over time.
- Failures often emerge during planning rather than perception, with models losing track of structural relationships as scenes change or proposing actions that violate physical constraints.
- The findings highlight an important opportunity to advance AI systems for robotics and interactive environments, where understanding what stays connected, enclosed, ordered, or knotted is essential for reliable decision-making.
Can AI determine whether two rooms remain connected after a wall is added? Can it recognize whether an animal is inside a fence, distinguish a true knot from a tangled loop, or rearrange several ropes without allowing them to pass through one another?
These questions concern 3D topology, a form of spatial understanding based not on exact distances, angles, or shapes, but on structural relationships that persist as objects bend, stretch, or deform. Connectivity, enclosure, ordering, and knottedness are examples of topological properties. These properties are a foundational layer of human spatial understanding in Cognitive Science, yet they remain largely absent from how multimodal AI systems are evaluated.
In a new research study, we introduce MindTopo (opens in new tab), a benchmark designed to evaluate whether multimodal large language models possess this kind of topological intuition. Our findings reveal a substantial gap between recognizing topology in a static image and maintaining an innate understanding of that topology while planning and acting. Current models can sometimes identify a connected path, enclosed region, or knot in a single scene, but that understanding often breaks down once the model must manipulate the scene through a sequence of actions.
How MindTopo defines topological spaceMost spatial evaluations for multimodal models focus on Euclidean properties such as distance, direction, size, and relative position. Inspired by Piaget and other cognitive literature’s classification of topological ability, MindTopo organizes its tasks around the following five categories:
- Continuity asks whether a path or object remains unbroken.
- Separation asks whether nearby elements form one structure or distinct parts.
- Order tracks how elements are arranged along a path or through a transformation.
- Enclosure tests whether a boundary creates an inside and an outside.
- Knots tests whether ropes are truly knotted or linked rather than merely tangled in appearance.
Each category is evaluated at two cognitive levels. In reasoning tasks, a model examines one or more rendered scenes and answers a question about their topological structure: whether two points in a maze are connected, whether the sheep are inside the fence, whether a rope is truly knotted. In planning tasks, the model interacts with a simulated environment and selects actions that must create, preserve, or remove a particular relation, such as rotating pipe segments, drawing a separating path, rearranging blocks, trapping a moving agent, or untangling ropes. The environments enforce legal actions, so a model cannot solve a rope puzzle by passing one strand through another.
Figure 1. MindTopo pairs questions about static scenes with interactive tasks that require models to preserve or change the same topological relations.All scenes are generated from controlled simulators, which provide exact ground truth and adjustable difficulty. That control makes it possible to separate two failure modes that otherwise look alike: a model that fails because a scene is visually complex, and a model that fails because it cannot maintain the underlying relationship as objects move.
Figure 2. MindTopo maps reasoning and planning tasks to continuity, separation, order, enclosure, and knots. Seeing topology is not the same as acting on itAcross a broad set of proprietary and open-weight models, performance was consistently stronger on static reasoning than on interactive planning, and both remained well below human performance. The contrast was especially clear when success depended on preserving a relationship across many actions.
The error patterns help locate the problem. Static mistakes usually began with perception, such as missing a wall, opening, or crossing. Planning mistakes appeared after the scene had been understood. Models followed a locally plausible move without tracking its later consequences, lost the task over multiple turns, or proposed an action that violated the environment’s dynamics.
Spotlight: Event Series
Microsoft Research ForumJoin us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.
Watch on-demand Opens in a new tab What generative tools revealWe also tested whether image and video generation could help models maintain an understanding of topological relationships. Image generation sometimes helped when the relevant relation was visible in a single frame, but it remained unreliable across a sequence of crossings or moves. Video rollouts frequently altered topology or violated task dynamics. Visual simulation appeared useful only to the extent that it preserved structural constraints over time.
Building agents that preserve structureMindTopo is intended as a controlled diagnostic for this gap. Robots, accessibility tools, and interactive assistants must understand not only where objects are, but also what remains connected, enclosed, ordered, or knotted as actions unfold. Closing that gap may require models that carry an explicit topological state, or world models whose predictions preserve topology by construction.
Opens in a new tabThe post MindTopo reveals VLMs’ spatial reasoning abilities appeared first on Microsoft Research.
Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Research Note: CARE-X is a research model and not a Microsoft product offering or medical device. It has not been cleared or approved by any regulatory authority and is not intended for clinical diagnosis, screening, or patient care. The results described below are retrospective research findings and do not establish the safety, effectiveness, or suitability of CARE-X for any clinical use. References to potential workflows describe areas for future research, not currently available capabilities or recommended uses.
At a glance- The challenge: Chest X-ray interpretation spans diverse tasks that require both expressive report generation and calibrated diagnostic predictions.
- CARE-X is a unified chest X-ray VLM for diverse clinical interpretation tasks. It combines generation and structured prediction to provide both free-text reasoning and deterministic outputs.
- CARE-X uses reinforcement learning (DAPO) to reward clinical correctness in a multi-task setting.
- In a separate research experiment from CARE-X, we paired Qwen3-VL-4B-Instruct with deterministic measurement tools to evaluate whether direct computation could improve performance on measurement-dependent conditions compared with visual approximation alone.
- Validated on real-world Indian clinical data from Narayana Health, including rare ICU pathologies and CT-confirmed enlargement conditions.
A clinically useful radiology AI system must support a wide range of tasks, adapt to different workflows, and produce outputs that are medically accurate.
Radiologists and other clinicians use chest X-rays for many different purposes. A clinically useful AI system must be able to support that range of tasks. It may be asked to generate detailed findings and concise impressions for a report, answer questions about the presence, absence, or location of a finding, identify medical devices and assess their placement, or pinpoint exactly where an abnormality appears in an image.
These tasks also require different kinds of outputs, from narrative reports to calibrated diagnostic scores. And above all, they require clinical accuracy. A report could ostensibly be perfectly written yet clinically wrong if it misses a finding, reverses a negation, or misidentifies a location. Certain findings could be trivial in one context and vital to identify in another.
CARE-X was developed as a research model to explore how a unified approach can address these diverse demands. The system combines generative and discriminative capabilities, clinically aligned optimization, and tool-based reasoning to support a broader range of radiology workflows while maintaining clinical fidelity.
Spotlight: Event Series
Microsoft Research ForumJoin us for a continuous exchange of ideas about research in the era of general AI. Watch the latest episodes on demand.
Watch on-demand Opens in a new tab Gaps in current radiology vision-language modelsDespite the impressive task breadth of recent models, critical gaps remain between what radiologists need and what current systems deliver:
- No calibrated confidence for diagnostic decisions. Generative VLMs predict diagnoses as free text, but they typically do not provide calibrated confidence scores. In clinical settings, confidence matters. Clinicians cannot tune sensitivity–specificity trade-offs across clinical contexts—an important requirement for real-world deployment. Discriminative models provide these properties but lack the flexibility of open-ended generation.
- Cross-entropy loss does not optimize clinical fidelity. Standard training methods treat all token-level errors similarly, regardless of their clinical consequences. A coordinate mistake may be penalized no more than a harmless wording change. A “yes” can be flipped to a “no” even though the clinical meaning is completely different. Missing a life-threatening finding may carry the same training penalty as omitting a minor observation. As a result, models are not explicitly optimized for what matters most in patient care.
- No capability for measurement-dependent findings. Some radiological findings require more than visual recognition. Radiological signs such as cardiomegaly, mediastinal widening etc. depend on precise measurements. For example, a model may correctly recognize whether a chest radiograph was acquired using an AP or PA view. But determining cardiomegaly requires measuring the cardiac and thoracic widths and determining the cardiothoracic ratio. Those quantities should be measured and computed rather than visually approximated while considering variables such as type of view, exposure, rotation of the patient etc.
Together, these gaps call for more than a fluent generative model. The system must combine broad task coverage, structured predictions, clinically aligned optimization, and quantitative tools where direct measurement is required.
CARE-X: One model, flexible outputsCARE-X brings these diverse interpretation capabilities into one model, using generative or dual inference according to the needs of each task:
Task typeWhat CARE-X doesInference modeReport generation: FindingsProduces the detailed findings sectionGenerativeReport generation: ImpressionProduces the concise diagnostic impressionGenerativePresence and negation assessmentDetermines whether a pathology is present or absent and handles negationDual: generative + auxiliary headDisease location assessmentIdentifies where an abnormality appearsGenerativeFine-grained multilabel disease classificationCategorizes abnormalities across multiple labelsGenerativeMultilabel tubes and lines classificationIdentifies visible medical devicesGenerativeAbnormal placement detection of tubes and linesDetermines whether a device is positioned incorrectlyDual: generative + auxiliary headAbnormality phrase groundingLocalizes a described pathological findingDual: generative + auxiliary headAnatomical groundingLocalizes 29 anatomical regionsDual: generative + auxiliary headTable 1: CARE-X task coverage and inference modesDual inference means that a single forward pass produces both an autoregressive response and a structured auxiliary-head prediction with a confidence score. This provides free-text flexibility alongside threshold-adjustable outputs for tasks where operating-point control matters.
The CARE-X architecture and training approachCARE-X is built on a SigLIP2-so400M vision encoder and a Phi-4-mini-instruct (3.8B) language model connected through a lightweight adapter. To support both free-text generation and structured clinical predictions, the model augments the shared language backbone with task-specific auxiliary heads for classification and visual grounding. These heads provide calibrated diagnostic predictions and spatial localization signals while sharing representations with the generative language model. Rather than being trained independently, they are co-trained with the language-modeling objective, allowing structured supervision to enrich shared representations and improve generative performance on the same tasks.
Training. CARE-X uses a three-stage supervised fine-tuning pipeline (vision pre-training, adapter/head training, and LoRA adaptation) followed by DAPO-based reinforcement learning. DAPO optimizes task-specific rewards for clinical reporting, diagnostic accuracy, and spatial grounding quality.
Figure 1. The CARE-X model. (Left) Supervised fine-tuning with task-specific heads — classification, grounding, and language modeling — sharing the same Phi-4-mini-instruct backbone. The classification head outputs calibrated P(Yes)/P(No) scores; the grounding head outputs bounding box coordinate with confidence; the language modeling head generates free-text responses. (Right) DAPO with task-specific rewards for multi-task reinforcement alignment across report generation, grounding, and VQA. Auxiliary supervision: Structured prediction strengthens generationA central finding of this work is that co-training discriminative auxiliary heads with a generative VLM enriches shared representations, leading to stronger generative performance on the same tasks while also providing calibrated structured predictions.
Grounding improvementsThe auxiliary grounding head consistently improves localization over generative decoding. On anatomical grounding (Chest ImaGenome), mAP and mIoU increase by +28.2 pp and +6.2 pp, while the largest gains occur on phrase grounding (PadChest), with +24.6 pp mAP and +14.1 pp mIoU. The composite spatial loss enhances geometric precision in shared representations.
DAPO bridges the gap to dedicated detection headsDAPO-trained generative output approaches or exceeds the SFT auxiliary detection head. On Anatomy grounding, CARE-X generative (0.868 mAP) surpasses the SFT detection head (0.865). This is practically significant—it demonstrates that reward-aligned learning can bring autoregressive spatial decoding to parity with structured prediction, offering clinicians a single generative inference mode without requiring auxiliary heads at test time.
Calibrated classification with tunable operating pointsBeyond representation enrichment, the classification head offers a distinct deployment advantage: calibrated probability scores with tunable thresholds allow clinicians to shift between high-sensitivity screening and high-specificity confirmation from a single forward pass—a capability purely generative architectures cannot provide.
ModelInference SettingSensitivity ↑PPV ↑F1 ↑CARE-XGenerative0.9320.8950.913CARE-X (Th=0.5)Auxiliary Head0.9430.8850.913CARE-X (Th=0.6)Auxiliary Head0.8550.9270.890CheXOneGenerative0.8780.8540.866MedGemmaGenerative0.7980.8860.839Table 2: Abnormality classification performance on Chest ImaGenome. Adjustable thresholds enable operating-point selection. Strong report generation across four benchmarksWithin the paper’s comparison set, CARE-X achieves the strongest performance on most reported metrics across MIMIC-CXR, IU-Xray, CheXpert-Plus, and ReXGradient. CRIMSON, a held-out metric that evaluates abnormal findings and weights errors by clinical severity, suggests these gains reflect clinically meaningful improvements rather than reward-specific optimization.
Figure 2. CRIMSON scores (↑) for CARE-X against baseline report-generation models across four chest X-ray datasets — ReXGradient, MIMIC-CXR, IU-Xray, and CheXpert-Plus. CARE-X (highlighted) achieves the highest CRIMSON score on every dataset. CARE-X reaches 94% accuracy on ReXVQACARE-X ranks first on the ReXrank RexVQA leaderboard (opens in new tab) as of August 2026. On the ReXVQA benchmark (41,007 question–answer pairs across five clinically relevant categories), CARE-X reaches 94% overall accuracy, six percentage points above the next-best publicly reported model.
Figure 3: ReXVQA accuracy across five findings-quality dimensions — negation, presence, location, differential diagnosis, geometric information, and overall. CARE-X consistently outperforms CheXOne-R1 and MedGemma on every axis, with the largest margins in differential diagnosis, location assessment and negation. Tool-augmented measurement: Interleaving perception and computationSome radiological findings depend on quantitative measurements rather than visual patterns. In a separate research experiment from CARE-X, we built an inference-time pipeline that combines Qwen3-VL-4B-Instruct with deterministic measurement tools, allowing the model to alternate between image understanding and precise computation. Qwen3-VL-4B-Instruct retains visual access to the radiograph throughout inference, invoking tools to identify anatomical landmarks, compute measurements, and evaluate diagnostic thresholds as needed. This creates a multi-turn reasoning loop that interleaves perception and measurement, enabling the model to combine visual context with exact quantitative evidence before reaching a diagnosis.
Figure 4. Tool-augmented quantitative reasoning pipeline. The orchestrator mediates a multi-turn loop: the VLM reasons over the image (perception), emits structured tool calls, receives deterministic results, and synthesizes the final diagnosis.Despite requiring no task-specific training, this approach substantially outperforms perception-only inference across all evaluated measurement-based conditions. The results suggest that for threshold-dependent diagnoses, direct computation of clinically defined measurements is more reliable than visual approximation alone.
More broadly, this measurement-augmented approach could augment clinical workflows by expanding the set of quantitative assessments routinely derived from chest radiographs. For example, aortic dilation is not typically quantified on CXR and is often detected only incidentally on CT scans obtained for other indications. As delayed detection can contribute to adverse cardiovascular outcomes, reliable CXR-based screening could enable earlier identification and follow-up of aortic dilation.
ConditionPerception F1Tool F1Δ F1Cardiomegaly74.5696.00+21.4Mediastinal Widening72.6397.47+24.8Aortic Knob Enlargement60.3199.76+39.5Ascending Aorta Enlargement39.33100.00+60.7Descending Aorta Enlargement†28.57100.00+71.4Average+43.6Table 3: Perception-only versus tool-augmented measurement. The average F1 improvement is 43.6 percentage points across five conditions. Validation on Indian clinical data: Rare ICU conditions and CT-confirmed enlargementResearch ethics and data use: The Narayana Health evaluations used de-identified, retrospective clinical data under applicable institutional ethics review and data-use approvals. Narayana Health approved publication of the study results described here.
Study 1: Inpatient and ICU conditionsTo assess real-world generalizability in a research setting, we evaluated CARE-X on 1,047 de-identified chest radiographs from Narayana Health, annotated for five rare, high-acuity conditions with prevalence ranging from 2.6% to 5.2%—reflecting realistic clinical distributions where missed diagnoses carry severe consequences.
FractureMediastinal ShiftPneumoperitoneumPneumothoraxTubes & Lines Abnormal PlacementModelSens / SpecSens / SpecSens / SpecSens / SpecSens / SpecCheXOne0.41 / 0.900.80 / 0.780.67 / 0.980.85 / 0.720.03 / 0.97MedGemma0.05 / 1.001.00 / 0.530.00 / 1.000.52 / 0.730.18 / 0.87CARE-X0.62 / 0.640.83 / 0.860.89 / 0.940.83 / 0.750.66 / 0.77Table 4: ICU pathology classification on Indian hospital data. CARE-X achieves the most balanced performance.CARE-X achieves the highest sensitivity in three out of five conditions while maintaining reasonable specificity, demonstrating generalization to low-prevalence clinical settings.
Study 2: CT-confirmed enlargement conditionsIn a retrospective study to measure pure recall efficacy, we evaluated measurement-dependent conditions such as mediastinal widening findings including aortic enlargement, hilar mass, and pulmonary artery enlargement on a outpatient cohort of 122 positive cases with CT-confirmed ground truth, avoiding the subjectivity of radiologist consensus on borderline enlargement findings on CXR. In the overlay setting, the VLM receives the original radiograph alongside a second image with condition-relevant anatomical segmentation masks — offering spatial guidance without direct access to measurement tools.
The tool-augmented variant reached 94.26% recall, a +10.65 percentage-point gain over the best perception-only baseline. Where CT or echocardiography access is limited, reliable triage from a widely available modality like chest X-ray can cut both unnecessary referrals and missed diagnoses.
Figure 5: Recall on the CT-confirmed enlargement cohort across perception-only, overlay-assisted, and tool-augmented inference. (Study 2)In a related study (accepted at EACTS conference 2026), for mild aortic dilation, the measurement-driven reasoning approach detected 40 of 43 CT-confirmed cases (93% sensitivity), compared to just 5 of 43 (12%) identified on the initial radiology reads, where aortic enlargement is usually not the primary indication for the chest X-ray. This corresponds to 35 additional mild cases that were surfaced but missed during the initial CXR interpretation. These results suggest that explicit quantitative measurements may help identify borderline enlargement that is difficult to assess through visual inspection alone.
What this does and doesn’t showThese numbers are all recall, i.e., how many true positives we catch. This was the focus of the initial study because, in triage, a missed diagnosis is typically the costlier failure mode, and CT-confirmed ground truth gave us a clean way to measure it without relying on radiologist consensus for the difficult cases.
Recall, however, captures only one dimension of diagnostic performance. A model that flags everything achieves perfect recall and is useless in practice. An extended study is underway that includes CT-confirmed negative cohorts as well. Preliminary results are promising, and further studies are planned to explicitly evaluate the viability of quantitative aortic measurements on chest X-ray as a screening tool for aortic dilation.
CARE-X: Toward clinically useful radiology AICARE-X demonstrates that discriminative and generative objectives can be effectively combined within a unified radiology AI model. By jointly training classification, grounding, and language capabilities, the model supports both flexible report generation and calibrated, threshold-adjustable predictions. The separate measurement study further highlights a practical division of labor between learned reasoning and deterministic computation: the VLM provides visual understanding and identifies relevant evidence, while measurement-dependent diagnoses are computed through transparent, tool-based calculations. Retrospective evaluation on clinically challenging Narayana Health cohorts provides encouraging evidence of the potential of this approach for real-world radiology applications. The clinical relevance of this research is underscored by the selection of the AI-based aortic dilatation screening application as a finalist for showcase at the IHF Innovation Hub, World Hospital Congress 2026, recognizing its potential to support earlier detection and clinical decision-making in cardiovascular care.
Looking ahead, CARE-X can be extended beyond its current capabilities through structured report generation, richer differential diagnosis support, and tighter integration of tools within the model itself. The framework could also benefit from incorporating broader clinical context, including laboratory results and patient history, enabling more comprehensive clinical reasoning.
CARE-X is a research model, not a Microsoft product offering or medical device. It has not been cleared or approved by any regulatory authority and is not intended or validated for clinical diagnosis, screening, patient care, or clinical decision-making. The results described are retrospective research findings and do not establish safety, effectiveness, or suitability for clinical use.
Paper co-authors:
Mercy Ranjit, Anirban Porya (opens in new tab), Niharika Vadlamudi (opens in new tab), Nikhilesh E (opens in new tab), Sathvik Joel (opens in new tab), Prasanth V V (opens in new tab), Tanuja Ganu, Abhyuday Swamy (opens in new tab), Pranay Umredkar (opens in new tab), Pradeep Narayan (opens in new tab), Vivek Rajagopal (opens in new tab)
Collaborators: Medha AI (opens in new tab), Narayana Health (opens in new tab)
Opens in a new tabThe post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.
Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement
Research Note: CARE-X is a research model and not a Microsoft product offering or medical device. It has not been cleared or approved by any regulatory authority and is not intended for clinical diagnosis, screening, or patient care. The results described below are retrospective research findings and do not establish the safety, effectiveness, or suitability of CARE-X for any clinical use. References to potential workflows describe areas for future research, not currently available capabilities or recommended uses.
At a glance- The challenge: Chest X-ray interpretation spans diverse tasks that require both expressive report generation and calibrated diagnostic predictions.
- CARE-X is a unified chest X-ray VLM for diverse clinical interpretation tasks. It combines generation and structured prediction to provide both free-text reasoning and deterministic outputs.
- CARE-X uses reinforcement learning (DAPO) to reward clinical correctness in a multi-task setting.
- In a separate research experiment from CARE-X, we paired Qwen3-VL-4B-Instruct with deterministic measurement tools to evaluate whether direct computation could improve performance on measurement-dependent conditions compared with visual approximation alone.
- Validated on real-world Indian clinical data from Narayana Health, including rare ICU pathologies and CT-confirmed enlargement conditions.
A clinically useful radiology AI system must support a wide range of tasks, adapt to different workflows, and produce outputs that are medically accurate.
Radiologists and other clinicians use chest X-rays for many different purposes. A clinically useful AI system must be able to support that range of tasks. It may be asked to generate detailed findings and concise impressions for a report, answer questions about the presence, absence, or location of a finding, identify medical devices and assess their placement, or pinpoint exactly where an abnormality appears in an image.
These tasks also require different kinds of outputs, from narrative reports to calibrated diagnostic scores. And above all, they require clinical accuracy. A report could ostensibly be perfectly written yet clinically wrong if it misses a finding, reverses a negation, or misidentifies a location. Certain findings could be trivial in one context and vital to identify in another.
CARE-X was developed as a research model to explore how a unified approach can address these diverse demands. The system combines generative and discriminative capabilities, clinically aligned optimization, and tool-based reasoning to support a broader range of radiology workflows while maintaining clinical fidelity.
Azure AI Foundry LabsGet a glimpse of potential future directions for AI, with these experimental technologies from Microsoft Research.
Azure AI Foundry Opens in a new tab Gaps in current radiology vision-language modelsDespite the impressive task breadth of recent models, critical gaps remain between what radiologists need and what current systems deliver:
- No calibrated confidence for diagnostic decisions. Generative VLMs predict diagnoses as free text, but they typically do not provide calibrated confidence scores. In clinical settings, confidence matters. Clinicians cannot tune sensitivity–specificity trade-offs across clinical contexts—an important requirement for real-world deployment. Discriminative models provide these properties but lack the flexibility of open-ended generation.
- Cross-entropy loss does not optimize clinical fidelity. Standard training methods treat all token-level errors similarly, regardless of their clinical consequences. A coordinate mistake may be penalized no more than a harmless wording change. A “yes” can be flipped to a “no” even though the clinical meaning is completely different. Missing a life-threatening finding may carry the same training penalty as omitting a minor observation. As a result, models are not explicitly optimized for what matters most in patient care.
- No capability for measurement-dependent findings. Some radiological findings require more than visual recognition. Radiological signs such as cardiomegaly, mediastinal widening etc. depend on precise measurements. For example, a model may correctly recognize whether a chest radiograph was acquired using an AP or PA view. But determining cardiomegaly requires measuring the cardiac and thoracic widths and determining the cardiothoracic ratio. Those quantities should be measured and computed rather than visually approximated while considering variables such as type of view, exposure, rotation of the patient etc.
Together, these gaps call for more than a fluent generative model. The system must combine broad task coverage, structured predictions, clinically aligned optimization, and quantitative tools where direct measurement is required.
CARE-X: One model, flexible outputsCARE-X brings these diverse interpretation capabilities into one model, using generative or dual inference according to the needs of each task:
Task typeWhat CARE-X doesInference modeReport generation: FindingsProduces the detailed findings sectionGenerativeReport generation: ImpressionProduces the concise diagnostic impressionGenerativePresence and negation assessmentDetermines whether a pathology is present or absent and handles negationDual: generative + auxiliary headDisease location assessmentIdentifies where an abnormality appearsGenerativeFine-grained multilabel disease classificationCategorizes abnormalities across multiple labelsGenerativeMultilabel tubes and lines classificationIdentifies visible medical devicesGenerativeAbnormal placement detection of tubes and linesDetermines whether a device is positioned incorrectlyDual: generative + auxiliary headAbnormality phrase groundingLocalizes a described pathological findingDual: generative + auxiliary headAnatomical groundingLocalizes 29 anatomical regionsDual: generative + auxiliary headTable 1: CARE-X task coverage and inference modesDual inference means that a single forward pass produces both an autoregressive response and a structured auxiliary-head prediction with a confidence score. This provides free-text flexibility alongside threshold-adjustable outputs for tasks where operating-point control matters.
The CARE-X architecture and training approachCARE-X is built on a SigLIP2-so400M vision encoder and a Phi-4-mini-instruct (3.8B) language model connected through a lightweight adapter. To support both free-text generation and structured clinical predictions, the model augments the shared language backbone with task-specific auxiliary heads for classification and visual grounding. These heads provide calibrated diagnostic predictions and spatial localization signals while sharing representations with the generative language model. Rather than being trained independently, they are co-trained with the language-modeling objective, allowing structured supervision to enrich shared representations and improve generative performance on the same tasks.
Training. CARE-X uses a three-stage supervised fine-tuning pipeline (vision pre-training, adapter/head training, and LoRA adaptation) followed by DAPO-based reinforcement learning. DAPO optimizes task-specific rewards for clinical reporting, diagnostic accuracy, and spatial grounding quality.
Figure 1. The CARE-X model. (Left) Supervised fine-tuning with task-specific heads — classification, grounding, and language modeling — sharing the same Phi-4-mini-instruct backbone. The classification head outputs calibrated P(Yes)/P(No) scores; the grounding head outputs bounding box coordinate with confidence; the language modeling head generates free-text responses. (Right) DAPO with task-specific rewards for multi-task reinforcement alignment across report generation, grounding, and VQA. Auxiliary supervision: Structured prediction strengthens generationA central finding of this work is that co-training discriminative auxiliary heads with a generative VLM enriches shared representations, leading to stronger generative performance on the same tasks while also providing calibrated structured predictions.
Grounding improvementsThe auxiliary grounding head consistently improves localization over generative decoding. On anatomical grounding (Chest ImaGenome), mAP and mIoU increase by +28.2 pp and +6.2 pp, while the largest gains occur on phrase grounding (PadChest), with +24.6 pp mAP and +14.1 pp mIoU. The composite spatial loss enhances geometric precision in shared representations.
DAPO bridges the gap to dedicated detection headsDAPO-trained generative output approaches or exceeds the SFT auxiliary detection head. On Anatomy grounding, CARE-X generative (0.868 mAP) surpasses the SFT detection head (0.865). This is practically significant—it demonstrates that reward-aligned learning can bring autoregressive spatial decoding to parity with structured prediction, offering clinicians a single generative inference mode without requiring auxiliary heads at test time.
Calibrated classification with tunable operating pointsBeyond representation enrichment, the classification head offers a distinct deployment advantage: calibrated probability scores with tunable thresholds allow clinicians to shift between high-sensitivity screening and high-specificity confirmation from a single forward pass—a capability purely generative architectures cannot provide.
ModelInference SettingSensitivity ↑PPV ↑F1 ↑CARE-XGenerative0.9320.8950.913CARE-X (Th=0.5)Auxiliary Head0.9430.8850.913CARE-X (Th=0.6)Auxiliary Head0.8550.9270.890CheXOneGenerative0.8780.8540.866MedGemmaGenerative0.7980.8860.839Table 2: Abnormality classification performance on Chest ImaGenome. Adjustable thresholds enable operating-point selection. Strong report generation across four benchmarksWithin the paper’s comparison set, CARE-X achieves the strongest performance on most reported metrics across MIMIC-CXR, IU-Xray, CheXpert-Plus, and ReXGradient. CRIMSON, a held-out metric that evaluates abnormal findings and weights errors by clinical severity, suggests these gains reflect clinically meaningful improvements rather than reward-specific optimization.
Figure 2. CRIMSON scores (↑) for CARE-X against baseline report-generation models across four chest X-ray datasets — ReXGradient, MIMIC-CXR, IU-Xray, and CheXpert-Plus. CARE-X (highlighted) achieves the highest CRIMSON score on every dataset. CARE-X reaches 94% accuracy on ReXVQACARE-X ranks first on the ReXrank RexVQAleaderboard (opens in new tab) as of August 2026. On the ReXVQA benchmark (41,007 question–answer pairs across five clinically relevant categories), CARE-X reaches 94% overall accuracy, six percentage points above the next-best publicly reported model.
Figure 3: ReXVQA accuracy across five findings-quality dimensions — negation, presence, location, differential diagnosis, geometric information, and overall. CARE-X consistently outperforms CheXOne-R1 and MedGemma on every axis, with the largest margins in differential diagnosis, location assessment and negation. Tool-augmented measurement: Interleaving perception and computationSome radiological findings depend on quantitative measurements rather than visual patterns. In a separate research experiment from CARE-X, we built an inference-time pipeline that combines Qwen3-VL-4B-Instruct with deterministic measurement tools, allowing the model to alternate between image understanding and precise computation. Qwen3-VL-4B-Instruct retains visual access to the radiograph throughout inference, invoking tools to identify anatomical landmarks, compute measurements, and evaluate diagnostic thresholds as needed. This creates a multi-turn reasoning loop that interleaves perception and measurement, enabling the model to combine visual context with exact quantitative evidence before reaching a diagnosis.
Figure 4. Tool-augmented quantitative reasoning pipeline. The orchestrator mediates a multi-turn loop: the VLM reasons over the image (perception), emits structured tool calls, receives deterministic results, and synthesizes the final diagnosis.Despite requiring no task-specific training, this approach substantially outperforms perception-only inference across all evaluated measurement-based conditions. The results suggest that for threshold-dependent diagnoses, direct computation of clinically defined measurements is more reliable than visual approximation alone.
More broadly, this measurement-augmented approach could augment clinical workflows by expanding the set of quantitative assessments routinely derived from chest radiographs. For example, aortic dilation is not typically quantified on CXR and is often detected only incidentally on CT scans obtained for other indications. As delayed detection can contribute to adverse cardiovascular outcomes, reliable CXR-based screening could enable earlier identification and follow-up of aortic dilation.
ConditionPerception F1Tool F1Δ F1Cardiomegaly74.5696.00+21.4Mediastinal Widening72.6397.47+24.8Aortic Knob Enlargement60.3199.76+39.5Ascending Aorta Enlargement39.33100.00+60.7Descending Aorta Enlargement†28.57100.00+71.4Average+43.6Table 3: Perception-only versus tool-augmented measurement. The average F1 improvement is 43.6 percentage points across five conditions. Validation on Indian clinical data: Rare ICU conditions and CT-confirmed enlargementResearch ethics and data use: The Narayana Health evaluations used de-identified, retrospective clinical data under applicable institutional ethics review and data-use approvals. Narayana Health approved publication of the study results described here.
Study 1: Inpatient and ICU conditionsTo assess real-world generalizability in a research setting, we evaluated CARE-X on 1,047 de-identified chest radiographs from Narayana Health, annotated for five rare, high-acuity conditions with prevalence ranging from 2.6% to 5.2%—reflecting realistic clinical distributions where missed diagnoses carry severe consequences.
FractureMediastinal ShiftPneumoperitoneumPneumothoraxTubes & Lines Abnormal PlacementModelSens / SpecSens / SpecSens / SpecSens / SpecSens / SpecCheXOne0.41 / 0.900.80 / 0.780.67 / 0.980.85 / 0.720.03 / 0.97MedGemma0.05 / 1.001.00 / 0.530.00 / 1.000.52 / 0.730.18 / 0.87CARE-X0.62 / 0.640.83 / 0.860.89 / 0.940.83 / 0.750.66 / 0.77Table 4: ICU pathology classification on Indian hospital data. CARE-X achieves the most balanced performance.CARE-X achieves the highest sensitivity in three out of five conditions while maintaining reasonable specificity, demonstrating generalization to low-prevalence clinical settings.
Study 2: CT-confirmed enlargement conditionsIn a retrospective study to measure pure recall efficacy, we evaluated measurement-dependent conditions such as mediastinal widening findings including aortic enlargement, hilar mass, and pulmonary artery enlargement on a outpatient cohort of 122 positive cases with CT-confirmed ground truth, avoiding the subjectivity of radiologist consensus on borderline enlargement findings on CXR. In the overlay setting, the VLM receives the original radiograph alongside a second image with condition-relevant anatomical segmentation masks — offering spatial guidance without direct access to measurement tools.
The tool-augmented variant reached 94.26% recall, a +10.65 percentage-point gain over the best perception-only baseline. Where CT or echocardiography access is limited, reliable triage from a widely available modality like chest X-ray can cut both unnecessary referrals and missed diagnoses.
Figure 5: Recall on the CT-confirmed enlargement cohort across perception-only, overlay-assisted, and tool-augmented inference. (Study 2)In a related study (accepted at EACTS conference 2026), for mild aortic dilation, the measurement-driven reasoning approach detected 40 of 43 CT-confirmed cases (93% sensitivity), compared to just 5 of 43 (12%) identified on the initial radiology reads, where aortic enlargement is usually not the primary indication for the chest X-ray. This corresponds to 35 additional mild cases that were surfaced but missed during the initial CXR interpretation. These results suggest that explicit quantitative measurements may help identify borderline enlargement that is difficult to assess through visual inspection alone.
What this does and doesn’t showThese numbers are all recall, i.e., how many true positives we catch. This was the focus of the initial study because, in triage, a missed diagnosis is typically the costlier failure mode, and CT-confirmed ground truth gave us a clean way to measure it without relying on radiologist consensus for the difficult cases.
Recall, however, captures only one dimension of diagnostic performance. A model that flags everything achieves perfect recall and is useless in practice. An extended study is underway that includes CT-confirmed negative cohorts as well. Preliminary results are promising, and further studies are planned to explicitly evaluate the viability of quantitative aortic measurements on chest X-ray as a screening tool for aortic dilation.
CARE-X: Toward clinically useful radiology AICARE-X demonstrates that discriminative and generative objectives can be effectively combined within a unified radiology AI model. By jointly training classification, grounding, and language capabilities, the model supports both flexible report generation and calibrated, threshold-adjustable predictions. The separate measurement study further highlights a practical division of labor between learned reasoning and deterministic computation: the VLM provides visual understanding and identifies relevant evidence, while measurement-dependent diagnoses are computed through transparent, tool-based calculations. Retrospective evaluation on clinically challenging Narayana Health cohorts provides encouraging evidence of the potential of this approach for real-world radiology applications. The clinical relevance of this research is underscored by the selection of the AI-based aortic dilatation screening application as a finalist for showcase at the IHF Innovation Hub, World Hospital Congress 2026, recognizing its potential to support earlier detection and clinical decision-making in cardiovascular care.
Looking ahead, CARE-X can be extended beyond its current capabilities through structured report generation, richer differential diagnosis support, and tighter integration of tools within the model itself. The framework could also benefit from incorporating broader clinical context, including laboratory results and patient history, enabling more comprehensive clinical reasoning.
CARE-X is a research model, not a Microsoft product offering or medical device. It has not been cleared or approved by any regulatory authority and is not intended or validated for clinical diagnosis, screening, patient care, or clinical decision-making. The results described are retrospective research findings and do not establish safety, effectiveness, or suitability for clinical use.
Paper co-authors:
Mercy Ranjit, Anirban Porya (opens in new tab), Niharika Vadlamudi (opens in new tab), Nikhilesh E (opens in new tab), Sathvik Joel (opens in new tab), Prasanth V V (opens in new tab), Tanuja Ganu, Abhyuday Swamy (opens in new tab), Pranay Umredkar (opens in new tab), Pradeep Narayan (opens in new tab), Vivek Rajagopal (opens in new tab)
Collaborators: Medhai AI, Narayana Health (opens in new tab)
Opens in a new tabThe post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.


