An agent browsing a site takes dozens of actions before anything succeeds or fails. Scoring only the ending teaches it very little. Our NeurIPS 2025 Spotlight paper builds a reward model that reads every step.
A web agent asked to book a flight takes thirty or forty actions before anything can be called success or failure. It reads a page, decides a link is the right one, waits, reads again, fills a field, misreads a date picker, recovers, continues.
The standard way to train and evaluate that agent is to look at the end. Did it book the flight. That single bit of feedback has to explain forty decisions, and it is a poor teacher: it cannot distinguish an agent that made one fatal mistake at step 31 from one that was lost from step 3 and stumbled into the right answer.
Web-Shepherd, published as a Spotlight at NeurIPS 2025, is a process reward model built for that problem. It reads every step.
Reinforcement learning from a terminal reward works when the horizon is short or the credit assignment is easy. Web navigation is neither. The reward is sparse, arrives late, and is the same for a near-miss as for a total failure.
The practical consequence is not just slow learning. It is that the agent has no signal to distinguish a productive detour from an unproductive one, so it never learns the difference.
A process reward model scores intermediate steps rather than final outcomes. For each action the agent proposes, it asks whether that action makes progress toward the instruction, given everything that has happened so far.
The difficulty is that "progress" is not well defined for an arbitrary web task. A single scalar score per step is easy to produce and hard to trust, because the judge has no explicit account of what progress would consist of.
The central design decision in Web-Shepherd is to decompose an instruction into an explicit checklist of subgoals first, and then evaluate each step against that checklist.
This changes the judgement from an opaque preference into a set of specific, checkable claims: has the destination been entered, has a date been selected, has the correct fare class been chosen. The judge is now doing something a human can audit, and its failures become legible rather than mysterious.
Making the reward model explain what it is scoring against is not only an interpretability win. It measurably improves the score.
The work introduces a training corpus of step-level annotations with checklists attached, and an evaluation benchmark for judging web-navigation trajectories. Both are necessary: without a step-level benchmark there is no way to tell whether a process reward model is better than a well-prompted general model.
The result that matters commercially is that a purpose-built process reward model outperforms substantially larger general-purpose models used as judges, at a fraction of the inference cost.
That is the shape of result that changes deployment decisions. Judging every step of every trajectory is only affordable if the judge is small, and the assumption had been that a small judge would be too weak to be useful.
Two are worth naming. Agreeable graders: a judge trained on preference data drifts toward approving whatever it is shown, and needs explicit negative supervision to stay useful. And reward hacking: an agent optimised against any fixed judge will eventually find the actions that score well without accomplishing anything, which is an argument for keeping outcome evaluation alongside process evaluation rather than replacing it.
The reason this line of work sits on an implementation firm’s site rather than only in a proceedings volume is that the same problem appears the moment an agent is pointed at an internal system.
An agent processing an invoice, reconciling a shipment or updating a record takes many steps, and the business cannot wait for the final outcome to discover it went wrong at step four. Step-level evaluation is what makes an agent auditable, and auditability is the precondition for letting one touch anything that matters.
Web-Shepherd was accepted as a Spotlight at NeurIPS 2025 and was listed as the top paper of the day on Hugging Face on publication.
Every operation has documentation that stopped being true at some point. This is a method for finding that point: extracting entities from records, building a graph of what should be, and reading the divergences against what is.
The pilot worked. A year later nothing is in production. The failure is almost never model quality. It is data nobody owns, permissions nobody granted, and a workflow nobody changed.
A seaweed supply operation running from Korean farms and factories to U.S. warehouses kept its truth in four disconnected spreadsheets. Discovery found 25 places the sources disagreed and 34 questions nobody had answered.