Large Language Models as Robot Task Planners in Unstructured Environments

A robot built for the factory floor fails the moment its task moves off the floor and into a world that changes shape. The subject of this piece is why large language models have become the leading candidate for fixing that failure, and the angle is specific: LLMs work as planners only when their semantic knowledge is coupled to perception and affordance layers that keep plans tied to what a robot's body can actually do.

Why unstructured environments break conventional robot planning

Classical task-and-motion planning built modern industrial automation, but it runs on handcrafted rules and fixed schedules. These pipelines assume the world holds still: the same bin, the same part, the same starting pose, every time. That assumption held for decades on automotive lines, where a weld point was always in the same place relative to the same chassis. It collapses the moment the task requires handling things that differ from one instance to the next.

Battery disassembly for electric vehicles makes the collapse concrete. Erdogan et al. (2026) point out that battery packs vary across makes, models, and years, so no fixed teardown sequence can cover the range of packs a facility will see. What the task needs instead is a system that can look at whatever pack arrives, figure out its layout, and plan a disassembly sequence on the spot.

The same pressure appears anywhere a robot has to work on material it hasn't seen configured in that exact way before: sorting mixed waste streams, handling parts inside a nuclear decommissioning site, working in a home. What breaks classical planning in these settings isn't the mechanics of moving an arm. A person gives an instruction in ordinary language, loose and incomplete by the standards of a formal planner, and turning that instruction into a sequence of physically valid steps is exactly the step rigid, rule-based systems cannot do.

Semantic World-Knowledge at Scale

Large language models bring something no hand-built planner has: broad knowledge about how objects, tasks, and human goals relate to each other, learned from far more text than any engineer could encode as rules. A transformer model trained on this scale can reason about what "clean up after the party" probably involves, without anyone having written out the steps in advance. It can infer that cups go in the sink, that trash goes in a bag, that surfaces get wiped, because that structure is present throughout the language it learned from.

Work on LLM-based task planning for service robots (ScienceDirect, 2026) frames the contribution this way: the model converts unstructured natural language into an executable sequence of actions, using the semantic understanding built into the transformer architecture. That is a different capability than the fixed-schedule automation that dominated automotive manufacturing from the late 1970s through the 1980s. LLM-based planning succeeds by absorbing ambiguity directly, taking a vague instruction and producing a concrete, if imperfect, first guess at what to do.

That guess carries a real limitation. An LLM can know, as a fact encoded in its training, that a cup holds liquid and that liquid spills if tipped. Semantic knowledge describes the world in general. Closing that distance is a separate architectural problem, and it's the one the field has spent the most engineering effort solving.

Grounding: Closing the Gap Between What an LLM Says and What a Robot Can Do

The fix is to pair the LLM's language output with two more components: a perception system that describes the scene in terms the model can reason over, and a set of learned affordance functions that score whether a given action is actually possible right now. Together they turn a plausible-sounding sentence into a plan the robot's body can carry out.

SayCan, developed at Google in 2022, is the clearest demonstration of this architecture working. Researchers ran it on 101 real-world tasks with a mobile robot in an actual kitchen, and the conditions were zero-shot, so the robot had not been specifically trained on those exact task instructions beforehand. The "Say" half of the system proposes actions useful for a stated goal; the "Can" half, trained as a language-conditioned value function through reinforcement learning, judges which of those actions the robot can presently execute.

None of this works without a prior step: the scene has to be translated into a description the language model can use, naming what objects are present, where they sit, and what state they're in. Affordance filtering has nothing to filter until perception has named the objects in play. Erdogan et al. (2026) build this pipeline out in a multi-robot cell, where a perception back-end produces a structured, object-level description of the scene, an ensemble of language models proposes candidate action sequences from that description, a verifier checks formatting and the order operations need to happen in, and a separate deterministic filter throws out any step that references an object the perception system never saw.

The SayCan results carry a second lesson beyond the raw numbers. Planning mistakes do not stay contained to the plan, either. If planning introduces an error, it compounds once the robot starts moving, so grounding quality works as a safety variable in these systems, not merely a score on a benchmark.

The Closed-Loop Feedback Requirement

A plan generated once, before the robot moves, and then executed without revision is called open-loop control, and it loses consistently to systems that check in on themselves during execution. An unstructured environment does not hold still for the length of a task. An object gets moved, a door turns out to be locked, a drawer the plan assumed was empty has something in it. No plan written in advance can anticipate every one of these, so the planner has to be able to hear about them as they happen.

Inner Monologue, one of the systems built around this requirement, feeds success signals, updated scene descriptions, and direct human feedback back into the language model's reasoning process, so the model can revise its plan mid-task. The Mitsubishi Electric Research Laboratories controller takes the same idea and splits it into two distinct roles: an LLM Planner that proposes the action sequence, and a separate LLM Evaluator that reviews the history of actions and observations, flags what information is missing, and passes an updated picture of the state back to the Planner. That evaluation step is what informs the replanning that follows, not failure alone triggering it blindly.

A further distinction matters here: reacting to failure after it happens is not the same as anticipating trouble before it arrives. Peña-Narvaez et al. (Robotics, 2026) build a system oriented around the second mode. Their architecture uses a semantic evaluation heuristic to track features of the environment continuously, forecast hazards before they cause a failure, and decide when the current plan needs a targeted repair. That targeted approach disrupts a task already in progress less than scrapping it outright would.

BrainBody-LLM, a closed-loop framework built on GPT-4, shows what this architecture can do on a hard benchmark: it improves average task success by a meaningful margin over prior state-of-the-art methods on the VirtualHome Embodied benchmark, and it carried over to a Franka Research 3 robotic arm performing real-world tasks, not just simulated ones. Closed-loop feedback is a load-bearing part of what makes these systems work.

Irreversibility and Predictive Monitoring

Closed-loop feedback assumes something that is not always true: that a failure, once detected, can be undone. Plenty of failures in the physical world do not work that way. A dropped battery cell in a disassembly line can create a hazard that makes the original task impossible to complete safely, no matter how good the next plan is.

A 2026 paper on arXiv builds a risk-aware architecture around this gap and evaluates it against a benchmark called TRUST-NAV. Measured against conventional LLM planners, it detects ambiguous instructions and rejects semantically conflicting plans far better, so it catches the problem before the robot acts on it, not after.

Predictive vision-language monitoring watches a scene continuously and tries to flag developing risk instead of waiting for a discrete failure event, and it has improved task completion in simulated dynamic robot tasks. Peña-Narvaez et al. (Robotics, 2026) again anchor the forecasting side of this: their semantic evaluation heuristic tracks environmental features continuously enough to flag a dangerous situation before it turns into a failure rather than after the damage is done. The literature treats safety guarantees on the robot's basic action primitives, along with corrective strategies validated in simulation ahead of time, as requirements for deployment in any setting where a mistake carries real consequences, not features to add once the system already works.

Hallucination and physical constraint violations as the two persistent failure modes at the plan level

Grounding and feedback together still leave two distinct kinds of errors sitting inside the plans these systems generate. The first is hallucination: the model references an object that isn't in the scene, or proposes a step that assumes a capability the robot doesn't have, a knowledge problem rooted in the fact that the model is generating plausible-sounding text and will sometimes generate something plausible that happens not to be true of the room in front of it. The risk compounds with plan length: the longer the sequence of steps, the higher the chance that at least one step references something that was never there.

Erdogan et al. (2026) attack this failure mode directly with a deterministic filter that checks every proposed plan step against the perception system's object list and rejects any step naming something the robot cannot see. Their ablation testing, which compares the system with and without that filter, shows it produces a meaningful improvement in how often the full plan sequence comes out correct.

The second failure mode is different in kind: a plan that names only real objects and valid-sounding steps but still asks the robot to do something physically impossible, like reach a location outside its arm's range or move through a path that ends in a collision. Smaller models built with explicit constraint-awareness outperformed larger, unconstrained models on multi-robot control tasks in that work, so the fix looks architectural, not a matter of scaling up. Language models also run into a separate ceiling around precise numerical computation, the kind needed for trajectory optimization and mid-level motion planning, where getting the general idea right isn't the same as getting the numbers right.

These two failure modes call for different remedies because they come from different places: hallucination is solved by checking plan content against what perception actually sees, while constraint violation is solved by building physical-constraint awareness into the reasoning process itself. The ensemble approach from Erdogan et al. (2026), running several language models in parallel and using a separate verifier to check formatting and ordering, shows one way to push full-sequence correctness higher than a single model achieves working alone, though it addresses the hallucination side of the problem more directly than the constraint-violation side.

The Kambhampati Critique of Whether LLMs Can Plan

Subbarao Kambhampati and his colleagues have raised an objection that targets a shared assumption running through every system described above. Their claim, presented at ICML 2024, is that auto-regressive language models cannot do planning or self-verification on their own terms. In their framing, LLMs function as universal approximate knowledge sources, useful and genuinely valuable for that role, but not planners in any sense that should be trusted to verify its own output. If that claim holds, every closed-loop architecture discussed so far, including ones that split planning and evaluation into separate LLM roles, is still built on a model checking its own homework, dressed up in a more convincing-looking wrapper.

Kambhampati's answer is the LLM-Modulo Framework: it keeps the language model in the loop as a generator of candidate plans, but it routes every candidate through an external, model-based verifier before anyone trusts it, in a tight back-and-forth between the two. The LLM proposes, and a verified symbolic method decides. A follow-on study, by Bhambri, Verma, and Kambhampati in 2025, extends the same skepticism to reasoning-style prompting techniques built around "think tag" steps, questioning whether this kind of prompting makes a plan genuinely more reliable or only makes the model's reasoning look more reliable on the page.

The disagreement is not academic in the narrow sense. It points the field toward one of two different research programs: keep scaling language models further into the planning role directly, or treat them as proposal generators and hand verification to symbolic methods that can actually guarantee correctness. Those two programs produce different systems, built on different assumptions about what a language model can be trusted to do alone.

LLM+P sits as a working answer to that fork, translating a natural-language problem into PDDL, the formal representation classical planners use, and then handing the actual search for a valid plan to a classical planner built for that job. It keeps the LLM's flexibility with language and intent, and it hands the part requiring guaranteed correctness to a method built to provide it. Seen against Kambhampati's critique, the ensemble-and-verifier design in Erdogan et al. (2026), and the split between LLM Planner and LLM Evaluator in the Mitsubishi Electric system, read less like isolated engineering choices and more like partial, independently-arrived-at agreement with the same underlying point: a language model should not be asked to verify its own plan.