Erase whatever is on the whiteboard that's bringing down the mood, then put the thing you used to erase back on the table next to the burger. You can use the loofah. The check after the first wipe found the board still marked, so the agent wiped again. Spot, 5 min 23 s of real time, sped up.
Move all the plush toys into the plastic container that's close to the buttons. There are two plastic containers in the room. The agent picks the one by the buttons and leaves the one behind the door. 6 min 45 s of real time, sped up.Move the panda bear to the cabinet in the middle of the room, pressing the button that matches the cabinet's color once before the move and once after. The first attempt to place the bear found nowhere to put it, so the agent tried again with a more specific spot. 2 min 42 s of real time, sped up.
Abstract
A mobile manipulator asked to “put the toys away in the toy box” must decide which objects are toys and which container the user means, then find base poses, grasps, and placements it can execute. A scene representation built before the instruction is known, such as a 3D scene graph, may lack the attributes, affordances, or concepts the task needs. We present Feijoa, an embodied language agent that performs task-driven grounding and reasoning: it queries a grounding tool over a posed RGB-D scan for task-relevant objects, inspects the returned views and 3D positions, and refines its queries until they resolve the instruction. The agent calls navigation and manipulation skills through an embodiment-aware harness that checks each skill's preconditions, samples feasible base poses and action parameters, and verifies the skill's effect from observations before and after execution. On 103 open-ended grounding queries, Feijoa finds every target object with no false positives on 93.2% of queries, compared with 67.0% for the strongest baseline, a text-captioned 3D scene graph. Over 50 real-world trials on Spot, Feijoa completes 35 tasks, 17 of them after recovering from a failed tool call or verification check.
Grounding at test time
Asked for the “plush toy whose animal is drawn on the whiteboard”, the agent asks for whiteboards and plush toys, reads the two drawings off the board, asks again for a penguin and a panda, and answers with the panda.
Feasibility before execution
One navigate call, to place on a chair. The harness samples standing poses around the chair and refuses any where Spot's body would collide or its arm could not reach. The ghosts are the refused poses and the solid robot is the one it drove to. On the right, Spot at that pose.
