Plans with Gaps in the World:
Task and Motion Planning for Execution in the Real
Abstract: — Task and motion planning represents a powerful set of hybrid planning methods that combine reasoning over discrete task domains and continuous motion generation. Traditional reasoning necessitates task domain models and enough information to ground actions to motion planning queries. Gaps in this knowledge often arise from sources like occlusion or imprecise modeling. This work generates task and motion plans that include actions cannot be fully grounded at planning time. During execution, such an action is handled by a provided human-designed or learned closed-loop behavior. Execution combines offline planned motions and online behaviors till reaching the task goal. Failures of behaviors are fed back as constraints to find new plans. Forty real-robot trials and motivating demonstrations are performed to evaluate the proposed framework and compare against state-of-the-art. Results show faster execution time, less number of actions, and more success in problems where diverse gaps arise. The experiment data is shared for researchers to simulate these settings. The work shows promise in expanding the applicable class of realistic partially grounded problems that robots can address.
The authors are affiliated to the Department of Computer Science at Rice University, Houston, TX, USA. Rahul Shome is currently with Australian National University. Corresponding author can be contacted at Tianyang (dot) Pan (at) rice (dot) edu. The work was supported in part by NSF 1830549, NSF 2008720, and Rice University Funds.
Fig 1: A task to move three objects to the goal tray. The full states of the dotted objects are not known precisely. A task and motion plan will have gaps which can only be filled in during execution, e.g., the blue object’s pose is known after opening the drawer. The problem imposes constraints visualized in a block-world-like diagram. This work introduces behaviors to bridge gaps during execution, and recovers constraints from the task, motion, and execution domains.
Fig 2: The source of gaps can be diverse, including (from left to right) occlusions, imprecise simulations, and modeling gaps.
Fig 3: The proposed framework. We leverage a typical task and motion planner to plan as much as we can for the part of the task that we have enough information, and use given closed-loop behaviors to fill in the gaps in the plan where partial grounding exists. When such behaviors still fail to execute, we feed back the failure as symbolic constraints to replan.
Fig 4: Horizontal Stacking Benchmark: The task is to move the objects to the goal positions. The initial poses of the objects are designed to have some occlusion. Strict ordering constraints exist because smaller objects cannot be grasped in the presence of a nearby larger object. Here shows 10 problems in Horizontal Stacking. Each column is the start (top) and the end (bottom) of TAMPER’s execution in each problem.
Videos: Real-world Horizontal Stacking benchmark recordings for TAMPER (left) and the baseline (right).
Fig 5: Kitchen Arrangement Benchmark: Move the objects to the goal positions. The starting position of one of the objects is inside the drawer. In this case, there are both occlusion and uncertainty in execution outcomes. Here shows the 10 problems in Kitchen Arrangement. Each column shows the start (top) and the end (bottom) of the execution of TAMPER in each problem. Since the execution outcome of the push is stochastic, we run each scenario multiple times. From left to right, columns 1-3, 4-5, 6-8, and 9-10 show four different starting poses with either 3 runs or 2 runs each. Note that in all scenarios, one object is initially inside the drawer.
Videos: Real-world Kitchen Arrangement benchmark recordings for TAMPER (left) and the baseline (right).
Fig 6: The results of the simulated ground truth benchmark. Here we have an optimal baseline. Two variants of the proposed method are evaluated with belief updates after each action execution or after each problem instance. All the proposed methods converge but show tradeoffs arising from frequency of belief updates.
Fig 6: The results of the simulated ground truth benchmark. Here we have an optimal baseline. Two variants of the proposed method are evaluated with belief updates after each action execution or after each problem instance. All the proposed methods converge but show tradeoffs arising from frequency of belief updates.
Grocery Demo Video: — In this task, the goal is to move the grocery bags into the large container. It is required that the bag of eggs must be put on top of the bag of apples for safety. Here partial grounding exists as the robot cannot distinguish between the two bags (the egg and apple photos are attached to the bags only for visualization purpose). In other words, a typical planner cannot even ground the symbolic bag to a real bag. The robot can only know what one bag contains by moving it closer to the head camera, and scanning for its barcode. Our method leverages TAMP capabilities to discover the stacking constraints during planning, and uses an online behavior to fill in the gaps of the plan (i.e., the picking actions). The behavior of picking works by repeatedly executing: 1). Finding grasp poses on the segmented point cloud; 2). Grasping and taking the object closer to the head camera; 3). Placing it back if the object does not match.
Rice University Houston TX USA