United States

This paper presents a teleoperation system that includes robot perception and intent prediction from hand gestures. The perception module identifies the objects present in the robot workspace and the intent prediction module which object the user likely wants to grasp. This architecture allows the approach to rely on traded control instead of direct control: we use hand gestures to specify the goal objects for a sequential manipulation task, the robot then autonomously generates a grasping or a retrieving motion using trajectory optimization. The perception module relies on the model-based tracker to precisely track the 6D pose of the objects and makes use of a state of the art learning-based object detection and segmentation method, to initialize the tracker by automatically detecting objects in the scene. Goal objects are identified from user hand gestures using a trained a multi-layer perceptron classifier. After presenting all the components of the system and their empirical evaluation, we present experimental results comparing our pipeline to a direct traded control approach (i.e., one that does not use prediction) which shows that using intent 
prediction allows to bring down the overall task execution time.

IEEE RO-MAN 2021

A System for Traded Control Teleoperation of Manipulation Tasks using Intent Prediction from Hand Gestures

human intent prediction

teleoperation

machine learning

[![](https://assets.underline.io/uploads/markdown_image/1/image/21d13e03491d7e9acf6b8c46c7979c25.png)](https://docs.google.com/spreadsheets/d/1SCDDfkQk7es4J6RULKHVx1RzbB0Mutla--Sle5SVfb0/edit?usp=sharing)

**RoboCup 2021 will be Worldwide!**

We are excited to announce that RoboCup 2021 is a fully remote event with RoboCup competitions and activities taking place all over the world. Everyone is invited to participate in, or observe, the [competition leagues](https://2021.robocup.org/leagues) and the [symposium](https://2021.robocup.org/symposium)!

**RoboCup Symposium**

The 24th RoboCup International Symposium will be held in conjunction with RoboCup 2021 in a purely online setting. Scientific papers will be presented reporting innovative, original research with relevance to robotics and artificial intelligence. Within the described scope of topics we also encouraged submissions of high-quality overview articles, papers describing real-world research, and papers reporting theoretical results. In addition to the main track with regular research papers, there will be a special track focused on systems, data, and benchmarks featuring papers describing novel, open-source hardware and software systems, as well as datasets and benchmarks relevant to the community. In addition to research paper presentations, the RoboCup Symposium 2021 includes keynote talks by Dieter Fox, Jean-Paul Laumond, and Stefanie Tellex, and several interactive sessions.

**RoboCup Introduction Video**
<iframe width="995" height="516" src="https://www.youtube.com/embed/Cnh43cCsDKU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>


Registration is required for entrance into RoboCup 2021. <br><br>
**Fees:** 
* Major Faculty or Symposium Faculty - US$50
* Major Student, Junior Mentor, Junior Student, or Symposium Student - US$20 

***Members of qualified teams:** Please contact your league for your registration password and proceed to register with your team. *<br>
***All non-team members:** Please register as "Symposium Only".*

You need to be registered to explore this event!

RoboCup 2021

The 24th RoboCup International Symposium will be held in conjunction with RoboCup 2021 in a purely online setting. Scientific papers will be presented reporting innovative, original research with relevance to robotics and artificial intelligence. Within the described scope of topics we also encouraged submissions of high-quality overview articles, papers describing real-world research, and papers reporting theoretical results. In addition to the main track with regular research papers, there will be a special track focused on systems, data, and benchmarks featuring papers describing novel, open-source hardware and software systems, as well as datasets and benchmarks relevant to the community. In addition to research paper presentations, the RoboCup Symposium 2021 includes keynote talks by Dieter Fox, Jean-Paul Laumond, and Stefanie Tellex, and several interactive sessions.

technical paper

RO-MAN 2021's theme is trustworthiness and cooperation. We chose this term well before the pandemic put the world on edge and added a new reason for us to question our own personal safety when in the company of others. But what about in the company of social robots? What would cause us to trust, or not trust, them? How do we as researchers create that feeling, that critical component of any relationship? With social robots, it is of course largely in our power to design the physical form, the control code and the interface software to maximize trustworthiness, facilitate cooperation and enhance the quality of human-robot interaction. This theme also touches upon people's expectations and perceptions of robots, as well as other social, cognitive and cultural factors that will influence human-robot interactions. We will have a special session/track on this topic, invite targeted submissions and include plenary speakers who are specialist in this area. We sincerely hope that this focus will help enlighten the design-thinking of all of us as we fabricate and program our own various systems.

To access RO-MAN Virtual 2021, [please register here.](https://ro-man2021.org/registration/)

RO-MAN 2021's theme is trustworthiness and cooperation. We chose this term well before the pandemic put the world on edge and added a new reason for us to question our own personal safety when in the company of others. But what about in the company of social robots? What would cause us to trust, or not trust, them? How do we as researchers create that feeling, that critical component of any relationship? With social robots, it is of course largely in our power to design the physical form, the control code and the interface software to maximize trustworthiness, facilitate cooperation and enhance the quality of human-robot interaction. 

A novel Synthesized Trust Learning (STL) method was developed to model human trust in the collaboration. STL explored using heterogeneous human feedback to model human trust. Specifically, human operator’s trust level and trust preference.

A Laser Projection System for

To build an agent providing assistance to human rescuers in an urban search and rescue task, it is crucial to predict their strategies. Developing data-driven models to predict them requires costly data collection and training for each new environment. We investigate transfer learning to mitigate this challenge (a) from a source environment with smaller number of types of injured victims to one with more types and (b) from a simpler environment to a more complex one for navigation strategy.

A Layered Control Approach to Human-Aware Task and Motion

Illusory Control that we proposed is a new teleoperation method that aims to increase the sensory value of comfort and happiness in teleoperation of mobile robots by providing visual feedback that seems to trick the operator. By conducting subject experiments, we compared the proposed method with conventional teleoperation methods such as direct teleoperation and shared control, and showed the possibility of improving task efficiency and system acceptance.

Teleoperation Method by Illusion of Human Intention and Time

In virtual reality, analogously, we can be a super-powered virtual giant who can supervise a swarm of robots in a vast and remote environment by flying over or resizing the world, and coordinate them by picking and placing a robot or creating virtual walls. This work implements this idea by using Virtual Reality along with Leap Motion, which is then validated by proof-of-concept experiments using real and virtual mobile robots in mixed reality.

Development of a Shared Indoor Smart Mobility Platform Based on Semi-Autonomous Driving

This work builds a dual-master/dual-slave teleoperation system that includes comprehensive force feedback on both arms and fingers. The system greatly improves the teleoperation experience, and some demonstrations on daily tasks are presented. Besides, this paper proposes a unified control architecture for the teleoperation scenario. This modular architecture design greatly facilitates research exploration in the teleoperation system.

Recurrent Neural Networks for Inferring Intentions in Shared Tasks for Industrial Collaborative Robots

Based on a user-study with experts in motion planning algorithms, we propose a taxonomy of explanations, including kinds of questions/explanations/methods related to motion planning, and a set of design considerations for explainable motion planning.

How experts explain motion planner output: a preliminary user-study to inform the design of explainable planners

We propose adding null-space motion to an online MPC-generated trajectory reference to avoid joint stiction and thus facilitate kinesthetic teaching in a redundant robot. Consequently, a reduction of the friction torque dispersion has been experimentally observed with this method. The trajectory deviations observed when adding null-space motion have been corrected experimentally by performing a fast sampling of the manipulator’s Jacobian matrix and by using angular joint position sensor feedback for the online reference trajectory recalculation.

Meet Your Personal Cobot but don‘t touch it just yet!

Interpreting spatial language not only adds a perceptual modality for robots, but also reduces the barrier of interfacing with humans. However, spatial language is inherently subjective and ambiguous. We thus consider spatial language as stochastic observations. We propose SLOOP (Spatial Language Object-Oriented POMDP), a new framework for partially observable decision making with a probabilistic observation model for spatial language, and apply it to object search in city-scale environments. Results demonstrate successful leverage of spatial language for faster target search.

Spatial Language Understanding for Object Search in Partially Observed City-scale Environments

We present a method for proactive approaching of humans for human-robot cooperative tasks. The proposed method can deal robustly with the uncertainties in the robot’s perception while also ensuring socially acceptable behavior.

Pedestrian Density-Based Path Recognition and Risk Prediction for Autonomous Vehicles

Interactive Task Learning is a research approach that studies the design of intelligent agents that can acquire knowledge from human instruction. This research has largely been robot-centered. We study this problem from a human-centered perspective. We conduct an interactive teaching study (N=10), and conduct a cognitive task analysis to characterize various aspects of teaching actions executed in natural human teaching. We propose ITL robot design recommendations using a computational theory SharedPlans that can support different participant teaching styles.

Premium content

A System for Traded Control Teleoperation of Manipulation Tasks using Intent Prediction from Hand Gestures

Downloads

Next from IEEE RO-MAN 2021

A Laser Projection System for

Similar lecture

Engineering Features to Improve Pass Prediction in Soccer Simulation 2D Games

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES