As robotic systems become more interactive and complex, there is a need to standardise interfaces and simplify development processes. This is particularly pertinent in the field of manufacturing, where human-robot collaboration is on the increase, but where standards and proprietary software are key barriers to deployment and adoption. In this article we present the ROSIE Adapter, a generalpurpose, modular adapter developed in ROS designed to support the creation and connection of industry-ready digital twins. Together with our previous work on the modular CSI digital twin framework, we demonstrate how the ROSIE Adapter creates a versatile “plug-and-play” interface that simplifies the development of new robotic processes, and improves accessibility to novice users. Furthermore, the adaptor supports integration of intuitive interface devices, such as speech and augmented reality interfaces, which enable more natural collaboration. We describe the adaptor and its use in two real-world applications, demonstrate the ease of use via a three-day hackathon event, and provide results showing the faithfulness of the arising digital twins to their connected physical systems.

ROSIE: A ROS adapter for a Modular Digital Twinning Framework

Augmented Reality (AR) provides a method to superimpose
real-time information on the physical world. AR is
well-suited for complex robotic systems to help users
understand robot behavior, status, and intent. This paper
presents an AR system, Augmented Robot Environment (AugRE),
that combines ROS-based robotic systems with Microsoft
HoloLens 2 AR headsets to form a scalable multi-agent
human-robot teaming system for indoor and outdoor
exploration. The system allows multiple users to
simultaneously localize, supervise, and receive labeled
images from robotic clients. An overview of AugRE and
details of the novel system architecture that allows for
large-scale human-robot teaming is presented below. Studies
showcasing system performance with multiple robotic clients
are presented. Results show that AugRE can scale to 50
robotic clients with minimal performance degradation, due
in part to key components that leverage a recent
advancement in robotic client-to-client communication
called Robofleet. Finally we discuss new capabilities
enabled by AugRE

Visualizing Robot Intent for Object Handovers with Augmented Reality

Data sonification enhance and enrich information understanding with an additional sensory dimension. Sonifica- tion also opens the way to more creative applications, joining arts and sciences. In this study, we present sequences of chords obtained as auditory feedback from the trajectories of a robotic swarm. The swarm behavior is an emerging effect from simple local interactions and autonomous decisions of each robot. The swarm effect can be identified through sonification outcomes in terms of voice leading patterns. Thus, chord patterns represent behavior patterns. The convergence to the target is represented by the convergence to a specific pitch. The swarm decision process is based upon quantum computing. We first present logic gates and their implementations as quantum circuits, describing examples with 2- and 3-dimensional motion of a 3-robot toy swarm. The considered scenarios are ant foraging (2D) and underwater search and rescue (3D). Then, we provide and discuss some examples of the harmonic sequences that can be obtained as feedback from robotic motion.

Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing

Students with a learning disability (LD) generally require supplementary one-to-one instruction and support to acquire the foundational academic skills learned at school. Because learning is more difficult for students with LD, students can frequently display off-task behaviours to avoid attempting or completing challenging learning tasks. Re-directing students back to their learning task is a frequent strategy used by educators to support students. However, there have been limited studies investigating the use of assistive technology to support student re-direction, specifically in a ``real-world'' educational setting. In this in situ study, we investigate the impact of integrating socially assistive robot to provide re-direction strategies to students. A social robot, QT, was employed within the existing learning program during one-to-one remedial instruction sessions. The study comprised two phases, ``Instruction as usual" (IAU) and ``Robot-mediated instructions" (RMI). Both followed the students' one-to-one instructional program where students get personalised learning support from their instructors, except for the RMI phase which included a social robot as a tool. We investigated the impact of the robot on students' on-task behaviours and progress towards learning goals. The results of our mixed method analysis suggest that the robotic intervention supported students in staying on-task and completing their learning goal.

Reimagining RViz: Multidimensional Augmented Reality Robot Signal Design

While Wizard-of-Oz (WOz) techniques are frequently used to supplement a machine's abilities, extending this approach to human entities can increase experimental control in studies comparing evaluations of humans and machines in the same role. This article describes the design, implementation, and use of a WOz system for facilitating controlled verbal interactions between children and robot or human interviewers. A collaborative interface allows multiple remote wizards to combine participant responses with interaction-specific goals in order to direct a robot or human interviewer's behavior in a consistent manner. While robot interviewers are controlled directly, human interviewers receive direction through a tablet device or via a projection system concealed from participants. In addition to the system's technical design, we describe the division of responsibilities between wizards and insights from using the system across three extensive interview studies to facilitate a total of 217 interactions with children.

Multi-perspective human robot interaction through an augmented video interface supported by deep learning

The advent of robots in our daily life depends on
their ability to navigate and intervene efficiently in our
environment. Whether considering household or industrial
applications, one of the most important functions they should
have is the ability to grasp and manipulate a large amount of
objects, tools and machines that can vary in form, size and
weight, but share the fact that they were designed for humans
and have functional elements, e.g. buttons or handles, fitted to
the human hand. The development of biologically inspired
anthropomorphic robotic hands thus appears as a natural
research path to allow robots replicating these activities. Such
devices however prove to be complex to design and control, and
they remain in practice limited to date to laboratory
experiments. They hardly reach a sufficient simplicity,
robustness and cost allowing for their widespread adoption in
our houses or factories and industrial robots still make use of
simple bi-digital grippers or dedicated tools which in turn
suffer a poor versatility. To overcome this situation, novel
dexterous grippers are required, that are sufficiently versatile
to adapt to various situations and objects yet simple enough
and cost effective. This compromise is however difficult to
achieve and the specification of such grippers is still an open
issue. This paper introduces a human-centered manual
interaction patterns analysis methodology that intends to
contribute to fill this gap. After a presentation of our approach,
we apply it in different contexts and show how it can be used to
orient a robotic gripper design that will fit given use-case
requirements.

Augmented Environment Representations with Complete Object Models

Drowsiness is one of the major causes of accidents in
driving and other tasks. Therefore, finding effective ways
to prevent drowsiness and to maintain their alert state is
an important subject of study in human-machine systems such
as autonomous driving. 
In this paper, we show the potential to use a social robot
to prevent participants from becoming drowsy and to keep
them alert. Twenty-five participants were asked to perform
Sustained Attention to Response Tasks (SART) and report
their subjective drowsiness levels. When the system
detected drowsiness from the task reaction time or the
self-reports, one of the following awakening alarms was
triggered: (1) sound and the robot's movement (SRM), (2)
sound from (motionless) robot (SR), (3) sound only (no
robot present) (SO), or (4) no stimulus (NS), as a control.
The participants' task performance and self-reported
drowsiness were continuously recorded to evaluate the
effectiveness of each alarm condition. The experimental
results showed a significant difference in the
self-reported scores of drowsiness between the SRM and SR
conditions and the NS condition, while significance was not
found between the SO and NS conditions. In addition, the
response time was shorter for SRM. The difference between
SR and SO is only the presence of the social robot, so
these results indicate that the presence of the social
robot increases the participants’ alertness level.

AugRE: Augmented Robot Environment to Facilitate Human-Robot Teaming and Communication

Cognitive training is effective at retaining cognitive function and delaying decline for typically ageing older adults, individuals with mild cognitive impairment, and persons with dementia. Technological resources can address limiting factors that inhibit engagement and access to this treatment. We investigated how a socially assistive robot-facilitated memory task with sensory feedback was received by older adults. The impact of unimodal and multimodal administration of auditory and haptic feedback using two robot embodiments (Pepper and Nao) was evaluated in terms of user performance, usability, and workload. In contrast to sensory feedback research, auditory feedback resulted in significantly higher task accuracy. This was, however, supported by previous work from neurological literature. Auditory feedback also received significantly higher usability, and this preference was validated by qualitative feedback from participants. Regardless of robotic embodiment, this study demonstrates an advantage for auditory feedback (over haptic and multimodal) in cognitive training activities for older adults.

Investigating the Usability of a Socially Assistive Robotic Cognitive Training Task with Augmented Sensory Feedback Modalities for Older Adults

Humans are highly skilled in communicating their intent for when and where a handover would occur. However, even the state-of-the-art robotic implementations for handovers typically lack of such communication skills. This study investigates visualization of the robot's internal state and intent for Human-to-Robot Handovers using Augmented Reality. Specifically, we explore the use of visualized 3D models of the object and the robotic gripper to communicate the robot's estimation of where the object is and the pose in which the robot intends to grasp the object. We tested this design via a user study with 16 participants, in which each participant handed over a cube-shaped object to the robot 12 times. Results show communicating robot intent via augmented reality substantially improves the perceived experience of the users for handovers. Results also indicate that the effectiveness of augmented reality is even more pronounced for the perceived safety and fluency of the interaction when the robot makes errors in localizing the object.

Impacts of Teaching towards Training Gesture Recognizers for Human-Robot Interaction

From RViz to augmented reality (AR), a wide variety of robot \textcolor{black}{signal visualizations exist for conveying robot capabilities. Many of the visualizations designed for AR, however, have not isolated multiple salient Virtual Design Elements (VDEs) for a given signal and comparatively evaluated combinations of those VDEs. To address this, we identify multiple VDEs for AR signaling of the following core robot capabilities:} \textit{navigation}, \textit{light detection and ranging (LiDAR)}, \textit{camera}, \textit{face detection}, \textit{audio localization}, and \textit{natural language processing}. We evaluated each signal's VDE combinations with an Amazon Mechanical Turk study ($n$=150) where participants watched 4 videos for each signal (consisting of 2 independent VDE choices) and rated the clarity and visual appeal of each signal. The results define a set of the most clear and visually appealing signal visualization designs and inform about interaction effects among VDEs. The resulting VDEs offer design insights and a baseline for continued research into AR robot capability signalling.

Human-Robot Handovers using Task-Space Quadratic Programming

Safety is a fundamental requirement in any human-robot collaboration scenario. To ensure the safety of users, from both physical and psychological aspects, for such scenarios, we propose a novel Virtual Barrier system facilitated by an augmented reality interface. Our system provides two types of Virtual Barriers to ensure safety: 1) a Virtual Person Barrier which encapsulates and follows the user to protect them from collisions with the robot, and 2) Virtual Obstacle Barriers which users can spawn to protect objects or regions that the robot should not enter. Our system utilizes augmented reality to visually display these protective barriers to the user during operation. To enable effective human-robot collaboration, our system automatically replans the robot's motion when potential collisions are detected as a result of a barrier intersecting the robot's planned path. We compared our novel system with a standard 2D display interface through a user study, where participants performed a task mimicking an industrial manufacturing procedure. Results show that our system increases both physical and psychological safety and task efficiency, and makes the interaction more intuitive.


Explain yourself! Effects of Explanations in Human-Robot Interaction

While 2D occupancy maps commonly used in mobile robotics
enable safe navigation in indoor environments, in order for
robots to understand and interact with their environment
and its inhabitants representing 3D geometry and semantic
environment information is required. Semantic information
is crucial in effective interpretation of the meanings
humans attribute to different parts of a space, while 3D
geometry is important for safety and high-level
understanding. We propose a pipeline that can generate a
multi-layer representation of indoor environments for
robotic applications. The proposed representation includes
3D metric-semantic layers, a 2D occupancy layer, and an
object instance layer where known objects are replaced with
an approximate model obtained through a novel
model-matching approach. The metric-semantic layer and the
object instance layer are combined to form an augmented
representation of the environment. Experiments show that
the proposed shape matching method outperforms a
state-of-the-art deep learning method when tasked to
complete unseen parts of objects in the scene. The pipeline
performance translates well from simulation to real world
as shown by F1-score analysis, with semantic segmentation
accuracy using Mask R-CNN acting as the major bottleneck.
Finally, we also demonstrate on a real robotic platform how
the multi-layer map can be used to improve navigation
safety.

Underline digital video library

ROSIE: A ROS adapter for a Modular Digital Twinning Framework

Visualizing Robot Intent for Object Handovers with Augmented Reality

Virtual Barriers in Augmented Reality for Safe and Effective Human-Robot Cooperation in Manufacturing

Reimagining RViz: Multidimensional Augmented Reality Robot Signal Design

Multi-perspective human robot interaction through an augmented video interface supported by deep learning

Augmented Environment Representations with Complete Object Models

AugRE: Augmented Robot Environment to Facilitate Human-Robot Teaming and Communication

Investigating the Usability of a Socially Assistive Robotic Cognitive Training Task with Augmented Sensory Feedback Modalities for Older Adults

Impacts of Teaching towards Training Gesture Recognizers for Human-Robot Interaction

Human-Robot Handovers using Task-Space Quadratic Programming

Explain yourself! Effects of Explanations in Human-Robot Interaction

Designing Long-term Parent-child-robot Triadic Interaction at Home through Lived Technology Experiences and Interviews

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES