Androids that strongly but imperfectly resemble humans in shape can elicit negative emotions in people, a phenomenon known as the "uncanny valley," which has been replicated in laboratory experiments. Recently, the accuracy of face recognition utilizing machine learning has increased, raising the question of whether machine learning can replicate the uncanny valley. Using FaceNet as a representative face recognition algorithm, we examined the similarity of face recognition to human evaluation and its replication of the uncanny valley. The results revealed a strong correlation between machine learning and human evaluation of human-like shapes. However, it is evident that only certain aspects of the uncanny valley were replicated. Furthermore, visualization of the activation maps suggests that localized regions, such as the mouth and chin, acted as the basis for judgment. These findings support the idea that human and machine learning have distinct areas of attention.

Does Machine Learning Replicate the Uncanny Valley? An Example using FaceNet

Pre record assistants have become fixtures in everyday settings, but most research focuses on their development rather than their use following deployment. To facilitate study of their use in office settings, we introduce OfficeDial, a multimodal dataset containing audio recordings, transcriptions, eye tracking data, and screen recordings from conversations between humans and Pre record assistants in office environments. Conversations are paired with physical and behavioral measures of cognitive load. We study the associations between verbal behavior and noise level and reveal key relationships between verbal redundancy, disfluency, and noise level. We make our new dataset available to interested researchers to inspire further exploration.


Linguistic Cognitive Load Analysis on Dialogues with an Intelligent Pre record Assistant

From learning to play video games to using novel tools, humans are able to acquire a variety of complex mappings between their actions and arbitrary outcomes. In addition, once they have learned such mappings, they often have to use them sequentially to achieve goals, i.e., planning. In this work we study how the learning of a novel mapping interacts with planning in the context of grid navigation. In order to do so, we developed a computer-based game where subjects have to move a cursor from start to target locations using the keys of their keyboard. Importantly, to more closely resemble the complexity of the mappings that people acquire in their lives, the cursor movement was determined by a non-trivial rule inspired by the movement of the piece of chess known as the Knight. In Experiment 1, we show that participants were able to improve their performance in our task, though not arriving optimally to the targets in the majority of trials. Additionally, we proposed two computational models where the mapping-learning and planning interaction occurs in the form of an action value computation, and then evaluated their fit to our data. Finally, in Experiment 2, we showed that exposing participants to the mapping component of the task without having to plan, provides a performance improvement when exposed to the full task later. Crucially, this improvement does not occur if subjects are exposed to the planning component of the task prior to doing it fully.


Exploring human learning and planning on grid navigation using arbitrary mappings.

This study explores eye-markers for sub-categories of cognitive load. Experiments were conducted on 63 participants using Image Sliding Puzzle (ISP). NASA-TLX was administered post task completion as a measure of cognitive load. Total scanning duration, total fixation duration, fixation count and total saccadic duration were found to be significant, which is consistent with pre-existing literature. Next, we investigated whether sub-categories of cognitive load (mental demand, temporal demand, perceived performance, effort, and frustration) can be distinguished by characteristic eye-metrics. Our findings reveal signature eye-markers for specific sub-categories of cognitive load. Further, we explored the link between perceived performance and actual performance and established that mean fixation duration, peak velocity, mean saccadic duration, and skewness in saccadic velocity were significant markers for both objective and subjective markers of performance. To our knowledge this is the first study to compare the task-evoked eye measures for sub-categories of cognitive load.


Estimating Sub-categories of Cognitive Load: An Eye-tracking Study

As more of society uses virtual environments from collaborative learning environments to the now ubiquitous Zoom online meeting platform, there is an opportunity for artificial agents (AAs) to foster and support human collaboration. However, little appears to be known about whether people would actually want to collaborate with such an AA and what impact if any it has on cognitive processes underlying human collaboration. To begin to answer these questions, we devised a Wizard of Oz (WoZ) study where teams of 3 participants were tasked with having to locate and corral targets into a containment area in a virtual desert environment with the Wizard (confederate) playing the role of the Operator who could see the location of participants, targets and the containment area, and could communicate this information to participants. Although agent responsiveness had limited impact on performance and workload, as expected the responsiveness of the agent did influence participant's perceptions of whether the agent understand them and provided timely responses.

How do People Perceive Collaborative Conversational Agents?

We propose a novel approach that utilizes virtual reality (VR) and simulation environments to quantify the impact of visual impairments (VIs) on daily tasks, e.g., to what extent does glaucoma slow people down in wiping a table or chopping vegetables? We utilize clinical data from patients to develop visual field models, allowing VR to mimic the visual perception of VI patients as if it is seen through their eyes. Additionally, we leverage BEHAVIOR, the state-of-the-art simulation environment, to recreate a household environment for six daily activities. By measuring the disparity between the subjects' performance with and without VI, we can accurately quantify the impact of VIs. We further quantify the effects of VIs on body and eye movements, and model how movement strategies affect task performance under the influence of VIs. We hope our results can provide valuable insights into the challenges faced by individuals with VIs.

Quantifying the Effect of Visual Impairments on Daily Activities in Virtual, Interactive Environments

To inform Explainable AI (XAI) design for updating users’ beliefs about AI based on their mental models, we examined the similarities and differences between humans and AI in object detection in driving scenarios. In humans, individuals differed in adopting focused or explorative attention strategies, with better performance associated with the focused strategy. AI (Yolo-v5s) had higher similarity in attended features to the focused than the explorative strategy in humans, and achieved human-expert-level performance in vehicle detection even in difficult cases such as occlusion and degradation. In contrast, it performed much poorer than humans in detecting humans with low attended feature similarity due to humans’ attention bias for stimuli with evolutionary significance. Also, higher similarity to humans’ attended features was associated with better AI performance, suggesting that human attention may be used for guiding AI design. These findings have significant implications for both AI and XAI designs.

Humans vs. AI in Detecting Vehicles and Humans in Driving Scenarios

While saliency-based explainable AI (XAI) methods have been well developed for image classification models, they fall short in comparison with human explanations. Here we examined human explanation strategies for image classification and their relationship with explanation quality to inform better XAI designs. We found that individuals differed in attention strategies during explanation: Participants adopting more explorative strategies used more visual information in their explanations, whereas those adopting more focused strategies included more conceptual information. In addition, visual explanations were rated higher for effectiveness in teaching learners without prior category knowledge, whereas conceptual explanations were more diagnostic for observers with prior knowledge to infer the class label. Thus, individuals differ in the use of visual and conceptual information to explain image classification, which facilitate different aspects of explanation quality and suit learners with different experiences. These findings have important implications for adaptive use of visual and conceptual information in XAI development.

Individual differences in explanation strategies for image classification and implications for explainable AI

Technology to predict responses is a key element in human-to-human messaging that has increasingly been utilized to enable AI-mediated communication. When response suggestions from AI are incorporated into human messaging, it will have an impact on the flow and content of the dialogue. In this paper, we investigated the effect of AI response suggestion sentences used for chat messaging on dialogue flow from two aspects: dialogue act and initiative. Usage rates of response suggestions for different dialogue acts were measured with BLEU scores, and we found that the response suggestions contributed to the establishment of a proper dialogue flow, such as an answer in response to a question. The results of our case study indicated that users who take the initiative in the dialogue tend to utilize response suggestions less frequently. We also found that some written responses were based on the suggested sentence structure but conveyed different messages.

The Effect of Response Suggestion on Dialog Flow: Analysis Based on Dialogue Act and Initiative

Previous research found that loaded language (i.e., words, phrases, and overall rhetorical strategies that have strong emotional implications and intent to sway others) is used with significantly greater frequency by users of an online anti-vaccination forum compared to users of a vaccination-neutral forum. Our present research expands on these findings by examining the presence of loaded language on forums dedicated to discussing a broader range of conspiratorial topics. We compare loaded language usage on the subreddit r/conspiracy with usage on r/science and r/wallstreetbets. Results corroborate the previous literature. We additionally examined the presence of loaded language in a dataset from the alt-tech social networking platform Parler on January 6, 2021, when the U.S. Capitol was attacked because of conspiracy theories regarding the 2020 Presidential Election. Results show that loaded language, whose usage is linked to the cognitive motivations underlying belief in conspiracy theories, is a linguistic marker of conspiracy theorizing.

Loaded Language and Conspiracy Theories on Reddit and Parler

This study used XAI, which shows its purposes and attention as explanations of its process, and investigated how these explanations affect human trust in and use of AI. In this study, we generated heat maps indicating AI attention, conducted Experiment 1 to confirm the validity of the interpretability of the heat maps, and conducted Experiment 2 to investigate the effects of the purpose and heat maps in terms of reliance (depending on AI) and compliance (accepting answers of AI). The results of structural equation modeling (SEM) analyses showed that (1) displaying the purpose of AI positively and negatively influenced trust depending on the types of AI usage, reliance or compliance, and task difficulty, (2) just displaying the heat maps negatively influenced trust in a more difficult task, and (3) the heat maps positively influenced trust according to their interpretability in a more difficult task.

Modeling Reliance on XAI Indicating Its Purpose and Attention

Virtual Reality (VR) presents a novel method to study the mind by generating dynamic, immersive environments for participants to engage with. However, the adoption of VR in cognitive science has been slow---for one, in order to use VR as a methodological tool, we need to verify that studies conducted in VR reflect the same assumptions that are common in existing experimental methods. Here, we investigate this question by drawing on existing empirical work in the domain of intuitive physics. In Experiment 1, we found that participants' judgments on a physical scene prediction task largely correlated with the same judgments collected using conventional methods. In Experiment 2, participants preferentially explored objects that violated core principles of physics, even engaging in distinct hypothesis-testing behaviors in accordance with the violation. In sum, these findings suggest that VR can be an exciting, novel method to study human behavior.

Underline digital video library

Does Machine Learning Replicate the Uncanny Valley? An Example using FaceNet

Linguistic Cognitive Load Analysis on Dialogues with an Intelligent Pre record Assistant

Exploring human learning and planning on grid navigation using arbitrary mappings.

Estimating Sub-categories of Cognitive Load: An Eye-tracking Study

How do People Perceive Collaborative Conversational Agents?

Quantifying the Effect of Visual Impairments on Daily Activities in Virtual, Interactive Environments

Humans vs. AI in Detecting Vehicles and Humans in Driving Scenarios

Individual differences in explanation strategies for image classification and implications for explainable AI

The Effect of Response Suggestion on Dialog Flow: Analysis Based on Dialogue Act and Initiative

Loaded Language and Conspiracy Theories on Reddit and Parler

Modeling Reliance on XAI Indicating Its Purpose and Attention

Intuitions about physical scenes and objects are preserved in Virtual Reality (VR)

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES