In two experiments we examined the influence of ‘history-driven’ attentional biases on choice behavior. In Experiment 1 we used a value-modulated attentional capture procedure to induce an automatic reward-related attentional bias, and found that this bias shaped choice in a subsequent task in which participants were required to pick the highest number from a briefly displayed choice array. In Experiment 2 we investigated the influence of a working memory manipulation, and found that choice in the number-selection task was influenced by the current (and prior) contents of memory, consistent with an influence of memory-driven attentional bias on information encoding. Our findings indicate that history-driven attentional biases can translate to an influence on overt, downstream processes of behavioural choice, and should be incorporated into models of the interaction between attention and choice.


Reward-driven and memory-driven attentional biases automatically modulate rapid choice

Human memory does not simply function like an information
storage disk; instead, it flexibly reorganizes information. This
flexibility can sometimes produce false memories of items related
to those actually encountered–a possible byproduct of an
adaptive memory system that enables generalization across related
items or experiences. In the Deese-Roediger-McDermott
(DRM) task, participants often falsely remember seeing words
that are semantically related to presented words. Here, we propose
and test a new model of lexical (false) memory that predicts
these errors, made possible by integrating (i) theories of
memory that posit encoding of verbatim and gist-level information
with (ii) a computational framework adapted from the
perceptual false memory literature, and (iii) semantic relatedness
measures from word embeddings analysis of large-scale
text corpora. This Lexical Target Confusability Competition
(Lexical TCC) model successfully predicts false recognition
in the DRM task, with implications for understanding how and
when the mind produces false memories.


A Computational Model of Lexical False Memory based on Semantic Distance from Word Embeddings

After viewing motion events with a starting-point (Source) and end-point (Goal), people mention the Goal more often and remember it more accurately than the Source. This Goal privilege has been hypothesized to arise from an online attentional bias during event apprehension itself, yet no data exists that: (a) documents this online attentional bias and (b) correlates any online bias with offline memory and linguistic measures. Here we do just that: we recorded participants’ eye movements as they viewed or prepared to describe motion events and later tested their memory of Goals or Sources. We find an online attentional bias for Goals over Sources during initial encoding of events. Moreover, the extent of the attentional Goal bias is systematically related to both language production and memory: the bias is greatest when the Source is not mentioned in production or not remembered later at test.


The Goal Bias Emerges Early in Motion Event Inspection and Speech Planning: Evidence from Eye-Movements

As the complexity of pacemaker devices continues to grow, the importance of capturing its functional correctness requirement formally cannot be overestimated. The pacemaker system specification document by \emph{Boston Scientific} provides a widely accepted set of specifications for pacemakers. 
As these specifications are written in a natural language, they are not amenable for automated verification, synthesis, or reinforcement learning of pacemaker systems. This paper presents a formalization of these requirements for a dual-chamber pacemaker in \emph{duration calculus} (DC), a highly expressive real-time specification language.
The proposed formalization allows us to automatically translate pacemaker requirements into executable specifications as stopwatch automata, which can be used to enable simulation, monitoring, validation, verification and automatic synthesis of pacemaker systems. 
The cyclic nature of the pacemaker-heart closed-loop system results in DC requirements that compile to a decidable subclass of stopwatch automata. We present shield reinforcement learning (shield RL),  a shield synthesis based reinforcement learning algorithm, by automatically constructing safety envelopes from DC specifications. 


Correct-by-Construction Reinforcement Learning of Cardiac Pacemakers from Duration Calculus Requirements

Recently, improving the robustness of policies across different environments attracts increasing attention in the reinforcement learning (RL) community. Existing robust RL methods mostly aim to achieve the max-min robustness by optimizing the policy’s performance in the worst-case environment. However, in practice, a user that uses an RL policy may have different preferences over its performance across environments. Clearly, the aforementioned max-min robustness is oftentimes too conservative to satisfy user preference. Therefore, in this paper, we integrate user preference into policy learning in robust RL, and propose a novel User-Oriented Robust RL (UOR-RL) framework. Specifically, we define a new User-Oriented Robustness (UOR) metric for RL, which allocates different weights to the environments according to user preference and generalizes the max-min robustness metric. To optimize the UOR metric, we develop two different UOR-RL training algorithms for the scenarios with or without a priori known environment distribution, respectively. Theoretically, we prove that our UOR-RL training algorithms converge to near-optimal policies even with inaccurate or completely no knowledge about the environment distribution. Furthermore, we carry out extensive experimental evaluations in 6 MuJoCo tasks. The experimental results demonstrate that UOR-RL is comparable to the state-of-the-art baselines under the average-case and worst-case performance metrics, and more importantly establishes new state-of-the-art performance under the UOR metric.

User-Oriented Robust Reinforcement Learning

Safety comes first in many real-world applications involving autonomous agents. Despite a large number of reinforcement learning (RL) methods focusing on safety-critical tasks, there is still a lack of high-quality evaluation of those algorithms that adheres to safety constraints at each decision step under complex and unknown dynamics. In this paper, we revisit prior work in this scope from the perspective of state-wise safe RL and categorize them as projection-based, recovery-based, and optimization-based approaches, respectively.  Furthermore, we propose Unrolling Safety Layer (USL), a joint method that combines safety optimization and safety projection. This novel technique explicitly enforces hard constraints via the deep unrolling architecture and enjoys structural advantages in navigating the trade-off between reward improvement and constraint satisfaction. To facilitate further research in this area,  we reproduce related algorithms in a unified pipeline and incorporate them into SafeRL-Kit, a toolkit that provides off-the-shelf interfaces and evaluation utilities for safety-critical tasks. We then perform a comparative study of the involved algorithms on six benchmarks ranging from robotic control to autonomous driving. The empirical results provide an insight into their applicability and robustness in learning zero-cost-return policies without task-dependent handcrafting. The project page is available at \url{https://sites.google.com/view/saferlkit}.

Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks

Cooperative multi-agent reinforcement learning (c-MARL) is widely applied in safety-critical scenarios, thus the analysis of robustness for c-MARL models is profoundly important. However, robustness certification for c-MARLs has not yet been explored in the community. In this paper, we propose a novel certification method, which is the first work to leverage a scalable approach for c-MARLs to determine actions with guaranteed certified bounds. c-MARL certification poses two key challenges compared with single-agent systems: (i) the accumulated uncertainty as the number of agents increases; (ii) the potential lack of impact when changing the action of a single agent into a global team reward. These challenges prevent us from directly using existing algorithms. Hence, we employ the false discovery rate (FDR) controlling procedure considering the importance of each agent to certify per-state robustness and propose a tree-search-based algorithm to find a lower bound of the global reward under the minimal certified perturbation. As our method is general, it can also be applied in single-agent environments. We empirically show that our certification bounds are much tighter than those of the state-of-the-art RL certification solutions. We also run experiments on two popular c-MARL algorithms: QMIX and VDN, in two different environments, with two and four agents. The experimental results show that our method produces a meaningful guaranteed robustness for all models and environments. Our tool CertifyCMARL is available at https://github.com/TrustAI/CertifyCMARL.

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

Hippocampus (HPC) is a key structure in category learning, forming organized concepts from individual episodes to adaptively represent shared and distinct elements. Most of the evidence for the role of HPC in concept formation, however, relies on neural coding measured at the end of learning. Thus, it remains unclear how neural mechanisms early in learning support the formation of flexible knowledge. We explored this by combining human functional MRI with a computational model of learning, SUSTAIN. Participants learned to categorize complex visual objects with multiple features. A multidimensional rule linked features to categories with exception items in each category. We observed distinct HPC engagement along its primary axis for exception learning that also predicted learning outcomes. The model predicted that better learners use exceptions to update their knowledge, which was reflected in HPC activation. These findings suggest that HPC discriminates initial informative experiences during learning to support flexible knowledge formation.

Hippocampus Engagement Early in Learning Supports Concept Formation

Sensory-related word association differences in Parkinson's Disease for verb processing

Free word association tasks are tools to study spontaneous lexical organization in children and adults. Lexical organization resembles a network of words grounded on relevant sensory information; for example, verbs engage the most with motor information. Studying lexical networks in populations with motor disorders, like Parkinson's Disease (PD), will insight into the effect of relevant information deprivation in lexical organization.

We applied a free word association task with early-acquisition Spanish nouns as stimuli. Participants consisted of 20 with PD -divided into two groups depending on their cognitive state- and ten healthy older adults for control. Then, we used an artificial neural network to determine the semantic distance between stimuli-response dyads.

We found differentiated semantic distances for verb responses in all groups in an inverse u-shaped curve depending on their cognitive and PD state. Our findings suggest that lexical organization and meaning are not crystallized, but adapt from sensory information.

Motor-related word association differences in Parkinson Disease for verb processing

The “Direction and Orientation Strategy Questionnaire” (DOSQ) is a novel assessment based on giving directions around your hometown and how one remains oriented in real-world environments. The DOSQ produces three sub-scale scores: use of allocentric strategies, use of egocentric strategies, and sense-of-direction. Through an online survey, over one hundred Texas A&M undergraduates completed demographic questions, DOSQ, Santa Barbara Sense-of-Direction scale (Hegarty et al., 2002), Wayfinding Strategy Scale (Lawton, 1994), Questionnaire on Survey Representation (Pazzaglia & De Beni, 2006), Spatial Anxiety Scale (Lawton, 1994), and familiarity measures using the Texas A&M campus. Using factor analysis, we found that the DOSQ sub-scores were highly related to other measures of strategy use and sense-of-direction, establishing its convergent validity. DOSQ sub-scales were not related to the spatial anxiety or familiarity measures, establishing its discriminant validity. The DOSQ provides a method for briefly assessing spatial thinking strategies that support navigation through real-world environments.

Convergent and Discriminant Validity of the Direction and Orientation Strategy Questionnaire

From previous work we know that mechanical reasoning is associated with spatial skills, and that it could even be possible for mechanical reasoning training to lead to meaningful gains in spatial abilities. However, it is unknown whether the connection with mechanical reasoning applies to different types of spatial skills or is specific to only one or a subset of them. To answer this question, we presented participants with six different tasks, four of which corresponded to spatial assessments that were organized along a 2 (intrinsic/extrinsic) x 2 (static/dynamic) matrix of factors, and two evaluations that measured mechanical reasoning (gears and pulleys task and DAT-5 mechanical reasoning test). Three hundred adult participants from Chile responded from their personal computers to the stimuli presented on Open Lab, an online platform. The results shed light on the link between mechanical reasoning and the different subtypes of spatial skills.


What's the relationship between mechanical reasoning and different types of spatial skills?

Previous research has shown that proportions, ornateness, and lighting of indoor architectural spaces affect observers' mental states (Negami & Ellard 2021). However, most studies are limited to verbal self-reports, and focus either on photographs, or CAD-modeled rooms in which certain elements, (e.g. ceiling height), are experimentally manipulated. 
We study human psychological and physiological responses to historical indoor sites reconstructed in virtual reality (VR). Such spaces are often designed to evoke affective responses -- for example, sacred architecture is meant to evoke feelings of calm. Using drone footage, we record 3D geometry, visual and auditory sensory data of an indoor space.
We recreate the space in VR, and record humans’ eye-movements, heart-rate, galvanic skin response, and reports of affective states, during free exploration. We propose a cognitive model that interprets physiological responses as information-foraging, and identify the correlates of reported changes in affective states with specific properties of architectural space.


Underline digital video library

Reward-driven and memory-driven attentional biases automatically modulate rapid choice

A Computational Model of Lexical False Memory based on Semantic Distance from Word Embeddings

The Goal Bias Emerges Early in Motion Event Inspection and Speech Planning: Evidence from Eye-Movements

Correct-by-Construction Reinforcement Learning of Cardiac Pacemakers from Duration Calculus Requirements

User-Oriented Robust Reinforcement Learning

Evaluating Model-free Reinforcement Learning toward Safety-critical Tasks

Certified Policy Smoothing for Cooperative Multi-Agent Reinforcement Learning

Hippocampus Engagement Early in Learning Supports Concept Formation

Motor-related word association differences in Parkinson Disease for verb processing

Convergent and Discriminant Validity of the Direction and Orientation Strategy Questionnaire

What's the relationship between mechanical reasoning and different types of spatial skills?

Human Information Seeking in Architectural Spaces Simulated in Virtual Reality

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES