Translating imagined speech from human brain activity into voice is a challenging and absorbing research issue that can provide new means of human communication via brain signals. Endeavors toward reconstructing speech from brain activity have shown their potential using invasive measures of spoken speech data, however, have faced challenges in reconstructing imagined speech. In this paper, we propose NeuroTalk, which converts non-invasive brain signals of imagined speech into the user's own voice. Our model was trained with spoken speech EEG which was generalized to adapt to the domain of imagined speech, thus allowing natural correspondence between the imagined speech and the voice as a ground truth. In our framework, automatic speech recognition decoder contributed to decomposing the phonemes of generated speech, thereby displaying the potential of voice reconstruction from unseen words. Our results imply the potential of speech synthesis from human EEG signals, not only from spoken speech but also from the brain signals of imagined speech.

Towards Voice Reconstruction from EEG during Imagined Speech

Audio-Visual Question Answering (AVQA) is a sophisticated QA task, which aims at answering textual questions over given video-audio pairs with comprehensive multimodal reasoning. Through detailed causal-graph analyses and careful inspections of their learning processes, we reveal that AVQA models are not only prone to over-exploit prevalent language bias, but also suffer from additional joint-modal biases caused by the shortcut relations between textual-auditory/visual co-occurrences and dominated answers. In this paper, we propose a COllabrative CAusal (COCA) Regularization to remedy this more challenging issue of data biases.
Specifically, a novel Bias-centered Causal Regularization (BCR) is proposed to alleviate specific shortcut biases by intervening bias-irrelevant causal effects, and further introspect the predictions of AVQA models in counterfactual and factual scenarios. Based on the fact that the dominated bias impairing model robustness for different samples tends to be different, we introduce a Multi-shortcut Collaborative Debiasing (MCD) to measure how each sample suffers from different biases, and dynamically adjust their debiasing concentration to different shortcut correlations. Extensive experiments demonstrate the effectiveness as well as backbone-agnostic ability of our COCA strategy, and it achieves state-of-the-art performance on the large-scale MUSIC-AVQA dataset.

COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering

Brain-computer Interface (BCI) builds a neural signal to the motor command pathway, which is a prerequisite for the realization of neural prosthetics. However, a long-term stable BCI suffers from the neural data drift across days while retraining the BCI decoder is expensive and restricts its application scenarios. Recent solutions of neural signal recalibration treat the continuous neural signals as discrete, which is less effective in temporal feature extraction. Inspired by the observation from biologists that low-dimensional dynamics could describe high-dimensional neural signals, we model the underlying neural dynamics and propose a semantic-dynamic feature that represents the semantics and dynamics in a shared feature space facilitating the BCI recalibration. Besides, we present the joint distribution alignment instead of the common used marginal alignment strategy, dealing with the various complex changes in neural data distribution. Our recalibration approach achieves state-of-the-art performance on the real neural data of two monkeys in both classification and regression tasks. Our approach is also evaluated on a simulated dataset, which indicates its robustness in dealing with various common causes of neural signal instability.

Extracting Semantic-Dynamic Features for Long-Term Stable Brain Computer Interface

We propose a multimodal data fusion framework to systematically analyze complex human behavioral data from specialized domains that are inherently dynamic, sparse, and heterogeneous. We develop a novel two-tier architecture of probabilistic mixtures, where the lower tier leverages parametric distributions from the exponential family to extract significant behavioral patterns from each data modality. These patterns are then organized into a dynamic latent state space at the higher tier to fuse patterns from different modalities. In addition, our framework jointly performs pattern discovery and maximum-margin learning for downstream classification tasks by using a novel group-wise sparse prior that regularizes the coefficients of the maximum-margin classifier. Therefore, the discovered patterns are highly interpretable and discriminative to support downstream classification tasks. Experiments on real-world behavioral data from medical and psychological domains demonstrate that our framework discovers meaningful multimodal behavioral patterns with outstanding interpretability and prediction performance.

Sparse Maximum Margin Learning From Multimodal Human Behavioral Patterns

Deep artificial neural networks (ANNs) play a major role in modeling the visual pathways of primate and rodent. However, they highly simplify the computational properties of neurons compared to their biological counterparts. Instead, Spiking Neural Networks (SNNs) are more biologically plausible models since spiking neurons encode information with time sequences of spikes, just like biological neurons do. However, there is a lack of studies on visual pathways with deep SNNs models. In this study, we model the visual cortex with deep SNNs for the first time, and also with a wide range of state-of-the-art deep CNNs and ViTs for comparison. Using three similarity metrics, we conduct neural representation similarity experiments on three neural datasets collected from two species under three types of stimuli. Based on extensive similarity analyses, we further investigate the functional hierarchy and mechanisms across species. Almost all similarity scores of SNNs are higher than their counterparts of CNNs with an average of 6.6%. Depths of the layers with the highest similarity scores exhibit little differences across mouse cortical regions, but vary significantly across macaque regions, suggesting that the visual processing structure of mice is more regionally homogeneous than that of macaques. Besides, the multi-branch structures observed in some top mouse brain-like neural networks provide computational evidence of parallel processing streams in mice, and the different performance in fitting macaque neural representations under different stimuli exhibits the functional specialization of information processing in macaques. Taken together, our study demonstrates that SNNs could serve as promising candidates to better model and explain the functional hierarchy and mechanisms of the visual system.

Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse

While neuromorphic computing architectures based on Spiking Neural Networks (SNNs) are increasingly gaining interest as a pathway toward bio-plausible machine learning, attention is still focused on computational units like the neuron and synapse. Shifting from this neuro-synaptic perspective, this paper attempts to explore the self-repair role of glial cells, in particular, astrocytes. The work investigates stronger correlations with astrocyte computational neuroscience models to develop macro-models with a higher degree of bio-fidelity that accurately captures the dynamic behavior of the self-repair process. Hardware-software co-design analysis reveals that bio-morphic astrocytic regulation has the potential to self-repair hardware realistic faults in neuromorphic hardware systems with significantly better accuracy and repair convergence for unsupervised learning tasks on the MNIST and F-MNIST datasets.

Astromorphic Self-Repair of Neuromorphic Hardware Systems

Spiking Neural Networks (SNNs) have received extensive academic attention due to the unique properties of low power consumption and high-speed computing on neuromorphic chips. Among various training methods of SNNs, ANN-SNN conversion has shown the equivalent level of performance as ANNs on large-scale datasets. However, unevenness error, which refers to the deviation caused by different temporal sequences of spike arrival on activation layers, has not been effectively resolved and seriously suffers the performance of SNNs under the condition of short time-steps. In this paper, we make a detailed analysis of unevenness error and divide it into four categories. We point out that the case of the ANN output being zero while the SNN output being larger than zero accounts for the largest percentage. Based on this, we theoretically prove the sufficient and necessary conditions of this case and propose an optimization strategy based on residual membrane potential to reduce unevenness error. The experimental results show that the proposed method achieves state-of-the-art performance on CIFAR-10, CIFAR-100, and ImageNet datasets. For example, we reach top-1 accuracy of 64.32% on ImageNet with 10-steps. To the best of our knowledge, this is the first time ANN-SNN conversion can simultaneously achieve high accuracy and ultra-low-latency on the complex dataset. Code is available at https://github.com/hzc1208/ANN2SNN_SRP.

Reducing ANN-SNN Conversion Error through Residual Membrane Potential

Spiking neural networks (SNNs) have manifested remarkable advantages in power consumption and event-driven property during the inference process. To take full advantage of low power consumption and improve the efficiency of these models further, the pruning methods have been explored to find sparse SNNs without redundancy connections after training. However, parameter redundancy still hinders the efficiency of SNNs during training. In the human brain, the rewiring process of neural networks is highly dynamic, while synaptic connections maintain relatively sparse during brain development. Inspired by this, here we propose an efficient evolutionary structure learning (ESL) framework for SNNs, named ESL-SNNs, to implement the sparse SNN training from scratch. The pruning and regeneration of synaptic connections in SNNs evolve dynamically during learning, yet keep the structural sparsity at a certain level. As a result, the ESL-SNN can search for optimal sparse connectivity by exploring all possible parameters across time. Our experiments show that the proposed ESL-SNN is able to learn SNNs with sparse structures effectively while reducing the limited accuracy. The ESL-SNNs achieve merely 0.28% accuracy loss with 10% connection density on the DVS-Cifar10 dataset. Our work presents a brand-new approach for sparse training of SNNs from scratch with biologically plausible evolutionary mechanisms, closing the gap in the expressibility between sparse training and dense training. Hence, it has great potential for SNN lightweight training and inference with low power consumption and small memory usage.  

ESL-SNNs: An Evolutionary Structure Learning Strategy For Spiking Neural Networks

Brain-machine interfaces (BMIS) of the future will be used to treat neurological disorders for which a cure does not exist yet. For this futuristic view to become a reality, we must overcome many technological challenges. A major goal is to increase the number of neurons that we can simultaneously interact with. To do so, future systems will record from more than a thousand channels at the same time. These systems will create a massive amount of raw data that needs to be compressed on the implant to lower the power consumption of the wireless link. As custom signal specifications for BMI applications become clearer and are diverging from those needed for basic neuroscience, there is an opportunity for application-specific lossy compression that improves further the power efficiency of these implants. In this workshop, we will cover the basic aspects of compression for BMIs and describe two different compression schemes for a motor BMI and an artificial retina.

Near-sensor compression in high-bandwidth brain-computer interface

In this talk, we present emerging technologies for implantable CMOS optoelectronics for interfacing to the brain.

Implantable Optoelectronics for Neural Interfaces

We present the Py-based SYstem for Chronic monitoring of rOdent SpeciesPy-(PSYCO) Mouse Tracker (PNMT): an affordable, open-sourced, and customizable/scalable behavior recording software for small foot-print Raspberry Pi-based acquisition computers and subsequent offline analysis tools capable of recording and tracking multiple mice for extended periods of time, respectively. Video tracking coupled with radio frequency identification (RFID) was employed to track multiple mice in a modified home-cage setting for up to four days. Video and RFID recordings were completed utilizing a Raspberry Pi 3B+ or Pi4 micro-computer. During offline analysis performed on a PCdesktop computers, mice were tracked using You Only Look Once version 4 (Yolov4) algorithm coupled with the Simple Online and Realtime Tracking (SORT) for mice tracking, respectively. RFID readings recorded in the system were then associated with the best-matched mouse that was tracked. For evaluation, PMTNMT performance was compared to a human manual validation of video recordings containing one to four mice. PMTNMT maintained a minimum of 70% identities tracked among mice detected with an overall accuracy >above 90 %. Moreover, we also showed that PMTNMT can be adapted to other behavior apparatus such as an open field like arena or sociability chamber to track mice. Furthermore, the PSYCO tracker can incorporate inputs from popular open-sourced packages such as Deeplabcut and Traja for postures estimation and travel trajectory analysis, respectively. Overall, our group demonstrated the performance of a newly developed multiple animal tracking system capable of tracking mice for extended periods of time and adaptable to the use case of individual investigators.

PSYCO:Py-based SYstem for Chronic monitoring of rOdent Species, an open-sourced python based system for homecage mouse tracking and image processing

The set of behavioral strategies we develop to manage stress stems from social interactions. Similarly, chronic stress models in rich environments in mice models, simplify the analysis of environmental and social factors using an automated assessment and tracking system. In a large group, mice are so influenced by pressures from other members of the group that over time, the evolutionary structure is formed by the social hierarchy in the group. For example, in this evolutionary structure, in rich and complex environments, group members interact with each other, sometimes play, and sometimes fight with each other to prove their dominance to the group. The behavioral ability that each mouse exhibits during daily life to control stress determines its status in the colony. The goal of this project is to develop an automated behavioral platform using machine-based units and a radio-frequency identification system (RFID) to evaluate the behaviors resulting from chronic variable stress (CVS) in male and female mice in groups ethically. To achieve our goals, we have developed two smart vivariums (1m x1m) equipped with high-quality cameras including various components such as beddings, enrichments, food, nesting material, games and toys, bridges, and ropes for each group of male and female. We control the behavioral characteristics of different groups up to a maximum of 10 people for 6 weeks by these smart vivariums at the same time for both mice groups and finally archive them in cloud data storage. To that end, we created an extensive library of several thousand images of different mice to automatically detect these individual or group behavioral characteristics by tracking them through a hybrid machine system consisting of DeepLabCut and YOLO with an RFID system.

Underline digital video library

Towards Voice Reconstruction from EEG during Imagined Speech

COCA: COllaborative CAusal Regularization for Audio-Visual Question Answering

Extracting Semantic-Dynamic Features for Long-Term Stable Brain Computer Interface

Sparse Maximum Margin Learning From Multimodal Human Behavioral Patterns

Deep Spiking Neural Networks with High Representation Similarity Model Visual Pathways of Macaque and Mouse

Astromorphic Self-Repair of Neuromorphic Hardware Systems

Reducing ANN-SNN Conversion Error through Residual Membrane Potential

ESL-SNNs: An Evolutionary Structure Learning Strategy For Spiking Neural Networks

Near-sensor compression in high-bandwidth brain-computer interface

Implantable Optoelectronics for Neural Interfaces

PSYCO:Py-based SYstem for Chronic monitoring of rOdent Species, an open-sourced python based system for homecage mouse tracking and image processing

The smart vivarium v1.0: implementation and validation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES