**Abstract:**

Social learning is a key driver of the success of the human species (Henrich, 2015), enabling the preservation of technologies that individuals could not possibly have acquired on their own (Tennie, Call, & Tomasello, 2009). Presently, we are witnessing the rise of intelligent machines – “AI systems” – that are capable of making discoveries beyond the human horizon (Wang et al., 2023). This raises the question of whether social learning enables humans to also adopt and preserve inconceivable discoveries made by intelligent machines. In our behavioral experiment, participants’ success in a strategic task hinged on adopting a solution that humans struggle to discover due to a characteristic bias to avoid large losses. We contrast populations of humans transmitting solutions over multiple generations on their own with populations that had the opportunity to learn from AI agents in their first generation. Our findings show that an algorithmic model succeeds in discovering the adaptive strategy where humans fail on their own. Most importantly, humans are able to adopt this machine-introduced strategy and preserve it over multiple generations. Taken together, our results showcase the potential of intelligent machines to transform human cultural evolution, and an opportunity for designing them towards benefitting human progress.

**Speaker's social media:**

Twitter: @ThomasF_Mueller
Bluesky: @thomasfmueller.bsky.social

LIVE - Humans can learn and culturally preserve inconceivable strategies discovered by intelligent machines

Children engage in cultural knowledge exchange by actively sharing what they know with conspecifics– and do so selectively with respect to information type and its recipients. Previous research suggests that from preschool age, children tend to share information that is generalisable to a kind (“dogs have fur”), rather than specific (“this dog has spots”). However, the ontogeny of such generalisability preference and its systematic investigations are lacking. We aimed to probe this question further. 

In two experimental studies, children aged 2 (N = 49) and 6-9 (N = 144) were presented with novel information and then had an opportunity to share it with a naïve learner. We tested if children showed a preference for transmitting generalisable over specific information. In Study 1, such information was presented though action outcomes on novel objects and 2-year-olds did not display the hypothesised preference. In four experiments comprising Study 2, we found that older children selectively shared generalisable facts when those were of neutral nature, as well as health-related, but not when it was in a potentially implicitly threatening context. 

These results advance our understanding of children’s selective sharing of generalisable knowledge by uncovering the limits in early childhood and in certain learning contexts. 

LIVE - Information worth sharing: Do children transmit generalisable or specific information to naïve social partners?

Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method, we find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public to facilitate the reproduction of our work and advance research in this area.

Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST).
Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. 
However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. 
To address this limitation, we propose a U2S generator that incorporates a distillation with no label (DINO) self-supervised training strategy into it's pretraining process.
Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment.
Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments.

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public’s preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual’s preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system messages steering the LLM’s generation behavior to better align with the user's intentions. However, a naive adaptation of such an approach is non-trivial since LLMs are typically trained on a uniform system messages (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus demonstrates performance that is as good as or better than Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, 75.2%, 72.4%, and 66.4% of the time, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at https://anonymous.4open.science/r/janus.

Aligning to Thousands of Preferences via System Message Generalization

Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components

Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system's capabilities via strict user goals, namely "user familiarity" bias. This data bias deepens when it combines with data-driven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (open-goal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities. We discuss its characteristics and toxicity while showing recent large language models can also suffer from this behavior.

Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation

This paper presents an interpretable unsupervised morphological learning model, showing comparable performance to supervised models in learning complex morphological rules of Turkish as evidenced by its application to the problem of morphological inflection within the SIGMORPHON Shared Tasks. The significance of our unsupervised approach lies in its alignment with how humans naturally acquire rules from raw data without supervision. To achieve this, we construct a model with multiple codebooks of VQ-VAE employing continuous and discrete latent variables during word generation. We evaluate the model's performance under high and low-resource scenarios, and use probing techniques to examine encoded information in latent representations. We also evaluate its generalization capabilities by testing unseen suffixation scenarios within the SIGMORPHON-UniMorph 2022 Shared Task 0. Our results demonstrate our model's ability to distinguish word structures into lemmas and suffixes, with each codebook specialized for different morphological features, contributing to the interpretability of our model and effectively performing morphological inflection on both seen and unseen morphological features.

Unsupervised Learning of Turkish Morphology with Multiple Codebook VQ-VAE

While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce \textbf{G}enerative \textbf{P}re-trained \textbf{S}peech \textbf{T}ransformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. See \url{https://youngsheen.github.io/GPST/demo} for demo samples.

Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer

Integrated Speech and Large Language Models (SLMs) that can follow speech instructions and generate relevant text responses have gained popularity lately. However, the safety and robustness of these models remains largely unclear. In this work, we investigate the potential vulnerabilities of such instruction-following speech-language models to adversarial attacks and jailbreaking. Specifically, we design algorithms that can generate adversarial examples to jailbreak SLMs in both white-box and black-box attack settings without human involvement. Additionally, we propose countermeasures to thwart such jailbreaking attacks. Our models, trained on dialog data with speech instructions, achieve state-of-the-art performance on spoken question-answering task, scoring over 80% on both safety and helpfulness metrics. Despite safety guardrails, experiments on jailbreaking demonstrate the vulnerability of SLMs to adversarial perturbations and transfer attacks, with average attack success rates of 90% and 10% respectively when evaluated on a dataset of carefully designed harmful questions spanning 12 different toxic categories. However, we demonstrate that our proposed countermeasures reduce the attack success significantly.

SpeechGuard: Exploring the Adversarial Robustness of Multi-modal Large Language Models

The Spoken Language Understanding Evaluation (SLUE) suite of benchmark tasks was recently introduced to address the need for open
resources and benchmarking of complex spoken language understanding (SLU) tasks, including both classification and sequence generation tasks, on natural speech. The benchmark has demonstrated preliminary success in using pre-trained speech foundation models (SFM) for these SLU tasks. However, the community still lacks a fine-grained understanding of the comparative utility of different SFMs. Inspired by this, we ask: which SFMs offer the most benefits for these complex SLU tasks, and what is the most effective approach for incorporating these SFMs? To answer this, we perform an extensive evaluation of multiple supervised and self-supervised SFMs using several evaluation protocols: (i) frozen SFMs with a lightweight prediction head, (ii) frozen SFMs with a complex prediction head, and (iii) fine-tuned SFMs with a lightweight prediction head. Although the supervised SFMs are pre-trained on much more speech recognition data (with labels), they do not always outperform self-supervised SFMs; the latter tend to perform at least as well as, and sometimes better than, supervised SFMs, especially on the sequence generation tasks in SLUE. While there is no universally optimal way of incorporating SFMs, the complex prediction head gives the best performance for most tasks, although it increases the inference time. We also introduce an open-source toolkit and performance leaderboard, SLUE-PERB, for these tasks and modeling strategies.

On the Evaluation of Speech Foundation Models for Spoken Language Understanding

We present INDICVOICES, a dataset of natural and spontaneous speech containing a total of 7348 hours of read (9%), extempore (74%) and conversational (17%) audio from 16237 speakers covering 145 Indian districts and 22 languages. Of these 7348 hours, 1639 hours have already been transcribed, with a median of 73 hours per language. Through this paper, we share our journey of capturing the cultural, linguistic and demographic diversity of India to create a one-of-its-kind inclusive and representative dataset. More specifically, we share an open-source blueprint for data collection at scale comprising of standardised protocols, centralised tools, a repository of engaging questions, prompts and conversation scenarios spanning multiple domains and topics of interest, quality control mechanisms, comprehensive transcription guidelines and transcription tools. We hope that this open source blueprint will serve as a comprehensive starter kit for data collection efforts in other multilingual regions of the world. Using INDICVOICES, we build IndicASR, the first ASR model to support all the 22 languages listed in the 8th schedule of the Constitution of India.

Underline digital video library

LIVE - Humans can learn and culturally preserve inconceivable strategies discovered by intelligent machines

LIVE - Information worth sharing: Do children transmit generalisable or specific information to naïve social partners?

Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Aligning to Thousands of Preferences via System Message Generalization

Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components

Revealing User Familiarity Bias in Task-Oriented Dialogue via Interactive Evaluation

Unsupervised Learning of Turkish Morphology with Multiple Codebook VQ-VAE

Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer

SpeechGuard: Exploring the Adversarial Robustness of Multi-modal Large Language Models

On the Evaluation of Speech Foundation Models for Spoken Language Understanding

IndicVoices: Towards building an Inclusive Multilingual Speech Dataset for Indian Languages

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES