Canada

The ability of large language models (LLMs) to follow natural language instructions with human-level fluency suggests many opportunities in healthcare to reduce administrative burden and improve quality of care. However, evaluating LLMs on realistic text generation tasks for healthcare remains challenging. Existing question answering datasets for electronic health record (EHR) data fail to capture the complexity of information needs and documentation burdens experienced by clinicians. To address these challenges, we introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes clinician-written reference responses for 303 instructions, and provides 276 longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality of each LLM response. We found high error rates, ranging from 35% (GPT-4) to 68% (MPT-7B-Instruct), and 8.3% drop in accuracy moving from 32k to 2k context lengths for GPT-4. Finally, we report correlations between clinician rankings and automated natural language generation metrics as a way to rank LLMs without human review. We make MedAlign available under a research data use agreement to enable LLM evaluations on tasks aligned with clinician needs and preferences.

AAAI 2024

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

snlp: question answering

snlp: applications

medicine & wellness

app: healthcare

technical paper

We are pleased to announce the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24), which will be held in Vancouver, British Columbia at the Vancouver Convention Centre – West Building from 20-27 February, 2024.

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-24 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and a range of other activities to be announced.

We expect for AAAI-24 to be an in-person conference – one author of all accepted papers will be expected to present work in person unless there are exceptional circumstances that prevent this.<br><br><br><br>

In order to access the AAAI-24 event page you need to register [here](https://aaai.org/aaai-conference/registration/)

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. 

Fingerprint authentication is a popular security mechanism for smartphones and laptops. However, its adoption in web and cloud environments has been limited due to privacy concerns over storing and processing biometric data on servers. This paper introduces Blind-Touch, a novel machine learning-based fingerprint authentication system that leverages homomorphic encryption to address these privacy concerns. Homomorphic encryption allows for computations on encrypted data without decrypting it. Therefore, Blind-Touch can keep fingerprint data encrypted on the server while performing machine learning operations. Blind-Touch integrates three techniques to address the computational challenges of using homomorphic encryption for machine learning: (1) A distributed machine learning architecture that divides inference tasks between the client and server, thereby reducing encrypted computations on the server; (2) A data compression method that reduces client-server communication costs; and (3) A cluster architecture that improves scalability with the number of registered users. Blind-Touch achieves high accuracy on two benchmark fingerprint datasets, with a 93.6% F1-score for the PolyU dataset and a 98.2% F1-score for the SOKOTO dataset. Moreover, Blind-Touch can match a fingerprint among 5,000 in about 0.65 seconds. With its privacy-focused design, high accuracy, and efficiency, Blind-Touch is a promising alternative to conventional fingerprint authentication for web and cloud applications.

Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication | VIDEO

With the widespread adoption of IoT, wearable devices, and sensors, time series data from human subjects are significantly increasing in the healthcare domain. Due to the laborious nature of manual annotation in time series data and the requirement for human experts, self-supervised learning methods are attempted to alleviate the limited label situations. While existing self-supervised methods have been successful to achieve comparable performance to the fully supervised methods, there are still some limitations that need to be addressed, considering the nature of time series data from human subjects: In real-world clinical settings, data labels (e.g., sleep stages) are usually annotated by subject-level, and there is a substantial variation in patterns between subjects. Thus, a model should be designed to deal with not only the label scarcity but also subject-wise nature of data to ensure high performance in real-world scenarios. To mitigate these issues, we propose a novel self-supervised learning framework for human subject time series data: Subject-Aware Time Series Clustering (SA-TSC). In the unsupervised representation learning phase, SA-TSC adopts a subject-wise learning strategy rather than instance-wise learning which randomly samples data instances from different subjects within the batch during training. Specifically, we generate subject-graphs with our graph construction method based on Gumbel-Softmax and perform graph spectral clustering on each subject-graph. In addition, we utilize graph neural networks to capture dependencies between channels and design our own graph learning module motivated from self-supervised loss. Experimental results show the outstanding performance of our SA-TSC with the limited & subject-wise label setting, leading to its high applicability to the healthcare industry. The code is available at: https://github.com/DILAB-HYU/SA-TSC

Self-Supervised Framework Based on Subject-Wise Clustering for Human Subject Time Series Data

Hate speech (HS) is a growing concern in many parts of
the world, including India, where it has led to numerous instances of violence and discrimination. The development of
effective counter-narratives (CNs) is a critical step in combating hate speech, but there is a lack of research in this
area, especially in non-English languages. In this paper, we
introduce a new dataset, IndicCONAN, of counter-narratives
against hate speech in Hindi and Indian English. We propose a scalable human-in-the-loop approach for generating counter-narratives by an auto-regressive language model
through machine generation - human correction cycle, where
the model uses augmented data from previous cycles to generate new training samples. These newly generated samples
are then reviewed and edited by annotators, leading to further
model refnement. The dataset consists of over 2,500 exam- ˜
ples of counter-narratives each in both English and Hindi corresponding to various hate speeches in the Indian context. We
also present a framework for generating CNs conditioned on
specifc CN type with a mean perplexity of 3.85 for English
and 3.70 for Hindi, a mean toxicity score of 0.04 for English
and 0.06 for Hindi, and a mean diversity of 0.08 for English
and 0.14 for Hindi. Our dataset and framework provide valuable resources for researchers and practitioners working to
combat hate speech in the Indian context.

IndicCONAN: A Multilingual Dataset for Combating Hate Speech in Indian Context

Diffusion models have shown their effectiveness in generation tasks by well-approximating the underlying probability distribution. However, diffusion models are known to suffer from an amplified inherent bias from the training data in terms of fairness. While the sampling process of diffusion models can be controlled by conditional guidance, previous works have attempted to find empirical guidance to achieve quantitative fairness. 
To address this limitation, we propose a fairness-aware sampling method called \textit{attribute switching} mechanism for diffusion models. Without additional training, the proposed sampling can obfuscate sensitive attributes in generated data without relying on classifiers.
We mathematically prove and experimentally demonstrate the effectiveness of the proposed method on two key aspects: (i) the generation of fair data and (ii) the preservation of the utility of the generated data.

Fair Sampling in Diffusion Models through Switching Mechanism

The rise of AI-based and autonomous systems is raising concerns and apprehension due to potential negative repercussions arising from their behavior or decisions. These systems must be designed to comply with the human contexts in which they will operate. To this extent, Townsend et al. (2022) introduce the concept of SLEEC (social, legal, ethical, empathetic, or cultural) rules that aim to facilitate the formulation, verification, and enforcement of the rules AI-based and autonomous systems should obey. They lay out a methodology to elicit them and to let philosophers, lawyers, domain experts, and others to formulate them in natural language. To enable their effective use in AI systems, it is necessary to translate these rules systematically into a formal language that supports automated reasoning. In this study, we first conduct a linguistic analysis of the SLEEC rules pattern, which justifies the translation of SLEEC rules into classical logic. Then we investigate the computational complexity of reasoning about SLEEC rules and show how logical programming frameworks can be employed to implement SLEEC rules in practical scenarios. the result is a readily applicable strategy for implementing AI systems that conform to norms expressed as SLEEC rules.

Social, Legal, Ethical, Empathetic, and Cultural Rules: Compilation and Reasoning

In the digital age, cybercrimes, particularly cyber harassment, have become pressing issues, targeting vulnerable individuals like children, teenagers, and women. Understanding the experiences and needs of the victims is crucial for effective support and intervention. Online conversations between victims and virtual harassment counselors (chatbots) offer valuable insights into cyber harassment manifestations (CHMs) and determinants (CHDs). However, the distinction between CHMs and CHDs remains unclear. This research is the first to introduce concrete definitions for CHMs and CHDs, investigating their distinction through automated methods to enable efficient cyber-harassment dialogue comprehension. We present a novel dataset, Cyber-MaD that contains Cyber harassment dialogues manually annotated with Manifestations and Determinants. Additionally, we design an Emotion-informed Contextual Dual Attention Convolution Transformer (EmoConDuCT) framework to extract CHMs and CHDs from cyber harassment dialogues. The framework primarily: a). exploits inherent emotion features in the input via adjective-noun pairs modeled through an autoencoder, b). utilizes a novel Contextual Dual Attention Convolution Transformer to learn contextual information, and, c). employs a demarcation module that leverages task-specific emotional knowledge and a discriminator loss function to differentiate manifestations and determinants. EmoConDuCT outperforms the state-of-the-art systems on the Cyber-MaD corpus, showcasing its potential in the extraction of CHMs and CHDs. Furthermore, its robustness is demonstrated on the emotion cause extraction task using the CARES_CEASE-v2.0 dataset of suicide notes, confirming its efficacy in diverse cause extraction objectives. This research sheds light on cyber harassment comprehension, paving the way for improved support and intervention strategies.

Benchmarking Cyber Harassment Dialogue Comprehension through Emotion-Informed Manifestations-Determinants Demarcation

Multi-label classification is an arduous problem given the complication in label correlation. Whilst sharing a common goal with contrastive learning in utilizing correlations for representation learning, how to better leverage label information remains challenging. 
Previous endeavors include extracting label-level presentations or mapping labels to an embedding space, overlooking the correlation between multiple labels. 
It exhibits a great ambiguity in determining positive samples with different extent of label overlap between samples and integrating such relations in loss functions. 
In our work, we propose Multi-Label Supervised Contrastive learning (MulSupCon) with a novel contrastive loss function to adjust weights based on how much overlap one sample shares with the anchor. 
By analyzing gradients, we explain why our method performs better under multi-label circumstances. 
To evaluate, we conduct direct classification and transfer learning on several multi-label datasets, including widely-used image datasets such as MS-COCO and NUS-WIDE.
Validation indicates that our method outperforms the traditional multi-label classification method and shows a competitive performance when comparing to other existing approaches.

Multi-Label Supervised Contrastive Learning

Despite meaningful progress over the past few decades, advances in the self-driving industry have plateaued. Traditional approaches are incredibly labor- and capital-intensive, relying on hand-engineered systems and millions of real-world testing miles. Generative AI offers the much-needed new path to unlocking autonomous vehicles at scale and changing the way these systems work. In this lecture, Waabi CEO and Founder Raquel Urtasun will discuss how leveraging generative AI will evolve the way autonomous driving systems are developed and trained, resulting in faster, safer, and more scalable deployment of transformative self-driving technology worldwide.

2024 Robert S. Engelmore Memorial Lecture Award: Accelerating AVs with the Next Generation of Generative AI

The capabilities, autonomy, and efficiency of software and physical agents have increased remarkably in recent times. However, the rapid advancement of these intelligent systems presents significant challenges for humans when they need to interact, coordinate, or collaborate with them. Intelligent advising agents providing guidance to people in such situations can enhance their decision-making and alleviate cognitive load. Alongside identifying valuable advice, these assisting agents must also deliver explanations to convince humans that the provided advice is advantageous and trustworthy. In the presentation, we will highlight the challenges that emerge when humans need to collaborate with drones, autonomous vehicles, and intelligent systems for negotiation and mediation. Additionally, we will introduce several approaches for developing assisting agents aimed at enhancing human performance and satisfaction.

AAAI Invited Talk: Smart Advice: Intelligent Agents Assisting Humans in the Super AI Era | VIDEO

As the massive excitement about recent advances in AI indicates, AI is a transformative technology that likely will have a deep and systemic impact on the national and global economy and security. The recognition of the potential impact of AI has led many countries to launch national strategic research initiatives and policies about the development and deployment of AI. The goal of this panel is to discuss some of these strategic initiatives and policies from different perspectives and at different levels of aggregation. We will begin with an overview of the US NSF’s National AI Institutes Program (Donlon), illustrate it with a specific National AI Institute (Goel), describe the structures and processes in the US that result in a national strategic initiative (Littman), present current and potential AI-related policies in the US (Wagstaff), and compare with similar initiatives and policies across the world (Walsh).

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

Downloads

Next from AAAI 2024

Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication | VIDEO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from AAAI 2024

Blind-Touch: Homomorphic Encryption-Based Distributed Neural Network Inference for Privacy-Preserving Fingerprint Authentication | VIDEO

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads