We present the Adversarial Nibbler Challenge, a novel red-teaming methodology for crowdsourcing implicitly adversarial prompts (those that trigger models to generate unsafe images for non-obvious reasons). Using a suite of state-of-the-art text-to-image models, a simple user interface, and outreach to engage diverse populations, we have captured long-tail safety issues that may be overlooked in standard testing. The first challenge round resulted in over 12k prompt-image pairs with machine annotations for safety, with 1.5k samples containing rich human annotations of harm types and attack strategies. Our findings emphasize the need for continual auditing as new vulnerabilities emerge in text-to-image models.

Adversarial Nibbler - A novel crowdsourcing procedure for detecting harmful content in t2i models

Adversarially testing large language models (LLMs) is crucial for their safe and responsible deployment in practice. We introduce an AI-assisted approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AART AI-assisted Red-Teaming - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce significantly human effort and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within a new application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. This provides transparency of developers evaluation intentions and enables quick adaptation to new use cases and newly discovered model weaknesses. Compared to some of the state-of-the-art tools AART shows promising results in terms of concept coverage and data quality.

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012). Metrics such as Krippendorff’s alpha or Cohen’s kappa are typically required to be above a threshold of 0.6 (Landis and Koch, 1977). These absolute thresholds are unreasonable for crowdsourced data from annotators with high cultural and training variances, especially on subjective topics. We present a new alternative to interpreting IRR that is more empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen’s (1960) kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.

Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

HCOMP 2020 Closing Session

User engagement plays a central role in companies and organisations that aimed to personalise their online services. A main challenge is to leverage knowledge about the online interaction of users to understand what engages them. A common way that engagement is measured and understood is through the definition and development of metrics of user satisfaction, which can act as a proxy of user engagement. This talk will present various works and personal thoughts on how to measure user engagement. It will discuss the definition and development of metrics of user satisfaction that can be used as a proxy of user engagement. An important message is that, for personalisation to work both in the short and the long-term, it is important to consider the heterogeneity of both user and content to formalise the notion of engagement, and in turn design the appropriate metrics to capture these and optimize for. One way to achieve this is to follow these four steps: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity. The talk will illustrate these steps with some of the research at Spotify.

Engagement, Metrics and personalisation at Scale

Workshop 1: Data Excellence (DEW 2020)

Amplified Intelligence: Uniting Human and Machine Intelligence in Cultural Heritage Workflows

<head>
	<style>
		.b1 {
			color: #ffffff;
			background-color: #5b1cf4;
			font-size: 19px;
			font-family: system-ui, sans-serif;
			border-radius: 15px;
			padding: 8px 40px;
			cursor: pointer
		}
		.b1:hover {
			color: #5b1cf4;
			background-color: #ffffff;
		}
	</style>
</head>
<button name="button" class="b1" onclick="window.open('https://us06web.zoom.us/rec/share/9L3bjkxEATmacwrCSHFkKv--9Qj3pfW421-LU5MRDW-RWrHg9LLNIW1bpxf9tKEV.MpYRBA7uN_MA_MMP')">Live Session Recording</button>
 

**This workshop is sponsored by:** 
![](https://assets.underline.io/markdown_image/1/image/9930cd8119e86600f955f66e25a7a870.png)
&nbsp&nbsp&nbsp&nbsp
![](https://assets.underline.io/markdown_image/1/image/c0eb62676c8bf505a36d8ab2efaf4b22.png) 
**About this workshop:** The special theme for this year is Online Harms in the Age of Large Language Models [**Read more**](https://www.workshopononlineabuse.com/) [**Schedule**](https://www.workshopononlineabuse.com/)

W8: The 8th Workshop on Online Abuse and Harms (WOAH 2024) (WOAH)

workshop paper

## Welcome to 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics!
This year, the conference is in Mexico City. NAACL was actually already planned for Mexico City in 2021, but due to the pandemic the entire conference was moved online. This year, finally, we get to go! So it is my sincere pleasure to welcome you to Mexico City, whether in person or virtually. Having the conference in Mexico City is a good opportunity to emphasize that NAACL is our flagship conference for ACL members not only in North America but also in Central and South America, even though NAACL has been bearing “North” in its name. At this year’s conference, we have a theme to match, with a theme track on the Languages of Latin America to showcase the linguistic diversity of the region.
 
The opportunity to present at NAACL should not depend on a researcher’s travel budget, or their family status. This is why it is so important to make virtual participation at NAACL as good an experience as possible – but we want to also provide a good experience for in-person participants. As a community, we are still working out the best way to do that. This year at NAACL, we are trying out a big virtual poster session ahead of the conference, with the hope that this will make make for a lively and interactive experience. At the same time, we are reducing virtual oral presentations, which seem to be particularly tricky to make to work well. A big thanks to the NAACL program chairs and to Luciana Benotti for all their ideas and work to improve the virtual experience. And participants, virtual as well as in-person: Please let us know what worked for you and what didn’t, so we can continue to improve hybrid conferences. 
 
I have been lucky to work with many amazing people. Without their insight, dedication and patience, and without the many hours of work they put in, NAACL would not have been possible. A huge thank you to the program chairs Helena Gomez, Kevin Duh, and Steve Bethard – you are the best! 
 
Finally, I would like to thank all authors, invited speakers and panelists, area chairs and reviewers, the volunteers organizing and chairing sessions, and all attendees, in-person and virtual. Thank you for helping us make NAACL 2024 come to life. 
 
Welcome and hope you all enjoy the conference! 
Katrin Erk 
The University of Texas at Austin 
NAACL-2024 General Chair 
*You can read the full Welcome message in the [Conference Handbook (downloadable)](https://drive.google.com/file/d/1H1NvW0VASQjkSCYgw3mr-3l4yYDSxz6B/view?usp=sharing)*

To access the event page you need to register [**here.**](http://acl.swoogo.com/naacl2024) 
Your access to the event page is limited based on your registration type. If you registered for workshops only, you will gain access to full workshops content on the day of the workshops program.

Please register!

NAACL 2024

2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics

**Are you attending this poster session virtually?** 
In-person printed posters are available for in-person attendees only.
 
**Are you attending this poster session in person?** 
Hybrid posters are displayed in the East Foyer.

PS7 (In-person) Posters, Demo, Industry, Findings

poster

**Welcome to EMNLP 2023!**

On behalf of the EMNLP 2023 Organizing Committee, I extend a warm and heartfelt welcome to all of you to the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). It is with immense pleasure and excitement that we gather here in Singapore, a vibrant hub of innovation and technological advancement.

The conference program is packed with insightful presentations, thought-provoking workshops, and engaging networking opportunities. In addition to the technical sessions, we have also planned several social events that will provide you with the opportunity to connect with your colleagues.

The conference is held in person at the Resorts World Convention Centre in Singapore, and available on-line with the help of Underline.

For those in Singapore, I hope that you will find time to explore the exciting Sentosa Island, Gardens by the Bay, Singapore Botanic Gardens and the many other attractions unique to Singapore.

Of course, we are grateful to our sponsors and partners for their generous support of EMNLP 2023, whose contributions make it possible for us to host this world-class event.

Yuji Matsumoto (RIKEN AIP) 
EMNLP 2023 General Chair

To access the **EMNLP 2023** event page on Underline, you need to register for the Conference. 
Please follow **[this link](https://2023.emnlp.org/registration/)** for more details.

EMNLP 2023

EMNLP 2023 took place in Singapore from Dec 6th to Dec 10th, 2023.

**Session Chair: ** Bonnie Webber 
**Session Volunteer: ** Yubin Ge

16B: Resources and Evaluation #6

technical paper

**Welcome to ACL-IJCNLP 2021!**

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP). 

As in previous years, the program of the conference includes a poster session, tutorials, workshops and demonstrations in addition to the main conference.


We were able to keep the registration fees similar to those charged for the virtual ACL 2020. The one fee allows attendance at the main conference and any/all tutorials and workshops. These fees would be $125 Regular Early and $175 Regular Late; $50 Student Early and $75 Student Late. Early registration closes at midnight July 11, 2021 (Eastern Daylight Time).

**Reminder:** It is ACL’s policy that at least one author of each accepted paper (including ACL Finding papers) must register for the conference.

**Reminder2:** Underline site will open closer to the event. If you already registered you will receive access detail

Registration is now open

IJCNLP-AACL 2021

The great event is jointly organized by the Association for Computational Linguistics (ACL) and Asian Federation of Natural Language Processing (AFNLP).

**Session Chairs:** Elena Simperl and Lora Aroyo

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP 2020

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


**Session Chair:** Lora Aroyo



**TALK ABSTRACT**

User engagement plays a central role in companies and organisations that aimed to personalise their online services. A main challenge is to leverage knowledge about the online interaction of users to understand what engages them. A common way that engagement is measured and understood is through the definition and development of metrics of user satisfaction, which can act as a proxy of user engagement. This talk will present various works and personal thoughts on how to measure user engagement. It will discuss the definition and development of metrics of user satisfaction that can be used as a proxy of user engagement. An important message is that, for personalisation to work both in the short and the long-term, it is important to consider the heterogeneity of both user and content to formalise the notion of engagement, and in turn design the appropriate metrics to capture these and optimize for. One way to achieve this is to follow these four steps: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity. The talk will illustrate these steps with some of the research at Spotify.

**SPEAKER BIOGRAPHY**

Mounia Lalmas is a Director of Research at Spotify, and the Head of Tech Research in Personalization. Mounia also holds an honorary professorship at University College London. Before that, she was a Director of Research at Yahoo, where she led a team of researchers working on advertising quality for Gemini, Yahoo's native advertising platform. She also worked with various teams at Yahoo on topics related to user engagement in the context of news, search, and user generated content. Prior to this, she held a Microsoft Research/RAEng Research Chair at the School of Computing Science, University of Glasgow. Before that, she was Professor of Information Retrieval at the Department of Computer Science at Queen Mary, University of London. Her work focuses on studying user engagement in areas such as native advertising, digital media, social media, search, and now audio. She has given numerous talks and tutorials on these and related topics, including a WWW 2019 tutorial on 'Online User Engagement: Metrics and Optimization', which will also be given at KDD 2020. She is regularly a senior programme committee member at conferences such as WSDM, KDD, WWW and SIGIR. She was co-programme chair for SIGIR 2015, WWW 2018 and WSDM 2020. She is also the co-author of a book written as the outcome of her WWW 2013 tutorial on 'measuring user engagement.

Keynote talk: Mounia Lalmas: Engagement, Metrics and personalisation at Scale

keynote

**Organizers:** Praveen Paritosh, Google; Matt Lease, UT Austin and Amazon; Mike Schaekermann, Amazon and University of Waterloo; Lora Aroyo, Google

**Overview:** Human annotated data is crucial for operationalizing empirical ways for evaluating, comparing, and assessing the progress of ML/AI research. As human annotated data represents the compass that the entire ML/AI community relies on, the human computation (HCOMP) research community has a multiplicative effect on the progress of the field. Optimizing the cost, size, and speed of collecting data has attracted significant attention by HCOMP and related research communities. In the first to market rush with data, aspects of maintainability, reliability, validity, and fidelity of datasets are often overlooked. We want to turn this way of thinking on its head and highlight examples, case-studies, methodologies for excellence in data collection. Data excellence happens organically due to appropriate support, expertise, diligence, commitment, pride, community, etc. We will invite speakers and submissions exploring such case studies in data excellence, focusing on empirical and theoretical methodologies for reliability, validity, maintainability, and fidelity of data. Goals of the workshop:

* Gather case studies of data excellence;
* Help define data excellence;
* Build a catalog of best practices for data excellence;
* Empirical and theoretical methodologies for reliability, validity, maintainability, fidelity of data;
* How do we invest more, not less, in data? How do we justify that investment?


Workshop 1: Data Excellence (DEW 2020) 

**Session Chairs:** Lora Aroyo

**TALK ABSTRACT**

In the field of cultural heritage, artificial intelligence technologies are increasingly adopted to extract and interpret the ‘Big Data of the Past’ – the historical information that forms the basis of a society’s collective memory and identity. At the same time, these technologies still struggle with the complex nature of historical data, in particular when it comes to interpreting its meaning and relevance. Hence, the implementation of such computational techniques is often combined with facilities for engaging human intelligence, in the form of crowdsourcing or citizen science projects. This lecture reflects on the opportunities of AI and Citizen Science workflows for accessing, interpreting and reusing cultural heritage data, assessing the extent to which these solutions are responsive to the complexity of human culture and meaningful to its users. A range of examples from the practice of (audiovisual) archives will be discussed, including human-machine workflows for transcribing and understanding written and spoken text, analyzing images and generating new objects at the Amsterdam City Archives, EYE Film Museum, BBC, and the Netherlands Institute for Sound and Vision. Areas of use include the cultural heritage sector itself (European Time Machine project), the creative industries (AI generated compilation films at EYE and BBC) and digital humanities research (human-computer workflows in the CLARIAH Media Suite). Building on the history of ideas about how computational technologies can be assistive to human intelligence, I will argue we need to design frameworks for an ‘amplified intelligence’, a collaboration between humans and machines that is responsive to the cultural values our archives, museums and libraries uphold.

**SPEAKER BIOGRAPHY**

Julia Noordegraaf is professor of Digital Heritage in the department of Media Studies at the University of Amsterdam. Within the Faculty of Humanities she acts as director of the digital humanities research program and lab Creative Amsterdam (CREATE) that studies the history of urban creativity using digital data and methods. Noordegraaf’s research focuses on the preservation, exhibition and reuse of audiovisual and digital cultural heritage. She has published, amongst others, the monograph Strategies of Display (2004/2012) and, as principal editor, Preserving and Exhibiting Media Art (2013) and acts as principal editor of the Cinema Context database on Dutch film culture. She currently leads research projects on the conservation of digital art (in the Horizon 2020 Marie Curie ITN project NACCA) and on the reuse of digital heritage in data-driven historical research (besides CREATE in the project Virtual Interiors as Interfaces for Big Historical Data Research). She is a former fellow of the Netherlands Institute for Advanced Study in the Humanities and Social Sciences and acts as board member for Media Studies in CLARIAH, the national infrastructure for digital humanities research, funded by the Netherlands Organization for Scientific Research, NWO. Noordegraaf currently coordinates the realization of the Amsterdam Time Machine and acts as Vice President of the European Time Machine Organization that aims to build a simulator for 5.000 years of European history.

Lora Aroyo

7

14

Presentations

Adversarial Nibbler - A novel crowdsourcing procedure for detecting harmful content in t2i models

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

HCOMP 2020 Closing Session

Engagement, Metrics and personalisation at Scale

Workshop 1: Data Excellence (DEW 2020)

Amplified Intelligence: Uniting Human and Machine Intelligence in Cultural Heritage Workflows

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES