Netherlands

We present our work in progress platform Turkish Judge, a crowd-driven adjudication system for rejected work on Amazon Mechanical Turk. The Mechanical Turk crowdsourcing platform allows Requesters to approve or reject assignments submitted by Workers. If the work is rejected, then Workers aren&#39;t paid, and their reputation suffers. Currently, there is no built-in mechanism for Workers to appeal rejections, other than contacting Requesters directly. The time it takes Requesters to review potentially incorrectly rejected tasks means that their costs are substantially higher than the payment amount that is in dispute. As a solution to this issue, we present an automated appeals system called Turkish Judge which employs crowd workers as judges to adjudicate whether work was fairly rejected when their peers initiate an appeal.

HCOMP 2020

Turkish Judge: A Peer Evaluation Framework for Mechanical Turk Appeals

mechanical turk

crowdsourcing

fairness

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

The AAMAS conference series was initiated in 2002 in Bologna, Italy as a joint event comprising the 6th International Conference on Autonomous Agents (AA), the 5th International Conference on Multiagent Systems (ICMAS), and the 9th International Workshop on Agent Theories, Architectures, and Languages (ATAL).

Subsequent AAMAS conferences have been held in Melbourne, Australia (July 2003), New York City, NY, USA (July 2004), Utrecht, The Netherlands (July 2005), Hakodate, Japan (May 2006), Honolulu, Hawaii, USA (May 2007), Estoril, Portugal (May 2008), Budapest, Hungary (May 2009), Toronto, Canada (May 2010), Taipei, Taiwan (May 2011), Valencia, Spain (June 2012), Minnesota, USA (May 2013), Paris, France (May 2014), Istanbul, Turkey (May 2015), Singapore (May 2016), São Paulo (2017) and Stockholm, Sweden (2018), Montreal (May 2019), Auckland (May 2020, Virtual), London (May 2021, Virtual).
<br>
<br>



A ticket is required to attend this event, please register using the link below:

https://aamas2022-conference.auckland.ac.nz/attending/registration/

Registration Is Required

AAMAS 2022

AAMAS (International Conference on Autonomous Agents and Multiagent Systems) is the largest and most influential conference in the area of agents and multiagent systems. The aim of the conference is to bring together researchers and practitioners in all areas of agent technology and to provide a single, high-profile, internationally renowned forum for research in the theory and practice of autonomous agents and multiagent systems. AAMAS is the flagship conference of the non-profit International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS).

We present our work in progress platform Turkish Judge, a crowd-driven adjudication system for rejected work on Amazon Mechanical Turk. The Mechanical Turk crowdsourcing platform allows Requesters to approve or reject assignments submitted by Workers. If the work is rejected, then Workers aren't paid, and their reputation suffers. Currently, there is no built-in mechanism for Workers to appeal rejections, other than contacting Requesters directly. The time it takes Requesters to review potentially incorrectly rejected tasks means that their costs are substantially higher than the payment amount that is in dispute. As a solution to this issue, we present an automated appeals system called Turkish Judge which employs crowd workers as judges to adjudicate whether work was fairly rejected when their peers initiate an appeal.

technical paper

The 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2020) will be held October 26–28th virtually at The Netherlands Institute for Sound and Vision in Hilversum, The Netherlands.

While artificial intelligence (AI) and human-computer interaction (HCI) represent traditional mainstays of the conference, HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. We promote the exchange of advances in human computation and crowdsourcing not only among researchers, but also engineers and practitioners, to encourage dialogue across disciplines and communities of practice.

This year, we especially invite works that will generate new insights into the quality (or qualities) of human-annotated datasets, including elements such as reliability and replicability of human computation and crowdsourcing experiments, novel metrics for aggregation of results, and holistic approaches to deal with bias, fairness and interpretability.

HCOMP believes strongly in fostering and promoting broad, interdisciplinary research. Our field is particularly unique in the diversity of disciplines it draws upon and contributes to, including human-centered qualitative studies and HCI design, social computing, artificial intelligence, economics, computational social science, digital humanities, policy, and ethics. 


For many applications, human cognitive limitations are something that poses challenging constraints. In this paper, I argue that they also have their advantages: they enable new systems and functionality that would be impossible without them. This creates an opportunity for new and exciting interdisciplinary collaborations between cognitive science and computer science.

Using Human Cognitive Limitations to Enable New Systems

Crowd-powered innovation platforms act to a large degree as silos: they cater the cognitive surplus of a bespoke, self-selected audience to a limited amount of high-value clients in a model where both stakeholders typically have to jump through a series of hoops to enrol to the services. We propose a fundamentally disruptive way for discovery with distributed crowds, by orchestrating already established online audiences for serendipitous crowdsourcing. Two recent developments make our proposition, The Planetary Cognition Delivery Network, compelling right now. First, online properties are struggling due to declining advertising revenues, caused by the proliferation of ad-blockers and a few key Internet giants taking an increasingly larger cut of the available revenue. Second, and perhaps somewhat counter-intuitively to the immediate perception in the Western world, the Internet is just now becoming widely available in many corners of the world, which provides an opportunity for a truly worldwide reach during the next decade or two. COGNET is designed to offer a frictionless participation mechanism for all three key stakeholders: requesters, providers, and contributors. It essentially orchestrates a distributed network of human cognition pools for arbitrary discovery tasks that could benefit from the vast cognitive surplus available through the internet.


COGNET: The Planetary Cognition Delivery Network

The field of crowdsourcing and human computation has seen a steady invention of technology and new applications of technology to solve increasingly complex problems. We propose a new framework that serves as a complement to incremental innovation by assuming that some seemingly intractable social-scale problems can be solved, suggesting a crowd-ready process to subdivide the problem by parts and by phase and then seeking development of the needed technology. The framework outlines a technology-assisted human computation process which results in the participatory design of what we call a social problem-solving model—an actionable community story that enjoys the buy-in from the community because it is developed by the community. We propose reframing a given social problem in terms of a community-determined superordinate goal and then recursively subdividing obstacles to that goal into larger numbers of simpler obstacle subcomponents, which better lend themselves to community-proposed solutions. The question of resource matching to implement these solutions is handled as a separate step. Technology tools need to be developed to facilitate each phase of this process. We discuss potential beneficial characteristics of the problem-solving model designed by the community and the nature of the technology required to implement each phase of the process. Finally, we discuss limitations we foresee to this approach.

A Technology-Assisted Social Computing Framework for Solving Complex Social Problems

We live in the Information Age where information is vital to our work. This implies that the more information we have, the better equipped we are to perform work. Working under the constant drive to be more productive, it is a foreseeable future where everyone works in synergy and partnership with AI agents in order to do more. To achieve this, a preferably simple and robust framework should facilitate the collaboration across a network of different humans and machines. As advancements in AI (narrow or otherwise) models continue, we will invariably reach a stage where humans and AI need to coexist in an interactive and personalised manner different from today’s largely invisible AI mainly operating autonomously in the background. In this work, our main contribution is the creation of a type theoretic framework enabling the orchestration of human-expressed intent(s), associated data and implementation(s) in a distributed manner which mirrors real world complex working environments where work plans are evolving and even disjoint due to lack of bigger picture.

Group-Assign: Type Theoretic Framework for Human AI Orchestration

As a visual artist, datasets are a core part of both my process and output. I will talk about what I believe at dataset to be; the compiling and training machine learning on datasets; exploring how datasets are created and possibilities for overcoming bias the different ways they can be constructed and some of the the issues, problems, solutions with working with them. In particular I will be focusing on classification and the consequences of using it in datasets, machine learning, and artistic works.

Using Machine Learning in an Artistic Context: Classification and Consquences

User engagement plays a central role in companies and organisations that aimed to personalise their online services. A main challenge is to leverage knowledge about the online interaction of users to understand what engages them. A common way that engagement is measured and understood is through the definition and development of metrics of user satisfaction, which can act as a proxy of user engagement. This talk will present various works and personal thoughts on how to measure user engagement. It will discuss the definition and development of metrics of user satisfaction that can be used as a proxy of user engagement. An important message is that, for personalisation to work both in the short and the long-term, it is important to consider the heterogeneity of both user and content to formalise the notion of engagement, and in turn design the appropriate metrics to capture these and optimize for. One way to achieve this is to follow these four steps: 1) Understanding intents; 2) Optimizing for the right metric; 3) Acting on segmentation; and 4) Thinking about diversity. The talk will illustrate these steps with some of the research at Spotify.

Turkish Judge: A Peer Evaluation Framework for Mechanical Turk Appeals

Downloads

Next from HCOMP 2020

Using Human Cognitive Limitations to Enable New Systems

Similar lecture

REFORM: Reputation Based Fair and Temporal Reward Framework for Crowdsourcing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Turkish Judge: A Peer Evaluation Framework for Mechanical Turk Appeals

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from HCOMP 2020

Using Human Cognitive Limitations to Enable New Systems

Similar lecture

REFORM: Reputation Based Fair and Temporal Reward Framework for Crowdsourcing

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads