Spain

&quot;Competitions are a proven and cost-effective method to quickly develop new disruptive technologies for new markets. However, artificial intelligence and robotics competitions are usually organized independently, in silos. To get the most out of competitions, evaluation of robots should be modular (tasks are designed to assess independently different technological building blocks of the robotic systems) and open (evaluation tools and data must be publicly and freely available online). This not only meets the benchmarking needs of end users (mostly interested in performance measurements of the complete system), but also those of integrators and developers of intelligent components (camera, lidar, radar, actuators, etc.).
With this objective in mind, the H2020 METRICS project (2020-2023) organizes competitions in four application areas (Healthcare, Infrastructure Inspection and Maintenance, Agri-Food, and Agile Production) relying on both physical testing facilities (field evaluation campaign) and virtual testing facilities (data-based evaluation campaign) to mobilize, in addition to the European robotics community, the artificial intelligence one.
This presentation outlines this approach and paves the way for a new robotics and artificial intelligence competition paradigm.&quot;

ECAI 2020

AI evaluation campaigns during robotics competitions: the METRICS paradigm

ECAI2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI, including:




**An extensive programme of workshops, tutorials, and invited speakers**

**The Prestigious Applications of Intelligent Systems conference (PAIS)**

**The Frontiers in AI track sessions**

**The Lab To Market event**

**The Women in AI time**

**The EU Challenges forum**

… and many more!

ECAI 2020 featured a full programme of technical papers, as well as many AI-related events of interest for researchers, students and all attendants who are interested on contemporary AI.

"Competitions are a proven and cost-effective method to quickly develop new disruptive technologies for new markets. However, artificial intelligence and robotics competitions are usually organized independently, in silos. To get the most out of competitions, evaluation of robots should be modular (tasks are designed to assess independently different technological building blocks of the robotic systems) and open (evaluation tools and data must be publicly and freely available online). This not only meets the benchmarking needs of end users (mostly interested in performance measurements of the complete system), but also those of integrators and developers of intelligent components (camera, lidar, radar, actuators, etc.).
With this objective in mind, the H2020 METRICS project (2020-2023) organizes competitions in four application areas (Healthcare, Infrastructure Inspection and Maintenance, Agri-Food, and Agile Production) relying on both physical testing facilities (field evaluation campaign) and virtual testing facilities (data-based evaluation campaign) to mobilize, in addition to the European robotics community, the artificial intelligence one.
This presentation outlines this approach and paves the way for a new robotics and artificial intelligence competition paradigm."

technical paper

"The ROSE Challenge is the first robotics and artificial intelligence competition worldwide to implement a third-party performance evaluation of intra-row weeding robots in real conditions, under comparable conditions, to guarantee a credible and objective assessment of their effectiveness. This article reports on the design and
validation of testing facilities for this competition, which presents a particular complexity : the experiments take place outdoors and act on living things (crops and weeds). Moreover, it implies to guarantee repeatable experimental conditions for a comparable and equitable evaluation. The article also discusses the opportunity that these competitions represent to define testing facilities in a consensual way.
The method it proposes is very widely applicable to different fields of intelligent systems applications."

Design and validation of testing facilities for weeding robots as part of ROSE challenge

"Recent progress in machine learning, on predictive tasks in particular, is usually claimed on the basis of performance comparisons.
Due to the limitation of application scenarios and computational resources, most people cannot evaluate their work on a large variety of tasks and datasets, and it has become tricky for both reviewers and readers to verify individual performance gains for a given approach.
In this paper we investigate possible approaches to achieve better efficiency on model benchmarking. 
For a large collection of datasets, rather than training and testing a given approach on every individual dataset, we seek methods that allow us to pick only a few representative datasets to quantify the model's goodness, from which to extrapolate to performance on other datasets. 
To this end, we adopt existing approaches from psychometrics: specifically, Item Response Theory and Adaptive Testing. Both are well-founded frameworks designed for educational tests.
We propose certain modifications following the requirements of machine learning experiments, and present some initial results and insights in this paper."

Towards Efficient and Robust Model Benchmarks with Item Response Theory and Adaptive Testing

We consider the issue of AI (Artificial Intelligence) evaluation in the light of risk management for the proper marketing of machinery. When a physical device is driven by AI, the question arises as to how to manage the risks associated with the autonomous functions. Indeed, the risk of physical damage to persons or property in the event of a ``wrong'' decision must be properly assessed. The study presents the advances needed to carry out relevant risk analyses for the deployment of safe intelligent machines, and offers good practice that leverages the knowledge in AI evaluation to optimize the safety of these machines.

Risk assessment of artificial intelligence in autonomous machines

This paper argues that at a certain point in research toward AGI, the problem may become well-enough theorized that a clear roadmap exists for achieving it, such that a Manhattan Project- like effort could greatly shorten the time to completion. If state actors perceive that this threshold has been crossed, their incentives around openness and international cooperation may shift rather suddenly, with serious implications for AI risks and the stability of international AI governance regimes. The paper characterizes how such a ‘runway’ period would be qualitatively different from preceding stages of AI research, and accordingly proposes a research program aimed at assessing how close the field of AI is to such a threshold—that is, it calls for the formulation of a ‘roadmap to the roadmap.’

Roadmap to a Roadmap: How Could We Tell When AGI is a ‘Manhattan Project’ Away?

The widespread use of experimental benchmarks in AIresearch  has  created  new  competition  and  collaboration  dynamics that are still poorly understood. In this paper we provide an innovative methodology to explore this dynamics and analyse the way different entrants in these competitions, from academia to tech giants,behave and react depending on their own or others’ achievements.We perform an analysis of over twenty popular benchmarks in AI,linking their underlying research papers. We identify links between researchers and institutions (i.e., communities) beyond the standard co-authorship relations, and we explore a series of hypotheses about their behaviour as well as some aggregated results in terms of activity, breakthroughs and efficiency. As a result, we detect and characterise the dynamics of research communities at different levels of abstraction, including organisation, affiliation, trajectories, results and activity.

The Scientometrics of AI Benchmarks: Unveiling the Underlying Mechanics of AI Research

The performance of the machine learning algorithms can be empirically evaluated via test campaigns: conventional testing approaches allow to obtain, from annotated databases, a first estimate of the operating environment of artificial intelligence systems. However, as those data are hardly I .i.d, additional scores could be useful to offer complementary performance assessment typically for intelligent systems performing critical functions. This paper proposes an approach to generate, without annotation phase, realistic corner cases to complete the exploration of the operating environment. The results of the proof of concept provided for optical character recognition highlight the interest of test data generation and automatic scenario selection methods, including GAN and adversarial approaches, to address this evaluation need.

Generating corner cases for crash-testing deep networks

"The field of Artificial Intelligence (AI) and, in particular, the Machine Learning area, counts on a wide range of performance metrics and benchmark data sets to assess the problem-solving effec- tiveness of its solutions. However, the appearance of research centres, projects or institutions addressing AI solutions from a multidisci- plinary and multi-stakeholder perspective suggests a new approach to assessment comprising ethical guidelines, reports or tools and frame- works to help both academia and business to move towards a re- sponsible conceptualisation of AI. They all highlight the relevance of three key aspects: (i) enhancing cooperation among the different stakeholders involved in the design, deployment and use of AI; (ii) promoting multidisciplinary dialogue, including different domains of expertise in this process; and (iii) fostering public engagement to maximise a trusted relation with new technologies and practitioners. In this paper, we introduce the Observatory on Society and Artifi- cial Intelligence (OSAI), an initiative grew out of the project AI4EU aimed at stimulating reflection on a broad spectrum of issues of AI (ethical, legal, social, economic and cultural). In particular, we de- scribe our work in progress around OSAI and suggest how this and similar initiatives can promote a wider appraisal of progress in AI. This will give us the opportunity to present our vision and our modus operandi to enhance the implementation of these three fundamental dimensions.
"

Progressing Towards Responsible AI

This work presents an operational definition that facilitates a common understanding of AI and its measurability, paving the way for a transparent and comparable monitoring activity in the context of AI Watch, the European Commission’s knowledge service to monitor the development, uptake and impact of artificial intelligence (AI) for Europe. The operational definition is constituted by a taxonomy and a list of related keywords. The method that we propose is iterative, considering that AI is a dynamic field, so its definition should be updated over time to capture the rapid AI evolution. The method consists of the following steps: (i) qualitative analysis of AI definitions and subdomains emanating from reports with academic, industrial and policy perspectives, (ii) selection of definition as starting point, taxonomy formation, and identification of representative keywords in AI with a natural language processing method, and (iii) taxonomy and keywords validation. This results in a unique taxonomy that represents and interconnects all the AI subdomains from political, research and industrial perspective and enables the efficient mapping of the AI landscape of economic agents across different technological areas.

Setting the Boundaries of the AI Landscape: An Operational Definition for the European Commission’s AI Watch

This short paper aims at introducing some labelling concepts developed and used by the authors. Indeed, developing a conceptual framework for interdisciplinary research in history is a much-needed strategy in order to ensure that historians use all vestiges from the past regardless their origin or support for the construction of historical discourse. Fixing the semantics of historical knowledge is the first unavoidable step to build a new scenario in which NLP tasks enable more efficient data gathering and exploitation.

The Semantics of Historical Knowledge. Labelling Strategies for Interdisciplinary and Digital Research in History

Presentation is about our chatbot creation platform and its dialog management techniques. The framework enables creating chatbots without coding using intent-slot model. It contains intent detection and dialog act classifier modules to understand user utterances more accurately. Also the framework includes and analytics tool to monitor and cluster past conversation data to discover topics and training sentences.

Downloads

Next from ECAI 2020

Design and validation of testing facilities for weeding robots as part of ROSE challenge

Similar lecture

Design and validation of testing facilities for weeding robots as part of ROSE challenge

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from ECAI 2020

Design and validation of testing facilities for weeding robots as part of ROSE challenge

Similar lecture

Design and validation of testing facilities for weeding robots as part of ROSE challenge

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads