Italy

Activation functions and attention mechanisms are typically treated as having different purposes and have evolved differently.
However, both concepts can be formulated as a non-linear gating function. 
Inspired by their similarity, we propose a novel type of activation units called attentional activation (ATAC) units as a unification of activation functions and attention mechanisms.
In particular, we propose a local channel attention module for the simultaneous non-linear activation and element-wise feature refinement, which locally aggregates point-wise cross-channel feature contexts.
By replacing the well-known rectified linear units by 
such ATAC units in convolutional networks, we can construct fully attentional networks that perform significantly better with a modest number of additional parameters.
We conducted detailed ablation studies on the ATAC units using several host networks with varying network depths to empirically verify the effectiveness and efficiency of the units.
Furthermore, we compared the performance of the ATAC units against existing activation functions as well as other attention mechanisms on the CIFAR-10, CIFAR-100, and ImageNet datasets.
Our experimental results show that networks constructed with the proposed ATAC units generally yield performance gains over their competitors given a comparable number of parameters.

ICPR 2020

Attention as Activation

activation function

channel attention

attention mechanism

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


technical paper

Evaluation of generative AI is a difficult problem, especially in artistic domains in which aesthetic qualities of generated samples are to an extent subjective, such as in music. The most widely accepted method for evaluating such models is to conduct a survey of users, which is a resource intensive process. In this work we propose a framework for cheaply evaluating generative models in the symbolic music domain by utilizing tools from music theory, such as the circle of fifths, with the goal of producing quantifiable metrics which reflect the "musicality" of a written score or MIDI file.

Heuristics for Evaluation of AI Generated Music

Despite the success of deep neural networks (DNN) in many applications, their ability to model uncertainty is still significantly limited. For example, in safety-critical applications such as autonomous driving, it is crucial to obtain a prediction that reflects different types of uncertainty to address life-threatening situations appropriately. In such cases, it is essential to be aware of the risk (i.e.,~aleatoric uncertainty) and the reliability (i.e.,~epistemic uncertainty) that comes with a prediction. We present AE-DNN, a model allowing the separation of aleatoric and epistemic uncertainty while maintaining a proper generalization capability. AE-DNN is based on deterministic DNN, which can determine the respective uncertainty measures in a single forward pass. In analyses with synthetic and image data, we show that our method improves the modeling of epistemic uncertainty while providing an intuitively understandable separation of risk and reliability.

Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks

We use feature attention and transformation to address the nighttime pedestrian detection issue.

Nighttime Pedestrian Detection Based on Feature Attention and Transformation


We present DeepScoresV2, an extended version of the DeepScores dataset for optical music recognition (OMR). We improve upon the original DeepScores dataset by providing much more detailed annotations, namely (a) annotations for135classes including fundamental symbols of non-fixed size and shape, increasing the number of annotated symbols by 23%; (b) oriented bounding boxes; (c) higher-level rhythm and pitch information (onset beat for all symbols and line position for noteheads); and (d) a compatibility mode for easy use in conjunction with the MUSCIMA++ dataset for OMR on handwritten documents. These additions open up the potential for future advancement in OMR research. Additionally, we release two state-of-the-art baselines for DeepScoresV2based on Faster R-CNN and the Deep Watershed Detector. An analysis of the baselines shows that regular orthogonal bounding boxes are unsuitable for objects which are long, small, and potentially rotated, such as ties and beams, which demonstrates the need for detection algorithms that naturally incorporate objects angles.

The DeepScoresV2 Dataset and Benchmark for Music Object Detection

We investigate whether it is possible to classify the gender of a standing person based on a video sequence containing body sway recorded by an overhead camera. Existing methods that extract a feature from the movement of a walking person for gender classification cannot detect the slight movements of a standing person. In this paper, we propose a method for extracting a feature from the body sway of a standing person. We design a spatio-temporal feature for representing body sway using the frequency analysis of time-series signals derived from the local movements of the upper body. To evaluate the accuracy of our method, we acquired video sequences of body sway from 30 females and 30 males using an overhead camera. We found that our method obtained 90.3% accuracy for the gender classification of a standing person. We compared the accuracy of our method with that of parameters based on medical data. We found that the proposed spatio-temporal feature extracted from body sway significantly improved gender classification accuracy.

Gender Classification Using Video Sequences of Body Sway Recorded by Overhead Camera

Ureteroscopy is becoming the first surgical treatment option for the majority of urinary affections. 
This procedure is performed using an endoscope which provides the surgeon with the visual information necessary to navigate inside the urinary tract. 
Having in mind the development of surgical assistance systems, that could enhance the performance of surgeon, the task of lumen segmentation is a fundamental part since this is the visual reference which marks the path that the endoscope should follow. 
This is something that has not been analyzed in ureteroscopy data before. 
However, this task presents several challenges given the image quality and the conditions itself of ureteroscopy procedures. 
In this paper, we study the implementation of a Deep Neural Network which exploits the advantage of residual units in an architecture based on U-Net. 
For the training of these networks, we analyze the use of two different color spaces: gray-scale and RGB data images. 
We found that training on gray-scale images gives the best results obtaining mean values of Dice Score, Precision, and Recall of 0.73, 0.58, and 0.92 respectively. 
The results obtained shows that the use of residual U-Net could be a suitable model for further development for a computer-aided system for navigation and guidance through the urinary system.

A Lumen Segmentation Method in Ureteroscopy Images based on a Deep Residual U-Net architecture


The vascular structure of blood vessels is important in diagnosing retinal conditions such as glaucoma and diabetic retinopathy. Accurate segmentation of these vessels can help in detecting retinal objects such as the optic disc and optic cup and hence determine if there are damages to these areas. Moreover, the structure of the vessels can help in diagnosing glaucoma. The rapid development of digital imaging and computer-vision techniques has increased the potential for developing approaches for segmenting retinal vessels. In this paper, we propose an approach for segmenting retinal vessels that uses deep learning along with transfer learning. We adapted the U-Net structure to use a customized InceptionV3 as the encoder and used multiple skip connections to form the decoder. Moreover, we used a weighted loss function to handle the issue of class imbalance in retinal images. Furthermore, we contributed a new dataset to this field. We tested our approach on six publicly available datasets and a newly created dataset. We achieved an average accuracy of 95.60% and a Dice coefficient of 80.98%. The results obtained from comprehensive experiments demonstrate the robustness of our approach to the segmentation of blood vessels in retinal images obtained from different sources. Our approach results in greater segmentation accuracy than other approaches.

Transfer Learning Through Weighted Loss Function and Group Normalization for Vessel Segmentation from Retinal Images

Data is one of the most important keys for success when studying a simple or a complex phenomenon. With the use of deep learning exploding and its democratization, non-computer science experts may struggle to use highly complex deep learning architectures, even when straightforward models offer them suitable performances. In this article, we study the specific and common problem of data imbalance in real databases as most of the bad performance problems are due to the data itself. We cover two keys aspects. First, we propose ways to deal with the situation when a data set contains different levels of imbalance. Classical imbalanced learning strategies cannot be directly applied when using multi-attribute deep learning models, i.e., multi-task or multi-label architectures. Therefore, one of our contributions is a proposed adaptation to face each one of the problems derived from imbalance. Second, we demonstrate that with little to no imbalance, straightforward deep learning models work well. However, for non-experts, these models can be seen as black boxes, where all efforts are invested in pre-processing the data. To simplify the problem, we perform the classification task without features that are costly to extract, such as part localization which is widely used in the state of the art of attribute classification. We make use of three widely known attribute databases, CUB-200-2011 - CUB as our main use case due to its deeply imbalanced nature, along with two balanced databases: celebA and AwA2. All of them contain multi-attribute annotations. The results of very fine-grained attribute learning demonstrate that in the presence of imbalance, our proposed strategies make it possible to have competitive results against the state of the art while taking advantage of multi-attribute deep learning models. We also noticed an increase in performance while using a specialized loss function (Focal Loss). For CUB, we have competitive results, and for CelebA and AwA2 our strategies over-perform the state of the art.

Multi-Attribute Learning With Highly Imbalanced Data

Despite being very powerful in standard learning settings, deep learning models can be extremely brittle when deployed in scenarios different from those on which they were trained. Domain generalization methods investigate this problem and data augmentation strategies have shown to be helpful tools to increase data variability, supporting model robustness across domains. 
In our work we focus on style transfer data augmentation and we present how it can be implemented with a simple and inexpensive strategy to improve generalization. Moreover, we analyze the behavior of current state of the art domain generalization methods when integrated with this augmentation solution: our thorough experimental evaluation shows that their original effect almost always disappears with respect to the augmented baseline. This issue open new scenarios for domain generalization research, highlighting the need of novel methods properly able to take advantage of the introduced data variability.

Rethinking Domain Generalization Baselines

The research in reinforcement learning has achieved great success in strategic game playing. These successes are thanks to the incorporation of deep reinforcement learning (DRL) and Monte Carlo Tree Search (MCTS) to the agent trained under the self-play (SP) environment. By self-play, agents are provided with an incrementally more difficult curriculum which in turn facilitates learning. 
However, recent research suggests that agents trained via self-play may easily lead to getting stuck in local equilibria.
In this paper, we consider a population of agents each independently learns to play an alternating Markov game (AMG).
We propose a new training framework ---group practice--- for a population of decentralized RL agents.
By group practice (GP), agents are assigned into multiple learning groups during training, for every episode of games, an agent is randomly paired up and practices with another agent in the learning group.
The convergence result to the optimal value function and the Nash equilibrium are proved under the GP framework. Experimental study is conducted by applying GP to $Q$-learning algorithm and the deep $Q$-learning with Monte-Carlo tree search on the game of Connect Four and the game of Hex. 
We verify that GP is the more efficient training scheme than SP given the same amount of training.
We also show that the learning effectiveness can even be improved when applying local grouping to agents.

Attention as Activation

Downloads

Next from ICPR 2020

Heuristics for Evaluation of AI Generated Music

Similar lecture

Attention as Activation

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES