Italy

Explaining the behaviors of deep neural networks, usually considered as black boxes, is critical especially when they are now being adopted over diverse aspects of human life. Taking the advantages of interpretable machine learning (interpretable ML), this paper proposes a novel tool called Catastrophic Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual learning settings. We also introduce a new method called Critical Freezing based on the observations of our tool. Experiments on ResNet-50 articulate how catastrophic forgetting happens, particularly showing which components of this famous network are forgetting. Our new continual learning algorithm defeats various recent techniques by a significant margin, proving the capability of the investigation. Critical freezing not only attacks catastrophic forgetting but also exposes explainability.

ICPR 2020

Explaining How Deep Neural Networks Forget by Deep Visualization

ICPR 2020 is the premier international conference on Pattern Recognition, gathering leading researchers from around the world to present their latest achievements in Pattern Analysis and Recognition, applied Artificial Intelligence and Machine Learning. The topic of the year is " Putting Artificial intelligence to work on Patterns". 
For the first time, ICPR has employed a two-round review process similar to journal submissions, with new papers allowed to be submitted in either the first or second round and papers submitted in the first round and not accepted allowed to be revised and re-submitted for second round review. 

In this edition of the **ICPR 2020** Conference, there are 1407 Oral/Poster presentations organized in five Tracks: 
- Artificial Intelligence, Machine Learning for Pattern Analysis; 
- Biometrics, Human Analysis and Behavior Understanding; Computer Vision, Robotics and Intelligent Systems; 
- Document and Media Analysis; and 
- Image and Signal Processing. 

The **ICPR 2020** Main Conference program also includes 7 Keynotes from prominent world-renowned scientists, 8 Tutorials, 28 Demos, 8 Challenges, 2 Panels, as well as an Industry/Sponsor Exhibition. In addition, 41 Workshops on a range of important current and emerging research topics complete the ICPR 2020 program. 


technical paper

In this talk we present our work on a perplexity-based sparsity definition to derive and visualise layer-wise activation measures. These novel explainable AI strategies reveal a surprising relationship between activation sparsity and overfitting. Our methods reveal novel insights into the contradicting concepts of activation sparsity and network capacity by demonstrating that dense activations can enable discriminative feature learning while efficiently exploiting the capacity of deep models without suffering from overfitting, even when trained excessively.

The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks

Explanations for algorithmic decision systems can take different forms, they can target different types of users with different goals. One of the main challenges in this area is therefore to devise explanation methods that can accommodate this variety of situations. A first step to address this challenge is to allow explainees to express their needs in the most convenient way, depending on their level of expertise and motivation. In this paper, we present a solution to this problem based on a multi-layered approach allowing users to express their requests for explanations at different levels of abstraction. We illustrate the approach with the application of a proof-of-concept system called IBEX to two case studies.

A Multi-layered Approach for Tailored Black-box Explanations

Machine learning is currently successfully used for addressing several cybersecurity detection and classification tasks. Typically, such detectors are modeled through complex learning algorithms employing a wide variety of features. Although these settings allow achieving considerable performances, gaining insights on the learned knowledge turns out to be a hard task. To address this issue, research efforts on the interpretability of machine learning approaches to cybersecurity tasks is currently rising. In particular, relying on explanations could improve prevention and detection capabilities since they could help human experts to find out the distinctive features that truly characterize malware attacks. In this perspective, Android ransomware represents a serious threat. Leveraging state-of-the-art explanation techniques, we present a first approach that enables the identification of the most influential discriminative features for ransomware characterization. We propose strategies to adopt explanation techniques appropriately and describe ransomware families and their evolution over time. Reported results suggest that our proposal can help cyber threat intelligence teams in the early detection of new ransomware families, and could be applicable to other malware detection systems through the identification of their distinctive features.

Explanation-driven Characterization of Android Ransomware

This paper profiles the recent research work on eXplainable AI (XAI), at the Insight Centre for Data Analytics. This work concentrates on post-hoc ex-planation-by-example solutions to XAI as one approach to explaining black box deep-learning systems. Three different methods of post-hoc ex-planation are outlined for image and time-series datasets: that is, factual, counterfactual, and semi-factual methods). The future landscape for XAI solutions is discussed.

Post-Hoc Explanation Options for XAI in Deep Learning: The Insight Centre for Data Analytics Perspective

Neural networks are increasingly applied to high-stakes tasks, such as autonomous driving and medical applications. For these tasks, it is important to explain the contributions of data components to a model prediction. In recent times, mask-based methods have been proposed for visual explanations. They optimize masks that maximally affect the model output. In this work, we propose a novel mask-based saliency method for given black box classifiers. We jointly optimize positive and negative masks to achieve a faithful feature importance map. To optimize them effectively, we define a distance between them in terms of the selected activation maps; the corresponding kernels express the important features of an input image. Then, we impose the distance to the objective function of each mask as a regularizer. By forcing both masks to be dissimilar in terms of the influential features, they can focus on essential parts of an object while alleviating noises.

Jointly Optimize Positive and Negative Saliencies for Black Box Classifiers

In the astrophysics domain, the detection and description of gamma rays is a research direction for our understanding of the universe. Gamma-ray reconstruction from Cherenkov telescope data is multi-task by nature. The image recorded in the Cherenkov camera pixels relates to the type, energy, incoming direction and distance of a particle from a telescope observation. We propose γ-PhysNet, a physically inspired multi-task deep neural network for gamma/proton particle classification, and gamma energy and direction reconstruction. As ground truth does not exist for real data, γ-PhysNet is trained and evaluated on large-scale
Monte Carlo simulations. Robustness is then crucial for the transfer of the performance to real data. Relying on a visual explanation method, we evaluate the influence of attention on the variability due to weight initialization, and how it helps improve the robustness of the model. All the experiments are conducted in the context of single telescope analysis for the Cherenkov Telescope Array simulated data analysis.

Deep Learning for Astrophysics, Understanding the Impact of Attention on Variability Induced by Parameter Initialization

Machine Learning (ML) is becoming an increasingly critical technol-ogy in many areas such as health, business but also in everyday applications of significant societal importance. However, the lack of explainability or ability of ML systems to offer explanation on how they work, which refers to the model (related to the whole data) and sample explainability (related to specific sam-ples) poses significant challenges in their adoption, verification, and in ensur-ing the trust among users and general public. We present novel integrated Ran-dom Forest Model and Sample Explainer – RFEX. RFEX is specifically de-signed for important class of users who are non-ML experts but are often the domain experts and key decision makers. RFEX provides easy to analyze one-page Model and Sample explainability summaries in tabular format with wealth of explainability information including classification confidence, tradeoff be-tween accuracy and features used, as well as ability to identify potential outlier samples and features. We demonstrate RFEX on two case studies: mortality prediction for COVID-19 patients from the data obtained from Huazhong Uni-versity of Science and Technology, Wuhan, China, and classification of cell type clusters for human nervous system based on the data from J. Craig Venter Institute. We show that RFEX offers simple yet powerful means of explaining RF classification at model, sample and feature levels, as well as providing guid-ance for testing and developing explainable and cost-effective operational pre-diction models.

Random Forest Model and Sample Explainer for non-experts in Machine Learning – Two Case Studies

Emotions are very important in dealing with human decisions, interactions, and cognitive processes. Art is an imaginative human creation that should be appreciated, thought-provoking, and elicits an emotional response. The automatic recognition of emotions triggered by art is of considerable importance. It can be used to categorize artworks according to the emotions they evoke, recommend paintings that accentuate or balance a particular mood, and search for paintings of a particular style or genre that represent custom content in a custom state of impact. In this paper, we propose an attention-based multi-modal approach to emotion recognition that aims to use information from both the painting and title channels to achieve more accurate emotion recognition. Experimental results on the WikiArt emotion dataset showed the efficiency of the model we proposed and the usefulness of image and text modalities in emotion recognition.

Attention-based Multi-modal Emotion Recognition From Art

Document images dewarping is still a challenge especially when documents are captured with one camera in an uncontrolled environment. In this paper we propose a generic approach based on vanishing points (VP) to reconstruct the 3D shape of document pages. Unlike previous methods we do not need to segment the text included in the documents. Therefore, our approach is less sensitive to pre-processing and segmentation errors. The computation of the VPs is robust and relies on the a-contrario framework, which has only one parameter whose setting is based on probabilistic reasoning instead of experimental tuning. Thus, our method can be applied to any kind of document including text and non-text blocks and extended to other kind of images. Experimental results show that the proposed method is robust to a variety of distortions.

Generic 10 - 752

In previous works, a mobile application was developed using an unmodified commercial smartphone to recognize whole-body exercises. The working principle was based on the ultrasound Doppler sensing with the device built-in hardware. Applying such a lab environment trained model on realistic application variations causes a significant drop in performance, and thus decimate its applicability. The reason of the reduced performance can be manifold. It could be induced by the user, environment, and device variations in realistic scenarios. Such scenarios are often more complex and diverse, which can be challenging to anticipate in the initial training data. To study and overcome this issue, this paper presents a database with controlled and uncontrolled subsets of fitness exercises. We propose two concepts to utilize small adaption data to successfully improve model generalization in an uncontrolled environment, increasing the recognition accuracy by two to six folds compared to the baseline for different users.

Premium content

Explaining How Deep Neural Networks Forget by Deep Visualization

Downloads

Next from ICPR 2020

The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks

Similar lecture

Analyzing Zero-shot Cross-lingual Transfer in Supervised NLP Tasks

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES