United States

Assuming the common aphorism that “all models are wrong, but some are useful”, the natural question to ask is, how do we know a model is useful, and in particular, how useful is it? Uncertainty Quantification (UQ) is the scientific framework typically employed in search of answers to these types of questions. UQ is a multi-disciplinary field, borrowing tools from probability, statistics and numerical analysis; it has been around for a long time and has been applied, in one form or another, in various physics and engineering contexts to identify, quantify, and reduce uncertainties associated with models, algorithms and experimental outcomes. 

With the advent of Deep Learning (DL), substantial progress has been made in research areas dealing with the modeling and understanding of complex systems that had been otherwise challenging to achieve with conventional methods. Despite DL’s successes in many of these areas, the deployment of robust and reliable DL systems in safety-critical, high-risk, real world applications remains limited, in part due to the inherent fragility of basic DL models, as well as their tendency to deliver over- or under-confident prediction estimates, i.e. lack of proper calibration. To overcome this, in recent years there has been an increased interest in understanding and quantifying uncertainty in DL network models, and a variety of approaches have been proposed. The goal of this tutorial is to provide a gentle introduction and overview of Uncertainty Quantification in Deep Learning, focusing on some of the main techniques, applications and challenges. In the first half, George will provide a brief overview of UQ in Deep Learning, and will discuss some intrinsic UQ methods for DL, which treat uncertainty organically within a DL model. In the second half Ben will discuss extrinsic/post-hoc UQ methods for DL, which allow for the estimation of uncertainty and the calibration of existing non-probabilistic DL models, using conventional methods (e.g. temperature scaling), as well as techniques based on novel metrics (e.g. the recently introduced angular margin).


NAML 2022

Uncertainty Quantification in Deep Learning

model calibration

bayesian methods

uncertainty quantification

deep learning

# Welcome everyone to ACL 2022!

The 60th Annual Meeting of the Association for Computational Linguistics is taking place May 22-27, 2022 as a hybrid event, in Dublin and online. We are happy to welcome all of you to this anniversary edition with an almost 50-50 in-person and virtual participation. 
The main conference program features oral presentations, in-person and virtual posters and demo sessions, a plenary session for our best paper presentations and awards, three amazing keynote events and two new initiatives of invited talks: Spotlight Talks for Young Rising Stars (STIRS) and The Next Big Idea Talks. Posters (including Findings of ACL 2022) and demos are grouped by areas for both the in-person and the virtual sessions. For the virtual component, the talks will be on Zoom and the posters and the demos will be in GatherTown. The Student Research Workshop will have an oral session and a poster session as part of Poster Session 1. The program also features eight Tutorials and 28 Workshops. 

 
We wish you a wonderful conference! 
[**The ACL 2022 Organizing Committee**](https://www.2022.aclweb.org/organisers)
 
[**Conference Handbook**](https://drive.google.com/file/d/1_BUCMfhMVrjG9E2e71aHdHeE28KSje0l/view?usp=sharing) 
[**Mini Handbook**](https://drive.google.com/file/d/1qlBKl0wzmlVF1oCeMQl3BahLd9nLP5Ce/view?usp=sharing) 
[**Posters and Demo guides**](https://drive.google.com/file/d/1UucMAoCNncIOaH1rMMDa0owuG9qgvJTG/view?usp=sharing)

ACL 2022

The Association for Computational Linguistics (ACL) is the premier international scientific and professional society for people working on computational problems involving human language, a field often referred to as either computational linguistics or natural language processing (NLP). 

tutorial

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
 <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826 height=526 frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Machine Learning (ML) systems are able to complete many tasks more efficiently and accurately than humans, but their competency often drops when operating in untrained conditions.  The opaqueness of how most ML agents arrive at their output reduces trust of human operators or teammates even for conditions where the ML agent’s average capability exceeds that of their human counterparts.  The MindfuL™ System has been developed on DARPA’s Competency Aware Machine Learning (CAML) program and aims to solve both of these issues.
The MindfuL™ system characterizes incoming data and environmental conditions that drive competency of the ML system through Hierarchical Dirichlet Processes.  This very compact representation focuses on the elements of the ML agent’s experience that leads to specific approaches in the ML model to completing the task (strategies) and performance of the ML agent.  Leveraging input data as well as past performance and strategies of the ML agent, MindfuL™ software is able to predict future performance and strategy implementation and provides users with the context necessary to understand if an ML agent is suited to its task given the current environment and why.
The MindfuL™ system performance exceeded the Year 2 DARPA Competency Aware Machine Learning program metrics.  When applied to a Convolutional Neural Network aiming to predict obstacle depth from 2-D imagery, the MindfuL™ system was able to correctly identify more than 95% of competency controlling conditions present in the environment, to characterize the behavior of the neural network correctly 94% of the time, and to predict performance of the ML agent relevant to a human for 99% of images.
Continued development on MindfuL™ software aims to provide more semantically meaningful conditions and specific suggestions for how best an operator should use an ML agent under the current or proposed operating conditions.

MindfuL™ Software: Competency Aware Machine Learning

A key challenge in modern machine learning is how to build accurate explainable classifiers. A significant issue is the tradeoff between using expert-generated, explainable features and deeply learned features generated by neural networks. Deep learning (DL) achieves impressive performance in many domains, but the resulting networks function as “black box” systems, without the capability to explain their conclusions. The lack of explainability impedes their safe application to some of the very domains that could most benefit from AI, including military, intelligence, security, medical, and legal applications where decisions must be justified and the consequences of failure are too great.
Case-based reasoning (CBR) solves problems by retrieving and adapting cases encoding the solutions to prior similar problems. The CBR process leverages analogical reasoning with knowledge about similarity and about how to adapt solutions to new circumstances. CBR can exploit the intelligibility of cases to provide explanations in a form compelling to humans. CBR often uses expert-generated features to index cases. This is both a strength for interpretability and a weakness for knowledge acquisition.
We are developing methods for applying deep learning to generate case features to augment expert features, to improve performance while retaining the explainability of expert features. The goal of this hybrid approach is for cases to provide a human-intelligible vehicle for capturing and explaining decisions, while network learning methods support the CBR process by generating the similarity knowledge to select the right cases.

Deep Learning to Enhance Similarity Assessment for Case-based Reasoning

The US Navy relies on SONAR for detection and classification of contacts to support navigation and other mission related tasks. The field of deep learning (DL) could be useful in this problem space given that it is the current state of the art approach for image and speech recognition. However, supervised deep learning is a data hungry process and many Navy problems have limited labeled data. A sufficiently data rich underwater acoustic classification problem is recognizing marine mammals from individual echolocation clicks. The objective of this project is to apply DL to this problem space and determine how much training data is required for a DL classifier to compete with an existing baseline approach. This collaborative effort between the Naval Undersea Warfare Center (NUWC) Division, Newport, RI (NUWCDIVNPT) and the Naval Surface Warfare Center Carderock Division (NSWCCD) demonstrated multiple successful DL methods utilizing different input types (i.e. time-series or spectrogram-like images). Models investigated include a one dimensional convolutional neural network (1D CNN) that ingests raw time-series data previously isolated by a power law detector, as well as a Temporal Convolutional Network (TCN) that can detect as well as classify time-series data. This work also includes detailed performance comparisons between the different DL methods in terms of classification accuracy, precision, recall, and heat maps that portray one-to-one visualizations of classifications results for 1,500 common (hold out) test cases organized by approximate signal to noise ratio (SNR) and recording sensor location.

deepBioNet: Biologic Acoustic Transient Classifier

We propose MagNet, a simple and interpretable time-domain neural network for magnetic anomaly detection. We describe the results of a supervised learning application in which we train MagNet to identify the presence of magnetic dipoles as they pass by a magnetic vector gradiometer. The detection statistics show robust performance as a function of signal-to-noise ratio and the ability to provide detections before the closest point of approach. In addition, we use the interpretable layers of MagNet as a diagnostic tool to provide new insights into sensor data and bulk dynamics of the dipole.

Time-Domain Neural Network Detects Passing Dipole Targets Registered by a Magnetic Vector Gradiometer

Professional and technology skills are in high demand; so are human factors like creativity. With today’s top talent already engaged in meaningful work, the race is on to find emerging talent. Resumes present an inherent barrier to finding the top performers of the future. We developed an Artificial Intelligence (AI) based system, CVkeys, to overcome the challenges associated with finding top talent. As a result, we can match people with jobs in milliseconds. This is a demonstrated application of machine learning that has far-reaching applications and implications across the Navy information enterprise.


Artificial Intelligence/Machine Learning Ontologies

Automatic Modulation Classification (AMC) is an important front-end step in a number of applications such as Signals Intelligence, spectrum interference monitoring, link adaptation, dynamic spectrum access, radio fault detection, cognitive radio and 4G/5G cellular standards. We present results of Quantization-Aware Training (QAT) and Pruning of a Deep Convolutional Neural Network (DCNN) for Hardware-Efficient RF modulation classification. An 8-bit quantized MobileNeXt DCNN consisting of sandglass blocks of Depthwise and Pointwise convolution layers was trained on the DeepSig RadioML 2018.01A dataset as part of the 2021 ITU AI/ML in 5G Lightning-Fast Modulation Classification challenge organized by the International Telecommunication Union. The training was done using the Xilinx Brevitas QAT library and the PyTorch pruning library. We introduce the notion of Inference Cost whose key determinants are the number of multiply accumulate (MAC) operations, the number of weights and the bit precision of the weights and activations as compared to a reference 8-bit quantized VGG10 DCNN. We will discuss the next steps for deploying the model with hardware specific optimizations on the Zynq UltraScale+ RFSoC using the Xilinx FINN (Fast Scalable Quantized Neural Network Inference) library. Work was supported by FY21 NISE 219 funding.

Quantization-Aware Training and Pruning for Hardware-Efficient Neural Network Modulation Classification

Understanding data prior to using machine learning algorithms can help make machine learning (ML) results less opaque and inform ML design decisions. Characterizing the geometry (topology) of data can help identify (i) missing data and (ii) the complexity of the decision boundary between different classes of data. This can lead to the more efficient deployment of ML tools. In this poster we present new results on our use of topological data analysis for analyzing machine learning tools and show how to scale these methods to manage larger data sets. We illustrate this approach with a large data set from the particle physics community and compare topological results with deep learning results.

Extracting Properties of Separators from Larger Scale Data using Topological Data Analysis Methods

Can reproduction alone in the context of survival produce intelligence in our machines? In this work, self-replication is explored as a mechanism for the emergence of intelligent behavior in modern learning environments. By focusing purely on survival, while undergoing natural selection, evolved organisms are shown to produce meaningful, complex, and intelligent behavior, demonstrating
creative solutions to challenging problems without any notion of reward or objectives. Atari and robotic learning environments are re-defined in terms of natural selection, and the behavior which emerged in self-replicating organisms during these experiments is described in detail.


Evolutionary Self-Replication as a Mechanism for Producing Artificial Intelligence

Over the years, there has been a rising application of artificial intelligence (AI) in computing devices, and studies have been done to develop a cost-effective and energy-efficient approach to machine learning. Advancements in computing and communication technologies have led to the creation of more advanced processors, neuromorphic computers, quantum computing technologies, as well as the increasing usage of 5G networks in the industry. Together, these not only increase the amount of processing that can be done but also the amount of data that can be transmitted at a given time.
The field of AI is becoming very broad, and several studies so far have been able to achieve very high prediction accuracy using convolutional neural networks (CNNs), which is a very popular class of deep neural networks. Neurons in CNNs constantly perform a lot of computation as data is fed into the network. The size of the network (the number of neurons) determines the amount of computation that must be performed. This uses up a lot of energy due to the high number of computations when dealing with large networks.
Spiking neural networks (SNNs) are event-based networks that make use of spikes in learning. Neurons in SNNs are designed to operate like actual biological neurons. The signals in an SNN are propagated in the form of spike trains which are a timed representation of input signals. Neurons within the network only forward propagate signals/action potentials when the neuron potential of a given neuron crosses a predefined threshold value. This implies that the number computation becomes mainly dependent on the amount of dynamic data rather than the size of the network.
Several studies have investigated different forms of collaborative learning such as federated learning, but not many have been done with Spiking Neural Networks, given that SNN learning algorithms have been a bit challenging to develop and haven’t been able to achieve very high accuracy like CNNs.
A study on spiking neural networks has investigated what could happen when weight parameters gotten by training a ReLU Multilayer Perceptron Network (ReLU MLP) model, are shared with a model gotten by training a less-performant Spiking Neural Network model of the same size. The results were an improved performance of the spiking neural network.
The purpose of this project is to investigate the improvement in the machine learning model that can be achieved by employing/simulating a swarm of client marine robots functioning on a distinct image dataset. This can be done by training different instances of a given algorithm on separate groups of datasets representing each client robot, and only updating the parameters across the group whenever any one of the clients have been able to achieve a higher prediction accuracy. This is inspired by the fact that hyperparameters learned during the training process can easily be shared with clients.
The idea behind our design is to use a more powerful computing unit on the backend to perform training while using a less powerful device on the client-side for inference. Each client in our design also has the additional responsibility of supplying data to the backend computing unit where training is done, with each client having a different repository for its dataset to ensure that the data remains distinct.


Initial work in this research has involved employing a time-based backpropagation algorithm to train our spiking neural network and fractional stochastic gradient descent for weight update. We trained our algorithm on the MNIST dataset. Results show a maximum prediction accuracy of 97.03%, on a single client.
This is ongoing research and, in the future, we intend to study the changes or improvements in prediction accuracy that can be gotten by running the training algorithm on distinct MNIST datasets and finally applying the algorithm to a real collection of marine datasets.


Non-critical Marine Application of Collaborative Learning with Spiking Neural Networks

Increasingly, operational systems within the defence space are leveraging AI/ML models. These models introduce unique vulnerabilities that cannot be addressed by traditional cybersecurity methods because their inputs during both training and deployment hang outside of traditional security firewalls. Consequently, AI/ML models are susceptible to a variety of adversarial attacks, both naturally occurring and those of malicious intent used to influence model behaviour or defeat AI/ML systems entirely. This poster provides a high-level overview of some of these vulnerabilities and outlines current work aimed at securing these AI/ML systems within an operational context.

Premium content

Uncertainty Quantification in Deep Learning

Downloads

Next from NAML 2022

MindfuL™ Software: Competency Aware Machine Learning

Similar lecture

When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES