Assuming the common aphorism that “all models are wrong, but some are useful”, the natural question to ask is, how do we know a model is useful, and in particular, how useful is it? Uncertainty Quantification (UQ) is the scientific framework typically employed in search of answers to these types of questions. UQ is a multi-disciplinary field, borrowing tools from probability, statistics and numerical analysis; it has been around for a long time and has been applied, in one form or another, in various physics and engineering contexts to identify, quantify, and reduce uncertainties associated with models, algorithms and experimental outcomes. 

With the advent of Deep Learning (DL), substantial progress has been made in research areas dealing with the modeling and understanding of complex systems that had been otherwise challenging to achieve with conventional methods. Despite DL’s successes in many of these areas, the deployment of robust and reliable DL systems in safety-critical, high-risk, real world applications remains limited, in part due to the inherent fragility of basic DL models, as well as their tendency to deliver over- or under-confident prediction estimates, i.e. lack of proper calibration. To overcome this, in recent years there has been an increased interest in understanding and quantifying uncertainty in DL network models, and a variety of approaches have been proposed. The goal of this tutorial is to provide a gentle introduction and overview of Uncertainty Quantification in Deep Learning, focusing on some of the main techniques, applications and challenges. In the first half, George will provide a brief overview of UQ in Deep Learning, and will discuss some intrinsic UQ methods for DL, which treat uncertainty organically within a DL model. In the second half Ben will discuss extrinsic/post-hoc UQ methods for DL, which allow for the estimation of uncertainty and the calibration of existing non-probabilistic DL models, using conventional methods (e.g. temperature scaling), as well as techniques based on novel metrics (e.g. the recently introduced angular margin).


Uncertainty Quantification in Deep Learning

Fully analog neuromorphic computing systems offer tremendous potential for the future of edge-AI applications, with the possibility of mimicking the functionality and efficiency of the human brain under considerable energy and spatial constraints. However, training such neuromorphic systems with conventional machine learning methods has proven rather challenging. In recent years, novel approaches relying on so called Energy Based Models provide an alternative paradigm for training and inference in analog neural networks. A major breakthrough in that direction was achieved with the introduction of the Equilibrium Propagation method in a paper by Benjamin Scellier and Yoshua Bengio published in 2017. Since then multiple extensions and generalization have been been proposed, including an Equilibrium Propagation algorithm for nonlinear resistive networks by Kendall, Bengio et al, and an associated extension for Spiking Neural Networks by Martin et al, both published in 2020. The goal of this tutorial is to provide a gentle introduction to Energy Based Models and the Equilibrium Propagation method, both as a general framework and in the context of training analog neural networks. The self-contained exposition will be based on a survey of recent papers and workshop presentations, as well as basic review of relevant background material. We will also discuss ideas for possible research directions and applications to realistic edge-AI problems.

Energy Based Models for Analog Neural Networks



**Title:** Uncertainty Quantification in Deep Learning: A Tutorial 

**Speaker:**  George Stantchev, Naval Research Laboratory

![](https://assets.underline.io/uploads/markdown_image/1/image/eb45529ee91364ead9cdce23ef1409e9.png)

**Bio:**  George Stantchev received his Ph.D. in Applied Mathematics and Scientific
Computation from the University of Maryland in 2003. After several years as
postdoctoral researcher at the University of Maryland’s Center for Scientific
Computation and Mathematical Modeling (CSCAMM), as well as a research scientist at
Leidos (formerly SAIC), he joined NRL as computer scientist. Dr. Stantchev has broad
experience in computational mathematics, high-performance computing, statistical
analysis, pattern recognition, and signal processing. He has been at the forefront of
interdisciplinary research on a variety of topics, including machine learning in the RF
signal domain, and decentralized computing for collaborative autonomous multi-agent
systems.

**Speaker:** Ben Miglioiri, Los Alamos National Lab
![](https://assets.underline.io/uploads/markdown_image/1/image/e13bcf76c62d4d3befefe7a015108882.png)

**Bio:**

Benjamin Migliori is an interdisciplinary physicist-slash-neuroscientist who focuses on
biologically inspired systems – that is, using our understanding of computational
neuroscience to help machines perceive the world, and using our understanding of
machines to help us understand the brain. He is currently the project lead for multiple
DoD-sponsored research efforts in the area of defense-related machine learning at Los
Alamos National Laboratory. Previously, he was the PI for Biologically Inspired
Autonomous Sensing at the Naval Information Warfare Center, and the technical
oversight representative for DARPA's Neural Engineering Systems Design program. He
has a Ph.D. in Biophysics from the University of California, San Diego.

**Abstract:**

Assuming the common aphorism that “all models are wrong, but some are useful”, the natural question to ask is, how do we know a model is useful, and in particular, how useful is it?  Uncertainty Quantification (UQ) is the scientific framework typically employed in search of answers to these types of questions. UQ is a multi-disciplinary field, borrowing tools from probability, statistics and numerical analysis; it has been around for a long time and has been applied, in one form or another, in various physics and engineering contexts to identify, quantify, and reduce uncertainties associated with models, algorithms and experimental outcomes. 

With the advent of Deep Learning (DL), substantial progress has been made in research areas dealing with the modeling and understanding of complex systems that had been otherwise challenging to achieve with conventional methods. Despite DL’s successes in many of these areas, the deployment of robust and reliable DL systems in safety-critical, high-risk, real world applications remains limited, in part due to the inherent fragility of basic DL models, as well as their tendency  to deliver over- or under-confident prediction estimates, i.e. lack of proper calibration. To overcome this, in recent years there has been an increased interest in understanding and quantifying uncertainty in DL network models, and a variety of approaches have been proposed. The goal of this tutorial is to provide a gentle introduction and overview of Uncertainty Quantification in Deep Learning, focusing on some of the main techniques, applications and challenges. In the first half, George will provide a brief overview of UQ in Deep Learning, and will discuss some intrinsic UQ methods for DL, which treat uncertainty organically within a DL model. In the second half Ben will discuss extrinsic/post-hoc UQ methods for DL, which allow for the estimation of uncertainty and the calibration of existing non-probabilistic DL models, using conventional methods (e.g. temperature scaling), as well as techniques based on novel metrics (e.g. the recently introduced angular margin).

               

   Uncertainty Quantification in Deep Learning: A Tutorial 

tutorial

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

NAML 2022

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



These live Q&A sessions will take place in our Gather.Town lounge. Once you click the gathertown button below, you will need to navigate to the areas located to the left in the lounge. Enter the designated area listed below each Q&A.

**Date & Time:** Wednesday, March 24th 8:15-9:00am PDT

**Location:** **AREA 1 **

**Title:** Using Quantum-Inspired Algorithms to Solve Optimization Problems

**Time:** 8:15am PDT

**Speaker:** [Monica DeZulueta](https://www.underline.io/speakers/77571-monica-dezulueta)

[![](https://assets.underline.io/uploads/markdown_image/1/image/c7e88505bb752c494a40c43a7f560168.png)](https://gather.town/app/5IAViMZXC9ZF4E4l/NAML)

**Location:** **AREA 2 **

**Title:** Energy Based Models for Analog Neural Networks

**Time:** 8:30am PDT

**Speaker:** [George Stantchev](https://www.underline.io/speakers/77547-george-stantchev)

[![](https://assets.underline.io/uploads/markdown_image/1/image/722aea2c86d9dba9fcf527b54d20e092.png)](https://gather.town/app/5IAViMZXC9ZF4E4l/NAML)

**Location:** **AREA 3 **

**Title:** Cloud AI/ML for Disaster Response

**Time:** 8:45am PDT

**Speaker:** [Hayes Magnuson](https://www.underline.io/speakers/70516-hayes-magnuson)

[![](https://assets.underline.io/uploads/markdown_image/1/image/e35db3b90d4905c814cf082dceae6913.png)](https://gather.town/app/5IAViMZXC9ZF4E4l/NAML)





Tutorial Q&A / Demo Session 1A

technical paper

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

George Stantchev

2

SHORT BIO

Presentations

Uncertainty Quantification in Deep Learning

Energy Based Models for Analog Neural Networks

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES