
**Introduction:** Dr. Katie Rainey, Co-Chair, NAML

**Speaker:** RADM Douglas Small, Commander, NAVWAR
![](https://assets.underline.io/uploads/markdown_image/1/image/71a3103f0669cb735104f60a540f714a.jpg)

**Bio:**

Rear Adm. Doug Small is a native of Birchwood, Wisconsin. He is a 1988 graduate of Marquette University in Milwaukee, Wisconsin, where he received a Bachelor of Science in Physics. He also holds a doctorate degree in physics from the Naval Postgraduate School in Monterey, California.

Small’s operational tours included mechanical division officer and main propulsion assistant on USS Camden (AOE 2), in which he deployed in support of Operation Desert Storm, as well as combat systems officer (plankowner) on USS Iwo Jima (LHD 7). During Operation Iraqi Freedom he served as the first technical director of Joint Crew Composite Squadron (JCCS) 1, assembled by the Navy to assist with defeating then-prevalent radio-controlled improvised explosive devices (IED).

An engineering duty officer since 1997, Small had a number of tours in acquisition, starting at the Naval Surface Warfare Center, Dahlgren Division and then the Missile Defense Agency. Moving to the Program Executive Office for Integrated Warfare Systems (PEO IWS), Small was the electronic warfare assistant and then the major program manager for above water sensors (PEO IWS 2), overseeing the development of improvements for the Navy in electronic warfare and radar. Following this tour he served as the executive assistant to the Assistant Secretary of the Navy for Research, Development and Acquisition.

Small’s first flag assignment, in 2016, he was the program executive officer for Integrated Warfare Systems, where he led the team responsible for development, delivery, and sustainment of all surface navy combat and weapon systems. He assumed command of the Naval Information Warfare Systems Command in August 2020, leading a global workforce of 11,000 civilian and military professionals who design, develop, install, and support Navy’s networking, communications, information, and cyber capabilities and systems.

Small’s personal awards include the Distinguished Service Medal, Legion of Merit, and Bronze Star; unit awards include the Army Meritorious Unit Commendation and various campaign and service awards.

Keynote 1: RADM Douglas Small, Commander, NAVWAR


**Introduction:** Nicole Isoda, Co-Chair, NAML

**Speaker:** Lt. Gen. Matthew G. Glavy, Deputy Commandant for Information, United States Marine Corps

![](https://assets.underline.io/uploads/markdown_image/1/image/d514fcd2dbeab1a6035c868c0005ebe3.png)

**Bio:** 

Lieutenant General Glavy was commissioned in May 1986 through the United States Naval Academy. Upon receiving his wings in September of 1988, he was selected to fly the CH-46 Sea Knight. 

Assignments in the Operating Forces include: Weapons and Tactics Instructor and Embark Officer, HMM-261 (REIN); Aircraft Maintenance Officer, and Executive Officer, HMM-265 (REIN); Current Operations Officer, G-3, 1st MAW; Commanding Officer, HMM-265 (REIN); Commanding General, 2d Marine Aircraft Wing, and Commanding General, U.S. Marine Corps Forces Cyberspace Command. 

Assignments in the Supporting Establishment include: Platoon Commander, Officer Candidate School; Operational Test Director, White House Liaison Officer, Presidential Command Pilot and Commanding Officer, HMX-1.
Headquarters and Staff Assignments: Plans, Policy and Operation, HQMC; Information Operations, Joint Staff J3; HQMC Aviation, Expeditionary Enablers Branch; Assistant Deputy Commandant for Aviation, Marine Corps Staff; Deputy Director of Current Operations, United States Cyber Command.

Military Education: The Amphibious Warfare School, Marine Corps Command and Staff College and Marine Corps War College. Master’s degrees in both Military Studies and Strategic Studies and a Bachelor of Science in Systems Engineering from the United States Naval Academy


Keynote 2: Lt. Gen. Matthew G. Glavy, Deputy Commandant for Information, USMC


**Introduction:** Dr. Shibin Parameswaran, Co-Chair, NAML

**Speaker:** Dr. Gurpartap "GP" Sandhoo, D.SC., Deputy Director Intelligence Advanced Research Projects Activity (IARPA)
![](https://assets.underline.io/uploads/markdown_image/1/image/b9bd7c00acf9acce5bc57fe40c5fcf51.png)

**Bio:**

Dr. Sandhoo became deputy director of the Intelligence Advanced Research Project Activity in December 2021. Dr. Sandhoo previously occupied the Distinguished Visiting Professor Robert A. Heinlein Endowed Chair in Astronautics at the U.S. Naval Academy since 2020. 

At the U.S. Naval Research Laboratory, he headed the Spacecraft Engineering Division, and was also the acting director of the Naval Center of Space Technology. He provided executive direction and technical leadership in conducting research of space systems with advanced technologies.

He was also a flight controller at NASA’s Johnson Space Center from 2000-2005, and was a research & development engineer at Johns Hopkins University’s Applied Physics Laboratory from 1997-2000, as well as in industry. Since 1986, he has served in uniform in the U.S. Marine Corps and the U.S. Navy. Dr. Sandhoo is a Captain in the U.S. Navy Reserve as an Engineering Duty Officer. 

He holds a Bachelor’s degree in mechanical engineering from the University of Maryland, a Master’s degree in electrical engineering from Johns Hopkins University, Master’s from the U.S. Naval War College, a Master’s and a Doctorate in Aeronautics and Astronautics from George Washington University, and is a MIT Seminar XXI fellow.

Keynote 3:  Dr. Gurpartap "GP" Sandhoo, D.SC., IARPA Deputy Director


**Title:** A contemporary approach to oceanic acoustic propagation for ML dataset generation

**Speaker:**  Milo Prisbrey, Los Alamos National Laboratory

**Bio:** Milo Prisbrey is a Postdoctoral Research Associate at Los Alamos National Laboratory (LANL). Milo’s research focuses primarily on acoustic signal propagation. Currently, Milo is investigating machine learning applications involving oceanic acoustic signal propagation.

**Speaker:** Eric Michalak, Los Alamos National Laboratory

**Bio:** Eric Michalak is a research scientist in the Advanced Research in Cyber Systems group at Los Alamos National Laboratory (LANL). Eric’s research is at the intersection of machine learning and cyber security exploring topics such as deepfake generation and detection and adversarial artificial intelligence. Currently, Eric is investigating robust mitigations for deepfakes in multiple domains including radio frequency, acoustic wave propagation, and three-channel and multispectral imagery and video.
               
        


A contemporary approach to oceanic acoustic propagation for ML dataset generation


**Title:** Juneberry, a machine learning experiment framework tutorial

**Speaker:** Andrew Mellinger, Carnegie Mellon University Software Engineering
![](https://assets.underline.io/uploads/markdown_image/1/image/d16f8ab1cb6c87831d582b3187f4c46b.png)

**Bio:**  Andrew Mellinger is a practicing software architect, principal software engineer and developer at the Software Engineering Institute's AI Division.  Mellinger has a passion for transitioning and operationalizing software and hardware research, with experiences across all phases of the software development life cycle in many different software domains. Mellinger enjoys the challenge of balancing between people, research, design, development and process to achieve mission success and provide value to all the involved stakeholders. In addition he enjoys mentoring and teaching students via the Master of Software Engineering programs at CMU. Mellinger has his undergraduate degree in physics from the College of Wooster, and his Master of Software Engineering from CMU.

**Notes:** 

## IMPORTANT:

Before the tutorial the users will need to have docker installed on their platform (note, we do NOT support Apple Silicon) and download a specific docker image. The directions can be found here:

https://github.com/cmu-sei/juneberry/blob/main/docs/vignettes/vignette1/Replicating_a_Classic_Machine_Learning_Result_with_Juneberry.md#working-with-juneberry-docker-containers

For more information about Juneberry, see the following links:

https://github.com/cmu-sei/juneberry

https://github.com/cmu-sei/juneberry/blob/main/juneberry/documentation/vignettes/Replicating_a_Classic_Machine_Learning_Result_with_Juneberry.md
							 
        
**Abstract:**

In this session, attendees will participate in a live tutorial of Juneberry, an experimental machine learning framework designed to manage the training and evaluation of machine learning models. Each participant will be able to download a fully configured Docker container and follow along with a live exercise. The exercise is based on the tutorial “Replicating a Classic Machine Learning Result with Juneberry”, which guides users through the steps required to replicate a ResNet from the He et al. 2015 paper.

Juneberry aims to reduce the costs associated with having to deal with multiple sets of machine learning models. The framework facilitates experimentation and model comparison through a JSON-based config file system, which manages input data files, models, and the underlying calls of machine learning platforms such as PyTorch, TensorFlow, Detectron2, and MMDetection. By relying on this config pipeline, a machine learning researcher can focus more on the design of their model, while spending less time implementing and iterating upon their design.

Juneberry also offers an “experiment outline” feature, which allows users to vary aspects of their base model and compare how the model’s performance reacts to those changes. Experiments also allow users to compare the performance of multiple models, enabling researchers to choose the model that is the best match for their design goals.
        


Juneberry, a machine learning experiment framework tutorial


**Title:**  Strategic and Technical Landscape in 2022

**Speaker:** George Galdorisi, Naval Information Warfare Center Pacific
![](https://assets.underline.io/uploads/markdown_image/1/image/25539c5baae9b8f4e665518449e2cf37.png)

**Bio:** 

George Galdorisi is a career naval aviator whose thirty years of active duty service included four command tours and five years as a carrier strike group chief of staff. He began his writing career in 1978 with an article in the U.S. Navy’s professional magazine, U.S. Naval Institute Proceedings.

In addition to his Rick Holden thrillers published by Braveship Books, he has written thirteen other books distributed by mainstream publishers, including several bestselling novels in the rebooted Tom Clancy’s Op-Center series, including Out of the Ashes, Into the Fire, Scorched Earth, and Dark Zone.  With his longtime collaborator, Dick Couch, he coauthored the New York Times best seller, Tom Clancy Presents: Act of Valor the novelization of the Bandito Brothers/Relativity Media film.  He is also the author of The Kissing Sailor, which proved the identity of the two principals in Alfred Eisenstaedt’s famous photograph; as well as over three-hundred articles in professional journals and other media.

George has received a number of national and international writing awards, including: the Navy League of the United States Alfred Thayer Mahan Award for Literary Achievement, the Surface Navy Association Literary Award, the Navy League of Australia’s Annual Essay Competition, the Naval Helicopter Historical Association Mark Starr Pioneer Literary Award, and the Military Writers Society of America Silver Medal Award, among others. George speaks frequently at writing classes and seminars including: San Diego State University Writers Conference, La Jolla Writers Conference, San Diego Writer’s Ink, Southern California Writers Association, Coronado Writer’s Workshop, and other venues.

George is the Director of Strategic Assessments and Technical Futures at the Navy’s Command and Control Center of Excellence in San Diego, California.  He and his wife Becky live in Coronado, California. Other than writing thrillers, he likes nothing more than connecting with readers. You can follow him on Facebook and Twitter, and learn more about his books, blogs and other writing on his website: http://georgegaldorisi.com/ – especially his “Writing Tips,” – which offer useful advice for all writers from established authors to future best-selling writers.

**Click [here](https://docs.google.com/document/d/1ewpNLgbNvv02Z4krP062e64VM-ZQwnnp/edit?usp=sharing&ouid=103659049538309722979&rtpof=true&sd=true) to access handout.**
        
        

   Strategic and Technical Landscape in 2022


**Title:** Level of Rigor - How to develop quality AI products

**Speaker:** Bruce Nagy, Naval Air Warfarce Center Weapons Division, China Lake
![](https://assets.underline.io/uploads/markdown_image/1/image/23804bbe23ffd472c00bb8c48c456ded.jpeg)

**Bio:** 

Bruce Nagy is a Senior Scientist and Systems Engineer at the Naval Air Warfare Center, Weapons Division at China Lake. His research focuses on advanced game theory techniques, artificial intelligence and machine learning applications as applied to naval weapon systems and tactical decision aids. Nagy received an BS in Biology and BA in Mathematics from the Citadel. He went on to receive an BS and MS in Electrical Engineering from the Naval Postgraduate School. He continued his neural network interest with post graduate work in modeling brain stem activities to muscle fibers at UCLA, in cooperation with NIH. Nagy is a former Engineering Duty Officer.
        
**Abstract:**

Artificial Intelligence (AI)/Machine Learning (ML) integration is currently in the form of upgrades to existing programs or new program acquisitions being implemented throughout the Department of Defense (DoD). How do we know these AI/ML-enabled systems will perform as intended? To make this determination, AI/ML product development/acquisition needs a unique evaluation process compared to other traditional software development/acquisition projects. In response, the Naval Ordnance Safety and Security Activity (NOSSA) funded the following research to investigate unique AI/ML policies, guidelines, tools and techniques to assess safety concerns within identified critical functions. From that effort, fourteen key Levels of Rigor (LOR) were developed, by means of: best practice discussions, definitions, measurements, justification documentation, and hazard analysis formats specific to AI/ML systems. These fourteen LORs also provide clarity as to why AI/ML software development requires special consideration by the acquisition community. Moreover, this research has the potential of affecting the acquisition community with regard to: how they define requirements, create architecture, produce AI/ML algorithm designs, develop AI/ML algorithm code and perform test and evaluation.  The need for the fourteen LOR became evident during the development of an “acquisition sandbox” that researched a route planner deploying AI/ML autonomous systems, as well as having those systems deliver packages with a focus on assessing safety in critical function behavior. The sandbox was designed using an DoD Architecture Framework (DoDAF) and Unified Modeling Language (UML) diagrams that incorporate a variety of AI/ML techniques. When faced with this level of complexity and/or uncertainty, the fourteen LORs represent a cohesive set of recommendations/questions that provide focus in regards to current naval AI/ML acquisition issues. These guidelines also provide organizations involved with safety, such as NOSSA and Airworthiness, as well as Acquisition Professionals, including Program Managers and System Engineers, a step-by-step “how-to” evaluation approach that ensures the creation of quality AI-embedded products.
               
        


Level of Rigor - How to develop quality AI products



**Title:** Uncertainty Quantification in Deep Learning: A Tutorial 

**Speaker:**  George Stantchev, Naval Research Laboratory

![](https://assets.underline.io/uploads/markdown_image/1/image/eb45529ee91364ead9cdce23ef1409e9.png)

**Bio:**  George Stantchev received his Ph.D. in Applied Mathematics and Scientific
Computation from the University of Maryland in 2003. After several years as
postdoctoral researcher at the University of Maryland’s Center for Scientific
Computation and Mathematical Modeling (CSCAMM), as well as a research scientist at
Leidos (formerly SAIC), he joined NRL as computer scientist. Dr. Stantchev has broad
experience in computational mathematics, high-performance computing, statistical
analysis, pattern recognition, and signal processing. He has been at the forefront of
interdisciplinary research on a variety of topics, including machine learning in the RF
signal domain, and decentralized computing for collaborative autonomous multi-agent
systems.

**Speaker:** Ben Miglioiri, Los Alamos National Lab
![](https://assets.underline.io/uploads/markdown_image/1/image/e13bcf76c62d4d3befefe7a015108882.png)

**Bio:**

Benjamin Migliori is an interdisciplinary physicist-slash-neuroscientist who focuses on
biologically inspired systems – that is, using our understanding of computational
neuroscience to help machines perceive the world, and using our understanding of
machines to help us understand the brain. He is currently the project lead for multiple
DoD-sponsored research efforts in the area of defense-related machine learning at Los
Alamos National Laboratory. Previously, he was the PI for Biologically Inspired
Autonomous Sensing at the Naval Information Warfare Center, and the technical
oversight representative for DARPA's Neural Engineering Systems Design program. He
has a Ph.D. in Biophysics from the University of California, San Diego.

**Abstract:**

Assuming the common aphorism that “all models are wrong, but some are useful”, the natural question to ask is, how do we know a model is useful, and in particular, how useful is it?  Uncertainty Quantification (UQ) is the scientific framework typically employed in search of answers to these types of questions. UQ is a multi-disciplinary field, borrowing tools from probability, statistics and numerical analysis; it has been around for a long time and has been applied, in one form or another, in various physics and engineering contexts to identify, quantify, and reduce uncertainties associated with models, algorithms and experimental outcomes. 

With the advent of Deep Learning (DL), substantial progress has been made in research areas dealing with the modeling and understanding of complex systems that had been otherwise challenging to achieve with conventional methods. Despite DL’s successes in many of these areas, the deployment of robust and reliable DL systems in safety-critical, high-risk, real world applications remains limited, in part due to the inherent fragility of basic DL models, as well as their tendency  to deliver over- or under-confident prediction estimates, i.e. lack of proper calibration. To overcome this, in recent years there has been an increased interest in understanding and quantifying uncertainty in DL network models, and a variety of approaches have been proposed. The goal of this tutorial is to provide a gentle introduction and overview of Uncertainty Quantification in Deep Learning, focusing on some of the main techniques, applications and challenges. In the first half, George will provide a brief overview of UQ in Deep Learning, and will discuss some intrinsic UQ methods for DL, which treat uncertainty organically within a DL model. In the second half Ben will discuss extrinsic/post-hoc UQ methods for DL, which allow for the estimation of uncertainty and the calibration of existing non-probabilistic DL models, using conventional methods (e.g. temperature scaling), as well as techniques based on novel metrics (e.g. the recently introduced angular margin).

               

   Uncertainty Quantification in Deep Learning: A Tutorial 


**Title:** Challenges and Opportunities in the Navy AI Space: An Operational Perspective from the AI Task Forces

**Panel Description:**

The Navy AI Task Force Panel includes participants from the AI Task Forces based at each Navy Type Command (TYCOM). SUBFOR, SPECWAR, and EOD were unable to attend. The panel will discuss the unique operational and technical challenges, and opportunities, of developing and fielding AI algorithms in their respective communities. The task forces are unique in that they sit at the intersection of policy, aquisions, requirements building, and technical development.

**Moderator:** Artem Sherbinin, U.S. Navy
![](https://assets.underline.io/uploads/markdown_image/1/image/1274b1b14b9d87c1047209cb4008ba05.png)

**Bio:**

LTJG Artem Sherbinin is a Surface Warfare Officer. He previously established TF Hopper, the Surface Navy's AI Accelerator. His first AI role in DoD was supporting Undersecretary for Defense (Intelligence) Project Maven. He is passionate about bringing the right governance and infrastructure to operationalize AI applications in the Department of the Navy.

**Panelists:**

LT Brandon Strain (TF Hopper, COMNAVSURFOR)<br>
LCDR Jesse Nerius (TF Elyson, COMNAVAIRFOR)<br> 
Dr. Andrew Miller (TF Station Hypo, ONI)<br>
LT Jay Faylo (TF 59, C5F Unmanned and AI)<br>
Maj. Jack Long, USMC<br>



Keynote Panel: Challenges and Opportunities in the Navy AI Space: An Operational Perspective from the AI Task Forces


**Title:** Algorithms

**Moderator:**  Mr. Keith Sullivan, Naval Research Laboratory



        
        

  Algorithms -  Q&A

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

NAML 2022

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



Do you have any other suggestions or comments?

Your vote has been successfully recorded.