United States

As the digitization of modern life continues full steam ahead, criminals
and state-sponsored actors continue to target government systems. The impact of these attacks on the Navy range from the exposure of highly sensitive data to the disruption of services and the exploitation of back doors for future attacks. Modern cybersecurity teams have to defend on-premises and multi-cloud footprints. Effective threat detection and response are challenged by the cost and complexity of siloed tools that neither scale nor provide capabilities for advanced threat detection. These advanced threats are causing federal organizations and agencies to reevaluate their security posture and architectures being used to enable the
analysis of petabytes of data in real time and apply advanced analytics to reduce their cyber risk and operational costs.

NAML 2023

Databricks - Defending Hybrid Cloud Environments with Advanced Threat Detection through Machine Learning

data lakehouse

cyber

machine learning

The purpose of the IC3K is to bring together researchers, engineers and practitioners on the areas of Knowledge Discovery, Knowledge Engineering and Knowledge Management.

IC3K 2021

poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

Please log in to explore this event. You need to log in with the email address you registered with. Access credentials have been sent to your email. 

Please be sure to check your spam and other email folders if you do not see an email confirmation right away.

Please log in to explore this event.

Registration for this workshop has now closed. We appreciate your interest and hope to welcome you at a future event. If you’d like to hear about similar upcoming events, please check our website or join our mailing list.

Registration for this workshop has now closed.

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Automatic target recognition (ATR) is a critical task in a wide range of naval applications such as surface vessel protection and port security via radar, sonar, and optical sensor data processing. Data-driven methods, such as deep learning (DL), have been successfully applied to a variety of ATR tasks. Despite their strong performance, current state-of-the-art DL ATR applications are reliant upon aggregating data from several devices to a central server for training. This process can be computationally costly and presents a potential security risk if the data itself is sensitive. Moreover, current state-of-the-art DL ATR models also produce point estimates at test time with no associated measure of uncertainty. Uncertainty quantification (UQ) on model predictions is critical for accurately characterizing and assessing the reliability of model outputs. In this research, under the Office of Naval Research (ONR) Threat and Situational Understanding with Networked-Online Machine Intelligence (TSUNOMI) program, we develop a novel ATR framework, which is capable of both producing UQ measurements and eliminating the need for data centralization. This novel approach begins by considering an ensemble of ATR models that are jointly utilized to produce model predictions and UQ estimates. The prediction given by the ensemble is characterized by the distribution of outputs from the individual models rather than a point estimate produced from a single DL ATR model. Characteristics of the ensemble, such as its prediction interval and coverage probability are used to express model uncertainty. Using Federated Learning, the ensemble is trained in a distributed manner, where each vessel with contributing data trains a local ensemble using its own computational resources. Local model parameters are periodically transmitted to a centralized location for aggregation with the other ensembles. These parameters are generally much smaller-sized communications payloads than the raw data itself by several orders of magnitude and are used to perform the same shared learning task as centralized DL. We evaluate the efficacy of this ATR framework on a prototype task and share preliminary results.

An Uncertainty Quantification-Based Automatic Target Recognition Framework Over Distributed Surface Vessels

With the proliferation and rapidly improving endurance of next-generation unmanned undersea systems, long-term deployment of human-on-the-loop subsea and seabed resident fleets are fast-approaching. More specifically, wave-generated power and other renewable power sources, autonomous docking capabilities, and increased edge processing capabilities enable subsea residence of both static and mobile underwater systems. In this research, we evaluate human-on-the-loop adaptive mission planning algorithms for managing a fleet of unmanned understeer systems in simulation. These systems could provide persistent monitoring, maintenance, and repair of critical undersea infrastructure vulnerable to grey zone operations; such a system could estimate and track infrastructure health, including detection and sensor data acquisition of significant changes. Moreover, subsea resident systems could perform continuous mine survey of a particular (even mobile) area of responsibility and conduct environmental surveys over extended periods of deployment. It is often impractical or impossible for these systems to maintain consistent contact with human operators, necessitating novel approaches to mission planning that bridges the gap between constant human oversight and fully autonomous operation of the platforms. 
These persistent, autonomous systems are expected to adapt to increasingly complex environments and tasking, collaborate with other autonomous systems, and to respond appropriately to the preferences of human operators without requiring unnecessary oversight. The mission planning algorithms used by these systems must decompose high level objectives into discrete tasks that can be described in terms of their cost to complete, and their efficacy can be evaluated by the computational time required to create a solution, as well as the efficiency of the solution itself. To study these algorithms, a multi-agent problem is posed in simulation, wherein a simulated group of systems (agents) complete a number of (potentially mobile) tasks in the shortest time possible without colliding with other agents, or with dynamic obstacles drifting through the environment. By implementing a multi-stage approach to the problem decomposition, this work proposes an algorithm that outperforms state-of-the-art algorithms, such as the Consensus-Based Bundle Algorithm, with respect to both the computational time to create the plan and the efficiency of the plan itself. 


Human On the Loop Adaptive Mission Planning 

The need for abundant, diverse, and relevant training data continues to increase as deep machine learning based computer vision models grow in complexity. We have seen that tasks such as image classification and object detection benefit from large and varied datasets. Unfortunately, the cost of obtaining accurate labeled data at scale is prohibitive especially for more complex tasks such as object detection which require both categorical labels and accurate bounding box annotations. Active learning (AL) seeks to alleviate the need for brute-force data annotation by analyzing and automatically sampling a small but informative subset of data for labeling. With an effective sampling strategy, the trained model can achieve comparable performance to fully supervised methods at a fraction of the labeling cost. Sampling strategies generally fall into three categories: uncertainty-based, diversity-based, and hybrids of the two. For satellite imagery applications of computer vision, AL is an attractive potential solution: vast volumes of unlabeled satellite imagery exist, but annotation is expensive and tedious. For object detection in synthetic aperture radar (SAR) imagery, labeling is particularly challenging. Labeling SAR images is more costly than electro-optical (EO) images because the average person lacks the domain expertise to interpret and label SAR imagery. Because of this, crowdsourcing data annotation is not a viable option. Thus, we must intelligently allocate scarce labeling resources to the data most likely to improve the performance and robustness of the model being developed. The goal of AL is to automate and optimize this dataset curation process. We propose to evaluate active learning methods applied to maritime object detection in SAR imagery. Recent work has shown the efficacy of active learning for common EO image datasets like PASCAL VOC and COCO, but how do these methods perform in SAR imagery with unique image properties and much smaller objects? We consider two active learning methods: CALD selects images based on model uncertainty cues, and VAAL selects diverse images from the full dataset independent of the task or model being trained. With uncertainty-based methods the uncertainty cues are gathered from the model under training, so there is an inherent coupling between the model and the selected data. When a labeled dataset acquired for one model (the acquisition model) is used to train another model of a different architecture (a successor model), what are the model performance impacts? Given the costs of labeling datasets, it is desirable for the labeled set to transfer well across a variety of model architectures, since datasets are likely to be reused throughout model development. Our evaluation comprises two parts:
• We measure AL algorithm performance across EO and SAR image datasets and three deep object detection models. 
• We examine transferability of AL-acquired data for training different successor models.

Evaluating Active Learning Methods for Synthetic Aperture Radar Maritime Object Detection

Model cards organize and portray information
about a model’s training data, hyper parameters, and behavior,
such as predictive accuracy and bias for machine learning (ML)
algorithms or models. Consumers use these cards to determine
whether a model is well suited for a particular use case.
However, the current design of model cards does not include
resource utilization metrics, which are important when models
are being optimized for specific hardware. The main objective
of this study was to determine which set of hardware-centric
performance metrics provided the most value to users when
comparing among different algorithms for Internet of Things
(IoT) edge devices. The Department of Defense (DoD) can utilize
these metrics to select the best model for key mission tasks
within the Command and Control (C2) missions that require
AI/ML deployment on edge devices. Our study focused on
finding correlations between resource availability and machine
learning models’ computational footprint using three key metrics:
energy consumption, execution time, and memory utilization.
The data was gathered from simulated environments hosted
by Docker containers (i.e., lightweight, stand-alone, executable
package of software that includes everything needed to run an
application) with various memory sizes. Two different series of
experiments were performed. The first series of experiments
simulated environments with 256 MB to 8GB (steps of the
powers of 2) of available memory. For each memory size, we
deployed 50 different containers, calculated the metrics from
each container, and then recorded the average result of each
metric. Two ML models from Microsoft’s EdgeML library that
are designed to run more efficiently on edge devices were used:
Bonsai, a decision tree based model, and ProtoNN, a KNN based
model. Using Spearman’s correlation, we found that none of
the metrics showed promising correlations for the Bonsai model,
while the ProtoNN model demonstrated a correlation of 0.5 for its
computational time. In order to investigate the lack of adequate
results, we ran a second series of experiments in a similar fashion
but with smaller memory sizes from 250MB to 950MB (in steps of
50MB). Using Spearman’s correlation for the Bonsai model with
smaller memory sizes, we found that RAM energy consumption
showed the most promising correlation value of 0.75. While for
ProtoNN model with smaller memory sizes, both the RAM energy
consumption and total energy consumption showed promising
correlation values of 0.63 and 0.71 respectively.

Incorporating Hardware Metrics to Model Cards to Evaluate the Utilization of Machine Learning Models on IoT devices

From 2021 to 2022 participation at Carderock in the Congressionally mandated Defense Organizational Climate Survey (DEOCS) increased 76%!  Great for us, right?!  The survey is mostly multiple choice, but there are ten free response questions on the survey as well.  Five questions are the same for everyone in the Navy, and five are specific to each Command.  The 2,974 responses are 365 pages long when printed out.  Historically, a team from the Commander’s office must read through all of those responses and assign classes to each one.  The categories for each comment are: Requires Redaction, Sentiment (Positive, Negative, Neutral), Requires Follow-up Action by Command, and Risk Factor.  Then there are also 20 Sub-Factors and 27 Sub-Categories.  Sub-Factors include, Cohesion, Inclusion, Stress, Morale, etc.  Sub-Categories include Compliance, Diversity, Workload, Food, Gym, Parking, etc.  Therefore, the Commander’s Office initiated a project to evaluate the feasibility of classifying the DEOCS responses using natural language processing (NLP) models.  We successfully used the 2021 survey results and classifications to implement transfer learning on a variety of BERT, RoBERTa, and Distillbert pretrained NLP models using the Hugginface API.  We found that a model pretrained on 5 million Tweets performed best for sentiment classification.  Sentiment and all the binary classifications performed well (>90% accuracy).  We identified that if we corrected errors in only the smallest classes of each category that we can further improve the final accuracy with at 70% reduction in labor.  We found that we do not yet have enough training data to achieve acceptable acuracy for the Sub-Factor (19%) and Sub-Category (66%) multiple classification tasks. We would like to work with other Warfare centers to combine datasets for the five common DEOCS questions in order to improve performance with more data. 

DEOCS Survey Response NLP Analysis

There are large selections and offerings of online courses in the field of machine learning, computer science, programming language, database management, cloud computing, and many more. In addition to the selection of topics for courses, the courses are tailored for specific skill level and data competency, which includes whether the course provides expert level knowledge or working level knowledge in the domain to the individual completing the courses. Identifying relevant information from course description is important to recommend the correct courses to an individual for workforce development. To this end Naval Surface Warfare Center, Carderock Division (NSWCCD) implemented a machine learning and python approach that finds various online courses, classifies the course based on skill level and data competency, and recommends a selection of courses for workforce development within a given data competency and skill level. NSWCCD started with scraping the internet using python package Beautiful Soup to find courses and downloading the associated text from the webpages. The skill level required for the course was extracted from the web scraped text, but to determine data competency requires human in-the-loop and this was accomplished through Watchful. Watchful.io is an interactive solution that uses regular expression and statistical model to speed up the labeling of text data. Once the courses and the associated web-scraped text were labeled, then various NLP models were employed to classify the courses to the data competency level. The authors evaluated training separate NLP models per classification label and using single NLP model to classify for any label. In addition, the authors compared various deep learning models’ accuracy to a random forest classifier model.

Data Science Course NLP Classification

The Power Projection Using Process Technologies Tool (PPUPTT) enables data-driven tactics, techniques, and procedures (TTP) for enduring 21st Century Competition. Developed originally for exploiting fleeting opportunities in space ecosystems at scale, PPUPTT aligns with the Space Force’s “Combat Power Projection'' strategic capability. Importantly, however, PPUPTT may be adapted for use within any ecosystem of interest. Governments and military organizations need to transition from linear, kinetic approaches to a continual competition mindset. Better, faster data-driven decisions advantage one player over another. PPUPTT uses explainable process technologies which predict complex systems next-state, and identify fleeting opportunities to maneuver and increase adversary costs of operations. Process technologies are core to PPUPTT’s architecture. Developed originally in a business context for organizations to model the processes which run the business, PPUPTT uses process mining to model complex systems as temporal, unfolding processes. Using process models outputs, PPUPTT derives time series information entropy measures and depicts entropy with intuitive, engaging dashboards fueled by explainable artificial intelligence--making it easy for those without extensive data science training to use the software, understand the analyses, and communicate recommendations. Information entropy, which is linked to Ashby’s Law of Requisite Variety, is a measure of the state of possibilities within complex systems. Changes in information entropy reflect changes in the distribution (variety) of activities, which uncover planning and execution cycles inferences, cognitive state comparisons among agents, and ecosystem stimulus-response dynamics. Creative TTPs built using these measures improve the ability for blue to pursue goals by increasing blue variety of TTPs while simultaneously tarnishing red’s variety of activities. It does this by monitoring ecosystems of interest continually and providing operators with actionable insights tailored to their specific needs. Finally, PPUPTT speeds decision making and automates rote analyses, which frees up time for human sense making. PPUPTT operationalizes an aspect of complexity science by synthesizing research from noteworthy thinkers. For example, Shannon’s Information Theory and simple entropy calculation is used to identify surprising confluences of events which can be used for persistent monitoring, alerting, and decision making based upon recent and historical activity. Information entropy describes the variety of activities or state of possibilities within complex systems. Inputs and outputs are in open data standards which facilitates collaboration with adjacent systems. PPUPTT enables maneuver warfare grounded in complexity science, can be operationalized quickly, and will increase in value as more statistical insights are discovered and TTPs developed.

There is a pressing need to exploit fleeting opportunities continually--real time anticipatory ecosystem entropy offers a lens for these kinds of exploitations. We want to measure ecosystem entropy continually to make better, faster data-driven decisions which advantage one player over another. Combining entropy measures with process models, we seek to understand how dissipative structures interact with one another. Continual sliding window entropy measurements identify exploitation opportunities when a player’s ecosystem is especially chaotic or orderly. Process models provide turnkey simulation parameters which predict ecosystem next-states and identify exploitation actions. This concept may be applied to space, social media, critical infrastructure, urban, supply chain, macro-economic, natural, and geo-political ecosystems, as well. Efforts to date have neither examined ecosystems in a temporal process mining context using categorical data event logs nor analyzed interactions between ecosystem players in order to anticipate outcomes and improve decision making. A highly adaptable, domain agnostic information maneuver capability is possible. It will be possible to drive entropy purposefully by anticipating ecosystem changes or inflection points and make decisions which favor one party over the other.

Power Projection Using Process Technologies Tool (PPUPTT)

Modeling for safety risk often involves infrequent events with sparse data, and there is simultaneously a low tolerance for both type 1 and type 2 errors. Frequent type 1 errors will cause a numbness to develop with respect to any warnings generated by the model, while frequent type 2 errors are a symptom of chronic underestimation of risk. Managing type 1 and type 2 errors in a balanced way is the objective of ensemble modeling. This method can help reduce type 1 errors so that a more sensitive model can be produced, which also in turn remediates the chronic risk underestimation. Ensemble modeling can leverage multiple techniques that tend to agree on the true positive and true negative results, while tending to have a diverse response with respect to false positive and false negative results. When these results are unified under a top layer model, each lower layer model's classification results are interpreted in context with all lower layer model results. The outcome of this process is a more robust model which amplifies the true positive and negative results that are common between ensemble components while deemphasizing the results that the components disagree on. This paper will review some generally accepted ensemble modeling strategies and provide a practical walkthrough of a layered ensemble process using Naval safety data and the R programming language, along with an evaluation of the limits associated with the improvement that can be expected from this technique.

Ensemble Modeling: Using Advanced Learning Techniques for Naval Risk Analysis and Prediction

What we want to aim for is a navy where the autonomous systems are allowed to do what machines do best. And leave humans for things that they do best, like making decisions and doing analysis and things like that. We may argue the fact over how, how good we are humans are at that. But, but we'll just focus on what do machines do best.

Premium content

Next from NAML 2023

An Uncertainty Quantification-Based Automatic Target Recognition Framework Over Distributed Surface Vessels

Similar lecture

Integrating Big Data, Data Science and Cyber Security with Applications in Cloud and Internet of Things

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES