United States

&quot;Automatically detecting and classifying objects in remote sensing imagery can significantly improve intelligence analysis efficiency and throughput. Object detectors are typically trained on a specific set of object classes aligned with the intelligence problem of interest. The detection of unexpected, unusual, or novel objects is also a core intelligence task routinely performed by imagery analysts, but current detectors cannot perform this task. Novel objects are usually ignored by the detector, if they are not sufficiently similar to any known class, or are misclassified as a known class.

Novel object detection addresses this capability gap by discovering objects that are not within the set of known classes but are of intelligence significance. Remote sensing images often contain dozens or hundreds of object types, many of which are not in the training data. Many of these objects are irrelevant, while others may be quite important. Our method automatically alerts analysts to novel objects that are similar to the training set but are sufficiently different such that they may be a new class or a novel variant of a known class. Balancing these two constraints enables the detection of salient novelties, such as a new type of wing on a previously-known aircraft model, without overwhelming the analyst with nuisance novelties such as changes in vegetation.

The xView dataset contains WorldView-3 imagery with 60 object classes labeled in a two-level ontology, with 8 general classes in the top level, such as “Truck”, and 52 specific classes in the lower level, such as &quot;&quot;Truck with Flatbed&quot;&quot;. We adapted this dataset for realistic novelty detection by considering 18 lower level classes as novel and removing images containing them from the training set. These novel classes are distributed across all 8 of the general categories. During testing, the system should detect instances from these 18 categories as novel objects, while also detecting all instances from the 34 known classes, declaring them to be non-novel, and correctly classifying them.

Our method is mathematically grounded in Extreme Value Theory (EVT) and utilizes the Extreme Value Machine (EVM) that we developed on the DARPA Science of Artificial Intelligence and Learning for Open-world Novelty project. EVM creates an EVT distribution over the distances between deep-learned feature embeddings for each class to provide a novelty score. We train an object detector, classifier, and EVM on the known classes, then estimate the novelty of each detection at test time. On XView this approach achieves a 16% AUC increase over a baseline method that thresholds the classifier scores.

Our novelty detection method can be used in various applications of interest to the Navy, such as detecting novel aircraft classes or attributes on the ground, in the air, or on a carrier ship; novel ship or vehicle detection; or detection of novel aircraft attributes, including novel ammunition or damage.
&quot;

NAML 2023

Novel Object Detection in Remote Sensing Images for Naval Applications

"Automatically detecting and classifying objects in remote sensing imagery can significantly improve intelligence analysis efficiency and throughput. Object detectors are typically trained on a specific set of object classes aligned with the intelligence problem of interest. The detection of unexpected, unusual, or novel objects is also a core intelligence task routinely performed by imagery analysts, but current detectors cannot perform this task. Novel objects are usually ignored by the detector, if they are not sufficiently similar to any known class, or are misclassified as a known class.

Novel object detection addresses this capability gap by discovering objects that are not within the set of known classes but are of intelligence significance. Remote sensing images often contain dozens or hundreds of object types, many of which are not in the training data. Many of these objects are irrelevant, while others may be quite important. Our method automatically alerts analysts to novel objects that are similar to the training set but are sufficiently different such that they may be a new class or a novel variant of a known class. Balancing these two constraints enables the detection of salient novelties, such as a new type of wing on a previously-known aircraft model, without overwhelming the analyst with nuisance novelties such as changes in vegetation.

The xView dataset contains WorldView-3 imagery with 60 object classes labeled in a two-level ontology, with 8 general classes in the top level, such as “Truck”, and 52 specific classes in the lower level, such as ""Truck with Flatbed"". We adapted this dataset for realistic novelty detection by considering 18 lower level classes as novel and removing images containing them from the training set. These novel classes are distributed across all 8 of the general categories. During testing, the system should detect instances from these 18 categories as novel objects, while also detecting all instances from the 34 known classes, declaring them to be non-novel, and correctly classifying them.

Our method is mathematically grounded in Extreme Value Theory (EVT) and utilizes the Extreme Value Machine (EVM) that we developed on the DARPA Science of Artificial Intelligence and Learning for Open-world Novelty project. EVM creates an EVT distribution over the distances between deep-learned feature embeddings for each class to provide a novelty score. We train an object detector, classifier, and EVM on the known classes, then estimate the novelty of each detection at test time. On XView this approach achieves a 16% AUC increase over a baseline method that thresholds the classifier scores.

Our novelty detection method can be used in various applications of interest to the Navy, such as detecting novel aircraft classes or attributes on the ground, in the air, or on a carrier ship; novel ship or vehicle detection; or detection of novel aircraft attributes, including novel ammunition or damage.
"

technical paper

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


The US Navy systems may have unexpected significant cost growth for many reasons. Navy systems may have unexpected significant cost growth for many reasons. The Office of the Chief of Naval Operations (OPNAV) is charged, among other responsibilities, with executing the planning, programming, budgeting, and execution (PPBE) process through a series of concurrent annual planning cycles guided by a Program Objectives Memorandum (POM). The objective of the research is to leverage advanced analytics to help the OPNAV understand the common elements and causes of existing Navy systems that have significant cost growth from historical data, requirements documents, and open-source media. 

This research explores two questions:
1. What are the common elements of requirements that create excessive cost growth in Navy systems?
2. Assuming the elements are identified, what is the risk (likelihood and magnitude) of cost growth from common elements for both procurement and sustainment costs? 

We applied two categories of methods: 1. classic data sciences and business intelligence tools and 2. an artificial general intelligence framework to address the needs and research questions to analyze structured and unstructured data together and correlate them with excessive cost or cost growth of Navy systems. Specifically, we applied LLA, a semantic network analyzer, anomaly detection, and causal learning to discover patterns and deep causes that can lead to high cost or cost growth. 

The OPNAV’s Program Budget Information System (PBIS) has been modernized as an authoritative knowledge system including historical data of planned and executed POM information and spending each year. Data relevant to PBIS include structured data and unstructured data. For example, structured data include number of platforms procured and procurement and sustainment costs for Navy systems. Program elements (PEs) or Budget Exhibits (BEs) contain PPBE information as well as unstructured data of unclassified high-level program descriptions and their elements. 
We analyzed two unclassified data sets provided by the topic sponsors. We found patterns and deep causes for high cost or cost growth programs using lexical link analysis, natural language processing (NLP) tools, a semantic network analyzer, anomaly detection, and causal learning concepts. Programs with anomalous characteristics can lead to high costs or high growth. These tools provide counterfactual and drill-down discovery of the key words that explain the deep causes of cost growth. 

We recommend to apply these tools for the total benefits of analyzing Navy programs and requirements of post mortem data, towards modernizing the PBIS to become a knowledge system that can effectively learn from historical data to make better risk predictions and decisions for the future POMs.

Structured and Unstructured Data Sciences and Business Intelligence for Analyzing Navy Programs and Requirements of Post Mortem Data

Two requirements for maintaining our lead in artificial intelligence (AI) are widespread development and the adoption of AI applications.  Both are risky and not yet common. However we have achieved them with many large organizations including Pfizer, Schlumberger, and GE Aviation.  We call it “Everyday AI”: embedding AI development and apps in everyday tasks and processes throughout an enterprise.  A key challenge is balancing data/AI/ML decentralized development with governance.  If there’s too much decentralization then apps aren’t trusted or adopted.  If there’s too much governance then decentralization doesn’t take hold and everyday tasks aren’t addressed.  This presentation covers eight “plays” organizations can run to achieve Everyday AI, a few of which are a data/AI/ML literacy program for all skill levels, catalog the top 20% data/AI/ML products as-is, and an industrial-scale AI operations automation workbench.

Everyday AI Playbook

Monitoring potential maritime powers and/or adversaries is essential to the Department of Defense’s (DoD) mission and to the United States’ security. An essential tool for accomplishing this is the use autonomous object detection in electro-optical imagery. Currently, this is done by way of supervised machine learning. This approach has proven to be effective when applied to detection of objects in the labeled dataset, but struggles when tasked to detect objects that are not in the labeled dataset. Additionally, obtaining and/or generating labeled data is often a time consuming and expensive process. Therefore, our group at Penn State’s Applied Research Laboratory has been exploring the use of unsupervised machine learning for object detection. That is, we want to train a machine learning model to find and classify objects in an image without the use of labeled training data. There has been recent research in the use of variational autoencoders coupled with spatial transformers to accomplish this task on simple datasets. We will demonstrate results of these methods along with relevant data preprocessing algorithms for unsupervised object detection of maritime objects. Our results show promise for the ability to extract objects without the use of labeled training data.

Unsupervised Detection of Naval Objects in Electro-Optical Imagery

"The 21st century has shown the proliferation of algorithms, tools, and code runtimes for machine learning applications. In order to get started, a user or scientist requires a development environment with installed libraries, dependencies to run code, and access to training and test data. This development environment also needs to run on compatible hardware, and have a location to persistently store the source code, model, and output artifacts.

This presentation addresses best practices to create machine learning environments. An exploratory approach was used to find processes, tools, and methods to reduce administrative burden from installing software, and lower the time required to start cleaning, training, and testing data. Users need the capability to share research code and diagrams without conflicting with environment dependencies and packages of other users.

Using these best practices, users will be able to lower the time to get started with machine learning workloads. Reducing setup time allows users to emphasize more of their time on cleaning data, tuning hyperparameters, and improving algorithms. Additionally, by improving the methods to share environments, projects will be able to iterate faster by improving collaboration among their collogues and team."

Methods and Tools for Management of Repeatable Machine Learning Development Environments

The United States coastline spans 95,471 miles; a distance that cannot be effectively patrolled or secured by manual human effort alone. Unmanned Aerial Vehicles (UAVs) equipped with infrared cameras and deep-learning based algorithms represent a more efficient alternative for identifying and segmenting objects of interest - namely, ships. However, standard approaches to training these algorithms require large-scale datasets of densely labeled infrared maritime images. Such datasets are not publicly available and manually annotating every pixel in a large-scale dataset would have an extreme labor cost. In this work we demonstrate that, in the context of segmenting ships in infrared imagery, weakly-supervising an algorithm with sparsely labeled data can drastically reduce data labeling costs with minimal impact on system performance. We apply weakly-supervised learning to an unlabeled dataset of 7055 infrared images sourced from the Naval Air Warfare Center Aircraft Division (NAWCAD). We find that by sparsely labeling only 32 points per image, weakly-supervised segmentation models can still effectively detect and segment ships, with a Jaccard score of up to 0.756.

Weakly-Supervised Semantic Segmentation of Ships Using Thermal Imagery

Predictive analytics for Naval safety data often involves small and imbalanced sets. To cope with categorical data sets of this nature, the usual remedies often include replicating or oversampling the minority class to bring the numbers into balance, trimming the larger categories to achieve the desired balance or simply choosing a modeling method that is less sensitive to the imbalance. However, by creating a synthetic version of the data based on the characteristics of the original set, the new problems created by oversampling or using less data can be avoided. A more diverse set of data can ultimately be used in model training that will allow use of a broader range of modeling options and marginally increase the generalizability of the resultant solution. This paper will walk through a method for constructing a synthetic data set using an adversarial generative technique which generates a data set of arbitrary size with the same apparent modeling characteristics of the original set. This generated data can also be used for other purposes aside from directly modeling, including as a highly effective anonymization technique for the purpose of collaboration with other researchers while using sensitive data sets.

Generation and Utilization of Synthetic Data to Enhance Type I and Type II Error Resistance for Small Datasets

 Federated learning (FL) enables clients to collaborate with a server to train a machine learning model. To ensure privacy, the server performs secure aggregation of model updates from the clients. Unfortunately, this prevents verification of the well-formedness (integrity) of the updates as the updates are masked. Consequently, malformed updates designed to poison the model can be injected without detection. In this talk, I will formalize the problem of ensuring both update privacy and integrity in FL and present a new system,  EIFFeL, that enables secure aggregation of verified updates. EIFFeL is a general framework that can enforce arbitrary integrity checks and remove malformed updates from the aggregate, without violating privacy. Further, EIFFeL is practical for real-world usage. For instance, with 100 clients and 10% poisoning, EIFFeL can train an MNIST classification model to the same accuracy as that of a non-poisoned federated learner in just 2.4s per iteration.

EIFFeL: Ensuring Integrity for Federated Learning

Recreational water quality monitoring for fecal indicator bacteria (FIB) is widely used to notify the public about hazardous conditions. FIB monitoring is also used to track water quality over time and to identify hot spots that should be prioritized for remediation. However, FIB monitoring is generally an expensive, manually-intensive process with results often not available for at least 24 hours. Machine learning approaches have been used to generate real-time estimates of FIB levels without the costs and time delays of sample collection and analysis. Machine learning can make use of weather and hydrologic data which are readily available in near-real time and publicly available at no cost to users. Although many jurisdictions around the US and other high-income countries routinely collect water samples for FIB monitoring, many others do not, or only do so infrequently. Likewise, in many regions of the world, FIB monitoring is not done at all, although weather, wave, and tide data are readily available. It would be useful if ML models developed using FIB data from a given beach (“source beach”) could be applied to other beaches (“target beaches”). The application of ML models from data-rich settings to data-poor settings may be useful for public health and environmental purposes. However, ML models of FIB levels developed using data from a specific setting during a specific time period may not be applicable to other settings. This could be due to: (1) Different distributions of FIB and other variables (“features”) between “source” and “target” beaches or even at the same beach during different “beach seasons.” (2) Different microbial sources, hydrologic factors, and bacterial fate and transport across different beaches. (3) Additionally, beach samples can be analyzed for different FIB (E. coli, enterococci, fecal coliforms, total coliforms) and using different methods (culture, quantitative polymerase chain reaction [qPCR], ENT, etc.). Jurisdictions that make public health decisions (such as issuance of beach advisories or swam bans) at a target beach may use different FIB and analytic approaches than the source beach. To account for such differences, we study domain adaptation approaches that make it possible to apply ML models trained on sources with labeled data (i.e., weather/hydrologic variables and corresponding FIB levels) to predict FIB levels for targets with limited or no labeled data (i.e., targets where only weather/hydrologic variables are available). Using publicly available data, we experiment with four domain adaptation approaches: feature augmentation, balanced weighting, correlation alignment and subspace alignment on the task of predicting FIB levels, using two base learners, random forests and artificial neural networks, in a scenario where the same weather/hydrologic variables are available at both source and target beaches. We compare the domain adaptation approaches with supervised learning approaches that use either source labeled data or a very small amount of target labeled data. Experimental results show that domain adaptation approaches have the ability to transfer knowledge and predict FIB at beaches with limited or no existing FIB data. These results suggest that domain adaptation can potentially aid beach managers, especially in cases when traditional FIB testing methods are unavailable, deemed too costly, or when the past FIB data required to build reliable traditional machine learning models are scarce or unavailable.

A Domain Adaptation Approach for Predicting Recreational Water Quality at Data-Scarce Coastal Locations

"This talk will present the three-stage pipeline that we developed for runtime knowledge discovery and continual learning for SAR image processing. First, novel objects were detected using out-of-distribution (OOD) detection. Images of the detected objects were then clustered and labels were assigned to the clusters. Finally, new classes were added to the existing model using class incremental learning. The pipeline explores the energy efficiency of the state-of-the-art neuromorphic hardware by implementing the learning and inference on the Intel Loihi processor.
Techniques to improve the performance of each pipeline stage will be discussed. Outlier exposure is adopted during the training with the Convolutional Block Attention Module (CBAM) to improve OOD detection. Our method separates the SoftMax score distribution between in- and out-of-distribution images, which allows for more effective detection. 
Opposed to more traditional offline learning paradigms, the iterative nature of class-incremental learning leads to an issue of catastrophic forgetting, where accuracy on previously exposed classes is lost as new classes are introduced.  Understanding this challenge, we incorporate the idea of Neuro-Inspired Stability-Plasticity Adaptation (NISPA) with the biologically plausible spiking neural network learning algorithm, error-modulated spike-timing dependent plasticity (EMSTDP), to overcome the catastrophic forgetting. After learning a new task, inspired by the critical period in brain development, we will select certain neurons as the stable units. The weights between stable units will not be changed during learning. At the same time, rewiring will be implemented among weights between plastic units. Using stable and plastic units, stable paths are formed to preserve learned knowledge of older tasks. 
Exemplars of the previously learned classes must be stored in a limited memory and be trained again to refresh the model’s memory. We found that more robust networks may be generated by selecting exemplars close to decision boundaries. The impact of varied methods of exemplar insertion during training will also be presented in this talk."

Towards Runtime Knowledge Discovery and Continual Learning

"Natural Language Processing (NLP) is a branch of AI that gives computers the ability to understand text and spoken words in much the same way humans do.  NLP in our lives today includes, Amazon Alexa, Siri, ChatGPT, and Google.  Using open-source technology, NIWC PAC Code 584 has developed two classifiers, and two others are in design/requirements stage with prospective sponsors.  The classifiers automatically analyze and structure the text, quickly and efficiently, so NIWC PAC managers can automate processes and discover insights that ultimately lead to better decision-making.
"

Downloads

Next from NAML 2023

Structured and Unstructured Data Sciences and Business Intelligence for Analyzing Navy Programs and Requirements of Post Mortem Data

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

.css-70qvj9{display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}Downloads

Next from NAML 2023

Structured and Unstructured Data Sciences and Business Intelligence for Analyzing Navy Programs and Requirements of Post Mortem Data

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES

Downloads