United States

This talk consists of four parts as follows: (1) (15min) An overview of the Navy sponsored, ORNL administrated AI ATAC Prize Challenges 1 &amp; 2 will be given. These two challenges were large-scale experiments to evaluate market-leading cybersecurity tools that leverage AI/ML for endpoint malware detection (Challenge 1) and network traffic analysis to detect multi-step adversarial campaigns (Challenge 2). (2) (10min) The second part of this talk gives a deep-dive on how CORR (ORNL&#39;s Cyber Operation Research Range), a cybersecurity testbed, was designed and stood-up to host these and other evaluations. (3) (5min) The third portion of the talk gives a brief overview of the machine learning and software engineering task to create D2U, the Data-Driven User Emulator. This allows orchestration of hundreds of VMs on CORR that, guided by trained probabilistic generative models, take actions as real users. (4) (10 min) Finally, the fourth and final portion gives an overview of a cost-benefit framework for simulating how SOCs will use the cybersecurity tools in the test. It allows integration of the many statistical measures (precision, recall, time-to-detect, etc.) into a single comparable metric (cost) for ranking and reasoning about tradeoff&#39;s when considering cybersecurity tools.

NAML 2021

Artificial Intelligence Applications to Cybersecurity (AI ATAC) Prize Challenges I&amp;II

evaluation and test

intrusion detection

cybersecurity

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

This talk consists of four parts as follows: (1) (15min) An overview of the Navy sponsored, ORNL administrated AI ATAC Prize Challenges 1 & 2 will be given. These two challenges were large-scale experiments to evaluate market-leading cybersecurity tools that leverage AI/ML for endpoint malware detection (Challenge 1) and network traffic analysis to detect multi-step adversarial campaigns (Challenge 2). (2) (10min) The second part of this talk gives a deep-dive on how CORR (ORNL's Cyber Operation Research Range), a cybersecurity testbed, was designed and stood-up to host these and other evaluations. (3) (5min) The third portion of the talk gives a brief overview of the machine learning and software engineering task to create D2U, the Data-Driven User Emulator. This allows orchestration of hundreds of VMs on CORR that, guided by trained probabilistic generative models, take actions as real users. (4) (10 min) Finally, the fourth and final portion gives an overview of a cost-benefit framework for simulating how SOCs will use the cybersecurity tools in the test. It allows integration of the many statistical measures (precision, recall, time-to-detect, etc.) into a single comparable metric (cost) for ranking and reasoning about tradeoff's when considering cybersecurity tools.

Artificial Intelligence Applications to Cybersecurity (AI ATAC) Prize Challenges I&II

technical paper

Forecasting the weather over the oceans of the Earth is a necessary but difficult requirement of US Navy operations around the globe. Before computer science formally existed as a discipline, meteorologists were using the first computers to explore the possibilities of Numerical Weather Prediction (NWP) to support this forecasting. The foundation of NWP is a fluid dynamics model of the atmosphere interacting with a multitude of physical processes that drive the spatiotemporal vector fields of wind and scalar fields of pressure, temperature, and water vapor mixing ratio. Because the spatial domain of the model is essentially the entire Earth and practical forecasting operates on a cadence of updating every six hours, the computational constraints are formidable and require a number of compromises in formulating the model.
The Naval Research Laboratory (NRL) has over a few decades developed the Navy’s primary NWP model called the Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS). COAMPS has effectively supported naval operations at the Fleet Numerical Meteorology and Oceanography Operations Center (FNMOC) for the past 20 years but is going to be replaced by a more advanced model called Neptune in the next few years. As a part of this process, NRL has a project to understand how Machine Learning (ML) might be used to improve the forecasting accuracy and computational efficiency of this new system. In particular, there has recently been significant progress in applying ML to model turbulence in fluid dynamics. Since proper representation of turbulence in NWP models has been identified as a weakness, the opportunity of using ML to improve weather forecasting has focused on this aspect of the problem.
To support a data-driven ML approach to improving the NWP model, a series of Large Eddy Simulations (LES) have been performed at spatial resolutions not possible with operational NWP models to provide “ground truth” data for how turbulence behaves in the atmosphere. The goal is to learn both the relevant combinations of state variables and the partial differential equations of the spatial dynamics for insertion into the NWP model. This approach requires data and model compression to find useful state variables (or feature vectors) via unsupervised learning since not all quantities available in the LES data can be represented by the low resolution NWP model. Given a suitable space of feature vectors, a supervised learning step finds the partial differential equations for the NWP model that match the dynamics of the LES data. The talk will emphasize the broader issues of how ML can be used in the scientific domain when complex modeling choices can be better resolved with a data-driven approach.


Using Machine Learning to Improve Weather Forecasting for Naval Operations

The traditional approach to photonic design (the inverse problem) is to generate a solution based on intuition and expert knowledge, evaluate how closely it approximates the desired result by electromagnetic simulation (forward problem), and optionally optimize over a few input parameters. However, this approach is limited in what areas of the design space it can reach.. Alternatively, we may computationally optimize over many parameters simultaneously, effectively parameterizing the entire design space, but algorithms for doing so require many costly evaluations of the forward problem. In this work we apply a neural network to approximate simulation output, making the forward problem less costly to solve and the inverse problem more tractable. The neural network is used to assist traditional gradient descent in finding the optimum solution for a given design target. We consider the design of a non periodic grating as a proof of concept. Combining the neural network with gradient-based optimization, our algorithm designs a grating with a spectrum matching a target from the test set with R^2=.99 and an arbitrary spectrum with R^2=.91. These results serve as useful proof of concept for this technique and an opportunity to apply it to new problems in photonic design. 


Deep Inverse Design of a Photonic Structure

Simulating a fluid flow involving chemical reactions is critical to many scientific problems, from climate modeling to propulsion system design. Since chemical reactions can occur very quickly, these simulations typically solve the fluid state separately from the chemical state, syncing up at regular intervals. Unfortunately, the chemistry is a significant bottleneck in many practical problems: it scales very poorly as reactions get more complex. As a result, many useful simulations are computationally prohibitive, forcing researchers to rely on low-fidelity reduced order models, very small-scale simulations, or even forego the problem altogether. Deep learning is gaining traction as a way to develop data-driven surrogate models that are accurate and fast enough to overcome these bottlenecks; however, little work has been done to apply this to chemical kinetics. Today, we will present our work evaluating a broad spectrum of surrogate modeling approaches for stiff chemical kinetics. We discuss the strengths and weaknesses of different modeling and training strategies, highlighting the unique challenges of stiff time-series forecasting and recommending several promising paths forward. Finally, we will discuss the potential for machine learning for developing surrogate chemistry solvers in practical CFD.

Introducing Machine Learning Solutions for Chemically Reacting Flow Simulations

SpaceNet 6 was a geospatial data science challenge centered on foundational mapping from multimodal data. It was one in a series of challenges organized by SpaceNet, which accelerates geospatial deep learning by releasing open datasets, organizing deep learning challenges, and open-sourcing the resulting algorithms and analysis.

SpaceNet 6 introduced a new dataset covering the vicinity of the port of Rotterdam in the Netherlands. The multimodal dataset includes building footprint polygons, electro-optical imagery, and synthetic aperture radar (SAR). The high resolution of the SAR (processed at 0.5m for the challenge, the same resolution as the corresponding optical data) makes the dataset especially valuable. SAR works under nighttime and cloudy conditions when visible satellite imagery cannot be acquired. To make the challenge mimic such a real-world scenario, participants were permitted to use the SAR and electro-optical data for training but were only provided SAR for testing. The specific task was to identify building footprints in the SAR imagery.

An analysis of the winning models from SpaceNet 6 provides many lessons about using deep learning with SAR. Winning models significantly outperformed a baseline model by using EfficientNet architectures and incorporating pretrained weights. The ability to detect buildings varies with building size and height, with similar trends seen across models. Layover, a geometric effect intri

SpaceNet 6: Results from a Geospatial Deep Learning Challenge with Synthetic Aperture Radar Imagery

Threat Evaluation algorithms are designed for aiding in identifying which incoming tracks are threats to the surface fleet. The use of Artificial Intelligence (AI) and Machine Learning (ML) algorithms will speed up the threat identification process, allowing more time for coordinating defensive measures. Currently, the threat evaluation process is performed through verbal communication by a set of human operators making real time decisions on tracks as seen on the radar console. There are many factors that can be considered in threat evaluation; the data load and data processing rate required for an optimal real-time assessment far exceed human capacity. The conversion of as-received data to informed decisions can be performed faster with AI/ML. One such AI/ML approach that can be used is the Random Forest Classifier algorithm. This is not only more efficient but also more accurate than the current threat evaluation process because it can be accomplished with a series of measurements happening at machine speeds instead of human processing time. The Random Forest Classifier is chosen for this analysis because unlike most AI/ML algorithms, its process is transparent. It consists of a series of decision trees, so an operator can look at the decisions being made by the algorithm and make a final decision. In Random Forest implementations, each decision tree runs on a subset of all the features that are provided and evaluates the split of data made on those features using a mathematical equation. The equation implemented in this iteration of Random Forest is a Gini Impurity split measurement. This measurement takes into consideration the probability of a particular feature being wrongly classified when the data is split. This equation is able to handle both numerical and categorical data and performs well with large data sets. As with any AI model, Random Forest needs to be trained. The current iteration of the Random Forest Classifier was first run on a known set of Sonar Data obtained from open sources and correctly identified those data points 84% of the time. Next, training data was obtained by using the Ship Air Defence Model (SADM) and the Joint Cognitive Opertional Research Environment (JCORE) running multiple scenarios containing both threat and non-threat air entities to create radar and Electronic Support track data. This paper provides the results of running the Random Forest Classifier on test sets selected from the training data. The results are then discussed and plans for future work are presented. 
Classification: UNCLASSIFIED
Distribution Statement: Distribution Statement A



Evaluation of Threat Level Using Artifical Intelligence

Techniques to identify an RF emitter and key characteristics of the waveform being transmitted are critical when determining whether a transmission is intended to interfere with communications. In the specific case of space communications, there are many internationally owned transmitters attempting to communicate with spacecraft in a congested frequency band. There are approximately 6-8 thousand active spacecraft on orbit with many new commercial constellations being launched where all are competing to use the same ~100 MHz of bandwidth. For this reason, it’s critical to be able to distinguish between various signals and identify distinguishing characteristics within those signals. Techniques such as fingerprinting transmit signals of interest, machine learning for modulation classification of the observed signal, and pattern matching and identification methods on the resulting data stream are important for distinguishing signals.

The work presented here focuses on applying machine learning to unknown signals in order to classify the modulation. We have focused on three of the most common modulation types - Frequency Shift Keying (FSK), Phase Shift Keying (PSK), and Quadrature Amplitude Modulation (QAM). In particular, software-defined radio (SDR) systems are used to transmit and then receive signal bursts. We then use Mean-Teacher, a method that averages model weights, for semi-supervised training of four signal characteristics (Frequency, Angle, Magnitude, and Power Spectral Density (PSD)) of the received signals. We use a tiered approach, first training the high-level modulation types (FSK, PSK, and QAM) and then training for the specific modulations within each type (GMSK, MSK, 2FSK, 4FSK, 8FSK, OQPSK, QPSK, 8PSK, BPSK, 16QAM, 64QAM, 256QAM). We will present where we have found it beneficial to use semi-supervised learning, and compare the results of the 12-class classifier vs the tiered approach.

Mean Teacher Technique Applied to RF Modulation Classification

A Signal of Interest (SOI) is a signal that has been recorded for further analysis. This is driven by mission requirements for both known and anomaly signals. Identifying anomalies/SOIs is reliant on the system operator’s knowledge which can be prone to human error. The objective of this project is to improve situational awareness by automating the identification of SOIs with Machine Learning/Artificial Intelligence techniques. In year one, data science techniques were applied and proved the daily emitter and frequency distributions in the Hampton Roads area have a consistent subset of emitter traffic. Anomalies were identified based on this fingerprint, and an ANN algorithm was implemented in an application which provides a graphic highlighting anomalies/SOIs. During year two, the algorithm was validated, the data collection method was improved, and prototype functionality was successfully integrated into the tactical system that streams live Radio Frequency into a real-time Graphical User Interface and predicts potential anomalies/SOIs in real-time.

The Application of Machine Learning to Signal Processing for the Detection and Identification of Signals of Interest and Anomalies

In recent years, the concept of Digital Twin (DT) has come up to play a massive role in the efficient functioning of various Industry and Manufacturing sectors. Originally designed for the aerospace sector, the benefits of the framework were soon recognised for other arenas. One particularly exciting avenue for this concept is for healthcare where DTs of Human Beings can have applications for the prognosis of muscle injuries and diseases and rehabilitation of patients, while also providing the ability of virtually testing solutions before they are actually implemented. A type of a DT of Human Beings are those of Human Musculo-Skeletal (MSK) systems which require multiple muscle parameters and muscle excitation signals, traditionally found by solving a system identification problem through optimisation. In this work, a framework for the system identification of a DT of a Human MSK system is proposed where the problem is solved using Physics Informed Recurrent Neural Networks (PI-RNNs) to identify the equation parameters and predict the muscle excitation signals, given the recorded motion data and the system of equations governing the rigid body movement of the Human MSK system. PI-RNNs have been formulated by enhancing traditional Recurrent Neural Networks (RNNs) with physics-based constraints such as residuals of the differential equation and/or time integration schemes for differential equations. RNNs are extensively used to model sequential time series data and adding the physics-based residuals as a regularisation term to the loss function allows for a robust system identification procedure while modern machine learning architectures lend a computationally efficient framework to solve for system parameters and signals. The future work in uniting various data streams of the subject including motion capture data (position, velocity, and acceleration) and wearable sensor data (kinetic and kinematic data of the muscle groups) for a more accurate solution and connecting the system identification of the multi-body dynamics of the MSK DT to the patient specific muscle mechanical properties is also highlighted.

System identification in Digital Twins of Human Musculo-Skeletal systems.

Wideband signal detection is often achieved by careful design of signal-specific matched filters, requiring expert knowledge of signal protocols to develop optimal, matched filter design, which is both expensive and time-consuming. Learning a detector for new or unknown signals in a timely manner has important operational implications. Furthermore, the ability of autonomously monitoring a wideband in order to find signals-of-interest (SOI) or anomalous signals can increase operator efficiency and their ability to quickly react to new threats. In this work, we present CLAWS — Control Learning from Anomalies in a Wideband Spectrum — a machine learning system for wideband spectrum awareness and autonomous radio control able to achieve a demonstrated real-time throughput of 1 Gigasample per second. Our system can accurately detect SOIs and anomalies and make intelligent control decisions to maximize probability of intercept given available system resources. The core components of CLAWS are a Deep-Learning (DL) based SOI detector, anomalous signal detector, and a radio controller based on joint utility optimization. The DL detectors are realized with a multi-scale U-Net inspired architecture which consumes wideband complex-valued short-time Fourier transform slices, processes these slices with a combination of complex-valued and real-valued layers, and outputs time/frequency bounding boxes and signal type classification labels for signals of interest. The network is fully-convolutional, thus allowing training and testing to happen at different sample rates. This multi-rate property can significantly speed up the training process by reducing the footprint of the training set both in memory and on disk. The DL detector can be trained without expert signal knowledge and only requires time-frequency boxed examples of signals in I/Q form. In order to enable autonomous radio control, we build a Gaussian Mixture probabilistic model of the environment that serves as a prior distribution of SOI. The joint-utility based radio controller is trained to maximize the probability of intercepting the SOI in the wideband by tuning the radio center frequency and beam angle, trading-off the exploitation of the environment model with random exploration. We present results of both our SOI and anomaly detectors as well as the full CLAWS system using both government provided data as well as over-the-air collects, demonstrating its operational relevance.

Control Learning from Anomalies in a Wideband Spectrum (CLAWS)​

In November of 2017, the Consortium for Robotics and Unmanned Systems Education and Research (CRUSER) at the Naval Postgraduate School (NPS) conducted the Multi-Thread Exercise on San Clemente Island. The experiment provided a platform to test the collaboration of an autonomous network of UxVs with manned assets. The scenario was built around a NSW unit tasked with landing on San Clemente Island and traversing the island to act on a known target. The NSW unit was supported by both manned and unmanned assets operating together to provide ISR support, transportation, and battlespace awareness. Previous work took real world data, the NSW team’s path from their insertion point to their objective, and introduced a Disruption-Tolerant Network into the NCS to loosen the communication constraints and increase mission effectiveness. A simple, novel algorithm was introduced that leveraged distributed submodularity to arrive at near-optimal network topologies in a decentralized manner by utilizing local searches by and limited information sharing between the agents in the network. This algorithm presents a novel, viable method for using distributed submodularity to near-optimally position network agents in real-time. This research analyzes how well an adversary is able to predict an individual UxV future path and the location of a point of interest based on the paths of one or more UxV. It also explores how well an adversary would be able to predict the location of the NSW team at various time steps if the adversary assumed that a ground unit was present and controlling the UxV through some local communication platform. If an adversary would be able to predict the future paths of one or more UxV and the NSW, we investigate how well the adversary also predict the location of the ultimate objective, thereby providing the adversary with advanced warning of a raid or other operation. Similarly, we investigate how an adversary could also use such experimental data for the purposes of identifying tactics, techniques, and procedures of the Naval Surface Warfare units and UxV operations. And finally, we develop machine learning algorithms which can disguise the network’s intent. We use a two-stage approach to predicting UxV and NSW positions. The first stage consists of a time series analysis to develop ARIMA models for the X and Y coordinates of each UxV. Once the ARIMA model was created, we forecast the future positions of each UxV. The second stage of the model uses the predicted UxV locations to model possible NSW locations. We use other analytical methods to determine if the probability of accurately predicting the path or target is within acceptable limits or if adjustments should be made to the network configuration.

Artificial Intelligence Applications to Cybersecurity (AI ATAC) Prize Challenges I&II

Downloads

Next from NAML 2021

Using Machine Learning to Improve Weather Forecasting for Naval Operations

Similar lecture

Feature Evaluation of Anomaly Detection Models for Cyber Data

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES