United States

Model cards organize and portray information
about a model’s training data, hyper parameters, and behavior,
such as predictive accuracy and bias for machine learning (ML)
algorithms or models. Consumers use these cards to determine
whether a model is well suited for a particular use case.
However, the current design of model cards does not include
resource utilization metrics, which are important when models
are being optimized for specific hardware. The main objective
of this study was to determine which set of hardware-centric
performance metrics provided the most value to users when
comparing among different algorithms for Internet of Things
(IoT) edge devices. The Department of Defense (DoD) can utilize
these metrics to select the best model for key mission tasks
within the Command and Control (C2) missions that require
AI/ML deployment on edge devices. Our study focused on
finding correlations between resource availability and machine
learning models’ computational footprint using three key metrics:
energy consumption, execution time, and memory utilization.
The data was gathered from simulated environments hosted
by Docker containers (i.e., lightweight, stand-alone, executable
package of software that includes everything needed to run an
application) with various memory sizes. Two different series of
experiments were performed. The first series of experiments
simulated environments with 256 MB to 8GB (steps of the
powers of 2) of available memory. For each memory size, we
deployed 50 different containers, calculated the metrics from
each container, and then recorded the average result of each
metric. Two ML models from Microsoft’s EdgeML library that
are designed to run more efficiently on edge devices were used:
Bonsai, a decision tree based model, and ProtoNN, a KNN based
model. Using Spearman’s correlation, we found that none of
the metrics showed promising correlations for the Bonsai model,
while the ProtoNN model demonstrated a correlation of 0.5 for its
computational time. In order to investigate the lack of adequate
results, we ran a second series of experiments in a similar fashion
but with smaller memory sizes from 250MB to 950MB (in steps of
50MB). Using Spearman’s correlation for the Bonsai model with
smaller memory sizes, we found that RAM energy consumption
showed the most promising correlation value of 0.75. While for
ProtoNN model with smaller memory sizes, both the RAM energy
consumption and total energy consumption showed promising
correlation values of 0.63 and 0.71 respectively.

NAML 2023

Incorporating Hardware Metrics to Model Cards to Evaluate the Utilization of Machine Learning Models on IoT devices

machine learning internet of things hardware

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


From 2021 to 2022 participation at Carderock in the Congressionally mandated Defense Organizational Climate Survey (DEOCS) increased 76%!  Great for us, right?!  The survey is mostly multiple choice, but there are ten free response questions on the survey as well.  Five questions are the same for everyone in the Navy, and five are specific to each Command.  The 2,974 responses are 365 pages long when printed out.  Historically, a team from the Commander’s office must read through all of those responses and assign classes to each one.  The categories for each comment are: Requires Redaction, Sentiment (Positive, Negative, Neutral), Requires Follow-up Action by Command, and Risk Factor.  Then there are also 20 Sub-Factors and 27 Sub-Categories.  Sub-Factors include, Cohesion, Inclusion, Stress, Morale, etc.  Sub-Categories include Compliance, Diversity, Workload, Food, Gym, Parking, etc.  Therefore, the Commander’s Office initiated a project to evaluate the feasibility of classifying the DEOCS responses using natural language processing (NLP) models.  We successfully used the 2021 survey results and classifications to implement transfer learning on a variety of BERT, RoBERTa, and Distillbert pretrained NLP models using the Hugginface API.  We found that a model pretrained on 5 million Tweets performed best for sentiment classification.  Sentiment and all the binary classifications performed well (>90% accuracy).  We identified that if we corrected errors in only the smallest classes of each category that we can further improve the final accuracy with at 70% reduction in labor.  We found that we do not yet have enough training data to achieve acceptable acuracy for the Sub-Factor (19%) and Sub-Category (66%) multiple classification tasks. We would like to work with other Warfare centers to combine datasets for the five common DEOCS questions in order to improve performance with more data. 

DEOCS Survey Response NLP Analysis

There are large selections and offerings of online courses in the field of machine learning, computer science, programming language, database management, cloud computing, and many more. In addition to the selection of topics for courses, the courses are tailored for specific skill level and data competency, which includes whether the course provides expert level knowledge or working level knowledge in the domain to the individual completing the courses. Identifying relevant information from course description is important to recommend the correct courses to an individual for workforce development. To this end Naval Surface Warfare Center, Carderock Division (NSWCCD) implemented a machine learning and python approach that finds various online courses, classifies the course based on skill level and data competency, and recommends a selection of courses for workforce development within a given data competency and skill level. NSWCCD started with scraping the internet using python package Beautiful Soup to find courses and downloading the associated text from the webpages. The skill level required for the course was extracted from the web scraped text, but to determine data competency requires human in-the-loop and this was accomplished through Watchful. Watchful.io is an interactive solution that uses regular expression and statistical model to speed up the labeling of text data. Once the courses and the associated web-scraped text were labeled, then various NLP models were employed to classify the courses to the data competency level. The authors evaluated training separate NLP models per classification label and using single NLP model to classify for any label. In addition, the authors compared various deep learning models’ accuracy to a random forest classifier model.

Data Science Course NLP Classification

The Power Projection Using Process Technologies Tool (PPUPTT) enables data-driven tactics, techniques, and procedures (TTP) for enduring 21st Century Competition. Developed originally for exploiting fleeting opportunities in space ecosystems at scale, PPUPTT aligns with the Space Force’s “Combat Power Projection'' strategic capability. Importantly, however, PPUPTT may be adapted for use within any ecosystem of interest. Governments and military organizations need to transition from linear, kinetic approaches to a continual competition mindset. Better, faster data-driven decisions advantage one player over another. PPUPTT uses explainable process technologies which predict complex systems next-state, and identify fleeting opportunities to maneuver and increase adversary costs of operations. Process technologies are core to PPUPTT’s architecture. Developed originally in a business context for organizations to model the processes which run the business, PPUPTT uses process mining to model complex systems as temporal, unfolding processes. Using process models outputs, PPUPTT derives time series information entropy measures and depicts entropy with intuitive, engaging dashboards fueled by explainable artificial intelligence--making it easy for those without extensive data science training to use the software, understand the analyses, and communicate recommendations. Information entropy, which is linked to Ashby’s Law of Requisite Variety, is a measure of the state of possibilities within complex systems. Changes in information entropy reflect changes in the distribution (variety) of activities, which uncover planning and execution cycles inferences, cognitive state comparisons among agents, and ecosystem stimulus-response dynamics. Creative TTPs built using these measures improve the ability for blue to pursue goals by increasing blue variety of TTPs while simultaneously tarnishing red’s variety of activities. It does this by monitoring ecosystems of interest continually and providing operators with actionable insights tailored to their specific needs. Finally, PPUPTT speeds decision making and automates rote analyses, which frees up time for human sense making. PPUPTT operationalizes an aspect of complexity science by synthesizing research from noteworthy thinkers. For example, Shannon’s Information Theory and simple entropy calculation is used to identify surprising confluences of events which can be used for persistent monitoring, alerting, and decision making based upon recent and historical activity. Information entropy describes the variety of activities or state of possibilities within complex systems. Inputs and outputs are in open data standards which facilitates collaboration with adjacent systems. PPUPTT enables maneuver warfare grounded in complexity science, can be operationalized quickly, and will increase in value as more statistical insights are discovered and TTPs developed.

There is a pressing need to exploit fleeting opportunities continually--real time anticipatory ecosystem entropy offers a lens for these kinds of exploitations. We want to measure ecosystem entropy continually to make better, faster data-driven decisions which advantage one player over another. Combining entropy measures with process models, we seek to understand how dissipative structures interact with one another. Continual sliding window entropy measurements identify exploitation opportunities when a player’s ecosystem is especially chaotic or orderly. Process models provide turnkey simulation parameters which predict ecosystem next-states and identify exploitation actions. This concept may be applied to space, social media, critical infrastructure, urban, supply chain, macro-economic, natural, and geo-political ecosystems, as well. Efforts to date have neither examined ecosystems in a temporal process mining context using categorical data event logs nor analyzed interactions between ecosystem players in order to anticipate outcomes and improve decision making. A highly adaptable, domain agnostic information maneuver capability is possible. It will be possible to drive entropy purposefully by anticipating ecosystem changes or inflection points and make decisions which favor one party over the other.

Power Projection Using Process Technologies Tool (PPUPTT)

Modeling for safety risk often involves infrequent events with sparse data, and there is simultaneously a low tolerance for both type 1 and type 2 errors. Frequent type 1 errors will cause a numbness to develop with respect to any warnings generated by the model, while frequent type 2 errors are a symptom of chronic underestimation of risk. Managing type 1 and type 2 errors in a balanced way is the objective of ensemble modeling. This method can help reduce type 1 errors so that a more sensitive model can be produced, which also in turn remediates the chronic risk underestimation. Ensemble modeling can leverage multiple techniques that tend to agree on the true positive and true negative results, while tending to have a diverse response with respect to false positive and false negative results. When these results are unified under a top layer model, each lower layer model's classification results are interpreted in context with all lower layer model results. The outcome of this process is a more robust model which amplifies the true positive and negative results that are common between ensemble components while deemphasizing the results that the components disagree on. This paper will review some generally accepted ensemble modeling strategies and provide a practical walkthrough of a layered ensemble process using Naval safety data and the R programming language, along with an evaluation of the limits associated with the improvement that can be expected from this technique.

Ensemble Modeling: Using Advanced Learning Techniques for Naval Risk Analysis and Prediction

Keynote Talk: Rear Admiral Douglas Small

Automating Safety of Navigation with Machine Learning Technologies

Our project focuses on using machine learning and sensor data to predict quantities such as the refractive-index structure constant (Cn2), scattering, and extinction. Meteorological sensor data including temperature, relative humidity, wind speed, and surface pressure was input as predictors into RegressionLearner in MATLAB, with Cn2 and RSSI being the response variables. We provide a comparison of all the available machine learning algorithms. Principal Component Analysis (PCA) was included to determine the number of variables that explain the most variance. This presentation covers the initial results for our optical turbulence prediction trained on datasets from our collaborators at the Hellenic Naval Academy. 

Initial Results for Optical Turbulence Prediction Using a Bagged Decision Tree Model Trained on Geographically Diverse Datasets

First-principles computer models or simulations are often used as forward models for training reinforcement learning (RL) agents because training on real-world systems is too expensive, slow, or otherwise impractical. Even the highest-fidelity models can, however, fail to account for all physical variables and these modeling errors can lead to poor performance when the simulation-trained agent is exposed to the real-world. A potential solution is to use data from the physical system to directly train a deep neural network as an approximation (surrogate) of the system’s underlying state-transition function. Given difficulties inherent in acquiring sufficient real-world training data, one advantageous approach is to first train a neural network surrogate using a first-principles model and then use transfer learning and a smaller set of real-world data to fine-tune the surrogate’s approximations. An RL agent can then be quickly and cheaply trained using the well-trained surrogate and can exhibit better real-world performance compared to an agent trained on a first-principles model alone. We propose a unique algorithm for generating training data appropriate for this type of surrogate, explain several important considerations in posing the learning objective function, describe the training techniques used to achieve outstanding performance for several surrogate models before and after transfer learning, and present results for an RL agent trained using a surrogate as the forward model. More specifically, we consider the problem of training RL agents to control an Unmanned Underwater Vehicle (UUV) in an environment characterized by current flow fields. We include discussion of how our novel data generation algorithm can be extended to work for a UUV navigating such an environment and explain how an egocentric state-space definition can improve the performance of an RL agent trained to control the UUV.

Reinforcement Learning for UUV Control Using Stable and Accurate Neural Network Approximations of Motion Dynamics

"Models which predict optical turbulence (as measured by Cn2) from macro-meteorological data are expensive to develop and often perform poorly when deployed in new environments.
Hybrid models combine these existing models with local measurement and a Gradient Boosted Decision Tree (GBDT) architecture to improve predictive power.
This architecture achieved approximately a 29% improvement over the baseline using only one day of measurement, and 68% using 180 days of observation.
"

Hybrid Optical Turbulence Models Using Machine Learning and Local Measurements

Internet has become a ubiquitous influence in today’s age; however, internet activities have become vulnerable and detrimental due to malicious actions if not secured properly. For example, large scale data breaches such as those indicted at Yahoo and Equifax, essentially exploited the weaknesses of the traditional username/password authentication mechanism. Therefore, it has become imperative for us to find additional ways to protect our data, which also needs to be low-cost and unobtrusive. Existing authentication mechanisms can be categorized into what you know (knowledge-based), what you have (possession of physical things) and what you are (biometrics). Biometrics-based authentication can be further categorized into physiological and behavioral biometrics. Physiological biometrics are based on physiological characteristics of a person such as fingerprints, iris, facial recognition and palmprints, which can cumbersome due to the cost and usability. On the other hand, behavioral biometrics use the behavior of a person interacting with a device such as mouse dynamics (i.e., different mouse movements on a graphical user interface (GUI)), widget interactions (i.e., mouse cursor hovering over different icons or buttons on a GUI), keystroke dynamics, swipe, motion, walking or gait, all of which are inexpensive and unobtrusive techniques that can be leveraged to strengthen existing authentication without interfering with day-to-day computer operations. The experimental work of this proposal includes two novel contributions: 1) a survey of the state-of-the-art literature on mouse dynamics and widget interactions and 2) designing and validating fusion of mouse dynamics and widget interactions’ modalities on a realistic and popular website (i.e., PayPal and Facebook) utilizing our novel datasets. We performed literature survey on mouse dynamics with widget interactions along the following dimensions: tasks and experimental settings for data collection, taxonomy of raw attributes, feature extractions and mathematical definitions, publicly available datasets, algorithms (statistical, machine learning, and deep learning), fusion and performance, and limitations. Secondly, we experimented with trajectory driven window mapping to fuse mouse dynamics and widget interactions modalities using a total of 10 users for PayPal with an average of 12 minutes of data per user and a total of 15 users for Facebook with an average of 2 hours of data per user. Our mean EER rates with a score-level fusion of mouse dynamics and widget interactions are 7.64\% (SVM-rbf) and 3.25\% (GBM), for PayPal, and 5.49\% (SVM-rbf) and 2.54\% (GBM), for Facebook. We further improve the EER rates by combining decision scores from multiple consecutive trajectories, which yields a 0\% mean EER after 11 decision scores across all the users for both PayPal and Facebook.

Premium content

Incorporating Hardware Metrics to Model Cards to Evaluate the Utilization of Machine Learning Models on IoT devices

Next from NAML 2023

DEOCS Survey Response NLP Analysis

Similar lecture

Project Hermes – Naval Messaging

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES