United States

The Financial Management Systems (FMS) Financial Management Data &amp; Digital Transformation (FMDDT) Division under the Assistant Secretary of the Navy’s Financial Management and Comptroller’s Office (ASN FM&amp;C) along with Deloitte GPS (Government &amp; Public Services), has developed a data science solution for automating the mapping of financial management systems’ vendor data across several systems, identifying erroneous and missing data, and detecting duplicate information.
FMDDT is working with our partners to establish clear data governance in the ingestion, use, and modeling of data. We are working in close partnership with DON CDO and OSD’s ADVANA to leverage tools, data streams, and the analytics environment while establishing governance policies that foster trust in data through consistency and standardization.
As the Navy moves all financial systems under the Navy Enterprise Resource Planning (ERP) system there will be a need to improve the data quality and mapping as it is integrated from Navy and Marine Corp Standard Accounting, Budgeting, and Reporting System (SABRS) and Standard Financial &amp; Reporting System – Field Level (STARS-FL).
The current Navy ERP vendor master list needed consolidation and mapping across SABRS and STARS-FL to provide insight into budget and financial analytics. As these systems migrate into a single financial system in FY22, an understanding of the historical data and how it maps between systems will be critical to the development of future data analytics and machine learning applications.
While using the System for Award Management (SAM) data as the authoritative data source and vendor naming convention, the Navy ERP, SABRS, and STARS-FL vendor names were matched to the SAM data source. The effort entails matching almost 3 million unique ERP, SABRS, and STARS-FL vendor name records against millions of SAM vendor name records resulting in over 3 trillion unique comparisons.
The effort used the Jupiter Databricks environment which leverages Apache Spark to optimize the vendor matching algorithm using techniques such as Local Similarity Hashing, Cosine Similarity, and Euclidean distance theorem which quickly processed millions of records and assigned a similarity score for each record. This approach clustered vendor names using cosine similarity into smaller groups with lower degrees of accuracy. This ensures a smaller likelihood of missing potential matches. The Euclidian distance of these clustered vendors provides a “smaller” group (&lt; 3.6 trillion) and compares in greater detail by generating more features and using the cosine similarity approach.
This effort established data pipelines to ingest financial data on a regular cadence into a vendor identification and mapping algorithm to correct and detect deficiencies, enforce compliance and governance, and report new findings. FMDDT and our partners are leaning forward to increase the number of customer-focused, innovative, and easy-to-use data visualizations and tools.



NAML 2022

Vendor Name Fuzzy Matching using Spark

spark

data science

text similarity

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

The Financial Management Systems (FMS) Financial Management Data & Digital Transformation (FMDDT) Division under the Assistant Secretary of the Navy’s Financial Management and Comptroller’s Office (ASN FM&C) along with Deloitte GPS (Government & Public Services), has developed a data science solution for automating the mapping of financial management systems’ vendor data across several systems, identifying erroneous and missing data, and detecting duplicate information.
FMDDT is working with our partners to establish clear data governance in the ingestion, use, and modeling of data. We are working in close partnership with DON CDO and OSD’s ADVANA to leverage tools, data streams, and the analytics environment while establishing governance policies that foster trust in data through consistency and standardization.
As the Navy moves all financial systems under the Navy Enterprise Resource Planning (ERP) system there will be a need to improve the data quality and mapping as it is integrated from Navy and Marine Corp Standard Accounting, Budgeting, and Reporting System (SABRS) and Standard Financial & Reporting System – Field Level (STARS-FL).
The current Navy ERP vendor master list needed consolidation and mapping across SABRS and STARS-FL to provide insight into budget and financial analytics. As these systems migrate into a single financial system in FY22, an understanding of the historical data and how it maps between systems will be critical to the development of future data analytics and machine learning applications.
While using the System for Award Management (SAM) data as the authoritative data source and vendor naming convention, the Navy ERP, SABRS, and STARS-FL vendor names were matched to the SAM data source. The effort entails matching almost 3 million unique ERP, SABRS, and STARS-FL vendor name records against millions of SAM vendor name records resulting in over 3 trillion unique comparisons.
The effort used the Jupiter Databricks environment which leverages Apache Spark to optimize the vendor matching algorithm using techniques such as Local Similarity Hashing, Cosine Similarity, and Euclidean distance theorem which quickly processed millions of records and assigned a similarity score for each record. This approach clustered vendor names using cosine similarity into smaller groups with lower degrees of accuracy. This ensures a smaller likelihood of missing potential matches. The Euclidian distance of these clustered vendors provides a “smaller” group (< 3.6 trillion) and compares in greater detail by generating more features and using the cosine similarity approach.
This effort established data pipelines to ingest financial data on a regular cadence into a vendor identification and mapping algorithm to correct and detect deficiencies, enforce compliance and governance, and report new findings. FMDDT and our partners are leaning forward to increase the number of customer-focused, innovative, and easy-to-use data visualizations and tools.



technical paper

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community



A rapidly approaching issue is a lack of decision support for Test and Evaluation, Verification, Validation (TEVV/LVC) facilities that have difficulty replicating adequate complexity and are therefore challenged with creating “realistic” battle engagement test scripts for autonomous products teaming/interacting with humans. The Automated White Cell (TAWC) is based on multi-model data integration, using AI/ML/Game Theory solutions, enhancing the predictive fidelity of TEVV/LVC facilities in assessing Autonomous technology strengths and weaknesses. The TAWC adjudication approach will be able to standardize existing TEVV/LVC facilities by providing a cost effective, common TEVV/LVC simulation environment through interface adaptation. The end goal is for TAWC to test autonomous systems that are designed to dynamically learn and adapt during a manned-unmanned teaming operation. TAWC is developed to provide real-time decision support and test script manipulation in the form of adjudication based on the simulation of events or discrete time. This research describes how TAWC is designed to create simulated environments augmenting TEVV/LVC facility capabilities. The design structure focuses on modeling manned and unmanned teaming opposing players that include: (1) Mixed manned and unmanned synchronous operations over complex battle engagement exchanges; (2) Varying offensive and defensive kinetic and non-kinetic assets moving at varying speeds and lethality; (3) Deception and counter deception techniques and related technology; (4) Human to autonomy interaction “teamwork” challenges; (5) Cyber tactical mixes; (6) Cyber vulnerabilities associated with security operations; (7) Logistics requirements for replenishment and repair; and (8) Crew readiness human factors, with the goal of representing greater realism and potential deployment challenges within a simulated environment.

White cell automated adjudication for TEVV/LVC facilities in assessing Autonomous technology strengths and weaknesses

Abstract: The goal of the proposed research is to make a theoretical contribution to understanding artificial intelligence/machine learning (AI/ML) in a naval fleet architecture to speed up the decision-making process and show commanders confidence in the predictions within a command and control (C2) headquarters afloat or ashore. Predictability means that models are reliable and data are representatively complete. The end state is merging the Future Knowledgebase System of Systems (FKSS) with AI/ML Engines to maximize system reliability and data availability for decision support. This knowledge architecture can offer a diffusion of ideas in an adaptive learning environment. The naval fleet unit will need to dynamically adapt its System of System (SoS) reliability and availability management methods with AI/ML agents. These analytic data problems involve three dimensions to meet the challenges—People, algorithms, and machines. The decision-makers can address overall risk management values, and the AI/ML algorithms can adjust to the individual machines and their data. The overall reliability and availability of the FKSS will be gauged by the ability to access individual systems when required to migrate authoritative data into one location or a few corresponding locations where AI/ML can act on it. The "gauge" is assisted by information sciences, data science, and risk management.


Maximizing Data Capabilities for Decision Making with Artificial Intelligence (AI) / Machine Learning (ML) in a Naval Future Knowledgebase System of Systems (FKSS)

GTRI is deploying Artificial Intelligence (AI) and Machine Learning (ML) via a model-based systems engineering (MBSE) approach to optimize aircraft maintenance scheduling and risk management in DoD and commercial applications. Leveraging big data analytics and multivariable optimization methods, GTRI is coupling industry best practices with AI/ML to maximize aircraft readiness and minimize costs at global enterprise scale.

Understanding relationships between aggregated aircraft maintenance and routing data is critical to inform and automate business decisions in aerospace operations. GTRI is using AI/ML methods, particularly random forest regression and classification, on curated DoD and commercial data sets to predict maintenance costs and operational risks of aircraft assignments on flown and unflown flight schedules. By correlating aircraft risk values, maintenance resource requirements, and fleet reliability metrics, GTRI is developing an MBSE logistics framework and model to optimize readiness costs and profits/losses in a near-real time, automated, and dynamic manner.   

Specific technical material will include data collection, exploration, and processing, AI/ML technique selection (e.g., deciding between elastic net regularized regression, gradient boosting, and random forest), and advanced scheduling theory algorithms such as P||Cmax to minimize operational loads on aircraft.

This model will provide operations managers/planners recommended schedules and quantitative data to globally optimize the tactical execution of operations. Additionally, the models wholistic insight into operations and logistics complexities enable innovation, experimentation, and simulation for Condition Based Maintenance, irregular operations, and integration of applied technologies and commercial best practices.  
    
GTRI’s efforts directly support the Joint AI Center's Joint Logistics National Mission Initiative and commercial partnerships, specifically with Delta Air Lines, on advancing industry standards on maintenance and reliability programs (e.g, Maintenance Steering Group-4). Specific opportunities to apply this research to Naval/Marine aviation fleets in collaboration with NAVAIR will be discussed. Other GTRI research referenced in support of this abstract will include AI/ML virtual reconstruction of H-60 IVHMS data and AI/ML characterization and modeling of rocket plume performance.


AI Driven Cost-Wise Readiness: Machine Learning Applications in Aircraft Maintenance Scheduling Optimization and Risk Management

The Naval Safety Center has shown in previous modeling efforts that longitudinal analysis of personnel experience can indicate at-risk units. However, data acquisition processing delays result in out-of-date input data, which leads to time-late and, sometimes, misleading predictions. Previous NAML presentations have discussed the high correlation amongst chronological longitudinal experience features, and the unique challenges these types of features present when fitting regression models. However, this multicollinearity is an asset for generating reliable predictions of feature values in future time periods with the added inclusion of a decision model that considers patterns in the variation. This paper goes through the process of building an ensemble model that predicts feature values for future time periods, and then uses those predicted feature values as inputs for risk predictions. This paper also shows that accurate modeling predictions are possible for future time periods, even when that future time period’s input data is not yet available.

Ensemble Machine Learning to Facilitate High Confidence Chronological Augmentation of Input Data Sets Used for Predictive Risk Modeling

In this presentation, we will explore the general challenges to developing predictive analytics for mission readiness applications, from data sourcing to interface design, and how a contested logistics environment can alter the technical requirements and architectural considerations.

Improving Mission Readiness using AI/ML Enabled Predictive Analytics

Understanding and improving readiness is a critical task for Naval Aviation. The Digital Aviation Readiness Technology Engine (DARTE) is a proven paradigm for predicting readiness metrics such as mission capable (MC) status and expected quarterly flight hour (FH) execution for F/A-18 squadrons. The next generation of DARTE implements cutting-edge AI/ML techniques such as Long Short Term Memory networks with Attention and Hyper Deep Ensembles to extend MC prediction time horizons. We explore adding new capabilities to DARTE by utilizing Topological Data Analysis to yield new data insights, Natural Language Processing/Understanding to incorporate new data sources, Reinforcement Learning to recommend maintainer movements across the fleet, and expanding the MC prediction model to the MH-60. Additionally, we show how the products of these models can be delivered to the warfighter.

Daily Readiness Predictions with the Digital Aviation Readiness Technology Engine

Coding medical data involves analyzing unstructured text fields, including lengthy clinical narratives, physician and nurse notes, lab reports, discharge summaries, scanned documents, and medications. These clinical data are found in Subjective, Objective, Assessment, and Plan (SOAP) notes. Unstructured text data typically requires extensive resources to annotate, describe, analyze, and convert into meaningful and actionable information. This research project’s long-term objective is to transition translated coded data using machine learning and natural language processing into an automated clinical encounter repository to augment the clinical repositories that have missing or miscoded data. Machine learning model development involved a two-step process. The initial model was obtained using 11,304 records from the Expeditionary Medical Encounter Database (EMED) data and matched to the TMDS data which resulted in an overall accuracy of 73% when estimating 3-digit codes just using the clinical data. An additional step included the true positive cases from the TMDS data and machine learning predictions which significantly improved the accuracy and number of records from 73% to 89% and 11,304 to 44,257 records. The final assessment of the machine learning model and the main objective of this abstract will be to describe the accuracy of the 10,286 records from the TMDS that do not have a valid ICD-9 code. Preliminary results indicate that several thousand records will be automatically coded at the 3-digit level. A random selection of records will be reviewed and described. The limitation of this study is the lack of training data, which limits the prediction of billable ICD-9 codes. However, providing 3-digit ICD-9 codes is still a major finding that will greatly assist medical coders and augment the medical data repositories such as EMED and TMDS. 

Disclaimer: I am a military service member or employee of the U.S. Government. This work was prepared as part of my official duties. Title 17, U.S.C. §105 provides that copyright protection under this title is not available for any work of the U.S. Government. Title 17, U.S.C. §101 defines a U.S. Government work as work prepared by a military service member or employee of the U.S. Government as part of that person’s official duties.
This work was supported by Defense Medical Research and Development Program under work unit no. N1214. The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, nor the U.S. Government. 
The study protocol was approved by the Naval Health Research Center Institutional Review Board in compliance with all applicable Federal regulations governing the protection of human subjects. Research data were derived from an approved Naval Health Research Center, Institutional Review Board protocol number NHRC.2003.0025.


Development of the Automated Clinical Encounter Repository: Using natural language processing algorithms to medically code clinical notes in the electronic health record

ARISE - Using Similarity for Recommendations
Scott, Alicia

Data analysts such as Reliability Engineers and Operations Research Analysts
manually group records to identify trends and summarize subsets of their
data. However, when datasets grow beyond human capability to effectively
track and organize, the number of errors and time to analyze data also
increase. To reduce human error and manual work, the Automated
Recommendation Investigator for Systems Engineering (ARISE) was developed to
identify similarity between records within a dataset to assist in providing
possible solutions to maintenance records using Natural Language Processing
(NLP) and machine learning methods. Using specific data preprocessing
techniques for text data as well as other data types separately and cosine
similarity ranking, ARISE is able to find potential solutions from already
created maintenance records with a full perspective of the data and not just
a few or single column sorting.

ARISE - Using Similarity for Recommendations

The Operational Safety Risk Indicator (OSRI) project, led by Commander, Naval Surface Forces (CNSF), seeks to aggregate disparate data sources across the Navy into DoN Jupiter, which is the Navy’s enterprise data environment enclave of Advana, and deploy predictive risk analytics/modeling to identify significant risk factors and enable better decision-making across all echelons of leadership. By leveraging advanced machine learning and artificial intelligence methodologies, coupled with robust data visualization tools, the OSRI project seeks to enhance operational safety and readiness across the Surface Force. 
Although the there is an abundance of relevant enterprise data to serve the OSRI project, the team has tapped into one of the most pertinent and relevant data sources available to date: Tactical Messages. These messages provide a wealth of information across many naval operations, including logistics, operations, personnel, medical, and maintenance. One such form of information within Tactical Messages that directly relates to operational safety risk indicators is destructive behaviors data. Currently this information is manually extracted in multiple ways from an enormous amount of Tactical Message traffic. How this information is extracted also depends on the command or operational function, which creates data management concerns. In addition, Tactical Messages vary widely in format, are largely unstructured, and may contain errors due to manual entry. Not only does the current data aggregation process result in redundancies and potential information loss, but it inhibits the ability to make real-time decisions and analysis, thus potentially impacting the Navy’s operational safety risk posture. As a result, the OSRI team developed a machine learning application that will enable faster processing and more timely delivery of insights drawn from a wealth of unstructured data. This application takes the raw Tactical Message data as input and—using Natural Language Processing techniques—classifies the type of destructive behaviors incident. This classifier provides a predictive capability that not only enables robust analytics in a streamlined fashion, but also helps to identify the factors that contribute to the occurrence of these events. Through timely delivery of this actionable information to leadership, the Navy can make better informed decisions that enact effective counter measures against destructive behaviors. The long-term gain for such a system is enhanced operational safety and improved performance outcomes.

Operational Safety Risk Indicators

The term “insider threat” can take many forms, ranging from an information security risk to the threat of an active shooter. Accordingly, it is beneficial to researchers and practitioners to understand the relationship between psychological factors and the different types of threats an insider may pose to an organization. This research advances this understanding. Specifically, we investigate the three-way relationship between user-generated text, psychological factors espoused in insider threat literature, and risk indicator categories used by the U.S. Government. We employ advancements in machine learning and Natural Language Processing to investigate this relationship. Specifically, we use Bidirectional Encoder Representations from Transformers (BERT) for word embedding and vector space modeling. Our results indicate that there are indeed associations between established risk categories and the psychological factors seen as predictive of malicious insiders. Our exploratory research also reveals that further research is warranted to advance the predictive capability of insider threat modeling.

Premium content

Vendor Name Fuzzy Matching using Spark

Downloads

Next from NAML 2022

White cell automated adjudication for TEVV/LVC facilities in assessing Autonomous technology strengths and weaknesses

Similar lecture

NAVWAR Digital Transformation Learning Continuu

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES