United States

The task of parsing, reading and analyzing the textual and structured information contained in scanned/digital-born documents, namely Visual Document Understanding (VDU) enabled by Optical Character Recognition (OCR), is an incredibly challenging task due to the diversity of layouts and formats, the complexity of template structures, and lack of support for low resource languages, dialects, and scripts. Traditionally, methods for VDU/OCR have relied on rule-based or conventional machine learning approaches, which does not only produce inferior results but also is extremely hard to generalize and fine-tune the solution for documents of interest. To address these shortcomings, our team proposes a VDU/OCR solution enabled by a unified text-image multimodal transformer to learn cross-modal representations. Our approach leverages pre-trained model families such as Language-Independent Layout Transformer (LiLT) a for both text-based document understanding using language models, and image-based document understanding using Visual Transformers, and fine-tune them for the languages, scripts, and document formats of interest. 

NAML 2023

Deep Learning for Visual Document Understanding in Low Resource Languages

technical paper

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


While the recent boom in Artificial Intelligence (AI) has given rise to the technology’s use and popularity across many domains, the same boom has exposed vulnerabilities of the technology to many threats that could cause the next “AI winter." AI is no stranger to “winters," or drops in funding and interest in the technology and its applications. Many in the field consider the early 1970’s as the first AI winter with another proceeding in the late 1990’s and early 2000’s. There is some consensus that another AI winter is all but inevitable in some shape or form, however, current thoughts on the next winter do not consider secure and robust AI and the implications of the success or failure of these areas. The emergence of AI as an operational technology introduces potential vulnerabilities to AI’s longevity. The National Security Commission on AI (NSCAI) report outlines recommendations for building secure and robust AI, particularly in government and Department of Defense (DoD) applications. However, are they enough to help us fully secure AI systems and prevent the next “AI winter”? An approaching “AI Winter” would have a tremendous impact in DoD systems as well as those of our adversaries. Understanding and analyzing the potential of this event would better prepare us for such an outcome as well as help us understand the tools needed to counter and prevent this “winter” by securing and robustifying our AI systems. In this talk, Harguess will introduce the following four pillars of AI assurance, that if implemented, will help us to avoid the next AI winter: security, fairness, trust, and resilience.

Is the Next Winter Coming for AI? Elements of Making Secure and Robust AI

"Social media and conversational messaging have created the largest untapped asset for public health enrichment and commercialization. This data is unstructured which makes it difficult to analyze with conventional technologies. Current natural language analytic methodologies are static, focusing on discerning what conversations are about. The next generational leap in analyzing human language is temporal trajectory modeling, offering the ability to both understand and predict where a conversation is going in the future. Temporal trajectories could enable the automated tracking of a person’s mental or emotional state over time to predict trends and outcomes like acts of aggression, suicide, attention deficit disorders, bipolar disorders, schizophrenia, and the early onset of Alzheimer’s. The goal is to show that these temporal trajectories can be recognized and predicted with accuracy that matches or improves upon human interpretations of the same data. The empirical evidence generated from this research will support the ultimate goal of early detection and/or prediction of future behaviors or mental health conditions. 

At the core of the approach being considered is the use of document vectors to model sentence similarities over time to track the trajectories of human thought and predict future behaviors and outcomes. The use of document vectors has already been proven to achieve high accuracy, particularly in the classification of documents. Document vectorization enables the classification of a discrete piece of a conversation with no relation to its entirety or forward path. A temporal trajectory classification model is an umbrella-like technology that predicts the trajectory of these discrete elements.  In essence, we will be creating a new ensemble learning model from discrete models that generate predictive outcomes and mindset directionality.  Key hurdles include the ability to incorporate linear proximity models into a multi-dimensional space while predicting semantic trajectories. This approach represents the next evolutionary step in natural language comprehension in the fields of mental health, public safety, national security and law enforcement.

The outcomes from the successful creation and deployment of such a technology could have far-reaching positive societal impacts, as well as strong commercial value. Imagine the improvements (saved lives, improved outcomes, efficiencies) that could be made in the diagnosis and treatment of a wide range of mental health issues plaguing our communities and institutions including suicidal ideation, radicalization, and homicidal ideation. Additionally, this research will establish a foundational technology for a wide range of other use cases including the trajectory of corporate communications to predict future bankruptcies, risk of non-compliance, shifts in employee morale, hostile work environments, corporate malfeasance, etc."

Mindset Trajectory Tracker

As AI continues to become part of our lives, it will become essential for people and AI to collaborate effectively. A key problem with current-generation AI is that it cannot always be trusted, and users do not know when they can or should trust it. In this presentation, I'll talk about the Trusted AI project, which is a major 3-year  collaboration between Indiana University, Notre Dame, Purdue University, and Naval Surface Warfare Center Crane Division. The project has two goals: to conduct novel research related to understanding how to improve AI and how to help people know when to trust (and not to trust) AI, and to develop the AI workforce by educating students at all levels about AI and security. The project involves 15 faculty PIs at the 3 universities and almost 50 students at all levels. 

Trusted AI Research and Workforce Development

K-Means Based Automated Behavior Recognition Using Random Forest Learning

"The San Diego Supercomputer Center was founded in 1985 as one of two national centers for supercomputing research and applications. It's mission is to ""translate innovation into practice"". Innovations range across computing, information, and data sciences, broadly defined, and practice includes any and all domains, with a special focus on STEM fields in addition to solving pressing societal challenges.
In this talk we will provide a brief overview of the center, followed by details on some of its AI & ML systems and activities. "

AI & ML at the San Diego Supercomputer Center (SDSC)

"Automatically detecting and classifying objects in remote sensing imagery can significantly improve intelligence analysis efficiency and throughput. Object detectors are typically trained on a specific set of object classes aligned with the intelligence problem of interest. The detection of unexpected, unusual, or novel objects is also a core intelligence task routinely performed by imagery analysts, but current detectors cannot perform this task. Novel objects are usually ignored by the detector, if they are not sufficiently similar to any known class, or are misclassified as a known class.

Novel object detection addresses this capability gap by discovering objects that are not within the set of known classes but are of intelligence significance. Remote sensing images often contain dozens or hundreds of object types, many of which are not in the training data. Many of these objects are irrelevant, while others may be quite important. Our method automatically alerts analysts to novel objects that are similar to the training set but are sufficiently different such that they may be a new class or a novel variant of a known class. Balancing these two constraints enables the detection of salient novelties, such as a new type of wing on a previously-known aircraft model, without overwhelming the analyst with nuisance novelties such as changes in vegetation.

The xView dataset contains WorldView-3 imagery with 60 object classes labeled in a two-level ontology, with 8 general classes in the top level, such as “Truck”, and 52 specific classes in the lower level, such as ""Truck with Flatbed"". We adapted this dataset for realistic novelty detection by considering 18 lower level classes as novel and removing images containing them from the training set. These novel classes are distributed across all 8 of the general categories. During testing, the system should detect instances from these 18 categories as novel objects, while also detecting all instances from the 34 known classes, declaring them to be non-novel, and correctly classifying them.

Our method is mathematically grounded in Extreme Value Theory (EVT) and utilizes the Extreme Value Machine (EVM) that we developed on the DARPA Science of Artificial Intelligence and Learning for Open-world Novelty project. EVM creates an EVT distribution over the distances between deep-learned feature embeddings for each class to provide a novelty score. We train an object detector, classifier, and EVM on the known classes, then estimate the novelty of each detection at test time. On XView this approach achieves a 16% AUC increase over a baseline method that thresholds the classifier scores.

Our novelty detection method can be used in various applications of interest to the Navy, such as detecting novel aircraft classes or attributes on the ground, in the air, or on a carrier ship; novel ship or vehicle detection; or detection of novel aircraft attributes, including novel ammunition or damage.
"

Novel Object Detection in Remote Sensing Images for Naval Applications

Two requirements for maintaining our lead in artificial intelligence (AI) are widespread development and the adoption of AI applications.  Both are risky and not yet common. However we have achieved them with many large organizations including Pfizer, Schlumberger, and GE Aviation.  We call it “Everyday AI”: embedding AI development and apps in everyday tasks and processes throughout an enterprise.  A key challenge is balancing data/AI/ML decentralized development with governance.  If there’s too much decentralization then apps aren’t trusted or adopted.  If there’s too much governance then decentralization doesn’t take hold and everyday tasks aren’t addressed.  This presentation covers eight “plays” organizations can run to achieve Everyday AI, a few of which are a data/AI/ML literacy program for all skill levels, catalog the top 20% data/AI/ML products as-is, and an industrial-scale AI operations automation workbench.

Everyday AI Playbook

Monitoring potential maritime powers and/or adversaries is essential to the Department of Defense’s (DoD) mission and to the United States’ security. An essential tool for accomplishing this is the use autonomous object detection in electro-optical imagery. Currently, this is done by way of supervised machine learning. This approach has proven to be effective when applied to detection of objects in the labeled dataset, but struggles when tasked to detect objects that are not in the labeled dataset. Additionally, obtaining and/or generating labeled data is often a time consuming and expensive process. Therefore, our group at Penn State’s Applied Research Laboratory has been exploring the use of unsupervised machine learning for object detection. That is, we want to train a machine learning model to find and classify objects in an image without the use of labeled training data. There has been recent research in the use of variational autoencoders coupled with spatial transformers to accomplish this task on simple datasets. We will demonstrate results of these methods along with relevant data preprocessing algorithms for unsupervised object detection of maritime objects. Our results show promise for the ability to extract objects without the use of labeled training data.

Unsupervised Detection of Naval Objects in Electro-Optical Imagery

"The 21st century has shown the proliferation of algorithms, tools, and code runtimes for machine learning applications. In order to get started, a user or scientist requires a development environment with installed libraries, dependencies to run code, and access to training and test data. This development environment also needs to run on compatible hardware, and have a location to persistently store the source code, model, and output artifacts.

This presentation addresses best practices to create machine learning environments. An exploratory approach was used to find processes, tools, and methods to reduce administrative burden from installing software, and lower the time required to start cleaning, training, and testing data. Users need the capability to share research code and diagrams without conflicting with environment dependencies and packages of other users.

Using these best practices, users will be able to lower the time to get started with machine learning workloads. Reducing setup time allows users to emphasize more of their time on cleaning data, tuning hyperparameters, and improving algorithms. Additionally, by improving the methods to share environments, projects will be able to iterate faster by improving collaboration among their collogues and team."

Methods and Tools for Management of Repeatable Machine Learning Development Environments

The United States coastline spans 95,471 miles; a distance that cannot be effectively patrolled or secured by manual human effort alone. Unmanned Aerial Vehicles (UAVs) equipped with infrared cameras and deep-learning based algorithms represent a more efficient alternative for identifying and segmenting objects of interest - namely, ships. However, standard approaches to training these algorithms require large-scale datasets of densely labeled infrared maritime images. Such datasets are not publicly available and manually annotating every pixel in a large-scale dataset would have an extreme labor cost. In this work we demonstrate that, in the context of segmenting ships in infrared imagery, weakly-supervising an algorithm with sparsely labeled data can drastically reduce data labeling costs with minimal impact on system performance. We apply weakly-supervised learning to an unlabeled dataset of 7055 infrared images sourced from the Naval Air Warfare Center Aircraft Division (NAWCAD). We find that by sparsely labeling only 32 points per image, weakly-supervised segmentation models can still effectively detect and segment ships, with a Jaccard score of up to 0.756.

Premium content

Deep Learning for Visual Document Understanding in Low Resource Languages

Downloads

Next from NAML 2023

Is the Next Winter Coming for AI? Elements of Making Secure and Robust AI

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES