United States

&quot;With the rapid progression of technology, it becomes difficult for individuals to navigate the amount of data and systems at their disposal. To lessen the burden, technology can be leveraged to create an ambient intelligent environment where humans can naturally utilize systems with little prior training.  In this work, we demonstrate various technical applications that contribute to the effort of creating an environment that supports users by providing systems that enhance their performance. The features we developed allow the user to interact and control systems via hand gestures, action recognition, and verbal cues.

The capabilities have been implemented using various machine learning models and frameworks. For example, hand gesturing has been implemented with the MediaPipe posing framework, which allows for key point estimation on live camera feed. These key points allow us to identify the position of an individual’s hand in a relative space and can use this data to create interactive actions between the user and the system. Action recognition data is generated via an action classification model. This model estimates confidence scores for relevant action classification labels based on the real-time camera footage it receives as input. This data provides our system with more context about the individuals in our command centers, allowing us to tailor system interactions to specific actions. Lastly the verbal cues are implemented using a live transcript utilizing an automatic speech recognition model to transcribe live audio. This is fed into a dialogue manager which then determines the intent of the speaker using an intent classification model, which will determine what actions the dialogue manager should take or how the conversation should continue. Our goal is to expand our Human Computer Interaction (HCI) capabilities in order to achieve a fully ambient intelligent environment, which would allow its users to easily interact with the system and provide a more effective interactive experience.&quot;

NAML 2023

Developing Interactive Ambient Intelligent Environments

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

**Watch this video to hear about the AFCEA purpose.**

<iframe width=1280 height=706 frameborder="0" scrolling="no" src="https://screencast-o-matic.com/player/creleVVcvRh?v=6&ff=1&title=0&controls=1" allowfullscreen="true"></iframe>

NAML 2021

"With the rapid progression of technology, it becomes difficult for individuals to navigate the amount of data and systems at their disposal. To lessen the burden, technology can be leveraged to create an ambient intelligent environment where humans can naturally utilize systems with little prior training.  In this work, we demonstrate various technical applications that contribute to the effort of creating an environment that supports users by providing systems that enhance their performance. The features we developed allow the user to interact and control systems via hand gestures, action recognition, and verbal cues.

The capabilities have been implemented using various machine learning models and frameworks. For example, hand gesturing has been implemented with the MediaPipe posing framework, which allows for key point estimation on live camera feed. These key points allow us to identify the position of an individual’s hand in a relative space and can use this data to create interactive actions between the user and the system. Action recognition data is generated via an action classification model. This model estimates confidence scores for relevant action classification labels based on the real-time camera footage it receives as input. This data provides our system with more context about the individuals in our command centers, allowing us to tailor system interactions to specific actions. Lastly the verbal cues are implemented using a live transcript utilizing an automatic speech recognition model to transcribe live audio. This is fed into a dialogue manager which then determines the intent of the speaker using an intent classification model, which will determine what actions the dialogue manager should take or how the conversation should continue. Our goal is to expand our Human Computer Interaction (HCI) capabilities in order to achieve a fully ambient intelligent environment, which would allow its users to easily interact with the system and provide a more effective interactive experience."

poster

### Welcome to the 7th annual workshop on Naval Applications of Machine Learning!

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

* To spread awareness of current machine learning research relevant to Naval applications

* To connect machine learning researchers with experts in military requirements

* To build and strengthen collaborations within the defense research community

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

**General Chairs:**

Katie Rainey, Naval Information Warfare Center Pacific

Nicole Isoda, Naval Information Warfare Center Pacific<br><br>

<button name="button" onclick="window.open('https://sites.google.com/go.spawar.navy.mil/naml/','blank')">**NAML 2023 Website**</button>

<br><br><br>

To access the site please register [**here**](https://events.r20.constantcontact.com/register/eventReg?oeidk=a07ejmzkc5g71584a82&oseq=&c=&ch=). 

The 7th annual workshop on Naval Applications of Machine Learning is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team. The goals of NAML are to spread awareness of current machine learning research relevant to Naval applications, to connect machine learning researchers with experts in military requirements and to build and strengthen collaborations within the defense research community.


Keynote Talk: Dr. Kimberly Sablon

"Abstract for the 2023 NIWC Pacific 
Seventh Annual Workshop on Naval Applications of Machine Learning:

The Strategic and Technical Landscape in 2023


Our briefing will provide a summary of the strategic and technical landscape facing NIWC Pacific, our government, academic and industry partners, as well as the DoD and the DoN in 2023. The purpose of this briefing will be to provide a stage-setter and strategic, operational and technical grounding for the discussions throughout the 2023 NIWC Pacific Naval Applications of Machine Learning Workshop.

As an organizing high concept for this briefing, it is our belief as well as that of our colleagues in the military, industry, academia and the NRDE community that if we know who our warfighters will likely have to take on, and where they will fight them, and we have a sense of when and why this fight might begin, and understand how our warfighters will work to defeat an enemy, then we will know what C4ISR, cyber, and space platforms, systems, sensors and weapons NIWC Pacific needs to deliver to our warfighters today and tomorrow. Further, we will have a strong sense of the role big data, artificial intelligence and machine learning will play in providing our warfighters with an asymmetric advantage.

Our briefing will consist of three primary segments:

•        An overview of the strategic environment, especially the dramatic change in the United States strategic calculus as articulated in the National Security Strategy, the National Defense Strategy and the Nation’s Tri-lateral Maritime Strategy, Advantage at Sea. We will address peer competitors such as China and Russia, and other threats as articulated in the “2+3” national security construct.

•        A “burning platform” overview of the strides our principal adversaries are making in big data, artificial intelligence and machine learning. This will provide an objective review regarding how crucial it is for the DoD and the DoN to advance U.S. government efforts to fully leverage big data, artificial intelligence and machine learning for military applications. We will highlight the Report on the National Security Commission on Artificial Intelligence, as well as statements by SECDEF, DEPSECDEF and other senior officials highlighting the importance of these technologies.

•        Finally, we will address what big data, artificial intelligence and machine learning can do for U.S. warfighters. It is not, as some worry about, to launch armed autonomous weapons downrange without a human in the loop. Rather, as the JAIC Director, Chief of Naval Operations, and others have emphasized, the primary thing that big data, artificial intelligence and machine learning for military applications will do for the DoD and the DoN will to enhance speed of decisions. 

We have mined all relevant information regarding DoD and DoN plans to leverage big data, artificial intelligence and machine learning to support U.S. military platforms, systems, sensors and weapons, and especially the plans evolving under the stewardship of the DoD’s Joint Artificial Intelligence Center (JAIC).

Far from a dry recitation of threats, this briefing will energize NAML attendees and motivate them to accelerate their efforts in their day-to-day work with big data, artificial intelligence and machine learning. Since this briefing will be a stage-setter for the workshop, we recommend we brief early on day one of the event.



"

The Strategic, Operational and Technical Landscape in 2023

The future of Naval Aviation is unmanned and eventually autonomous. This presentation focuses on the generation of certification for a ML algorithm to complete a task without a human in or on the loop: The autonomous aerial refueling task. With the support and cooperation of the National Airworthiness Council Artificial Intelligence Working Group I have published multiple academic papers, white papers, and several presentations on this topic. This presentation highlights my previous work area, and details a possible path forward. Standards and methods of compliance do not exist for certification officials to allow ML algorithms to operate without human oversite. Until these can be established use of ML without a human in or on the loop will remain a demonstration vice a fleet fielded system.

Standards and Methods of Compliance for Autonomous Unmanned Aerial Refueling

"Baseline efforts in Performance in Noise indicated that tasks performed in a tactical situation break down when speech intelligibility decreases. Speech intelligibility decrements can be caused by excessive background noise or cluttered audio channel communications. The objective of this effort is to investigate the development of a communication aid using combined benefits of speech recognition, speech analysis, and Machine Learning (ML) tools for enhancing and filtering communications to achieve high speech intelligibility even when communications are poor due to the addition of noise and cluttered comms. The benefit to this technology is not only improved communications in hard to hear situations, but also allows for the appropriate use of hearing protection without sacrificing ability to hear verbal communications. 
The purpose of this effort is to investigate the development of assistive technology through speech recognition and Machine Learning (ML) tools to enhance and filter communications for Navy specific environments and test the technology on the ability of Sailors to successfully complete tasks in low speech intelligibility conditions. Traditional commercial off the shelf (COTS) speech recognition algorithms are not effective with the typical sentence structure of military communications. Our current efforts are building upon previous successes to train models capable of parsing and understanding our Warfighters to provide a useful assistive technology for improving their ability to communicate in lower speech intelligibility environments.
"

Performance in Noise, Automated Speech Recognition Communications Aid

"Concerns of trustworthiness, unintended social
harms, and unacceptable social and ethical viola-
tions undermine the promise of ML advancements.
Moreover, such risks in complex ML-driven sys-
tems are often difficult to foresee, as many only
arise over periods of time, across populations, and
at scale. These risks emerge through interactions
amongst ML development choices, the context of
model use, environmental factors, and the effects
of a model on its target. Systems safety engi-
neering is an established discipline with a proven
track record of identifying and managing such
risks in high-complexity sociotechnical systems.
In this work, we apply a state-of-the-art systems
safety approach to concrete ML applications with
notable social and ethical risks to demonstrate a
systematic means to argue for safe and trustwor-
thy ML in sociotechnical systems."

Applying Real Safety to Machine Learning Problems

The task of parsing, reading and analyzing the textual and structured information contained in scanned/digital-born documents, namely Visual Document Understanding (VDU) enabled by Optical Character Recognition (OCR), is an incredibly challenging task due to the diversity of layouts and formats, the complexity of template structures, and lack of support for low resource languages, dialects, and scripts. Traditionally, methods for VDU/OCR have relied on rule-based or conventional machine learning approaches, which does not only produce inferior results but also is extremely hard to generalize and fine-tune the solution for documents of interest. To address these shortcomings, our team proposes a VDU/OCR solution enabled by a unified text-image multimodal transformer to learn cross-modal representations. Our approach leverages pre-trained model families such as Language-Independent Layout Transformer (LiLT) a for both text-based document understanding using language models, and image-based document understanding using Visual Transformers, and fine-tune them for the languages, scripts, and document formats of interest. 

Deep Learning for Visual Document Understanding in Low Resource Languages

While the recent boom in Artificial Intelligence (AI) has given rise to the technology’s use and popularity across many domains, the same boom has exposed vulnerabilities of the technology to many threats that could cause the next “AI winter." AI is no stranger to “winters," or drops in funding and interest in the technology and its applications. Many in the field consider the early 1970’s as the first AI winter with another proceeding in the late 1990’s and early 2000’s. There is some consensus that another AI winter is all but inevitable in some shape or form, however, current thoughts on the next winter do not consider secure and robust AI and the implications of the success or failure of these areas. The emergence of AI as an operational technology introduces potential vulnerabilities to AI’s longevity. The National Security Commission on AI (NSCAI) report outlines recommendations for building secure and robust AI, particularly in government and Department of Defense (DoD) applications. However, are they enough to help us fully secure AI systems and prevent the next “AI winter”? An approaching “AI Winter” would have a tremendous impact in DoD systems as well as those of our adversaries. Understanding and analyzing the potential of this event would better prepare us for such an outcome as well as help us understand the tools needed to counter and prevent this “winter” by securing and robustifying our AI systems. In this talk, Harguess will introduce the following four pillars of AI assurance, that if implemented, will help us to avoid the next AI winter: security, fairness, trust, and resilience.

Is the Next Winter Coming for AI? Elements of Making Secure and Robust AI

"Social media and conversational messaging have created the largest untapped asset for public health enrichment and commercialization. This data is unstructured which makes it difficult to analyze with conventional technologies. Current natural language analytic methodologies are static, focusing on discerning what conversations are about. The next generational leap in analyzing human language is temporal trajectory modeling, offering the ability to both understand and predict where a conversation is going in the future. Temporal trajectories could enable the automated tracking of a person’s mental or emotional state over time to predict trends and outcomes like acts of aggression, suicide, attention deficit disorders, bipolar disorders, schizophrenia, and the early onset of Alzheimer’s. The goal is to show that these temporal trajectories can be recognized and predicted with accuracy that matches or improves upon human interpretations of the same data. The empirical evidence generated from this research will support the ultimate goal of early detection and/or prediction of future behaviors or mental health conditions. 

At the core of the approach being considered is the use of document vectors to model sentence similarities over time to track the trajectories of human thought and predict future behaviors and outcomes. The use of document vectors has already been proven to achieve high accuracy, particularly in the classification of documents. Document vectorization enables the classification of a discrete piece of a conversation with no relation to its entirety or forward path. A temporal trajectory classification model is an umbrella-like technology that predicts the trajectory of these discrete elements.  In essence, we will be creating a new ensemble learning model from discrete models that generate predictive outcomes and mindset directionality.  Key hurdles include the ability to incorporate linear proximity models into a multi-dimensional space while predicting semantic trajectories. This approach represents the next evolutionary step in natural language comprehension in the fields of mental health, public safety, national security and law enforcement.

The outcomes from the successful creation and deployment of such a technology could have far-reaching positive societal impacts, as well as strong commercial value. Imagine the improvements (saved lives, improved outcomes, efficiencies) that could be made in the diagnosis and treatment of a wide range of mental health issues plaguing our communities and institutions including suicidal ideation, radicalization, and homicidal ideation. Additionally, this research will establish a foundational technology for a wide range of other use cases including the trajectory of corporate communications to predict future bankruptcies, risk of non-compliance, shifts in employee morale, hostile work environments, corporate malfeasance, etc."

Mindset Trajectory Tracker

As AI continues to become part of our lives, it will become essential for people and AI to collaborate effectively. A key problem with current-generation AI is that it cannot always be trusted, and users do not know when they can or should trust it. In this presentation, I'll talk about the Trusted AI project, which is a major 3-year  collaboration between Indiana University, Notre Dame, Purdue University, and Naval Surface Warfare Center Crane Division. The project has two goals: to conduct novel research related to understanding how to improve AI and how to help people know when to trust (and not to trust) AI, and to develop the AI workforce by educating students at all levels about AI and security. The project involves 15 faculty PIs at the 3 universities and almost 50 students at all levels. 

Premium content

Developing Interactive Ambient Intelligent Environments

Next from NAML 2023

Keynote Talk: Dr. Kimberly Sablon

Similar lecture

Machine Learning Provides SeaVision with Maritime Behavior Recognition

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES