Addressing quality-of-service (QoS) requirements for software applications
in undersea acoustic networks is challenging. This is in
part due to the strong dependency of the characteristics on the
acoustic propagation environment and the lack of effective congestion
control protocols for undersea acoustic networks. In this
work we propose a deep reinforcement learning (DRL) based algorithm
for allocating bandwidth resources across different traffic
types. Bandwidth allocations are periodically updated based on
traffic prioritization, QoS delay requirements, and queue occupancy
levels. A DRL agent learns the optimal bandwidth allocation policy
via a Soft Actor-Critic (SAC) algorithm. In order to test the
effectiveness of our proposed algorithm, we developed a weighted
fair queuing (WFQ) module in NS3 to enable testing in a high fidelity
simulation environment. The WFQ module enables the use of
queueing disciplines together with an NS3 library for underwater
acoustic communications and networking. The interaction between
the DRL agent and the simulated environment is enabled by the
NS3-gym framework. Numerical tests illustrate the performance of
the DRL-based policy and a use case for the new WFQ NS3 module.

Active Queue Management in Underwater Network via Deep Reinforcement Learning

This work explores local congestion control methods for undersea acoustic networks. Congestion control policies are deployed and managed locally at each node acting as a router. Rather than accepting all traffic, intermediate routers can decide not to accept certain packets or to reduce the rate at which they receive packets while resolving the congestion they are experiencing. By not accepting packets, a backlog of packets is built in upstream queues. This backlog will eventually reach the packet sources and, similar to TCP, trigger a traffic rate reduction at the source application. Local packet queue management policies are also responsible for managing queue-space allocations among different traffic classes.These policies are developed based on a financial framework in which each traffic type is associated a commodity that is part of an investment portfolio. Each router, uses its local policy to optimize its portfolio based on the funds it has available, that is, its available data storage space. The space allocation profile is updated periodically via the solution of the celebrated Markowitz Portfolio Selection problem. A price model for packets based on local congestion indicators is used to quantify notions of return, profit and risk. Our framework uses a hybrid policy that combines the space allocation policy with a collection of traffic flow control policies implemented separately per traffic class. The traffic flow control policies are responsible for deciding what flows to pause and restart, and when to requests a reduced packet transmission from a neighboring transmitter. Traffic flow control policies use congestion indicators such as queue occupancy levels per traffic class, packet priority, and sojourn time statistics. These policies are learned via a reinforcement learning framework. The congestion control policy is coupled with a traffic shaping policy implemented at the same node. The traffic shaping policy is responsible for allocating available transmission bandwidth to each traffic type based on traffic priorities and quality-of-service (QoS) requirements. Initially our framework presumes an arbitrary traffic-shaping policy that does not coordinate with the local congestion control policy to decide what packets to prioritize. Numerical tests illustrating the behavior of several components of the proposed method will be discussed.

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Poster Session 2B

poster

##### For full functionality of Underline, please SHUT OFF your VPN, or use a personal device.


Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop.

The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community


<figure class="video_container">
  <iframe src="https://s3.amazonaws.com/pf-user-files-01/u-59356/uploads/2022-03-22/hk13mxo/NAML_ReceptionWithtitle.mp4" width=826  height=526  frameborder="0" allowfullscreen="true"> </iframe>
</figure>

NAML 2022

Naval Information Warfare Center Pacific is partnering with AFCEA San Diego chapter to host the workshop. The NAML workshop is an opportunity for researchers from Navy research and development centers and our partner organizations to exchange technical ideas and to come together as a team.

The goals of NAML are:

To spread awareness of current machine learning research relevant to Naval applications

To connect machine learning researchers with experts in military requirements

To build and strengthen collaborations within the defense research community




**Title:** Reinforcement Learning

**Moderator:**  Christopher Rosser, Naval Information Warfare Center Pacific


               
        


Pedro A. Forero

2

SHORT BIO

Presentations

Active Queue Management in Underwater Network via Deep Reinforcement Learning

Local Congestion Control in Underwater Acoustic Networks via Reinforcement Learning

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES