Sample-efficient offline reinforcement learning (RL) with linear function approximation has been studied extensively recently. However, prior work has focused on obtaining $\frac{1}{\sqrt{K}}$-type sub-optimality bound, which is also minimax optimal, with $K$ being the number of episodes in the offline data. In this work, aiming to understand instance-dependent bounds for offline RL with function approximation, we present an algorithm named Bootstrapped and Constrained Pessimistic Value Iteration (BCP-VI) which leverages data bootstrapping and uses a constraint optimization to further encode pessimism. With our algorithm, we show that under the optimal-policy concentrability, offline RL speeds up the rate to $\tO(\frac{1}{K})$ when there is a positive sub-optimality gap in the optimal $Q$-value function, even when the offline data were collected adaptively. Moreover, when the linear features of the optimal actions in the states reachable by an optimal policy spans those reachable by the behavior policy and the optimal actions are unique, offline RL achieves absolute zero sub-optimality error when $K$ exceeds an (finite) instance-dependent threshold. To the best of our knowledge, these are the first $\tO(\frac{1}{K})$-type bound and zero bound for offline RL with linear function approximation under the partial data coverage and adaptive data. We also provide instance-agnostic and instance-dependent information-theoretical bounds to complement our upper bounds and a numerical simulation to verify the algorithm. 

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

We consider the problem of learning a set of probability distributions from the empirical Bellman dynamics in distributional reinforcement learning (RL), a class of state-of-the-art methods that estimate the distribution, as opposed to only the expectation, of the total return. We formulate a method that learns a finite set of statistics from each return distribution via neural networks, as in the distributional RL literature. Existing distributional RL methods however constrain the learned statistics to predefined functional forms of the return distribution which is both restrictive in representation and difficult in maintaining the predefined statistics. Instead, we learn unrestricted statistics, i.e., deterministic (pseudo-)samples, of the return distribution by leveraging a technique from hypothesis testing known as maximum mean discrepancy (MMD), which leads to a simpler objective amenable to backpropagation.  Our method can be interpreted as implicitly matching all orders of moments between a return distribution and its Bellman target. We establish sufficient conditions for the contraction of the distributional Bellman operator and provide finite-sample analysis for the deterministic samples in distribution approximation. Experiments on the suite of Atari games show that our method outperforms the standard distributional RL baselines and sets a new record in the Atari games for non-distributed agents.

Distributional Reinforcement Learning via Moment Matching

Posters & Demos III

poster

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines. AAAI-23 is the Thirty-Seventh AAAI Conference on Artificial Intelligence. The theme of this conference is to create collaborative bridges within and beyond AI. Like previous AAAI conferences, AAAI-23 will feature technical paper presentations, special tracks, invited speakers, workshops, tutorials, poster sessions, senior member presentations, competitions, and exhibit programs, and two new activities: a Bridge Program and a Lab Program. Many of these activities are tailored to the theme of bridges and all are selected according to the highest standards, with additional programs for students and young researchers. 
AAAI is providing you with a conference planner, which you can use to help organize your itinerary of activities. This includes talks to attend in person, talks to attend remotely, breaks with colleagues and your site seeing activities. To access this conference planner, please go to [https://aaai-2023.takemobi.io](https://aaai-2023.takemobi.io).

In order to access this site, you need to register. If you haven't already, please register [here](https://aaai.org/Conferences/AAAI-23/registration/).


AAAI 2023

The purpose of the AAAI conference series is to promote research in Artificial Intelligence (AI) and foster scientific exchange between researchers, practitioners, scientists, students, and engineers across the entirety of AI and its affiliated disciplines.

Main Track

technical paper

The purpose of the AAAI conference is to promote research in artificial intelligence (AI) and scientific exchange among AI researchers, practitioners, scientists, and engineers in affiliated disciplines. 

Thanh Nguyen-Tang

2

SHORT BIO

Presentations

On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximation

Distributional Reinforcement Learning via Moment Matching

Stay up to date with the latest Underline news!

PRESENTATIONS

CONFERENCES

COMPANY

RESOURCES