Representation, Learning, and Reasoning (RLR) Lab

Large Scale Multiagent Deep Reinforcement Learning

Our increasingly interconnected urban environments provide several opportunities to deploy intelligent agents——from self-driving cars, ships to aerial drones——that promise to radically improve productivity and safety. Achieving coordination among agents in such urban settings presents several algorithmic challenges—ability to scale to thousands of agents, addressing uncertainty, and partial observability in the environment. Furthermore, in such complex transportation systems, only a simulator for the domain is available rather than a complete model.

My group is actively developing reinforcement learning (RL) based strategies as well as feature-rich traffic simulators (for autonomous vessels, drones) that enable coordination in such large multiagent systems with thousands of agents. E.g., the (left) figure shows a snapshot from our maritime traffic simulator. Accompanying paper develops deep RL based strategies to coordinate vessel traffic to increase safety of navigation by reducing traffic congestion, and also takes into account realistic domains constraints such as vessels cannot fully stop midwater, and movement of vessels is uncertain affected by several factors (such as weather).

Relevant publications along this direction are:

• Multiagent Decision Making For Maritime Traffic Management. AAAI 2019

• Credit Assignment For Collective Multiagent RL With Global Rewards. NeurIPS, 2018

• Successor Features Based Multi-Agent RL for Event-Based Decentralized MDPs. AAAI 2019

• Policy Gradient With Value Function Approximation For Collective Multiagent Planning. NeurIPS, 2017

Multiagent Planning and Decision Making Using Probabilistic Inference

Traditionally, the area of decision making and planning in agent-based models such as Markov decision processes (MDPs) and its multiagent extension (Dec-MDPs) has evolved relatively independently from the area of machine learning. Recently some intriguing connections have been developed between problems in machine learning (ML) such as likelihood maximization and that of planning and decision making in agent-based models. My ongoing work explores this connection in the area of multiagent systems leading to efficient and scalable algorithms for several problems. Applying ML techniques to that of decision making promises to be a very exciting research direction for the future.

Relevant publications along this direction are:

• Anytime Planning for Decentralized POMDPs using Expectation Maximization. Akshat Kumar and Shlomo Zilberstein, In UAI 2010.

• Probabilistic Inference Techniques for Scalable Multiagent Decision Making. Akshat Kumar, Shlomo Zilberstein and Marc Toussaint. In JAIR, 2015.

• Probabilistic Inference Based Message-Passing For Resource Constrained DCOPs. Supriyo Ghosh, Akshat Kumar and Pradeep Varakantham. In IJCAI 2015.

Decision Making and Learning for Computational Sustainability

Computational sustainability is an emerging interdisciplinary field that aims to bring together computational techniques from different areas of computer science and operations research for intelligent decision making to manage and conserve society and environment’s limited resources. Addressing important problems in sustainability such as network design for spatial conservation planning, understanding the population dynamics of an endangered species, and decision making for smart power grid management represent some of the core issues of my research agenda.

Relevant publications along this direction are:

• Lagrangian Relaxation Techniques for Scalable Spatial Conservation Planning. Akshat Kumar, Xiaojian Wu and Shlomo Zilberstein, In AAAI 2012.

• Collective Diffusion Over Networks: Models and Inference. Akshat Kumar, Daniel Sheldon and Biplav Srivastava, In UAI 2013.

• Robust Optimization for Tree-Structured Stochastic Network Design. Xiaojian Wu, Akshat Kumar, Daniel Sheldon, and Shlomo Zilberstein. , In AAAI 2017 (Best Paper Award, CompSust track)

Approximate Inference in Graphical Models

Probabilistic graphical models (PGMs) are a powerful framework that arise in a variety of practical applications such as multiagent systems, computer vision and bioinformatics. PGMs succinctly encode the structure present in real world problems by combining the language of graphs with probability theory. Such structure can then be exploited to develop efficient inference algorithms.

In my work, I have developed several message-passing based inference algorithm for the problem of maximum-a-posteriori (MAP) estimation in PGMs. I have been also active at addressing inference problems for collective graphical models (CGMs) that are graphical models defined over aggregate-level data rather than the individual model. CGMs are particularly useful when one has access to aggregate level data such as modeling bird migration, visitor diffusion in theme parks and learning from anonymized datasets.

Relevant publications along this direction are:

• MAP Estimation for Graphical Models by Likelihood Maximization. Akshat Kumar and Shlomo Zilberstein, In NIPS 2010.

• Message Passing Algorithms for MAP Estimation Using DC Programming. Akshat Kumar, Shlomo Zilberstein and Marc Toussaint. In AISTATS 2012.

• Message Passing For Collective Graphical Models. Tao Sun, Daniel Sheldon and Akshat Kumar. In ICML 2015.