Toward a Theoretical Foundation of Policy Optimization for Learning Control Policies

Annual Review of Control Robotics and Autonomous Systems · 2023 · 60 citations

• Computer Science
• Computer Science
• Mathematical optimization

Gradient-based methods have been widely used for system design and optimization in diverse application domains. Recently, there has been a renewed interest in studying theoretical properties of these methods in the context of control and reinforcement learning. This article surveys some of the recent developments on policy optimization, a gradient-based iterative approach for feedback control synthesis that has been popularized by successes of reinforcement learning. We take an interdisciplinary perspective in our exposition that connects control theory, reinforcement learning, and large-scale optimization. We review a number of recently developed theoretical results on the optimization landscape, global convergence, and sample complexityof gradient-based methods for various continuous control problems, such as the linear quadratic regulator (LQR), [Formula: see text] control, risk-sensitive control, linear quadratic Gaussian (LQG) control, and output feedback synthesis. In conjunction with these optimization results, we also discuss how direct policy optimization handles stability and robustness concerns in learning-based control, two main desiderata in control engineering. We conclude the survey by pointing out several challenges and opportunities at the intersection of learning and control.

<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" id="d1e148" altimg="si3.svg"><mml:msub><mml:mrow><mml:mi>L</mml:mi></mml:mrow><mml:mrow><mml:mn>2</mml:mn></mml:mrow></mml:msub></mml:math>-gain analysis for dynamic event-triggered networked control systems with packet losses and quantization

Automatica · 2021 · 57 citations

• Computer Science
• Computer Science
• Algorithm

Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms

Studies in systems, decision and control · 2021 · 1099 citations

• Computer Science
• Artificial Intelligence
• Computer Science

Multi-competitive viruses over time-varying networks with mutations and human awareness

Automatica · 2020 · 52 citations

• Computer Science
• Computer Science
• Mathematical optimization

Natural policy gradient primal-dual method for constrained Markov decision processes

Neural Information Processing Systems · 2020 · 66 citations

• Computer Science
• Computer Science
• Mathematical optimization

Approximate Markov-Nash Equilibria for Discrete-Time Risk-Sensitive Mean-Field Games

Mathematics of Operations Research · 2020 · 31 citations

• Mathematics
• Mathematical economics
• Mathematical optimization

In this paper, we study a class of discrete-time mean-field games under the infinite-horizon risk-sensitive optimality criterion. Risk sensitivity is introduced for each agent (player) via an exponential utility function. In this game model, each agent is coupled with the rest of the population through the empirical distribution of the states, which affects both the agent’s individual cost and its state dynamics. Under mild assumptions, we establish the existence of a mean-field equilibrium in the infinite-population limit as the number of agents (N) goes to infinity, and we then show that the policy obtained from the mean-field equilibrium constitutes an approximate Nash equilibrium when N is sufficiently large.

On the Stability and Convergence of Robust Adversarial Reinforcement Learning: A Case Study on Linear Quadratic Systems

Neural Information Processing Systems · 2020 · 28 citations

• Computer Science
• Artificial Intelligence
• Mathematical optimization

Quantifying Market Efficiency Impacts of Aggregated Distributed Energy Resources

IEEE Transactions on Power Systems · 2020 · 25 citations

• Computer Science
• Industrial organization
• Microeconomics

We focus on the aggregation of distributed energy resources (DERs) through a profit-maximizing intermediary that enables participation of DERs in wholesale electricity markets. Particularly, we study the market efficiency brought in by the large-scale deployment of DERs and explore to what extent such benefits are offset by the profit-maximizing nature of the aggregator. We deploy a game-theoretic framework to study the strategic interactions between an agreggator and DER owners. The proposed model takes into account the stochastic nature of the DER supply. We explicitly characterize the equilibrium of the game and provide illustrative examples to quantify the efficiency loss due to the strategic incentives of the aggregator. Our numerical experiments illustrate the impact of uncertainty and amount of DER integration on the overall market efficiency.

Modeling, estimation, and analysis of epidemics over networks: An overview

Annual Reviews in Control · 2020 · 146 citations

• Computer Science
• Computer Science
• Machine Learning