Learning-based control

Can a system learn how to control itself?

Classical control design usually starts from a mathematical model: we write equations describing how a system behaves, and then use those equations to design a controller.

But today we often have something else in abundance: data.

A robot can repeatedly interact with its environment. A building can record years of operational data. An industrial process can continuously measure its inputs and outputs. This raises a natural question: How can we learn a good control strategy directly from experience, without first identifying a complete model of the system?

Collecting data efficiently

First, collecting enough data can be expensive. We show that, for linear systems, a relatively small but sufficiently rich experiment [1] can be used to generate many additional Bellman inequalities offline. In other words, one carefully designed experiment can contain much more information than the individual measured transitions may initially suggest. We then expand this idea to affine systems [2], extending the famous Willems' Lemma to this class of dynamical systems, and design estimators for the Bellman inequalities when the dynamics is stochastic.

Solution guarantees

Second, the resulting optimization problems can become very large or even unbounded. We develop relaxed formulations [3] that substantially reduce their size and require less data collection,  especially for stochastic systems. We then focus on the geometry [4] behind these optimization problems to understand when a finite solution is guaranteed. More recently, we use moment-matching techniques [5] to extend boundedness guarantees to nonlinear problems and polynomial features.

Reducing complexity

We also work on the algorithms underlying dynamic programming itself. The mini-batch Bellman operator [6] provides a tunable compromise between sequential algorithms, which can converge quickly, and fully parallel algorithms, which exploit modern computing hardware. In a complementary direction, our PAGE-PG algorithm [7] reduces the variance of policy-gradient estimates, one of the main reasons why reinforcement learning can require so many interactions with the environment.