Imagine holding the plate in your hands. If the ball is left to the center, you will tilt the plate to the right, and vice versa. The further away the ball is from the center, the stronger you will tilt the plate. Linear feedback controllers do precisely that: They take the difference of the ball-position from the center and multiply it with a constant gain. The product is the plate angle, outputted as a voltage to the motors.

If the gain is chosen well, the ball automatically moves to any desired position on the plate. However, there is room for improving performance!

When the ball is standing still, static friction causes the ball to stick even when the plate is slightly tilted. This can greatly affect control performance: Recall that a linear controller multiplies the difference of the ball and target position with a constant number to compute the tilt angle. If this factor is chosen too small, the ball may never start rolling. As a human, we would carefully increase the tilt angle incrementally until the ball finally starts rolling. In control theory, this is called integral control: The algorithm sums the difference between the ball's current and desired position, multiplies this sum by a factor, and sets the plate's tilt angle to the resulting product. As long as the ball does not move towards the target, e.g., due to static friction, integral controllers keep increasing the tilt angle.

Once the ball starts rolling, humans typically try to counter this movement to reduce velocity and avoid overshooting the target. This is mimicked by acting against the change of error, i.e., we multiply the change of the error by a constant factor and set the tilt angle to the resulting product.

None of these controllers works well just by itself. Instead, all have their own strengths and weaknesses. A simple heuristics is to just use them all at the same time, and sum their signals, leading to so-called Proportional-Integral-Derivative-controllers (PID).

Sounds too simple? In fact, such controllers are the de-facto industry standard for simple automation tasks, and more than sufficient to balance the ball of our system in the center:

Check out portafilter machines at you local coffee machine store. Some particularly expensive machines advertise their fancy "PID temperature control". This means that there is a built-in temperature sensor, and a small computer that increases or decreases heating using a PID controller until a desired temperature is reached. Not really high-tech (particularly given the price of these machines) but it does the job!

PID controllers are simple to implement and intuitive, but they have their limitations. A simple example where PID controllers are often insufficient is in the face of safety critical constraints of operation (e.g., a car must not crash into a wall). Imposing such operational constraints via PIDs requires careful tuning, and often tinkering of rule-based heuristics that are prone to fail. Also, you may damage your system while trying to find safe gains for your PID controller.

Humans enforce safety by anticipating the consequences of their actions, and correcting their behavior before they even get close to a dangerous state. For example, when driving a car, we will slow down before a sharp turn. Model Predictive Control (MPC) mimics this philosophy. The idea is to predict the system evolution under a control signal for a few seconds into the future using a mathematical model of the system. The computer then crunches numbers to find control signals that are safe and yield good performance.

The animation shows the optimized predicted trajectory of the ball. The optimization is updated frequently to cope with the ball's true evolution.

Model Predictive Control requires very specific types of systems. For instance, it can not model the boundary of the plate and exploit bounces to collect points faster. A (somewhat brute force) alternative for optimizing input signals with predictions is Model Predictive Path Integral Control (MPPI). We start with some initial input sequence, copy it a thousand times, perturb every copy with random noise, and compute a weighted average based on the resulting performance of each perturbed signal. This approach is rather new, enabled by the abundance of computational resources, and particularly popular for robotics systems such as humanoids and quadrupeds.

The animation shows the many samples used to find an optimal trajectory for the ball. Similar to MPC, the prediction is regularly updated based on the ball's position.

MPC and MPPI only plan for a limited lookahead, which makes them less effective for planning long-term paths. In contrast, path planning algorithms aim to find complete paths that reach a target from the current state. Rapidly Exploring Random Trees (RRT) are popular path planning algorithms that find continuous paths from the current state to the target by randomly sampling the state space. This is similar to drawing lines with a pen in a maze until you find a path to the exit.

The animation shows the generation of such a tree. Once a path is found, it is tracked by any of the aforementioned controllers.

Usually one aims to find the path that is shortest, fastest, or most energy efficient. RRT does not guarantee any of that, neither does it ensure that the system is actually able to track the path. Dynamic programming algorithms address this limitation.

A bit mind-twisting: The algorithms starts at a future point in time, and then iteratively decides backwards in time on the best input from every state. More specifically, at every time-step, the algorithm assigns a cost-value to every state. If the target is easy to reach from a given state, or if states with low cost-values are within reach, the algorithm assigns a low cost-value to this given state. If it is hard to reach the target or states with low cost-values, a low cost-value is assigned. Once the value of all states is known, any previously discussed controller can be used to move the ball towards states with low cost-values, guiding it to the target.

We use similar intuitions everyday: Imagine you are trying to travel from Zurich to Berlin. Dynamic Programming would search for the optimal path starting from Berlin. The cost-value of a city is the minimum time it takes to travel to Berlin. Nürnberg to Berlin takes 6 hours, so a cost-value of 6 is assigned to Nürnberg. Würzburg to Berlin takes 7 hours, so cost-value of 7 is assigned to Würzburg. Munich to Würzburg takes 2.5 hours (leading to a total of 9.5 hours to Berlin), but Munich to Nürnberg takes only 2 hours (leading to a total of 8 hours to Berlin). Dynamic Programming chooses the least-cost option leading to a cost-value of 8 for Munich. Continuing this process for Stuttgart and Zurich leads to the optimal path from Zurich to Berlin via Munich and Nürnberg, with a total cost-value of 11 hours. In fact, these simple principles are used in apps like Google Maps and Waze (just highly optimized).

We use Dynamic Programming to navigate our ball through obstacles towards a target region. Once the value of all states is known, any previously discussed controller can be used to move the ball towards states with low cost-values, guiding it to the target. The animation shows the cost-values assigned to each state as a heatmap that is computed backwards in time, and the controllers' targeted ball position as green circles once all cost-values are computed.

Dynamic programming is a powerful tool for planning paths optimally, but it requires a perfect model of the system. Many real-world applications are subject to complex dynamics, where models are not readily available (e.g., robots with hundreds of motors and joints). Reinforcement learning (RL) mimics human learning through trial and error. Similar to Dynamic Programming, if a state led to a good outcome, it is assigned a low cost-value. It is assigned a high cost-value otherwise. The outcomes of every action from every state is memorized using a neural network. Simultanously, another neural network is trained to output plate angles that move the ball towards states with low cost-values.

The animation shows the activity of the neural network based on the current ball and desired target position.