Imagine holding the plate in your hands. If the ball is to the left of the center, you will tilt the plate to the right, and vice versa. The further away the ball is from the center, the steeper you will tilt the plate. Linear feedback controllers do precisely that: They take the difference of the ball-position from the center and multiply it with a constant (K_p). The product is the plate angle, which is achieved by outputting a respective voltage to the motors.

If the gain is chosen well, the ball automatically moves to any desired position on the plate. However, there is room for improving performance!

When the ball is standing still, static friction causes the ball to stick even if the plate is slightly tilted. This friction can greatly affect control performance: Recall that a linear controller multiplies the difference between the ball and target position with a constant number to compute the tilt angle. If this factor is too small, the ball may never start rolling. As a human, we would carefully increase the tilt angle incrementally until the ball begins to roll. In control theory, this is called integral control: The algorithm sums the difference between the ball's current and desired position, multiplies this sum by a factor, and sets the plate's tilt angle to the resulting product. As long as the ball does not move towards the target, e.g., due to static friction, integral controllers keep increasing the tilt angle.

Once the ball starts rolling, humans typically try to counter this movement to reduce velocity and avoid overshooting the target. This is mimicked by acting against the change of error, i.e., we multiply the change of the error by a constant factor and set the tilt angle to the resulting product.

None of these controllers works well just by itself. Instead, all have their own strengths and weaknesses. To combine these strengths, a common approach is to just use them all at the same time, and sum their signals, leading to so-called Proportional-Integral-Derivative-controllers (PID).

Sounds too simple? In fact, such controllers are the de facto industry standard for simple automation tasks, and more than sufficient to balance the ball of our system in the center:

Check out portafilter machines at you local coffee machine store. Some particularly expensive machines advertise their fancy "PID temperature control". This means that there is a built-in temperature sensor, and a small computer that increases or decreases heating using a PID controller until a desired temperature is reached. Not really high-tech (particularly given the price of these machines) but it does the job!

Similarly, we could implement PID controllers to program an airplane's autopilot. If the airplane looses altitude, the PID controller adjusts the elevators on the tail to manage the airplane's pitch and maintain a set altitude. This ensures that any deviations caused by turbulence or air pressure changes are corrected smoothly.

PID controllers are intuitive and simple to implement, but they have their limitations. A simple example where PID controllers are often insufficient is in the face of safety critical constraints of operation (e.g., a car must not crash into a wall). Imposing such operational constraints via PIDs requires careful tuning and thorough validation, which is time-consuming and error-prone. It is also possible to damage your system while trying to find safe gains for your PID controller.

Humans enforce safety by anticipating the consequences of their actions, and correcting their behavior before getting close to a dangerous state. For example, when driving a car, we will slow down before a sharp turn. Model Predictive Control (MPC) mimics this philosophy. The idea is to predict the system evolution under a control signal for a few seconds into the future using a mathematical model of the system. The computer then crunches numbers to find control signals that are safe and yield good performance.

The animation shows the optimized predicted trajectory of the ball. The optimization is updated frequently to update the ball's predicted position based on latest measurements.

Model Predictive Control works well on very specific types of systems, but can fall short when those systems become more complex. For instance, it cannot model collisions from the boundary of the plate and use bounces off the wall as part of the ball's trajectory to collect points faster. A (somewhat brute force) alternative for optimizing input signals with predictions is Model Predictive Path Integral Control (MPPI). We start with some initial input sequence, copy it a thousand times, randomly modify each copy slightly to obtain many diverse but similar input sequences, and compute a weighted average based on the resulting performance of each perturbed signal. This approach is rather new, enabled by the abundance of computational resources. It is particularly popular for robotics systems such as humanoids and quadrupeds since it can model arbitrary complex dynamics and physical constraints.

The animation shows the many samples used to find an optimal trajectory for the ball. Similar to MPC, the prediction is regularly updated based on the ball's position.

MPC and MPPI only plan optimal input sequences over a limited prediction horizon, which makes them less effective for planning long-term paths. For instance, if a goal is placed far from the current position, these methods may struggle to find a viable path.

In contrast, path planning algorithms keep search for complete paths that reach from the current state to the target, no matter how long it takes to compute this path. Rapidly Exploring Random Trees (RRT) are one such type of path planning algorithms. RRT starts at a random point in space and sprouts new branches from it toward other random points in space, progressively expanding until one of these branches reaches the target. This is similar to drawing lines with a pen in a maze until you find a path to the exit.

Such algorithms are particularly useful for navigating in complex environments with obstacles, e.g., for humanoids in buildings.

The animation shows the generation of such a tree. Once a path is found by RRT, any of the previously discussed controllers may be used to get the ball to actually follow it.

Usually one aims to find the path that is shortest, fastest, or most energy efficient. RRT does not guarantee any of that, neither does it ensure that the system is actually able to track the path. Dynamic programming algorithms address this limitation.

A bit mind-twisting: The algorithms starts at a future point in time, and then iteratively decides backwards in time on the best input from every state. More specifically, at every time-step, the algorithm assigns a cost-value to every state. If the target is easy to reach from a given state, or if states with low cost-values are within reach, the algorithm assigns a low cost-value to this given state. If it is hard to reach the target or states with low cost-values, a high cost-value is assigned to that state. Once the value of all states is known, any previously discussed controller can be used to move the ball towards states with low cost-values, guiding it to the target.

We use similar intuitions everyday: Imagine you are trying to travel from Zurich to Berlin. Dynamic Programming would search for the optimal path starting from Berlin. The cost-value of a city is the minimum time it takes to travel to Berlin. Nürnberg to Berlin takes 6 hours, so a cost-value of 6 is assigned to Nürnberg. Würzburg to Berlin takes 7 hours, so cost-value of 7 is assigned to Würzburg. Munich to Würzburg takes 2.5 hours (leading to a total of 9.5 hours to Berlin), but Munich to Nürnberg takes only 2 hours (leading to a total of 8 hours to Berlin). Dynamic Programming chooses the least-cost option leading to a cost-value of 8 for Munich. Continuing this process for Stuttgart and Zurich leads to the optimal path from Zurich to Berlin via Munich and Nürnberg, with a total cost-value of 11 hours. In fact, these simple principles are used in apps like Google Maps and Waze (just highly optimized).

We use Dynamic Programming to navigate our ball through obstacles towards a target region. Once the value of all states is known, any previously discussed controller can be used to move the ball towards states with low cost-values, guiding it to the target. The animation shows the cost-values assigned to each state as a heatmap that is computed backwards in time, and the controllers' targeted ball position as green circles once all cost-values are computed.

Dynamic programming is a powerful tool for planning paths optimally, but it requires a perfect model of the system. Many real-world applications are subject to complex dynamics, where models are not readily available (e.g., robots with hundreds of motors and joints). Reinforcement learning (RL) mimics human learning through trial and error. Similar to Dynamic Programming, if a state led to a good outcome, it is assigned a low cost-value. It is assigned a high cost-value otherwise. The outcomes of every action from every state is memorized using a neural network. Simultanously, another neural network is trained to output plate angles that move the ball towards states with low cost-values.

The animation shows the activity of the neural network based on the current ball and desired target position.