In our 3rd RL Debates presentation, Niels from the Thousand Brains Project shared how their team is reverse-engineering the cortex to build sensorimotor systems that learn and infer like the brain.

Niels explained the core thesis of the Thousand Brains theory: the neocortex is not a single learning system but is composed of thousands of semi-independent cortical columns, each learning complete models of the world. A key mechanism for this is the use of reference frames, which allow each column to understand the structure of objects through active sensing. As an agent interacts with the world, these columns “vote” to reach a consensus on what they are sensing, leading to robust inference.

Niels demonstrated this with their simulated agent, “Monty,” which learns object models rapidly by actively moving its sensors over them. This approach is highly data-efficient, saving a vast number of parameters and FLOPs compared to data-hungry deep learning methods. As a result, Monty achieves impressive robustness on challenging tasks like out-of-distribution (OOD) generalization and continual learning, where traditional models often struggle.

Watch the full meeting here:

Updated: