Latent test-time compute, exploring attractors in latent space

The talk examines how pretraining transformers to loop internal layers creates stable attractor points in latent space, revealing dynamics of test‑time computation.

Overview

Transformers are compositions of transformer blocks. In this talk, we dive in the internals of their latent space, and specifically take a look a twhat happens if we pretrain the model to learn to loop its inner layers”. we find that the model learns to converge to attractors in Latent space for each independent looping block.

Links

https://github.com/tomoqt/Latent_ttc.git
Modifies `nanoGPT` for recurrent transformer block looping, using specialized variance initialization.

Tech stack