steering machines with geometry

big models! we can train 'em, we can scale 'em. we're getting better at evaluating them. but it seems that we don't have a real theory of steering: moving a network's behavior from one place to another in a controlled, predictable way.

i think the language for this already exists. it was developed across the 1970s and 80s by people like roger brockett, héctor sussmann, and velimir jurdjevic in a field called geometric control theory. the core insight is elegant: if you want to understand what a dynamical system can do, study the geometry of the space it moves through. the reachable set of a system is a question about lie algebras and differential geometry.

the theory

brockett's key contribution was showing that controllability, whether you can steer a system from any state to any other state, is fundamentally a question about the lie bracket structure of your control vector fields. if the vector fields you can apply, together with all their lie brackets, span the full tangent space at every point, you can go anywhere. this is the lie algebra rank condition, and it's one of the most elegant results in modern control theory.

what makes this geometric is that the structure of reachable sets depends on curvature, on holonomy, on the topology of the underlying manifold. brockett's proof that certain systems can't be stabilized by smooth feedback (the topological obstruction result) shows that the shape of the space is what matters in permitting what you're trying to achieve.

it's kind of like being in a car, someone throws a random point on the map, and you're figuring out if there is a set of roads you can drive that get you there. sorta.

sussmann and jurdjevic extended this into optimal control: what is the cheapest way to steer between two states? their work connects to sub-riemannian geometry, where you're constrained to move only along certain directions at each point, which is exactly the situation you're in when you have limited control inputs, limited actions you can take. the optimal paths in these spaces are sub-riemannian geodesics (locally length-minimizing paths between two points on the manifold).

the connection to neural networks

during model training, or during inference-time steering (prompting, activation patching, fine-tuning, RLHF), you're applying a control input to a dynamical system. in a forward pass of the network or the trajectory through weight space during optimization, you're trying to reach some desired behavior.

the parameter space of a neural network is a high-dimensional manifold. the loss landscape defines a geometry on it. gradient descent is a particular control law, but it's a crude one. it follows the local slope without any understanding of the global structure of the space it's moving through.

analogies:

controllability. given a particular network architecture and a particular set of interventions (say, modifying activations at specific layers), what behaviors are reachable? this is the lie algebra rank condition question. if your interventions and their interactions don't span the relevant directions in behavior space, there are things you simply cannot make the network do. understanding this would tell us something fundamental about the limits of alignment.

observability. the dual question: given the outputs you can see, what can you infer about the internal state? brockett's work on nonlinear observability gives tools for this. when people talk about mechanistic interpretability, i think they're asking an observability question.

topological obstructions. brockett showed that some systems have topological reasons why smooth control laws can't stabilize them. are there analogous obstructions in neural network steering? are there behaviors that are reachable but not stably reachable? so, you can get there, but you can't stay there with any smooth intervention? i think this would have direct implications for the robustness of safety techniques.

optimal steering. if you want to move a network from behavior A to behavior B with minimal disruption to everything else it knows, you're solving an optimal control problem on a manifold. the sub-riemannian framework gives you the tools to think about this when your interventions are lower-dimensional than the full state space.

some project ideas

lie bracket analysis of activation interventions. take a transformer. define vector fields on the activation space corresponding to different intervention types (steering vectors, activation additions, ablations). compute their lie brackets. determine the reachable set. this gets at something about what activation engineering can actually control?

sub-riemannian geometry of fine-tuning. model the fine-tuning process as a control system on the weight manifold, where the "allowed directions" are determined by the structure of the training data and optimizer. characterize the sub-riemannian geodesics. this tells you, i think, the most efficient path between two behaviors, and what behaviors are unreachable from a given starting point.

topological obstructions to alignment. investigate whether there are topological reasons (in the brockett sense) why certain alignment properties cannot be maintained by smooth control laws. if such obstructions exist, this is an argument for discontinuous or hybrid approaches to alignment, and it would be important to know.

observability rank conditions for interpretability. apply nonlinear observability theory to determine what internal structure is in principle recoverable from external measurements (outputs, logits, attention patterns). this attempts to put mechanistic interpretability on a rigorous geometric foundation and identify fundamental limits on what can be inferred.

geometric optimal control for continual learning. use the riemannian and sub-riemannian optimal control framework to develop training procedures that navigate the weight manifold along geodesics, minimizing catastrophic forgetting by construction rather than by regularization heuristics. hard!!!

why it's hard

ok, sounds good, but geometric control theory is hard. it requires differential geometry, lie theory, functional analysis. the people who know this material are in control theory and mathematics departments.

also, the ML community has been so successful with empirical methods that there's a strong cultural bias against mathematical theorizing mumbo jumbo, maybe? but perhaps we're reaching the point where "try it and see" isn't sufficient for questions about what's reachable, what's observable, what's possible. we should definitely try things and find out! but hopefully it's still cool to discover fundamental understanding :)

brockett himself was always interested in the connection between control and computation. his 1991 paper on dynamical systems that sort lists is a cool example. he showed that a simple gradient flow on the space of orthogonal matrices produces a system that sorts eigenvalues, connecting continuous dynamics to discrete algorithms.

do you think this is cool?

if you're interested in this (or you think it's completely misguided) i'd like to hear from you! get in touch — spencer at workwithcypress dot com.

(a bibliography)

the classic

  • brockett, "dynamical systems that sort lists, diagonalize matrices, and solve linear programming problems" (Linear Algebra Appl., 1991). gradient flows on matrix lie groups that solve combinatorial problems. the intellectual ancestor of using geometric dynamics for computation — and the proof that the bridge between control geometry and algorithms has been there for thirty-five years.

geometric control meets deep learning

the most direct work connecting lie-algebraic controllability to neural networks:

  • tabuada & gharesifard, "universal approximation power of deep residual neural networks via nonlinear control theory" (ICLR 2021). proves universal approximation of deep resnets by reducing it to a controllability problem, then applying lie bracket rank conditions. the closest existing work to what i'm proposing.
  • agrachev & sarychev, "control on the manifolds of mappings with a view to the deep learning" (J. Dynamical and Control Systems, 2022). two foundational geometric control theorists tackle deep learning directly — controllability results on infinite-dimensional manifolds of diffeomorphisms as continuous-depth network analogs.
  • cuchiero, larsson & teichmann, "deep neural networks, generic universal interpolation, and controlled ODEs" (SIAM J. Math. Data Sci., 2020). views deep networks as discretizations of controlled ODEs. uses lie brackets and the hörmander condition to show universal interpolation is generic.
  • ruiz-balet & zuazua, "neural ODE control for classification, approximation, and transport" (SIAM Review, 2023). classification recast as simultaneous controllability — each data point is an initial condition, and a single control must steer all points to correct regions. constructive and genuinely nonlinear.

neural ODEs and the dynamical systems bridge

the work that made "neural network as dynamical system" concrete:

  • chen, rubanova, bettencourt & duvenaud, "neural ordinary differential equations" (NeurIPS 2018, best paper). the paper that launched it — continuous-depth networks via ODE solvers, opening the door to control-theoretic analysis.
  • weinan e, "a proposal on machine learning via dynamical systems" (Comm. Math. Stat., 2017). the foundational position paper connecting deep learning to dynamical systems and optimal control.
  • geshkovski, letrouit, polyanskiy & rigollet, "a mathematical perspective on transformers" (Bull. Amer. Math. Soc., 2025). the comprehensive treatment — transformers as interacting particle systems, connecting to nonlinear transport equations, wasserstein gradient flows, and kuramoto-type synchronization.

the dynamics of transformers, specifically

this is the most active thread right now. the picture: tokens are particles, depth is time, attention is interaction.

  • geshkovski, letrouit, polyanskiy & rigollet, "the emergence of clusters in self-attention dynamics" (NeurIPS 2023). proves tokens cluster toward limiting configurations as depth goes to infinity, with cluster locations determined by the spectrum of the value matrix. the self-attention matrix converges to a low-rank boolean matrix.
  • geshkovski, koubbi, polyanskiy & rigollet, "dynamic metastability in the self-attention model" (arXiv 2024). the key insight: tokens get trapped in multi-cluster configurations for exponentially long times before collapsing. computation happens in these metastable states, not at equilibrium. connects to the otto-reznikoff framework for slow motion of gradient flows.
  • rigollet, "the mean-field dynamics of transformers" (arXiv 2024). takes the continuum limit. unnormalized self-attention is a wasserstein gradient flow of an interaction energy. normalized attention corresponds to a hessian metric. identifies phase transitions for long-context attention.
  • bruno, pasqualotto & agazzi, "emergence of meta-stable clustering in mean-field transformer models" (NeurIPS 2024). rigorous analysis of the metastable phases — the multi-cluster configurations that persist before eventual global collapse. this is where the useful computation lives.
  • altafini, "multistability of self-attention dynamics in transformers" (arXiv 2025). connects self-attention to the oja flow (the dynamical system that computes principal eigenvectors). classifies four types of equilibria: consensus, bipartite consensus, clustering, polygonal. multistability in >50% of instances.
  • karagodin & polyanskiy, "clustering in causal attention masking" (NeurIPS 2024). extends the clustering analysis to the autoregressive setting. proves convergence to a single cluster for arbitrary key-query matrices with identity value, but metastable multi-cluster states dominate practical timescales.
  • wu, ajorlou, wang, jegelka & jadbabaie, "on the role of attention masks and layernorm in transformers" (NeurIPS 2024). layernorm isn't a training trick — it turns self-attention into a qualitatively different dynamical system. sparse attention masks provably slow rank collapse.
  • sander, ablin, blondel & peyre, "sinkformers: transformers with doubly stochastic attention" (AISTATS 2022). doubly stochastic attention enables interpreting self-attention as a wasserstein gradient flow. in the infinite-sample limit, sinkformers perform heat diffusion.
  • hutchinson et al., "LieTransformer: equivariant self-attention for lie groups" (ICML 2021). builds attention that's equivariant to arbitrary lie groups by construction — the most direct connection between transformer architecture and lie group structure.
  • jemley, "continuous-depth transformers with learned control dynamics" (arXiv 2026). replaces discrete layers with a neural ODE block, treating depth as a controllable continuous variable. explicitly frames transformer depth as a control problem.

the geometry of transformer representations

the manifold structure that steering would operate on:

  • su & you, "geometric and dynamic scaling in deep transformers" (arXiv 2026). depth-scaling failures are fundamentally geometric — residual connections don't constrain updates to the semantic manifold. proposes tangent-space-constrained residual updates.
  • "latent semantic manifolds in large language models" (arXiv 2026). the internal representation space as a riemannian submanifold with fisher-information metric. representations occupy only 1-3% of ambient space, with smooth curvature structure. universal hourglass intrinsic dimension profiles across architectures.
  • agarwal, dalal & misra, "the bayesian geometry of transformer attention" (arXiv 2025). value outputs lie on a low-dimensional manifold parameterized by posterior entropy. during training, this manifold unfurls while attention patterns remain stable.
  • freytes, "geometric attention: a regime-explicit operator semantics for transformer attention" (arXiv 2026). decomposes attention into four geometric components and shows that softmax emerges as a special "newtonian regime" under scalar relational-work and multiplicative compositionality.

optimal control formulations of training

training as a control problem, with real optimality conditions:

  • li, chen, tai & e, "maximum principle based algorithms for deep learning" (JMLR, 2018). applies pontryagin's maximum principle to derive training algorithms with convergence guarantees that backprop lacks.
  • e, han & li, "a mean-field optimal control formulation of deep learning" (Res. Math. Sci., 2019). population risk as mean-field optimal control, with hamilton-jacobi-bellman and pontryagin-type conditions.
  • haber & ruthotto, "stable architectures for deep neural networks" (Inverse Problems, 2018). relates exploding/vanishing gradients to ODE stability, uses control-theoretic stability to design provably stable architectures.
  • geshkovski & zuazua et al., "turnpike in optimal control of PDEs, ResNets, and beyond" (Acta Numerica, 2022). 130-page survey connecting turnpike theory to resnet training — optimal trajectories spend most time near a steady state, explaining why intermediate layers produce similar representations.

riemannian and information geometry

the geometry of parameter space:

  • amari, "natural gradient works efficiently in learning" (Neural Computation, 1998). the foundational paper — parameter space as a riemannian manifold with fisher information metric, and why steepest descent in this geometry vastly outperforms euclidean gradient descent.
  • martens & grosse, "optimizing neural networks with kronecker-factored approximate curvature" (ICML 2015). makes natural gradient practical for large networks via kronecker approximation of the fisher matrix.
  • benfenati & marta, "a singular riemannian geometry approach to deep neural networks" (Neural Networks, 2023). pulling back the output metric to intermediate layers yields a degenerate riemannian metric, capturing information loss through depth. the singular geometry is exactly what you'd expect from a control-theoretic perspective.

sub-riemannian geometry and neural networks

the newest and least developed thread — but the most relevant to steering under constraints:

  • celledoni, ehrhardt, etmann & owren, "sub-riemannian landmark matching and its interpretation as residual neural networks" (J. Computational Dynamics, 2025). directly connects sub-riemannian geometry to resnets. diffeomorphic shape matching with restricted vector fields creates a sub-riemannian structure; resnets are time-discretizations of the resulting optimal control problem.
  • smets, portegies, bekkers & duits, "PDE-based group equivariant convolutional neural networks" (J. Math. Imaging and Vision, 2023). CNNs whose layers are PDE flows on lie groups — sub-riemannian geometry emerges naturally when diffusion is directionally restricted.

topology of networks and loss landscapes

what the shape of the space tells you:

  • naitzat, zhitnikov & lim, "topology of deep neural networks" (JMLR, 2020). uses persistent homology to show networks operate by simplifying topology layer-by-layer — transforming complex input topology into simple output topology.
  • rieck et al., "a differential topological view of challenges in learning with feedforward neural networks" (arXiv 2018). morse theory and transversality applied to explain why certain dataset topologies create fundamental architectural obstacles.
  • kratsios & papon, "universal approximation theorems for differentiable geometric deep learning" (JMLR, 2022). curvature-dependent bounds on approximation and the first known homotopic obstructions to non-euclidean universal approximation.

where the ML community is on steering (without the geometry)

the empirical work this program would give foundations to:

  • zou et al., "representation engineering: a top-down approach to AI transparency" (arXiv 2023). derives steering vectors from activation contrasts to control honesty, harmlessness, power-seeking. powerful but has no theory of what's reachable.
  • turner et al., "activation addition: steering language models without optimization" (arXiv 2023). adds activation differences from contrastive prompts during forward passes. works surprisingly well, with no explanation of why or when it should work.
  • rimsky et al., "steering llama 2 via contrastive activation addition" (ACL 2024). scales activation steering with hundreds of contrast pairs. identifies that middle-to-late layers work best — a geometric fact begging for a geometric explanation.
  • li et al., "inference-time intervention: eliciting truthful answers from a language model" (NeurIPS 2023). identifies attention heads linearly informative about truthfulness, shifts activations along truthful directions. an observability result dressed in empirical clothing.
  • park, choe & veitch, "the linear representation hypothesis and the geometry of large language models" (ICML 2024). formalizes when concepts are linearly represented and identifies a non-euclidean causal inner product — the closest the steering community has come to the geometric framework i'm proposing.
  • tan et al., "analyzing the generalization and reliability of steering vectors" (NeurIPS 2024). the most rigorous failure-mode analysis: steering is highly variable, some inputs are "anti-steerable," and out-of-distribution generalization is brittle. exactly the kind of result a controllability analysis would explain.
  • skifstad et al., "local linearity of LLMs enables activation steering via model-based linear optimal control" (arXiv 2026). the paper that comes closest to the program i'm describing — models LLM inference as a linear time-varying system and applies LQR for closed-loop steering. state-of-the-art on toxicity, truthfulness, and refusal. but linear control theory is only the beginning.

control-theoretic interpretability

  • "from black-box to white-box: control-theoretic neural network interpretability" (arXiv 2025). treats trained networks as nonlinear state-space systems, computes controllability and observability gramians, uses hankel singular values to rank neuron importance. the right instinct, applied to small networks — needs the full geometric treatment to scale.