Can LLMs Be Computers? | Percepta

When LLMs Learn to Do Mental Math: How Percepta's 2D Attention Turns Transformers into Actual Computers

GPT can win a math olympiad gold medal but can't reliably multiply three-digit numbers. The contrast sounds absurd, yet it precisely captures a fundamental deficiency in today's large language models: they are brilliant reasoning coordinators, but terrible computation executors.

A recent technical blog post from the Percepta research team, Can LLMs Be Computers?, confronts this problem head-on. Rather than inventing a new architecture, they applied an elegant parametric constraint to the existing Transformer — restricting attention heads to two dimensions — then leveraged convex hull data structures from computational geometry to reduce attention lookup complexity from linear to logarithmic. The result: a standard PyTorch Transformer that can efficiently execute millions of steps of precise computation within its own inference loop.

Why does this matter? Because it touches a deeper question about what the relationship between AI models and "computation" should actually look like.

A Contrast Worth Paying Attention To

Today's LLMs handle computational tasks as dispatchers, fundamentally. The model writes a snippet of Python, hands it to an external interpreter, gets the result back, and continues the conversation. Or an agent framework breaks the task into small steps, repeatedly calling the model over short contexts. These approaches work, sure. But they all point to the same fact: the model itself doesn't compute. Computational capability always resides outside the model.

Percepta's original post uses a restrained analogy: humans can't fly, and building airplanes doesn't change that — it just means we built a machine that flies for us.

The power of this analogy lies in what it reveals: an architectural limitation, not merely an engineering inconvenience. When a model must hand off control to an external tool, it introduces latency, opacity, and system complexity. More importantly, it forecloses the possibility of deeply integrating computational ability with reasoning ability.

Tool use (external round-trip) vs. in-model execution

source: Percepta, "Can LLMs Be Computers?"

The figure above illustrates the two paradigms side by side. On the left, traditional tool calling: the model emits python -c "print(3+5)", pauses, waits for an external interpreter to return 8. On the right, Percepta's in-model execution: after emitting WebAssembly instructions, the model executes them step by step inside the Transformer, generating a complete execution trace in token form — never leaving the forward pass.

The Transformer's "Notebook Problem"

To appreciate why Percepta's approach is non-trivial, you first need to understand the structural difference between how Transformers and traditional computers work.

A conventional computer has memory it can read and write at will. Each instruction updates the relevant memory locations at roughly constant cost — whether the program has been running for 100 steps or a million.

Transformers are fundamentally different. Think of one as a notebook you can only write downward in: the first few lines are the input (the prompt), and each subsequent line is a new token the model generates. When writing a new line, you can look back at previous content, but you cannot edit it. The notebook only grows.

What does this mean in practice? Generating the -th token requires the model to interact with the KV cache of all preceding  tokens. Even with KV caching to avoid redundant computation, per-step work still grows linearly with sequence length, and total cost scales quadratically. By the 10,000th step, the model has to "flip through" 9,999 previous lines of notes just to decide what to write next.

Verb parity counting execution trace

source: Percepta, "Can LLMs Be Computers?"

But Percepta identifies a key insight: many algorithms can be rewritten as "append-only traces" where each step only needs to look back at a small, fixed number of positions. The figure above shows a simple example — determining whether the number of verbs in a sentence is odd or even. Each trace token only needs to attend to two positions: the corresponding input word (to check if it's a verb) and the previous trace token (to read the running parity state).

The problem isn't whether the Transformer can represent this kind of computation. The problem is that even when each step only needs a few relevant positions, the standard attention mechanism still scans all historical tokens to find them. That's the bottleneck.

I explored this  bottleneck in detail in a previous article on DeepSeek's sparse attention mechanism — it's the most well-known efficiency ceiling of the Transformer architecture. DeepSeek's answer was to sparsify the attention pattern. Percepta takes an entirely different path.

2D Attention: An Elegant Geometric Trick

Percepta's core idea can be stated in a single sentence: turn the attention lookup into a computational geometry problem, then solve it with efficient data structures that already exist.

What does standard attention actually do? Each historical token contributes a key vector  and value . The current step generates a query vector , computes relevance scores  against all keys, and returns a weighted sum. It's as if you're in a library holding a "question" (query) and need to check every shelf label (key) one by one to find the best match. More shelves, more time.

When key vectors are constrained to two dimensions, everything changes.

Each key becomes a point on a 2D plane. The query becomes a directional arrow. "Find the most relevant key" becomes equivalent to "find the point farthest in this direction" — a classic problem in computational geometry known as the support point query on a convex hull. And for this class of query, data structures exist that can answer in  time.

The geometric view of 2D attention

source: Percepta, "Can LLMs Be Computers?"

From linear to logarithmic. What does this jump mean concretely? Running a program for one million steps, each step requires roughly 20 comparisons () instead of one million.

A natural objection: is two dimensions really enough? Percepta provides an elegant mathematical construction. To implement a memory lookup operation ("give me the value at address "), encode address  as key  and query address  with direction . Expanding the dot product: . The quadratic penalty  ensures only the exact match  maximizes the dot product. Two dimensions — sufficient for precise memory addressing.

One misconception worth clearing up: "2D heads" does not mean "small model." Total parameter count can match a standard Transformer — with , the standard approach might use 4 heads of 64 dimensions each, while the 2D approach uses 128 heads of 2 dimensions each. Same parameter budget. The model itself is a completely standard PyTorch Transformer. No custom attention kernels. No sparse masks.

The Performance Gap, in Hard Numbers

Theoretical elegance is one thing. Practical performance is another.

HullKVCache vs. standard KVCache performance comparison

source: Percepta, "Can LLMs Be Computers?"

The numbers speak for themselves. On the same task, standard KV cache takes 258.9 seconds (20 tokens/s). Percepta's HullKVCache takes 0.1 seconds (10,390 tokens/s) — roughly a 500x speedup. The curve below makes the scaling behavior even more stark: standard KV cache shows accelerating growth (quadratic), while HullKVCache is practically flat.

This gap compounds as execution traces grow longer. For tasks requiring millions of computation steps — complex combinatorial optimization or constraint satisfaction problems — the standard approach becomes infeasible in practice. Percepta's approach maintains near-constant per-step cost.

Sudoku solving demo

source: Percepta, "Can LLMs Be Computers?"

Sudoku serves as an excellent stress test. Neural approaches perform reasonably on easy puzzles but collapse on hard ones. The common explanation is that autoregressive models commit to answers token by token — once an early mistake is made, there's no going back. Yet Percepta's system achieves 100% accuracy on benchmark puzzles. Not because it "guesses better," but because it executes a fully correct compiled solver step by step inside the model. If the solver code is correct, the execution result is guaranteed correct.

From "Can It Work" to "How to Deploy It"

The current Percepta system is a Transformer purpose-built as an "executor." The team envisions three deployment modes:

Standard Transformer vs. 2D-head Transformer architecture and deployment modes

source: Percepta, "Can LLMs Be Computers?"

Fast path — 2D heads handle heavy computation, functioning like a built-in coprocessor. Fast/slow hybrid — 2D heads execute, full-dimension heads reason, each playing to its strengths. Speculative decoding — the 2D model proposes tokens quickly, a larger model verifies and accepts, leveraging the speed advantage to accelerate overall inference.

This calls to mind DeepMind's work on adaptive computation — letting models "breeze through" easy tokens and "pause to think harder" on difficult ones. Percepta's fast/slow hybrid pursues a similar goal at heart: route to the fast path when precise computation is needed, fall back to the standard path when abstract reasoning is required. The difference is that Percepta's "fast" isn't about skipping computation — it's about making computation itself dramatically more efficient through geometric acceleration.

One technical advantage deserves emphasis: because the execution trace is part of the forward pass, the entire process remains differentiable. Gradients can propagate through the computation itself. This makes it fundamentally different from external tool calls — it's not a black box, but a computational component that can be jointly trained with the model.

A More Radical Possibility

In its closing section, Percepta's original post floats a rather bold conjecture: if a WebAssembly interpreter can be implemented in Transformer weights, then in principle, arbitrary programs could be compiled directly into the weights themselves.

C source code compiled into Transformer weights

source: Percepta, "Can LLMs Be Computers?"

If program logic can be compiled into weights, gradient descent ceases to be the only way to modify a model. We could directly "write" portions of a model's functionality. Taken to its logical conclusion, this means AI systems might not only learn from data but also directly modify and extend their own weights — effectively rewriting parts of their internal machinery.

AI systems growing by accumulating modules

source: Percepta, "Can LLMs Be Computers?"

The broader vision: future AI systems won't merely use software — they'll contain software. They'll evolve the way software ecosystems do, accumulating modules and reusable components over time. New computational capabilities compiled incrementally into the model's internal execution engine, wiring into existing circuits, no full retraining required.

This connects to an interesting thread from my earlier piece on The Geometry of Belief. That article, drawing on Columbia University research, showed that Transformer attention heads spontaneously learn to perform structured Bayesian inference — not pattern matching, but genuine computation. Percepta's work reads as a natural extension of that finding: if attention heads are already performing some form of computation, can deliberate parametric constraints make that computational capacity stronger, more general, more efficient?

The evidence suggests yes.

A Measured Takeaway

It should be noted that what Percepta has demonstrated remains a proof of concept. From a standalone 2D executor to deep integration with large-scale language models, there is still considerable distance to cover. How 2D heads perform after training at scale, how effectively they cooperate with full-dimension heads, and the engineering complexity of real-world deployment — these are questions that haven't been fully answered.

But the real value of this work isn't in how many Sudoku puzzles it solves today. It lies in reframing the question.

We used to ask: "How can we help LLMs call tools better?" Percepta invites a different question: "Can the LLM itself be the tool?"

From coordinator of computation to executor of computation — the distance may be shorter than we assumed. And the key that opens this door turns out to be an old friend from the computational geometry textbook: the convex hull.