How Large Language Models Solve Problems Without Introspection

LLM AI Models kinda hide their secret sauce….. Anthropic, the AI company behind the language model Claude, published their results on trying to understand how reasoning models work. The results, we are all left more confused. When the model was asked a question in English, using idea of circuit tracing to see which areas of the model were being used (similar to how you will do imaging of a brain to see which areas show activity when asked a question), it turns out the model did not use areas that it normally uses for learning English, Chinese or even French. It seems to use unrelated areas and then just before it spits out the answer, it chooses English to spit out the answer. Ask yourself, what language do you think in? Claude has none. The researchers again studied how Claude solves simple math problems and found something surprising. Instead of following the standard methods it was trained on, Claude seems to have developed its own quirky way of doing calculations. For example, if you ask it to add 36 + 59, it doesn’t just carry the one like we were taught in school. Instead, it takes a roundabout path—first adding rough estimates like "40ish + 60ish" or "57ish + 36ish" to get"92ish." Then, it focuses on the last digits (6 + 9) and realizes the answer must end in a 5. Combining these, it correctly lands on 95. But here’s the strange part: if you ask Claude how it got the answer, it doesn’t tell you about its weird estimation tricks. Instead, it gives a textbook-perfect explanation like, "I added the ones place (6+9=15), carried the 1, then added the tens (3+5+1=9) to get 95." That’s the method you’d find all over the internet but not what it actually did. This shows that large language models don’t always think the way they claim. They can come up with unexpected strategies but then hide them behind more "normal"-sounding answers. So while AI like Claude is impressive, it’s also… kind of sneaky. And that’s why we shouldn’t always take its explanations at face value.

When we talk about large language models “reasoning,” we’re often lumping together very different processes. In reality, there are two distinct types of reasoning that LLMs exhibit: implicit reasoning and explicit reasoning. 📣 Explicit reasoning This is the step-by-step style many of us encourage with prompts like “think step by step.” In the math problem below, the model writes out: → Step 1: Calculate total markers → Step 2: Subtract what was given away → Step 3: Subtract what was lost → Final answer = 48 Advantages: → Transparent and auditable → Better performance on complex, multi-step tasks Limitations: → Slower and more verbose → Can be harder to scale for large volumes of queries 💠 Implicit reasoning Here, the model “jumps” directly to the final answer without writing intermediate steps. The reasoning still happens, but it’s compressed in the model’s hidden states. It’s like solving a problem in your head vs. explaining it out loud. Advantages: → Fast and efficient → Feels more natural for tasks like translation or summarization Limitations: → Opaque- you can’t inspect how the model got there → Harder to debug or align behavior ⚖️ Why this matters → Implicit reasoning is good when efficiency matters. → Explicit reasoning is essential when correctness, safety, or interpretability are non-negotiable. The future of LLM systems will depend on balancing both: implicit shortcuts for fluid performance, and explicit reasoning when decisions must be explained and trusted. 📚 If you want to go deeper, I’d highly recommend, reading these two resources: - https://lnkd.in/dPJUR7dn - https://lnkd.in/dAD8AUjA 〰️〰️〰️ Follow me (Aishwarya Srinivasan) for more AI insight and subscribe to my Substack to find more in-depth blogs and weekly updates in AI: https://lnkd.in/dpBNr6Jg

Nobody knows how Large Language Models manage to display their impressive capabilities - they are incredibly powerful black boxes. Anthropic has now released methods to peek under the hood. It's an exciting step towards more interpretable LLMs. The transformer architecture contains very few building blocks, making it even harder to understand how they lead to these advanced, emergent capabilities. Here are two example challenges: 1) Polysemantic neurons: The LLM needs to represent more concepts than it has neurons available. As a result, one neuron represents parts of many unrelated concepts, making it hard to understand what it means when it fires. 2) Connectivity: Information passes sequentially through the transformer layers. It's unclear how concepts from earlier layers influence later ones. Anthropic's approach to untangle this is to create mechanistic interpretations of the transformer components in human-understandable language. They identify interpretable building blocks (features) the model uses, then describe processes (circuits) by which features interact to produce outputs. Concretely, Anthropic creates a second model (transcoder) that mimics the original LLM but with key differences: 1) More neurons: More neurons in specific layers allow separate concept representation in individual "features." 2) Direct connections: Transcoder layers receive direct input from all earlier transoder layers, passing features directly between layers. 3) Sparsity penalty: The loss penalizes activating too many features per layer. This encourages assigning information across independent features instead of creating concept superpositions in single neurons. Anthropic provides interesting insights based on this method. For example: LLMs produce coherent output over thousands of tokens while only predicting the next token. But how do they think ahead? The creation of poems illustrates this particularly well: If the first line is "He saw a carrot and had to grab it," the next sentence must rhyme with "it." Indeed, the model continues with "His hunger was like a starving rabbit.", and Anthropic's transcoder model shows how the "rabbit" concept builds up way before the word itself appears. What are problems with/questions around this new approach? 1) The transcoder isn't the original model. It's explanations might not apply to the LLM. This is a well-known problem with these "surrogate model" problem throughout ML. 2) Why not train the transcoder directly? Sparsity and connectivity make it much harder to train with lower accuracy. 3) Feature graphs are heavily pruned by humans, risking biased review and anthropomorphizing by the people analyzing the results. Despite this, it's exciting research with well-written, interactive papers and open-sourced analysis tools (see comments). The mechanistic approach to LLM interpretability is hard. But Anthropic has made great progress and I'm excited to see where the community goes next! #ai #genai #llm

Anthropic’s AI “brain scanner” reveals that large language models like Claude don’t think the way they claim. (1ish+2ish=3ish) When solving simple math, Claude doesn’t follow traditional arithmetic steps. Instead, it runs multiple parallel approximations, like adding “40ish and 60ish” or “57ish and 36ish” and merges these rough estimates. It also separately computes that 6+9 ends in a 5, then combines that with a 92-like estimate to land at 95. Yet when asked how it solved it, Claude replies with a textbook explanation: “I added the ones, carried the one…” a rationalization borrowed from training data, not its actual reasoning. This mismatch exposes a core issue: language models don’t reason like humans; they simulate plausible answers, even when the path to them is strange.

The next time an AI gives you an explanation about itself, remember: it’s not telling you the truth about its inner workings—it’s predicting what “someone in this situation” might say. It’s a story, not a confession. Large Language Models (LLMs) have no inner voice, no self-awareness, and no consistent personality. There’s nobody “home.” They don’t keep an internal diary of their actions. They can’t consult a log of their own mistakes. Instead, they generate plausible-sounding explanations based on patterns from training data—often confidently wrong. LLMs are incredible tools for generating ideas, summarizing, coding, and more—but they are not introspective agents. Treating them like self-aware beings is not just inaccurate—it can lead to bad decisions. https://lnkd.in/g7bHk_PJ