How to Use Memory Innovation in AI Hardware

I'm deep in the AI memory rabbit hole this week. Forget simple KV stores or fancy vector DBs acting like they've solved recall. Today’s deep dive is into MemOS, an open-source library that treats memory like a proper operating system framework with interfaces, operations, and infrastructure. Think of it as upgrading your agent's brain from sticky notes to a hypervisor managing cognitive resources. And yes, it's making my Qwen3 235B on-device runs significantly less... forgetful. Most projects out there often hyper-focus on external plaintext retrieval. MemOS integrates plaintext, activation, AND parameter memories – a proper memory hierarchy, not just a single-threaded fetch. It's like having RAM, cache, and disk, not just a single floppy drive. It doesn't just store memories; it manages them. Creation, activation (pulling into context), archiving (moving to cold storage), and expiration (the polite "forget this nonsense" signal). Full. Memory. Concierge. Service. Has fine-grained access control and versioning. Provenance tracking is baked into the data structure itself. No more wondering which hallucination spawned that terrible output or who gave the agent permission to recall your embarrassing internal docs. Audit trails are now a feature, not an afterthought. I’m watching MemOS automatically promote hot plaintext to faster activation memory (or demote cold activation back) based on usage patterns, and it’s pure sysadmin joy. It's like an LRU cache got a PhD in cognitive psychology and started optimizing itself. Efficiency? We got it. It works beautifully with serious on-device LLMs. I'm hammering it with Qwen3 235B locally, and the difference in coherent, context-aware persistence is noticeable. Less "wait, what were we talking about?", more "Ah yes, user, based on our conversation 47 interactions ago and the relevant archived parameter, I suggest..." Make sure you own your AI. AI in the cloud is not aligned with you; it’s aligned with the company that owns it.

Claude Code's source code leaked last week. 512,000 lines of TypeScript. Most people focused on the drama. I focused on the memory architecture. Here's how Claude Code actually remembers things across sessions — and why it's a masterclass in agent design: 𝗧𝗵𝗲 𝟯-𝗟𝗮𝘆𝗲𝗿 𝗠𝗲𝗺𝗼𝗿𝘆 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: 𝗟𝗮𝘆𝗲𝗿 𝟭 — 𝗠𝗘𝗠𝗢𝗥𝗬. 𝗺𝗱 (𝗔𝗹𝘄𝗮𝘆𝘀 𝗟𝗼𝗮𝗱𝗲𝗱) A lightweight index file. Not storage — pointers. Each line is under 150 characters. First 200 lines get injected into context at every session start. It points to topic files. It never holds the actual knowledge. Think of it as a table of contents, not the book. 𝗟𝗮𝘆𝗲𝗿 𝟮 — 𝗧𝗼𝗽𝗶𝗰 𝗙𝗶𝗹𝗲𝘀 (𝗢𝗻-𝗗𝗲𝗺𝗮𝗻𝗱) Detailed knowledge spread across separate markdown files. Architecture decisions. Naming conventions. Test commands. Loaded only when MEMORY. md says they're relevant. Not everything gets loaded. Only what's needed right now. 𝗟𝗮𝘆𝗲𝗿 𝟯 — 𝗥𝗮𝘄 𝗧𝗿𝗮𝗻𝘀𝗰𝗿𝗶𝗽𝘁𝘀 (𝗚𝗿𝗲𝗽-𝗕𝗮𝘀𝗲𝗱 𝗦𝗲𝗮𝗿𝗰𝗵) Past session transcripts are never fully reloaded. They're searched using grep for specific identifiers. Fast. Deterministic. No embeddings. No vector DB. Just plain text search when the first two layers aren't enough. But here's the part that blew my mind: 𝗦𝗸𝗲𝗽𝘁𝗶𝗰𝗮𝗹 𝗠𝗲𝗺𝗼𝗿𝘆. The agent treats its own memory as a hint, not a fact. Memory says a function exists? → Verify against the codebase first. Memory says a file is at this path? → Check before using it. And one more design principle hidden in the code: If something can be re-derived from source code — it doesn't get stored. Code patterns, conventions, architecture? Excluded from memory saves entirely. Because if it can be looked up, it shouldn't be remembered. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗯𝗲𝘆𝗼𝗻𝗱 𝗖𝗹𝗮𝘂𝗱𝗲 𝗖𝗼𝗱𝗲: This 3-layer pattern is model-agnostic. Any team building AI agents can steal this: → Keep your always-loaded context tiny → Reference everything else via pointers → Never persist what can be looked up → Treat memory as a hint, not truth The future of AI agents isn't about how much they remember. It's about how well they forget. What memory patterns are you using in your agent builds?

Launching Glance AI was never just an engineering challenge. It was a relentless tug-of-war between accuracy, user delight, and cost. Early on, we doubled down on delivering value and precision, knowing that at small scale, costs wouldn’t limit us. But as our user base exploded, and with every iteration running on GPUs, the team had to master every angle, from token optimization to splitting training from inference. The payoff? Glance AI now outperforms industry benchmarks for both cost-efficiency and fidelity. But our dependency on a single GPU class still lingered. So, when Jonathan Ross (the creator of Google’s pioneering TPU and now CEO of Groq) visited us, it gave us many more ideas. We’ve been experimenting with TPUs to streamline training and inference, but Groq’s chip, the Language Processing Unit (LPU), looks very promising. It’s a leap in AI hardware design, using colossal amounts of SRAM (up to 230MB per chip, nearly 10x more than top GPUs) and delivering unprecedented memory bandwidth (nearly 80TB/s, 25x higher than the best H100 GPUs). This means instantaneous data movement, blazing speeds, and a dramatic cut in bottlenecks. Here’s what blew me away about Groq: ➡ No DRAM bottleneck: All live data for inference stays in ultra-fast SRAM, eliminating DRAM/HBM delays and accelerating LLM responses. ➡ Single-core simplicity: Groq’s Tensor Streaming Processor ditches GPU multi-core complexity for streamlined, predictable workflows led by software instead of clunky hardware synchronizations. ➡ Assembly-line architecture: Compared to GPUs’ hub-and-spoke layout, data and compute flow seamlessly, making programming fast and dead times a thing of the past. ➡ Software-led execution: All planning is handled by software, liberating silicon for raw compute. Caching and hardware sync layers? Gone. More resources for solving problems, not shuffling data. ➡ Chip-level orchestration: Hundreds of Groq chips sync as one “virtual core," a feature that’s crucial for scaling huge LLMs. Beyond the pure speed and efficiency, Groq’s software-first approach is a paradigm shift, unlocking new possibilities for model deployment at scale. Now I am all for benchmarking Glance AI inference on Groq chips. Stay tuned: We will be sharing our results. Abhay Singhal | Arvind Jayaprakash | Debleena Das | Raman Srinivasan | Sudheer Bhat | Vivek Y S | Srikanth Sundarrajan InMobi InMobi Advertising Glance #AI #Hardware #Groq #Innovation #GlanceAI #EngineeringExcellence

I just finished reading three recent papers that every Agentic AI builder should read. As we push toward truly autonomous, reasoning-capable agents, these papers offer essential insights, not just new techniques, but new assumptions about how agents should think, remember, and improve. 1. MEM1: Learning to Synergize Memory and Reasoning Link: https://bit.ly/4lo35qJ Trains agents to consolidate memory and reasoning into a single learned internal state, updated step-by-step via reinforcement learning. The context doesn’t grow, the model learns to retain only what matters. Constant memory use, faster inference, and superior long-horizon reasoning. MEM1-7B outperforms models twice its size by learning what to forget. 2. ToT-Critic: Not All Thoughts Are Worth Sharing Link: https://bit.ly/3TEgMWC A value function over thoughts. Instead of assuming all intermediate reasoning steps are useful, ToT-Critic scores and filters them, enabling agents to self-prune low-quality or misleading reasoning in real time. Higher accuracy, fewer steps, and compatibility with existing agents (Tree-of-Thoughts, scratchpad, CoT). A direct upgrade path for LLM agent pipelines. 3. PAM: Prompt-Centric Augmented Memory Link: https://bit.ly/3TAOZq3 Stores and retrieves full reasoning traces from past successful tasks. Injects them into new prompts via embedding-based retrieval. No fine-tuning, no growing context, just useful memories reused. Enables reasoning, reuse, and generalization with minimal engineering. Lightweight and compatible with closed models like GPT-4 and Claude. Together, these papers offer a blueprint for the next phase of agent development: - Don’t just chain thoughts; score them. - Don’t just store everything; learn what to remember. - Don’t always reason from scratch; reuse success. If you're building agents today, the shift is clear: move from linear pipelines to adaptive, memory-efficient loops. Introduce a thought-level value filter (like ToT-Critic) into your reasoning agents. Replace naive context accumulation with learned memory state (a la MEM1). Storing and retrieving good trajectories, prompt-first memory (PAM) is easier than it sounds. Agents shouldn’t just think, they should think better over time.

Everyone's adding "memory" to their AI agents. Almost nobody's adding actual memory. Your vector database isn't memory. It's one Post-it note in an 8-drawer filing cabinet. Building Synnc's LangGraph agents taught us this the hard way. Here are 8 memory types — and the stack we actually use: 1) Context Window Memory ↳ The LLM's immediate working RAM ↳ We cap at 80% capacity to leave room for tool responses 2) Conversation Buffer ↳ Multi-turn dialogue persistence ↳ LangGraph checkpointers handle this natively 3) Semantic Memory ↳ Long-term user knowledge + preferences ↳ Mem0 gives us cross-session personalization out of the box 4) Episodic Memory ↳ Learning from past agent successes/failures ↳ Mem0 stores interaction traces → feeds few-shot examples 5) Tool Response Cache ↳ Stop paying for the same API call twice ↳ Redis gives us <1ms latency + native LangGraph integration 6) RAG Cache ↳ Embedding + retrieval deduplication ↳ Pinecone handles vector storage + similarity search 7) Agent State Store ↳ Time-travel debugging for complex workflows ↳ LangGraph + Redis checkpointing → rewind to any decision point 8) Procedural Memory ↳ Guardrails + consistent agent behavior ↳ Baked directly into our LangGraph node structure Our stack: LangGraph + Mem0 + Redis + Pinecone 4 products. 8 memory layers covered. The result? → 70% faster debugging (time-travel to any state) → 40% lower API costs (Redis caching) → Day-one personalization (Mem0 cross-session memory) Memory architecture isn't optional anymore. What's your agent memory stack?

We used to worry about mobile data limits - today, the tech world’s biggest anxiety is power. The skyrocketing energy consumption of GPUs during LLM inference isn't just an environmental concern, it's an engineering bottleneck. Standard infrastructure is incredibly wasteful. As someone deep in large-scale AI architecture, I knew we couldn't just keep throwing more GPUs at the problem. The real culprit isn't raw compute; it’s memory bandwidth and the KV Cache. When an LLM recalls conversation history, standard systems struggle with redundancy. They reload massive amounts of data or inefficiently access shared memory. Moving all that data between VRAM and the chip is exactly what drives up the wattage per token. We needed to rethink memory access entirely - that’s where my patent for Vectors and RadixAttention comes in. Instead of treating the KV cache as fragmented pages, RadixAttention uses a Radix Tree structure to index it. The game-changer? It recognizes shared context instantly. If multiple users query an LLM on the same document, that context is stored once and accessed by everyone, with zero redundant data movement. By fundamentally solving the KV cache redundancy problem, the impact was massive: ⚡️ Significantly Lower VRAM Usage: Eliminated duplicate storage, enabling larger models and more concurrent users on existing hardware. 🍃 Drastic Wattage Drop: Less data movement equals vastly less energy consumed per token. 🚀 Unprecedented Efficiency:Faster, radically more cost-effective inference at scale. The future of AI isn't just about building bigger models or faster chips, it's about designing smarter architecture. We can't ignore the energy bill of innovation. Proud to be building the infrastructure for a sustainable AI future.

You're in a Senior AI Engineer interview at Meta and the interviewer asks: "Instead of relying on 𝘗𝘺𝘛𝘰𝘳𝘤𝘩'𝘴 𝘣𝘶𝘪𝘭𝘵-𝘪𝘯 𝘢𝘶𝘵𝘰𝘨𝘳𝘢𝘥 𝘦𝘯𝘨𝘪𝘯𝘦, in what highly constrained production scenario does writing 𝘤𝘶𝘴𝘵𝘰𝘮 𝘧𝘰𝘳𝘸𝘢𝘳𝘥 𝘢𝘯𝘥 𝘣𝘢𝘤𝘬𝘸𝘢𝘳𝘥 𝘱𝘢𝘴𝘴𝘦𝘴 𝘧𝘳𝘰𝘮 𝘴𝘤𝘳𝘢𝘵𝘤𝘩 become an absolute engineering necessity?" Most candidates say: "It's to understand the underlying math better, or maybe to implement a custom mathematical function that isn't naturally differentiable." Wrong approach. Too academic. The reality is that standard dynamic computational graphs are memory hogs. In high-performance production environments, like training massive LLMs or serving real-time edge AI, you aren't bottlenecked by raw compute (FLOPs). You are memory bandwidth bound. Relying on standard autograd means storing massive intermediate activation tensors in VRAM during the forward pass just so the framework can compute gradients later. It's like renting a massive commercial warehouse just to store the empty cardboard boxes of the parts you're currently assembling. Here is exactly why we need to bypass autograd and write custom backward passes: 1️⃣ 𝘈𝘨𝘨𝘳𝘦𝘴𝘴𝘪𝘷𝘦 𝘒𝘦𝘳𝘯𝘦𝘭 𝘍𝘶𝘴𝘪𝘰𝘯: Standard autograd reads and writes intermediate tensors to global memory for every single operation. Writing a custom backward pass allows you to fuse operations at the hardware level, keeping data in the GPU's ultra-fast SRAM. (This is exactly the secret sauce behind architectures like FlashAttention). 2️⃣ 𝘝𝘙𝘈𝘔 𝘚𝘶𝘳𝘷𝘪𝘷𝘢𝘭: By manually defining the backward pass, you can mathematically recompute intermediates on the fly instead of saving them, drastically slashing your activation memory footprint to fit larger batch sizes. 3️⃣ 𝘕𝘢𝘬𝘦𝘥 𝘌𝘥𝘨𝘦 𝘋𝘦𝘱𝘭𝘰𝘺𝘮𝘦𝘯𝘵: If you are deploying to bare-metal embedded systems, microcontrollers, or extreme edge devices, dragging along the massive PyTorch runtime and its dynamic graph overhead is impossible. You need stripped-down, compiled C/C++ passes. 𝐓𝐡𝐞 𝐚𝐧𝐬𝐰𝐞𝐫 𝐭𝐡𝐚𝐭 𝐠𝐞𝐭𝐬 𝐲𝐨𝐮 𝐡𝐢𝐫𝐞𝐝: "We bypass standard autograd when we hit the memory bandwidth wall. Writing custom forward and backward kernels enables aggressive kernel fusion and VRAM optimization that standard dynamic graphs fundamentally cannot achieve." #MachineLearning #AIEngineering #PyTorch #DeepLearning #MLOps #DataScience #SoftwareEngineering

LLMs have hit a memory wall — and our ASPLOS 2026 paper tackles it head-on. This work is a proud outcome of the IBM-RPI Future of Computing Research Collaboration (FCRC) co-supervised with Prof. @liu liu. Congratulations to lead author Zehao Fan and all co-authors: Yunzhen Liu, Garrett Gagnon, Zhenyu Liu, Yayue Hou, and Hadjer Benmeziane. Read the full paper: https://lnkd.in/dRRU7T7k For a technical deep dive on this work, check out my blog at: https://lnkd.in/dYKzsCMF During decoding, LLMs perform one matrix-vector multiply per token. The GPU spends more time waiting for data than computing. Processing-in-Memory (PIM) brings computation closer to data, but existing PIM designs assume dense attention and struggle with the irregular access patterns of sparse token retrieval. STARC solves this with a simple idea: cluster semantically similar key-value pairs and co-locate them in contiguous PIM memory rows. This makes sparsity hardware-visible — enabling real computation skipping at the memory level. Results: up to 93% latency reduction and 92% energy reduction on the attention layer, with no loss in model accuracy. #ASPLOS2026 #LLMInference #ProcessingInMemory #AIHardware #EfficientAI #IBMResearch #RPI #FCRC

AI agents without proper memory are just expensive chatbots repeating the same mistakes. After building 50+ production agents, I discovered most developers only implement 1 out of 5 critical memory types. Here's the complete memory architecture powering agents at Google, Microsoft, and top AI startups: 𝗦𝗵𝗼𝗿𝘁-𝘁𝗲𝗿𝗺 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗪𝗼𝗿𝗸𝗶𝗻𝗴 𝗠𝗲𝗺𝗼𝗿𝘆) → Maintains conversation context (last 5-10 turns) → Enables coherent multi-turn dialogues → Clears after session ends → Implementation: Rolling buffer/context window 𝗟𝗼𝗻𝗴-𝘁𝗲𝗿𝗺 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗣𝗲𝗿𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗦𝘁𝗼𝗿𝗮𝗴𝗲) Unlike short-term memory, long-term memory persists across sessions and contains three specialized subsystems: 𝟭. 𝗦𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗕𝗮𝘀𝗲) → Domain expertise and factual knowledge → Company policies, product catalogs → Doesn't change per user interaction → Implementation: Vector DB (Pinecone/Qdrant) + RAG 𝟮. 𝗘𝗽𝗶𝘀𝗼𝗱𝗶𝗰 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗘𝘅𝗽𝗲𝗿𝗶𝗲𝗻𝗰𝗲 𝗟𝗼𝗴𝘀) → Specific past interactions and outcomes → "Last time user tried X, Y happened" → Enables learning from past actions → Implementation: Few-shot prompting + event logs 𝟯. 𝗣𝗿𝗼𝗰𝗲𝗱𝘂𝗿𝗮𝗹 𝗠𝗲𝗺𝗼𝗿𝘆 (𝗦𝗸𝗶𝗹𝗹 𝗦𝗲𝘁𝘀) → How to execute specific workflows → Learned task sequences and patterns → Improves with repetition → Implementation: Function definitions + prompt templates When processing user input, intelligent agents don't query memories in isolation: 1️⃣ Short-term provides immediate context 2️⃣ Semantic supplies relevant domain knowledge 3️⃣ Episodic recalls similar past scenarios 4️⃣ Procedural suggests proven action sequences This orchestrated approach enables agents to: - Handle complex multi-step tasks autonomously - Learn from failures without retraining - Provide contextually aware responses - Build relationships over time LangChain, LangGraph, and AutoGen all provide memory abstractions, but most developers only scratch the surface. The difference between a demo and production? Memory that actually remembers. Over to you: Which memory type is your agent missing?

This is the only guide you need on AI Agent Memory 1. Stop Building Stateless Agents Like It's 2022 → Architect memory into your system from day one, not as an afterthought → Treating every input independently is a recipe for mediocre user experiences → Your agents need persistent context to compete in enterprise environments 2. Ditch the "More Data = Better Performance" Fallacy → Focus on retrieval precision, not storage volume → Implement intelligent filtering to surface only relevant historical context → Quality of memory beats quantity every single time 3. Implement Dual Memory Architecture or Fall Behind → Design separate short-term (session-scoped) and long-term (persistent) memory systems → Short-term handles conversation flow, long-term drives personalization → Single memory approach is amateur hour and will break at scale 4. Master the Three Memory Types or Stay Mediocre → Semantic memory for objective facts and user preferences → Episodic memory for tracking past actions and outcomes → Procedural memory for behavioral patterns and interaction styles 5. Build Memory Freshness Into Your Core Architecture → Implement automatic pruning of stale conversation history → Create summarization pipelines to compress long interactions → Design expiry mechanisms for time-sensitive information 6. Use RAG Principles But Think Beyond Knowledge Retrieval → Apply embedding-based search for memory recall → Structure memory with metadata and tagging systems → Remember: RAG answers questions, memory enables coherent behavior 7. Solve Real Problems Before Adding Memory Complexity → Define exactly what business problem memory will solve → Avoid the temptation to add memory because it's trendy → Problem-first architecture beats feature-first every time 8. Design for Context Length Constraints From Day One → Balance conversation depth with token limits → Implement intelligent context window management → Cost optimization matters more than perfect recall 9. Choose Storage Architecture Based on Retrieval Patterns → Vector databases for semantic similarity search → Traditional databases for structured fact storage → Graph databases for relationship-heavy memory types 10. Test Memory Systems Under Real-World Conversation Loads → Simulate multi-session user interactions during development → Measure retrieval latency under concurrent user loads → Memory that works in demos but fails in production is worthless Let me know if you've any questions 👋