Zhongbo (Bobby) Chen

NVIDIA's DGX Spark started shipping this week. $3,999 for a petaflop on your desk. Research labs currently skip experiments because justifying cloud spend takes three approvals. Startups burn runway on compute before knowing if ideas work. Teams prototype in the cloud, then find their models behave differently in production. Run 200B parameter models locally. Fine-tune 70B models without cloud dependency. Link two units for 405B parameters. The development cycle tightens considerably. The GB10 Grace Blackwell chip is purpose-built for this—MediaTek collaborated on the CPU side, 128GB of memory sits between the CPU and GPU without the usual bottleneck. Different from retrofitting gaming hardware for AI workloads. Training frontier models still requires massive infrastructure. Production inference at scale needs proper compute. But prototyping, testing, fine-tuning—that friction just dropped significantly. Useful ideas die when the barrier to experiment is too high. The path from concept to test matters. $4,000 isn't nothing, but it's orders of magnitude less than building a cluster. NVIDIA's framing this as "democratizing AI"—partially marketing, partially true. The number of teams who can now iterate at this level just expanded considerably.

77

4 Comments

You spend months building the perfect PyTorch model. Then the real nightmare begins. Porting it. - One version for your flagship mobile app. - Another for that new wearable. - A third for the tiny IoT sensor. Each one needs different optimizations, different pipelines, different frameworks. It's a fragmented, time-sucking mess that kills your time-to-market. This is the single biggest bottleneck holding back true, at-scale edge AI. *Until now.* What if you could just... stop? What if you could use one unified workflow to deploy that one model across BILLIONS of devices? From ultra-efficient microcontrollers to flagship smartphones. From Arm Cortex-M CPUs to high-performance Ethos-U NPUs and Mali GPUs. *This isn't a "what if" anymore.* Meta and Arm just made it a practical reality. Introducing the ExecuTorch 1.0 GA (General Availability) release. This is the on-device runtime for PyTorch that developers have been waiting for. It's one toolset to rule them all. Developers can now author, export, optimize, quantize, and deploy using the same end-to-end PyTorch workflow. The best part? Your apps automatically benefit from performance and efficiency gains. Backend integrations with Arm KleidiAI, TOSA, and CMSIS-NN mean you get optimized performance "for free," with no need to modify your code. This is how we get the real promise of edge AI. Not just cloud-tethered apps, but... ➡️ Private, on-device assistants that run Llama 3. ➡️ Real-time audio generation (Stable Audio in <4 secs). ➡️ Smarter, power-efficient wearables. ➡️ Gaming experiences that adapt in real-time. Meta is already using this to power features for billions of users on Instagram, WhatsApp, and Facebook. Now, it's available to all developers. The fragmented, "port-it-again" days of edge AI are over. The "build-once, deploy-everywhere" era is here. Arm and Meta have dropped the full GA release, docs, tutorials, and pre-validated models. It's all in the blog post here: https://lnkd.in/ggj2rYCT I want to hear from the builders: - How will a single, unified PyTorch workflow change the way you develop for the edge? - What's the first on-device app you're excited to build with this? Drop your thoughts below 👇 and share this with every AI developer you know. This is a big one.

73

3 Comments

We (w/ Shansan Gong, Ruixiang ZHANG, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong) released a family of 7B diffusion language models, DiffuCoder, that specializes on code generation, with a focus on understanding and improving masked diffusion models. A core analysis of DiffuCoder is the autoregressiveness (AR-ness) score, a novel metric that quantifies the causal patterns in decoding, revealing how diffusion models break from strict left-to-right generation for more flexible, non-linear code planning. Recent advances in autoregressive (AR) models dominate code generation, but diffusion-based LLMs (dLLMs) like DiffuCoder offer a promising alternative, especially for complex programming tasks. DiffuCoder explores how these models decode differently—showing less global AR-ness in code tasks compared to math—and how temperature affects both token selection and generation order, unlike traditional AR models. We also introduce coupled-GRPO, a post-training RL method with a coupled-sampling scheme, to reduce performance drops during accelerated decoding, boosting parallelism and efficiency. We use a self-improvement pipeline that leverages AR-ness analysis, coupled-GRPO optimization, and evaluation on benchmarks like AceCode-89k to refine decoding strategies. This approach enables DiffuCoder to navigate diverse code generation pathways and enhance performance with modest computational overhead. Looking ahead, we aim to further leverage Reinforcement Learning to steer code generation through these decoding patterns, with the discrete nature of AR-ness scores providing a foundation for search-based strategies—ideal for the sparse rewards of optimizing complex code structures. Check out our full paper and code for a deeper dive! Paper: https://lnkd.in/gVWU3BDJ Code: https://lnkd.in/gmXTZ_6n Models: https://lnkd.in/gTcKCDr9 #MachineLearning #AI #CodeGeneration #DiffusionModels #NLP

220

5 Comments

I've been discussing this recently with Frances Buontempo in the context of coding assistants. We hear a lot from developers saying that LLM assistants in their fave IDEs are great for generating a lot of the (boring) boilerplate code needed to stand up a non-trivial system. However, most popular IDEs have been able to do exactly this for decades--whether with "wizards", code-block and project templates, or other various forms of meta-programming. The difference is that the "old" way is much less hungry of resources, while simultaneously being much more predictable and reliable at automatically generating the "boring" stuff. Coding assistants are also good at other things, of course, but many of the popular (in my conversations with developers) tasks are those I would much prefer to do myself, such as writing tests.

15

6 Comments

TL;DR: Cascade RL and even Cascade smaller LR SFT for mitigating catastrophic forgetting. Nvidia recently showed that Cascade RL can mitigate catastrophic forgetting. Really excellent and exciting work! (https://lnkd.in/gDwKxG5f) We also observed similar effects in healthcare domains (clinical trial tasks) in our recent paper: Developing Large Language Models for Clinical Research Using One Million Clinical Trials https://lnkd.in/gNCf9M4i. Beyond Cascade RL, we also find that Cascade SFT with a smaller learning rate can significantly suppress catastrophic forgetting, even across highly heterogeneous clinical trial tasks. This is based on the findings in my previous paper below: SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs https://lnkd.in/gqWJdvmX.

58

2 Comments