Neural Network Advancements

As AI+Science went more mainstream in 2025, our team’s seminal contributions to the field are getting wide recognition. Here are top professional contributions and achievements for 2025. 1. Medical Imaging: We applied Neural Operators as a universal AI scheme that can handle any subsampling scheme and can do zero-shot super-resolution and field of view, without the need for any retraining. We applied it to a range of modalities such as MRI, CT, ultrasound, and photo-acoustic imaging. 2. AI Weather and Climate Models: I led the creation of the first AI-based weather model FourCastNet, built on Neural Operators, back in 2021. This year we announced FourCastNet 3, the fastest AI based model to provide calibrated probabilistic answers, crucial for extreme weather events. This also serves as backbone for state of art AI-based climate models . 3. De-Novo Inverse Design of Physical Devices: We were able to design new devices that were previously out of reach in challenging systems such as gate design in quantum dots, controlling quantum systems and non-linear photonics, using Fourier Neural Operator (FNO). 4. Scientific Modeling: FNOs achieved modeling of bio-realistic neurons, quantum dynamics and black holes with significant speedups while maintaining fidelity. 5. Millennium prize in fluid dynamics: We developed high-precision physics-informed neural networks (PINN) to solve a key step in computer-assisted proofs of singularities. 6. Physics-informed chemistry: Orbitall is the first universal quantum-chemical AI model that can handle at any spin, charge and external fields, and can extrapolate to larger molecules than those in training data. Nucleusdiff improves structure-based drug design by generating physically plausible molecules that maintain proper atomic spacing. We were able to beat previously known natural and engineered enzymes in functionality and versatility using protein-language models (genSLM). 7. Neural Operator foundations: We developed a unified framework to convert many popular neural networks like convolutional and graph neural networks, transformers etc to Neural Operators. We improved generalization to different geometries and scales. We developed FunDPS, a diffusion based inverse problem solver on function spaces. It is a resolution-agnostic unified framework for both forward and inverse PDEs. We also established limitations of hybrid learning that combine numerical solvers with learned closures and superiority of operator learning. 8. Verified Learning in LLMs: We released LeanDojo v2, LeanAgent and LeanProgress for theorem proving. 9. TIME 100 Impact Award and IEEE Kiyo Tomiyasu Award. 10. Group members Zongyi Li and Miguel Liu-Schiaffini winning best graduate and undergraduate research at Caltech commencement for work on Neural Operators and alum Zhuoran Qiao winning the Tianqiao and Chrissy Chen Institute AI+Science prize.

🔥 Revolutionizing PINNs: Introducing Self-Adaptive Physics-Informed Neural Networks (SA-PINNs) 🌐 Excited to share insights from a fascinating research paper that takes Physics-Informed Neural Networks (PINNs) to the next level: "Self-Adaptive Physics-Informed Neural Networks" Authored by Levi McClenny and Ulisses Braga-Neto Why it matters: Traditional PINNs are powerful tools for solving PDEs, but they struggle with "stiff" problems that involve sharp transitions or fast dynamics. This paper introduces Self-Adaptive PINNs (SA-PINNs), a groundbreaking approach that uses trainable self-adaptive weights to prioritize difficult solution regions, enhancing accuracy and efficiency. 💡 Key Innovations Dynamic Focus: Adaptive weights automatically highlight stubborn areas in the solution, akin to attention mechanisms in computer vision. Improved Training: Concurrent optimization of network and adaptive weights ensures faster convergence and higher accuracy. Gaussian Process Regression: Enables robust training using stochastic gradient descent for challenging problems. Theoretical Insights: NTK analysis reveals how SA-PINNs smooth training dynamics and balance loss components effectively. 🚀 Results - Outperformed state-of-the-art PINN methods in L2 error across benchmarks. - Solved stiff PDEs with significantly fewer training epochs. - Showed exceptional robustness in handling sharp transitions and complex dynamics. This work demonstrates the potential of SA-PINNs to transform scientific computing, making them indispensable for solving challenging PDEs in physics, engineering, and beyond. 📖 Read the full paper: https://lnkd.in/djnVfpe6

Kolmogorov-Arnold Networks as an alternative to traditional Neural Networks! Researchers from MIT, Caltech, and Northeastern have introduced a new type of neural network architecture known as Kolmogorov-Arnold Networks (KANs), which presents a significant challenge to the traditional use of Multi-Layer Perceptrons (MLPs). KANs offer a novel approach to neural network architecture inspired by the Kolmogorov-Arnold representation theorem. This theorem essentially states that any multivariate continuous function can be represented as a composition of univariate functions and the addition operation. Translating this into neural network design, KANs uniquely place adaptable activation functions on the connections or edges between nodes rather than using standard fixed activation functions at the nodes themselves. This flexibility allows KANs to potentially model complex relationships and patterns more effectively, as they can tailor the transformation at each connection to better suit the specific data and task at hand, diverging from traditional networks where the choice of activation function at each layer is static and uniform across the network. In terms of accuracy, much smaller KANs can achieve comparable or better performance than larger MLPs on tasks such as data fitting and PDE solving. Moreover, KANs demonstrate faster neural scaling laws, meaning their performance improves more rapidly with increased model size compared to MLPs. KANs also excel in interpretability. They can be intuitively visualized and allow for easy interaction with human users. In case studies from knot theory and physics, KANs served as interactive "collaborators" to help scientists rediscover known mathematical and physical laws, showcasing their potential for scientific discovery. KANs could potentially serve as a foundation model for AI+Science applications and open opportunities to improve today's deep learning models that heavily rely on MLPs. Read the full paper for more details: https://lnkd.in/erEF6HbT :)

🧠 Titans and Transformers Unite! What if AI models could learn and memorize information in real-time, just like humans do? New research (link in comments) from Google introduces "Titans" - a new architecture that's challenging how we think about AI memory. The key innovation? A neural long-term memory module that learns to identify and store surprising or important information during inference, similar to how human memory prioritizes unexpected events. Three fascinating findings: - Titans outperformed both Transformers and modern recurrent models across multiple tasks, while scaling to massive 2M+ context windows - far beyond traditional limits. - The architecture introduces a "surprise-based" memory system, measuring both immediate surprise and the flow of information over time. This helps it determine what's truly worth remembering. - In needle-in-haystack tasks, Titans achieved 98.6% accuracy on 16K sequences - significantly outperforming GPT-4 and other large language models, despite using far fewer parameters. Titans introduces a two-tier memory system: - Short-term: Uses attention for precise, immediate understanding - Long-term: A neural memory module that learns what's worth remembering, just like our brains prioritize surprising or important events The real breakthrough? Titans can learn during deployment: - Adapts its memory in real-time - Uses "surprise metrics" to decide what to remember - Maintains fast training AND inference speeds The implications? We might be seeing the emergence of AI systems that can learn and adapt during deployment, rather than remaining static after training. What do you think - could this approach to AI memory revolutionize how we build adaptive systems? #MachineLearning #AI #DeepLearning #NeuralNetworks #Innovation

🧪 New Machine Learning Research: Optimizing Neural Networks with MetaMixer Researchers from the University of Seoul-서울시립대학교 have conducted a study on improving the efficiency and performance of neural networks through a new architecture called MetaMixer. - Research goal: Propose a new mixer architecture, MetaMixer, to optimize neural network performance by focusing on the query-key-value framework rather than self-attention. - Research methodology: They have developed MetaMixer by replacing inefficient sub-operations of self-attention with Feed-Forward Network (FFN) operations, and evaluated the performance across various tasks. - Key findings: MetaMixer, using simple operations like convolution and GELU activation, outperforms traditional methods. The study found that the new FFNified attention mechanism improves efficiency and performance in diverse tasks. - Practical implications: These advancements can lead to more efficient neural networks, reducing computational costs and improving the performance of AI models in applications such as image recognition, object detection, and 3D semantic segmentation. #LabelYourData #TechNews #DeepLearning #Innovation #AIResearch #MLResearch 

Most neural networks see the input data as a matrix or a vector, like pixels in rows and columns or words in a sequence. A lot of real-world data is better described as a graph: a set of nodes connected by edges, like social networks, power grids, or molecules. Graph neural networks try to process this kind of data, and the most common approach before this paper (GCNs) treated every neighbor of a node as equally important when aggregating information. This ICLR 2018 paper, published by Bengio's team and since then garnered more than 18,000 citations, introduces Graph Attention Networks (GATs), which borrow the idea of attention in language models to let each node learn which of its neighbors matter more for a given task. The attention weights are computed using a small shared neural network that looks at pairs of connected nodes, so the model doesn't need to know the full graph structure ahead of time and can generalize to entirely new graphs it's never seen during training. It's an exemplary AI research paper. The architecture is simple enough to describe in a few equations, runs efficiently in parallel, and at the time matched or beat everything else on standard benchmarks. Read online with an AI tutor: https://lnkd.in/e2a_RBht Read offline on your own: https://lnkd.in/eFVrT2kz

2026 - starting the year strong 💪 My colleagues at Google Research published a new paper in Science Advances that marks a significant step forward for large-scale precipitation forecasts. We’ve trained our hybrid AI-physics model, NeuralGCM, directly on NASA satellite observations to simulate global precipitation with a 40% average error reduction over land compared to leading climate models in multi-year runs. Precise precipitation forecasting is one of the "holy grails" of climate science—and it’s notoriously difficult because clouds are at smaller scales than traditionally modeled ☁️. Precipitation forecasts are so relevant in multiple scenarios: it's about knowing whether a farmer should plant seeds today or if a city needs to prepare for a 100-year storm. Here is why this development is a game-changer: ☁️ Smarter Tuning (compared to traditional models): Traditional models rely on fixed equations (parameterizations) that are difficult to tune perfectly for every scenario and rarely utilize the vast data available. NeuralGCM uses neural networks that are trained "online"—meaning they learn to work in harmony with the large-scale physics solver. ☁️ Learning Directly from Observations (compared to other hybrid models or ML models): While most AI models learn from "reanalysis" data (a mix of observations and model physics that can carry biases), NeuralGCM is trained directly on NASA satellite data. This allows the model to align its precipitation predictions with the best available record of actual rainfall. ☁️ Capturing Extremes:  NeuralGCM is significantly better at capturing extreme precipitation which traditional models often under-predict. ☁️ Correcting the Clock:  While many models predict peak rain too early in the day , NeuralGCM accurately reproduces the timing of peak precipitation, especially in complex regions like the Amazon. ☁️ Real-World Application:  This isn’t just theoretical. This past summer, a partnership with the University of Chicago and the Indian Ministry of Agriculture used NeuralGCM to provide AI-based monsoon forecasts for 38 million farmers. AI is learning the "parameterizations" of complex small-scale physics (like cloud formation) that have baffled traditional models for decades. A huge congratulations to Janni Yuval, Stephan Hoyer, Dmitrii Kochkov, Ian Langmore, Michael Brenner, Lizzie Dorfman, Olivia Graham, and the entire team for pushing the boundaries of what's possible for our planet’s resilience. Read the full story on the Google Research blog: https://lnkd.in/ga8V5jq8 Paper: https://lnkd.in/g3wfG4q2

Evaluating Neural Networks at the Speed of Light (with Light!). See live optical inference in the video below. Excited to share recent academic work on optical neural networks as a collection of computing elements embedded in the camera lens! These elements perform computation optically even before an image is captured, using the photons in the scene instead of GPU computation after the capture. We were able to achieve ImageNet classification more than two orders of magnitude faster than conventional neural networks on today's GPUs at almost no power consumption! To do this, we developed an array of metalenses that perform this computation on light from the scene. Project: https://lnkd.in/eaW5p9KT Paper: https://lnkd.in/eC_dt9RM Amazing collaboration with Kaixuan Wei, Xiao Li, Johannes Froech, Praneeth Chakravarthula, James Whitehead, Ethan Tseng, Arka Majumdar .

Imagine trying to predict how a car’s suspension reacts after a sudden jolt, how a skyscraper oscillates during an earthquake, or how the human heart maintains its rhythmic beat—each of these is governed by the physics of a harmonic oscillator, one of the most foundational systems in classical mechanics. Accurately modeling such behavior is not just a theoretical exercise—it’s essential for designing safe vehicles, resilient buildings, and advanced medical diagnostics. But in the real world, data is often incomplete, noisy, or expensive to collect. This is where Physics-Informed Neural Networks (PINNs) offer a revolutionary leap forward. PINNs don’t just learn from data; they learn from the laws of nature. For example, when applied to a harmonic oscillator, the PINN internalizes the dynamic interplay between inertia, damping, and restoring force. This means it can predict the system’s future behavior with high accuracy, even under unseen conditions. In real-life applications, this approach translates to smarter, faster, and safer solutions. Engineers can simulate how a bridge vibrates under wind stress without running countless expensive physical tests. Medical researchers can model how the heart or lungs respond to certain conditions without invasive procedures. Climate scientists can better understand oscillatory weather patterns even with limited historical records. By uniting physics and AI, PINNs go beyond conventional machine learning—they build trustworthy intelligence, rooted in science and ready for real-world impact. In a world where precision saves time, money, and lives, Physics-Informed Neural Networks are not just a tool—they are a paradigm shift. Feel free to share your thoughts 💭

The explosive growth in computation and energy cost of artificial intelligence has spurred interest in alternative computing modalities to conventional electronic processors. Photonic processors, which use photons instead of electrons, promise optical neural networks with ultralow latency and power consumption. However, existing optical neural networks, limited by their designs, have not achieved the recognition accuracy of modern electronic neural networks. In a recent work published in Science Advances, we bridge this gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture, before recording on the sensor. We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization. We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. The resulting setup is extremely simple: just replace a camera lens with our flat optics!! Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era. The paper can be found at: https://lnkd.in/gXUcn33Y