Evolutionary Algorithms

🧬 Reimagining Genetic Algorithms with LLMs in OpenEvolve I wanted to share some technical insights about how we've implemented evolutionary algorithms in OpenEvolve, our open-source implementation of Google DeepMind's AlphaEvolve system. Unlike traditional genetic algorithms with explicit mutation and crossover operators, OpenEvolve leverages the power of Large Language Models to create a more sophisticated evolutionary process: 🔄 Mutation via LLMs Instead of random bit flips or simple code transformations, OpenEvolve uses an ensemble of LLMs (Gemini-Flash-2.0-lite + Gemini-Flash-2.0) to generate sophisticated code modifications. These models understand programming concepts and can make targeted changes or complete rewrites based on the problem context. 🌿 Selection with MAP-Elites Our selection mechanism combines the MAP-Elites algorithm with an island-based population model. This maintains diversity across multiple feature dimensions while balancing exploration and exploitation - crucial for breaking through optimization plateaus. 🧩 Implicit Crossover Traditional crossover explicitly combines segments from two parent solutions. In OpenEvolve, "crossover" happens implicitly - we feed multiple high-performing programs as inspiration to the LLM, which then combines concepts in ways far more sophisticated than traditional bit-swapping could achieve. 📊 Cascade Evaluation Our fitness evaluation uses a multi-stage approach where promising solutions gradually undergo more intensive evaluation - similar to tournament selection but with progressive complexity. This architecture allowed us to successfully replicate AlphaEvolve's results, evolving from simple geometric patterns to sophisticated mathematical optimization in the circle packing problem (achieving a perfect match with DeepMind's reported 2.635 sum of radii). The most exciting finding? Traditional mutation operators would never discover scipy.minimize on their own, but our LLM-driven evolution did - showing how this approach can navigate complex solution spaces in ways classical genetic algorithms simply cannot. For those interested in the technical details, our implementation is available at https://lnkd.in/gbBgJRau - with database.py (selection) and controller.py (mutation) being particularly illustrative of how we've reimagined genetic algorithms for the LLM era. I'm excited to see how others might apply and extend this approach to their own domains. What other applications do you see for LLM-driven evolutionary algorithms? #MachineLearning #EvolutionaryAlgorithms #LLM #OpenSource #AIResearch #AlgorithmDiscovery

AI : The Future of Prompt Engineering? How PromptBreeder Uses Co-Evolution for Continual Improvement … ? Prompt engineering is key to unlocking the capabilities of large language models (LLMs). However, designing effective prompts remains more an art than a science. What if we could take a rigorous, automated approach to evolving better prompts? That's the promise of PromptBreeder, an intriguing new technique from DeepMind. PromptBreeder represents a major advance in prompt optimization. It uses an evolutionary algorithm to breed more effective prompts over successive generations. 👉 Here's how it works: 1. Initialize a population of "prompts" (instructions to the LLM) and "mutation prompts" (instructions for modifying prompts). This seeds genetic diversity. 2. Evaluate each prompt on a batch of training examples, scoring its fitness. Better prompts produce more accurate LLM responses. 3. Evolve the population over generations using a genetic algorithm. High-fitness prompts mutate and crossover to yield new prompt variations. 4. Crucially, mutation prompts also evolve, becoming better at generating useful prompt mutations over time. 5. The cycle repeats, with prompts and mutation prompts co-evolving in a self-referential loop. This technique outperformed state-of-the-art prompting methods like Chain-of-Thought on benchmarks in arithmetic, commonsense reasoning, and hate speech detection. 👉 For example, on a grade school math dataset (GSM8K), PromptBreeder evolved prompts like: "Show all your working. You should use the correct mathematical notation and vocabulary, where appropriate. You should write your answer in full sentences and in words. You should use examples to illustrate your points and prove your answers. Your workings out should be neat and legible." This prompt improved accuracy from 56% to 84% compared to the baseline prompt of simply "Solution:". Unlike other prompt optimization techniques, PromptBreeder improves continuously over generations rather than hitting diminishing returns. And it requires no parameter updates, making it scalable. The authors suggest it could become even more powerful when paired with larger foundation models. The paper provides a fascinating glimpse into the future where systems not only learn, but learn how to learn better. PromptBreeder shows language itself can be the substrate for open-ended self-improvement, no parameters required. 👉 I highly recommend reading the full paper for an in-depth look at this new technique: PromptBreeder: Self-Referential Self-Improvement via Prompt Evolution https://lnkd.in/g2QDDJZn This research opens exciting possibilities for automating prompt engineering through co-evolution. The self-referential approach is powerful yet simple and scalable. PromptBreeder demonstrates how recursive self-improvement grounded in language alone can unlock more of LLMs' latent capabilities. I'm eager to see future work building on these ideas!

This weekend I ran a customized version of Karpathy's autoresearch overnight. What I learned makes every enterprise process optimization strategy deck obsolete. When I checked in the morning, it had made 79 attempts at solving an optimization problem. Kept 11 improvements. After manual QA, adopted 7. No human in the loop until review. It mirrors exactly what we do in process and ops optimization and product iterations at scale — define success criteria, set constraints, let the agent iterate. Keep what works, discard what doesn't. Review what survived in the morning - evolutionary algorithms in practice. Financial services should be paying very close attention. "Overnight" doesn't mean anymore mainframe legacy batch jobs or ops teams doing processing and booking anymore until wee hours. "overnight" means turning zero - the idle time, into 100x compounding value. Today, an actuary builds one model. Tests it manually or supported by rudimentary software, gets interupted with urgent but low value tasks, goes back home, restarts on Monday and so on Iterates over multiple weeks. Tomorrow, and I mean this literally, that same actuary writes a strategy document and lets an agent test 100 variations overnight. Same rigour. 100x the throughput. Think about what this means for: - Claims triage — models that self-optimize against resolution speed and fraud, waste and abuse accuracy - Pricing — testing hundreds of risk factor combinations, not the 3 the team had time for - Fraud — pattern detection that evolves weekly, not annually Karpathy's insight is dead simple: fix the time budget, fix the success metric, let the agent loop. Run it during idle time creating 100x value. That's not a PhD-level theoretical model. That's good engineering applied to machine learning. And it works just as well for optimizing business processes as it does for training AI models. The question for enterprise isn't "should we do this?" — it's who knows in the company what is .md file and how do you make sure the governance process doesn't kill it before the evolutionary AI even starts? What's the biggest blocker to autonomous optimization in your org?

Exciting News to Kick Off 2025!  I'm happy to announce that our latest paper, titled 'Large Language Model-Based Evolutionary Optimizer: Reasoning with Elitism', has been published in Neurocomputing, Elsevier! This work explores the potential of Large Language Models (LLMs) as black-box optimizers, leveraging their remarkable reasoning capabilities for zero-shot optimization across a variety of scenarios, including multi-objective and high-dimensional problems. We introduce Language-Model-Based Evolutionary Optimizer (LEO), a novel, population-based method for numerical optimization. Applications include benchmark challenges and real-world engineering problems like, Supersonic nozzle shape optimization, Heat transfer optimization, Windfarm layout optimization. Key Highlights: 1. Comparable performance to state-of-the-art optimization methods Insights into leveraging LLMs creative potential while addressing challenges like hallucinations. 2. Practical guidelines for reliable optimization using LLMs 3. Limitations and exciting directions for future research A huge thanks to all the collaborators Shuvayan Brahmachary, Subodh Joshi, Kaushic K, Kaushik Koneripalli, Aniruddha Panda, Harshil Patel, PhD, et al.; and the reviewers for their support and feedback! If you're interested in cutting-edge intersections of AI, optimization, and engineering, I invite you to check out the paper: https://lnkd.in/e5hzJwhh Wishing everyone a joyful and prosperous New Year!

Bayesian Optimization (BO) is sample-efficient but exploration-limited. Evolutionary algorithms are exploration-rich but sample-hungry. What happens when you combine them? Multi-objective BO (MOBO) is well suited for autonomous discovery workflows, but its acquisition functions can be greedy, concentrating candidates in narrow regions of the Pareto front because they prioritize expected improvement over coverage. Evolutionary algorithms handle diversity well with population-based search, but require far more evaluations to converge because they rely on iterative selection pressure rather than a surrogate model. Evolutionary Guided BO (EGBO), introduced by Low et al. (https://lnkd.in/evanXmBx), combines both in a single loop, letting evolutionary and acquisition-driven proposals compete for selection within a MOBO framework. This gives the optimizer access to diverse candidates without sacrificing the sample efficiency of model-guided search. A new preprint systematically evaluates hybrid strategies across diverse problem types and introduces a novelty-aware ranking step that penalizes redundancy among selected candidates. The gains are consistent across benchmarks, and particularly strong on constrained problems where narrow feasible regions make diversity critical. Here is how the modified EGBO workflow operates: 🔹The BO acquisition function proposes a small set of candidates it considers most promising based on the surrogate model. 🔹An evolutionary algorithm independently proposes a larger set of candidates drawn from diverse regions of the design space. 🔹Both sets are pooled into a single merged candidate list and scored by the acquisition function, so exploitation-driven and exploration-driven proposals compete on equal footing. In standard EGBO, the top-scored candidates are selected directly. 🔹The novelty-aware modification adds one step: as candidates are selected sequentially, a novelty term penalizes those too close to already-selected batch members, reducing redundancy and promoting broader coverage within each batch. Across 10 synthetic benchmarks and 4 real-world datasets (reaction optimization, pharmaceutical formulations, industrial coatings, drug screening), novelty-aware EGBO consistently outperforms both standard EGBO and acquisition-only MOBO. The largest gains appear in many-objective and constrained settings, though in very high-dimensional feature spaces, standard acquisition-only MOBO performs better. A practical framework worth considering for campaigns with complex trade-offs or constrained feasible regions. 📄 Novelty-Aware Evolutionary Bayesian Optimisation for Multi-Objective Discovery Science, ChemRxiv, April 6, 2026 🔗 https://lnkd.in/eSEbHMp3

I’ve been developing a Monte Carlo simulation in Rust 🦀 that uses a genetic algorithm to optimize my Masters course selections for the first semester. By leveraging parallel processing, it can run through 1,000,000 populations in about a minute, though there’s still plenty of room for optimizations. What’s a genetic algorithm? Think of it like natural selection in biology. Imagine you’re trying to breed the perfect plant. You start with a variety of plants, select the best ones, and breed them together. Over many generations, you get plants that are better suited to your needs. Similarly, a genetic algorithm iteratively improves solutions to a problem by selecting the best “candidates” and combining them to create better ones. What’s a Monte Carlo simulation? Picture rolling a pair of dice thousands of times to see all the possible outcomes. Instead of rolling real dice, a Monte Carlo simulation uses random sampling to explore all potential scenarios, helping you understand the range of possible outcomes and their probabilities. What’s utility? In this context, utility measures how valuable a course is to me and ultimately to my entire schedule. For example, “Data Center Design” is high on my list with a utility of 200 because it’s both required and highly interesting to me. On the other hand, “Cloud Computing” is also required but overlaps significantly with material I’ve already covered in my Azure Architect exam, so it’s less appealing and has a utility of 40. Why combine genetic algorithms with Monte Carlo simulations? Each iteration of the Monte Carlo simulation uses a different course schedule generated by the genetic algorithm, based on the likelihood of courses being offered in a particular term. This helps balance the utility of each course with its future availability. So far, my top utility score is 1,530. I’ll share more soon about what that means, but feel free to ask any questions!

🧬 What if we could make LLMs think deeper by letting them... evolve? A new paper from Google DeepMind shows that evolutionary search strategies can help language models solve complex problems with 95%+ accuracy. Here's the game-changer: Instead of trying to formalize natural language problems (which is incredibly difficult), researchers developed "Mind Evolution" - a genetic search approach that lets LLMs explore and refine solutions through natural selection. The results are remarkable: - On travel planning tasks, Mind Evolution achieved 95.6% success using Gemini 1.5 Flash, beating both Best-of-N sampling (55.6%) and sequential revision (82.8%) - When combined with Gemini 1.5 Pro for unsolved cases, success rates hit 100% on validation sets - All this without requiring formal problem solvers or explicit rules Most intriguingly, the system works by mimicking human creativity: combining "divergent thinking" (exploring many possibilities) with "convergent thinking" (refining the best ideas). It's like having an AI brainstorming session with itself! Instead of the typical "agree with everything" approach of most LLMs, Mind Evolution creates an AI critics' room. Here's how it works: - A "critic" LLM analyzes potential solutions with a skeptical eye - An "author" LLM proposes refined solutions based on the criticism - Multiple solution candidates evolve over generations, combining the best elements of each What's next for evolutionary approaches in AI? Could this be a key to helping LLMs tackle even more complex real-world problems? Paper linked in comments 👇 #ArtificialIntelligence #MachineLearning #DeepLearning #LLMs #AIResearch

I used to think the Transformer was the best architecture to build LLMs. I was wrong. Let me explain ⬇️ Don’t get me wrong. The Transformer is 𝘁𝗵𝗲 𝗺𝗼𝘀𝘁 revolutionary architectural design in the deep learning space for the last 10 years. It has scaled > model size and training budgets > effective sequence length our models can process and use, and > conquered every LLM eval dataset benchmark out there. However, this is 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴. There is never a single-best-solution-for-every-problem-out-there. Including yours. Let me explain. 𝗪𝗵𝗮𝘁’𝘀 𝘁𝗵𝗲 𝗽𝗿𝗼𝗯𝗹𝗲𝗺? 🤔 In a real-world LLM project, you don’t care about model quality alone, you also care about model size as well. In general, your 𝗻𝗼𝗿𝘁𝗵-𝘀𝘁𝗮𝗿 𝗺𝗲𝘁𝗿𝗶𝗰 is a mix of (𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆, 𝗹𝗮𝘁𝗲𝗻𝗰𝘆, 𝗺𝗲𝗺𝗼𝗿𝘆 𝗰𝗼𝗻𝘀𝘂𝗺𝗽𝘁𝗶𝗼𝗻) that is determined by the hardware constraints where you plan to deploy this LLM, e.g. consumer laptop, iPhone, Android... So the question is, > How can you find 𝘁𝗵𝗲 𝗼𝗽𝘁𝗶𝗺𝗮𝗹 architecture for your task, evaluation metric and hardware constraints? 𝗧𝗵𝗲 𝘀𝗼𝗹𝘂𝘁𝗶𝗼𝗻 🧠 The research team at Liquid AI has developed (and shared!) > a universal representation of all possible neural network architectures you can think of (called the space of Linear Input-Varying Systems) and (more importantly!) > An algorithm to find the best set of candidates from this large space of architectures (aka the 𝗦𝗧𝗔𝗥 𝗮𝗹𝗴𝗼) It is an end-2-end evolutionary algorithm in which you set your target quality and model size metrics, and STAR iteratively 1️⃣ 𝗲𝘃𝗮𝗹𝘂𝗮𝘁𝗲𝘀 their fitness by running a quick pre-training run on a subset of the data, 2️⃣ 𝗿𝗲𝗰𝗼𝗺𝗯𝗶𝗻𝗲𝘀 the existing population to exploit what works. 3️⃣ 𝗺𝘂𝘁𝗮𝘁𝗲𝘀 a subset of candidate architectures to further explore the space of potential solutions until you hit the money and you get the architecture that has both a good enough quality and size metrics. This is how you find what works best for you. And as a mathematician I can only say that it is BEAUTIFUL ❤️ You can find the link to the original blog post by the Liquid AI team and the paper on the comments below ⬇️ ---- Follow me for more high-signal LLM engineering posts

This is significant. By combining LLMs and evolutionary algorithms, Google DeepMind has for the first time enabled new maths and science discoveries from AI. AlphaEvolve - in contrast to AlphaFold, applied to protein folding, and other AI for science tools - is that it is general-purpose and can be applied across domains. Google reports "We applied the system to over 50 open problems in mathematical analysis, geometry, combinatorics and number theory. The system’s flexibility enabled us to set up most experiments in a matter of hours. In roughly 75% of cases, it rediscovered state-of-the-art solutions, to the best of our knowledge. And in 20% of cases, AlphaEvolve improved the previously best known solutions, making progress on the corresponding open problems." An article in Nature describes the approach: "Each task starts with the user inputting a question, criteria for evaluation and a suggested solution, for which the LLM proposes hundreds or thousands of modifications. An ‘evaluator’ algorithm then assesses the modifications against the metrics for a good solution (for example, in the task of assigning Google’s computing jobs, researchers want to waste fewer resources). On the basis of which solutions are judged to be the best, the LLM suggests fresh ideas and over time the system evolves a population of stronger algorithms." AlphaEvolve has already been applied internally by Google to chip design and efficiencies across its computing ecosystem. Advances in maths and science are dramatically accelerating. It's hard to imagine the results of this. We'll begin to find out soon.

You really should consider using Differential Evolution in your AI Solutions. First time hearing about it? DE “ is a method that optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality”. It does so by, optimizing “a problem by maintaining a population of candidate solutions and creating new candidate solutions by combining existing ones according to its simple formulae, and then keeping whichever candidate solution has the best score or fitness on the optimization problem at hand”. Simply put, Differential Evolution will go over each of the solutions. If it matches criterion (meets minimum score for eg.), it will be added to the list of candidate solutions. New solutions might be found by doing simple math operations on candidate solutions. When iterations are finished, we take the solution with the highest score (or whatever criterion we want). Why use Differential Evolution. There are 4 reasons I can give you- Range- Since it doesn’t evaluate the gradient at a point, IT DOESN’T NEED DIFFERENTIALABLE FUNCTIONS. This is not to be overlooked. For a function to be differentiable, it needs to have a derivative at every point over the domain. This requires a regular function, without bends, gaps, etc. DE doesn’t care about the nature of these functions. They can work well on continuous and discrete functions. DEs can thus be (and have been)used to optimize for many real-world problems with fantastic results. Performance- Differential Evolution has one of the best bang for buck in the AI space. Many great innovations- such as the amazing One Pixel Attack- have leveraged DE for cost-effective exploration of the search space. Adaptability: Papers have shown a vast array of techniques that can be bootstrapped into Differential Evolution to create a DE optimizer that excels at specific problems. This is a trend with Evolutionary Algorithms in general, which are tacked on as optimizers for other techniques. Semi-Black Box when dealing with DNNs- DE is not a black-box algorithm. The functioning and process are very transparent. This makes it very good for tracing steps, and building on your system. It might require black-box feedback(probability labels)when dealing with Deep Neural Networks, but they can still provide great insight into your system. I think Evolutionary/Swarm techniques are often overlooked in today's AI landscape. We'll do a deep-dive on them eventually. If you come across any interesting use-cases, please do share them.