An AI system to help scientists write expert-level empirical software | alphaXiv

View 3 comments: good paper overall. promising big things here. there are milestones need to get there tho. The results of this paper could be something major like the transformers. This is what stood out to me,Geneti…

arXiv

AI-Driven Scientific Discovery: How Google Accelerates Empirical Software Development

In contemporary scientific research, software development plays an increasingly vital role, yet its time-consuming and complex nature often bottlenecks scientific discovery. Traditionally, developing software to support computational experiments has been slow and highly dependent on manual operations, severely limiting the speed at which scientists can explore new hypotheses and validate new theories. However, a groundbreaking AI system recently launched by Google is bringing revolutionary possibilities to accelerate scientific discovery by automating and optimizing the creation process of empirical software.

The Bottleneck of Scientific Discovery: Challenges of Empirical Software

First, we need to understand the concept of "empirical software." It is defined as software designed to maximize a definable or measurable quality score (often the fit to existing observations). If a task can be solved by empirical software, we call it a "scorable task." Such tasks are prevalent in the scientific domain; almost all subfields of science, applied mathematics, and engineering rely on software, and much core work involves empirical software. For instance, cutting-edge research in Density Functional Theory (1998 Nobel Prize in Chemistry), molecular dynamics simulation (2013 Nobel Prize in Chemistry), and protein structure prediction (2024 Nobel Prize in Chemistry) all depend on empirical software.

However, developing this scientific empirical software typically takes years and lacks systematic alternative method searches. Design choices are often based on intuition or expediency, which severely limits the exploration possibilities and unduly prolongs the cycle of scientific discovery.

AI-Driven Software Creation System: Working Principles and Advantages

To systematically and automatically create empirical software for scorable tasks, Google proposes an AI-based solution. The core mechanism of this method is to use Large Language Models to rewrite software to try and improve its quality score. The system generates multiple software candidates and employs Tree Search to decide which candidates are worth further exploration.

Crucially, the system inputs scorable tasks and research ideas (methods to solve the tasks) into the Large Language Model, which then generates code in a sandbox environment and evaluates it. The code's quality score is fed back to the Tree Search algorithm, guiding its selection of new nodes for exploration or exploitation. Through this iterative and tireless large-scale solution search, the system is capable of identifying high-quality solutions from an extensive and complex search space, thereby achieving performance that surpasses human experts. The injection of research ideas can be directly provided by users, automatically retrieved from literature via search engines, or even generated by AI agents (e.g., Gemini Deep Research and AI co-scientist).

AI System Schematic and Performance Overview

Source: Google Research Report

The system performed exceptionally well in the Kaggle Playground benchmark, with its average public leaderboard percentile performance significantly outperforming single Large Language Model calls and the best results from 1,000 Large Language Model calls. Notably, incorporating expert advice (such as Boosted Decision Tree) significantly enhances system performance. Specifically, the system achieved an average public leaderboard percentile performance of over 70% across 16 Kaggle competitions, far exceeding single Large Language Model calls (35%) and the best results from 1,000 Large Language Model calls (50%).

Breakthrough Applications Across Scientific Domains

Google's AI system has demonstrated its ability to surpass human experts in multiple scientific domains, including bioinformatics, epidemiology, geospatial analysis, neuroscience, time series forecasting, and numerical analysis.

Genomics: Single-cell RNA Sequencing Batch Integration

In the field of single-cell RNA sequencing (scRNA-seq), technological innovation has revolutionized cell heterogeneity analysis, but integrating multiple datasets faces computational challenges in removing complex batch effects while preserving biological signals. The system demonstrated excellent performance in the OpenProblems v2.0.0 batch integration benchmark. Evaluated on 6 datasets (comprising 1,747,937 cells), its solution is conceptually similar to ComBat but outperforms current OpenProblems leaderboards.

Batch Integration Task Schematic and Tree Search Performance

Source: Google Research Report

In pairwise comparisons with 9 existing methods (including 6 best-performing ones), the Tree Search method's overall score outperformed corresponding published results in 8 out of 9 cases. Among them, the best-performing model was Batch Balanced K-Nearest Neighbors (TS), implemented based on Tree Search, which achieved 14% higher overall performance than the best published method (ComBat) and met or surpassed the published BBKNN in 11 out of 13 metrics. Part of the performance improvement stems from combining two existing methods, ComBat and BBKNN; specifically, BBKNN (TS) computes neighbors on the ComBat-corrected PCA embedding, thereby removing global linear batch-related variance.

By programmatically generating "recombinations" of 55 existing methods, the system found that 24 (44%) recombined solutions outperformed both of their foundational methods, and 22 outperformed one of them. Additionally, 21 more ideas were generated and implemented through Gemini Deep Research and AI co-scientist. In total, 40 methods (6 foundational methods, 29 recombination methods, 4 Deep Research methods, and 1 AI co-scientist method) surpassed all currently published methods on the OpenProblems leaderboard.

Public Health: US COVID-19 Hospitalization Prediction

In public health, the primary benchmark for COVID-19 forecasting is the COVID-19 Forecast Hub coordinated by the CDC. Google's AI system conducted a rigorous retrospective study, optimizing and selecting models using data from the prior 6 weeks and making predictions within a rolling validation window for the 2024-2025 quarter.

COVID-19 Prediction Tree Search Performance

Source: Google Research Report

Results showed that the "Google Retrospective" model had an average Weighted Interval Score (WIS) of 26, outperforming the official CovidHub Ensemble's average WIS (29), and achieved lower WIS in most states. The system was able to replicate methods from existing CovidHub teams and surpassed the performance of original submissions in 6 out of 8 cases. Among the 28 generated hybrid models, 11 "recombination (Tree Search)" models achieved a better WIS score than their parent models.

In total, the systematic exploration produced 14 strategies that surpassed the official CovidHub Ensemble: 10 from recombination, 2 from Deep Research, 1 from AI co-scientist, and 1 replicated foundational model. Successful recombination models often merged different modeling paradigms, such as combining epidemiological models with statistical autoregressive models, or integrating machine learning models with climate baselines. New strategies generated by Deep Research and AI co-scientist represented significant conceptual leaps, such as introducing counterfactual simulations for uncertainty quantification or implementing more complex spatio-temporal graph neural networks.

Geospatial Analysis: Remote Sensing Image Segmentation

Semantic segmentation of remote sensing images is a computer vision task widely applied in land use monitoring, environmental impact assessment, and disaster response. The system used the DLRSD dataset (a densely annotated version of the UC Merced Land Use Dataset, containing 17 class labels) and was prompted to train a model to classify pixels.

Example Output of DLRSD Image Pixel Segmentation

Source: Google Research Report

The three best-performing solutions generated by Tree Search significantly outperformed recent academic papers reporting DLRSD benchmark results, with all having a "Mean Intersection over Union" (mIoU) greater than 0.80. Among them, Solution 2 (Tree Search) achieved the highest mIoU, reaching 0.82. These solutions utilized standard UNet++ and U-Net models, combined with powerful encoders pretrained on ImageNet (such as efficientnet-b7 and se-resnext101-32x4d), or employed state-of-the-art Transformer architecture SegFormer. Key differentiating factors included data augmentation and prediction strategies, with all solutions adopting extensive Test-Time Augmentation (TTA).

Neuroscience: Whole-Brain Neural Activity Prediction

In the field of neuroscience, the Zebrafish Activity Prediction Benchmark (ZAPBench) aims to test the prediction of whole-brain cell-resolution neural activity in zebrafish. Google's AI system was prompted to solve a multivariate time series forecasting problem, predicting the activity of all neurons for up to 32 future time steps, given the activity of the past 4 time steps as context.

ZAPBench Neural Activity Prediction Performance Comparison

Source: Google Research Report

The best model generated by Tree Search utilized a rich feature set from input windows, combining temporal convolutions, learned "global brain states," and neuron-specific embeddings, generating multi-step predictions through a series of residual blocks. This model outperformed all other baselines, including the best-performing video model (except for 1-step-ahead prediction). Even more impressively, these two solutions trained orders of magnitude faster than the best video model (less than 2 hours on a single T4 GPU compared to 36 hours on 16 A100 GPUs for the Unet model). The system also successfully integrated biophysical information, for example, by using the Jaxley library to simulate independent Hodgkin-Huxley model neurons and dynamically adjust their biophysical parameters.

Time Series Forecasting: GIFT-Eval

General Time Series Forecasting Model Evaluation (GIFT-Eval) is a time series forecasting benchmark comprising 28 datasets from 7 distinct domains, using "Mean Absolute Scaled Error" (MASE) as the scoring metric. The system applied a two-stage approach to this benchmark: finding independent solutions for each dataset, and creating a single, universal forecasting model.

In the "per-dataset solution" phase, results outperformed foundational models, deep learning models, and standard time series methods (such as ARIMA) on the leaderboard as of May 18, 2025. The discovered solutions strongly leaned towards gradient boosting and ensemble/decomposition models. In the "unified solution" phase, the system created a single, general forecasting library capable of spanning all 97 datasets. Through iterative development, the final MASE reached 0.734. The final solution sequentially modeled and removed the fundamental components of the series (baseline level, trend, seasonality, datetime features, and final residual correction), and featured 8 preset configurations.

Numerical Analysis: Numerical Evaluation of Difficult Integrals

Numerical integration is a core problem in numerical analysis, but existing standard methods (such as scipy.integrate.quad()) may fail when dealing with oscillatory integrals and infinite limits. The system used hill climbing optimization, aiming to build a general method superior to scipy.integrate.quad(), using 38 standard integrals where algorithms failed but analytical solutions were known as the benchmark set.

The best solution, built upon scipy.integrate.quad(), accelerates the convergence of slowly converging series by partitioning the infinite domain into a series of continuous finite sub-intervals and applying Euler's transformation. While scipy.integrate.quad() failed on every problem in the held-out set, the evolved code correctly evaluated 17 out of 19 held-out integrals, with a score error of less than 3%. The evolved code is as accurate as scipy.integrate.quad() in non-pathological cases, thus serving as a direct replacement.

System Innovation and Profound Impact on Scientific Progress

Google's AI system demonstrates remarkable innovation in several aspects:

1. Large Language Model-driven rewriting method: This system allows for flexible integration of domain knowledge and external research ideas, efficiently exploring research ideas. This contrasts sharply with the random mutation and structured recombination of traditional Genetic Programming, as Large Language Models can perform intelligent, semantically aware "mutations," leading to more complex and meaningful changes.
2. Tree Search generalizes the search process: Compared to systems like FunSearch, this system uses Tree Search, a robust algorithm, to explore large search trees and effectively integrate literature knowledge. Unlike traditional Large Language Models, which are often used for "one-shot" generation, this system uses Large Language Models in an iterative refinement loop, guided by the Tree Search algorithm and quality scores to rewrite existing code.
3. Versatility beyond AutoML: While AutoML focuses on ML models and hyperparameter optimization, this system is more general, capable of rewriting any software, including preprocessing, complex simulations, or mathematical heuristics.
4. General problem-solving capability: Unlike existing AI agents, which are often domain-specific automation tools or ideation assistants requiring human validation, this system demonstrates general problem-solving capabilities, achieving expert-level performance on public leaderboards and in academic literature across multiple domains.

Algorithm Schematic, Including Code Variation System

Source: Google Research Report

In summary, this method combines a Tree Search-based code variation system with the ability to integrate complex research ideas. Research ideas can come from published literature, AI research agents (e.g., Gemini Deep Research, AI co-scientist), or existing idea combinations discovered by the Large Language Model itself. The system has achieved expert-level performance in a wide range of scorable scientific tasks, capable of exploring and integrating complex research ideas.

Specifically, the system created 40 methods that outperformed the best-known methods for scRNA-seq batch integration, and developed 14 methods that surpassed the CDC epidemiology forecast ensemble model. Furthermore, the system achieved state-of-the-art performance in geospatial reasoning, neural activity prediction, time series forecasting, and computational mathematical algorithms.

This technology can rapidly generate expert-level solutions, reducing the time to explore a set of ideas from weeks or months to hours or days. This portends a revolutionary acceleration in scientific fields (particularly those where solutions can be machine-scored), promising to greatly advance the frontiers of human knowledge.