
At Living Models, we are a small team with big ambitions: to make breakthroughs in genomics research, starting with plants, through training our own foundation models.
Turning that ambition into actual models requires lots of data, compute and expertise. Notably, AI for biology is especially tricky: not only does biology have its own scaling laws, but simply assessing the performance of any AI model for biology is pretty much an open research question, particularly in genomics: high-quality experimental data generation is hard and costly, experiments are noisy (e.g., batch effects, where differences in laboratory conditions distort results, see the great piece by Bindwell [1]) and simple metrics can be misleading [2]. To borrow the terminology of frontier AI labs, biology cannot easily be “RLVR-ed” [3] (meaning one cannot simply use reinforcement learning with verifiable rewards (RLVR) [4], where models improve by repeatedly receiving automatically checkable feedback, as one does when training AI to solve math problems or to write code). As a result, one needs to rethink the entire training and evaluation stack in order to be able to produce AI models with biological value. Here we will talk mainly about how we manage the compute part.
You may have seen the impressive figures that are reported for training foundation models like Arc Institute’s Evo 2 [5], recently published in Nature, which took a few months to train on 2,000 H100 GPUs, or Basecamp Research’s EDEN, trained on over 1,000 H200 GPUs for 6 weeks [6]. Our first foundation model, Botanic0-L [7], was trained at a much smaller scale, but still required 10 days on 8 B200 GPUs (accelerators approximately 2.3 times as fast as H100). Recently, we released our second generation of foundation models, the Botanic1 series [8], with much better performance across the board, but we had to use twice the amount of compute to train its largest installment (4 nodes of 8 B200 for approximately half the time).
Importantly, those numbers cover what the community often refers to as “hero” training runs: the final, fully scaled models that are trained only once the underlying science and training recipes are considered sufficiently mature. Therefore, hero runs account for only a small fraction of the total compute required to produce a model. In practice, most of the resources are spent performing numerous smaller-scale experiments, which are needed to identify efficient training paradigms, validate scaling behavior, and develop the infrastructure and processes required to operate reliable model factories. (The exact ratio varies by lab and is rarely disclosed. As an example, DeepSeek-V3’s widely cited $5.576 million figure [9] for a fleet of 2,000 H800 GPUs (roughly equivalent to H100 in terms of performance) refers to the estimated rental cost of its official training, including pre-training, context extension, and post-training, but excluding prior research and ablation work. It is therefore a lower bound on the full model development cost [10].)
The AI training stack has made great strides in recent years, driven in large part by open-source software and openly shared knowledge. Projects like Lingua [11] have shown how to keep a large-scale pre-training stack research-friendly and customizable, while uv [12] (whose parent company, Astral, has now been acquired by OpenAI [13]) and free-threaded Python (pioneered by one Meta engineer [14]) have drastically improved the development experience and speed of software. Accelerators like NVIDIA’s B200 allow us to train larger models for longer, and tools like SkyPilot [15], torch.distributed [16], and CUTLASS [17] make that hardware easier to use. Meanwhile, open learning resources like GPU MODE [18] and the Smol Training Playbook [19] are making the expertise needed for large-scale training more widely available.
Even so, making full use of a large-scale cluster like ours, with 120 B200 GPUs, to conduct novel research with a technical team of just 5 people can still feel daunting.
At Living Models, while we make liberal use of AI agents both to ship code faster and to learn new concepts faster, in order to live up to our ambitions, we had to go a step further and started playing with the concept of using AI agents directly to manage experiment orchestration and infrastructure, accelerating the pace of our research experiments and thus of our research pipeline as a whole.
AI-assisted experimentation allowed us to train our most intelligent model thus far, Botanic1, in a more robust and efficient manner, making sure that the entirety of our reserved compute was used during our training runs, while still being dynamic in our decision-making to allow true serendipity and research to happen.
Frontier labs seldom share the way they approached structuring research, so we thought we would give our take for a change!
Big run preparation
Research planning: human 90%, AI 10%
At specific times of the year, we purposefully stop all isolated research initiatives in order to launch a coordinated research effort for a period of two weeks. This is what we internally call a “big run”.
In order to prepare a big run, we start with a whiteboarding session with the whole research team. The goal is to aim for exhaustivity over precision: no idea is too dumb!
Here is what it looks like at the end of the first part of our meeting:
We then collaboratively vote for a few research ideas to prioritize in order to end up with approximately 3 (magic number) phases, meaning 3 different kinds of research directions, some of which can be run in parallel on our cluster.
Importantly, we carefully plan gating both within and across phases: we do not run the same experiments depending on whether our starting hypotheses are confirmed or not.
Once the 3 phases are decided, we start to scope and size the experiments in terms of GPU-hours one could run within each phase to prove or disprove the general hypothesis the phase is supporting.
Each phase and the experiments within it can then be planned in detail: we open an AI assistant and start by asking it to perform back-of-the-envelope estimations on the amount of time and hardware a specific experiment will need, and use them to arbitrate between several experiments.
The output of this stage is a list of training configs: one for each planned experiment. These configs contain all of the required information for the run, from data substrate to model architecture to training parameters. At this point the data or the code may not exist yet to support all the experiments; placeholders are used and the missing parts become the focus of the pre-run sprint.
We found it especially useful to programmatically check every config beforehand to make sure everything will run both logically (make sure no error will be raised), as well as in terms of resources (nothing will run OOM and everything will run within the time constraints).
Planning research experiment orchestration with AI agents: human 90%, AI 10%
We then go on to write a skill. While the details of the skills are cluster-specific, below are the important bits of the skills of our AI orchestrator agent:
Your primary goal is to answer the questions asked in the research plan @research/plan/path
In order to do that, you have access to a cluster whose characteristics are to be found in @cluster/characteristics
The plan has been featurized into a set of experiment configs you can find in @experiments/configs
You will launch each experiment from the research plan using this command /command/to/use and set up a monitor each time you launch an experiment so that you check that everything is okay for the first 5–10 minutes of each experiment. Notably, check initialization or resumption of the model as well as the MLflow [20] connection: model training runs should populate training loss and system metrics. Before launching an experiment, synchronize the commit of the repository of the orchestrator and the commit of the code in the cluster.
If something fails, prioritize changing config parameters over making changes to the code or the dependencies. If you need to change the code, make sure the change will not affect running experiments. If you are not sure, escalate to a human via your Slack [21] bot and run another non-buggy experiment in the meantime if that is possible.
Set up a monitor to make sure that the cluster is fully used at all times and that runs are healthy by running a deep check every 30 minutes for cluster usage and convergence of training runs
Stop unhealthy or unpromising runs early and schedule other experiments from the plan to replace them
Whenever there is gating, set up a monitor when the gating experiment approaches its end so that you can smoothly make a decision
Log all surprises into a file
surprises.md
We use uv to manage dependencies and duplicate the uv environment across the cluster, using a single environment for all experiments synchronized with the orchestrator tower.
Monitoring the run: human 50%, AI 50%
Because at Living Models we are not simply training language models but AI models for biology, our evaluation during pre-training is more involved and more compute-intensive than traditional next-word LLM training (at equivalent sizes). Typically, it either involves fine-tuning and/or post-training each model checkpoint (supervised evals), or requires triggering multi-step computational biology pipelines orchestrated with Snakemake [22] (zero-shot evals).
To make the most of the high-end GPUs we use for training, we decouple training from evaluation and run evaluation on a separate cluster. All results are stored and shared with the orchestrator agent through a self-hosted MLflow [20] instance running on a remote machine.
Concretely, here is what helps us monitor the “big run” during the two-week reservation period:
A remote machine, a “gateway”, is provisioned to keep track of the experiment database and to host the orchestrator AI assistant for 24/7 monitoring.
The orchestrator is launched as an AI agent session in a tmux [23] session and explicitly loads the skill described above. All team members share the same user space and can access the tmux session and interact with the orchestrator if needed.
All experiments also trigger a Slack alert on job submission and exit, allowing us to have a quick vibe check on the runs by simply opening Slack.
In order not to run into subscription mode limits, we have a Bash script launching the tmux session and monitoring usage, allowing it to switch from subscription to API mode and back when needed, therefore achieving “immortality” as long as API credits do not run out… ($$$).
Auto evaluations: human 10%, AI 90%
As mentioned earlier, training runs uninterrupted on the GB200 GPUs and model evaluation takes place on a secondary, cloud-based cluster.
Our evaluation set-up is currently implemented using the following principles:
A scron [24] job regularly uploads pre-trained checkpoints from our training cluster to our cloud provider
On the eval cluster, a scron job scans the cloud bucket and queues new checkpoints to be consolidated and evaluated using Slurm [24]’s afterok utility. Each checkpoint is evaluated within a Slurm [24] job, which itself orchestrates several Python processes (cells) in parallel, to maximize CPU and GPU utilization. The number of concurrent cells and concurrent jobs depends on the size of the checkpoint (we currently train 300M to 3B models) as well as the nature of the cells. Since every “big run” potentially comes with new evaluation tasks, an early calibration period is usually required to maximize the throughput of the evaluation pipeline, which we measure in the number of checkpoints evaluated per GPU per hour.
All results are stored in a cloud DB, are accessible to the whole company via custom dashboards (CTOs love us!), and uploaded to MLflow [20]. These results are very important as they will be used for the aforementioned gating that shapes the training runs.
A second mini orchestrator runs on the eval cluster controller node and monitors for failed cells and missed checkpoints for the duration of the run. If any cells need re-running, the mini orchestrator automatically re-submits the partial evaluation jobs with a dedicated skill.
The whole workflow is summarized below:
Here is what our dashboards looked like at some point in time (the style changes at each iteration!)
Conclusion
We are still exploring how to better use our cluster and train models more quickly and more efficiently. Notably, we did not mention in this blog post optimizing MFU (Model FLOPs Utilization), which measures how efficient our training runs are with respect to the vendor’s theoretical max throughput for a certain dtype, but that is one of several key factors that we work a lot on as it directly impacts the throughput of the big run.
If you read this and thought, “That is definitely not how I would build it!” we definitely want to hear from you. Send us an email! Living Models is always looking for bright people with bold ideas and different ways of seeing the world.
Find more in:
If you use this in your research, please cite it using the BibTeX below. Click the Copy button to copy the citation:
@misc{OgierduTerrail2026agenticLab,
title = {Running a Frontier AI Research Lab
with Five Researchers … and Agents},
author = {Ogier du Terrail, Jean and
Rukhovich, Alexey and Cabeli, Vincent},
year = {2026},
howpublished = {Living Models (Substack)},
url = {https://livingmodels.substack.com/p/running-a-frontier-ai-research-lab},
note = {Blog post}
}References
[1] Anand, N., & Goel, R. (2026, October 18). The Bitter Lesson for Biology. Bindwell.
[2] Miller, H. E., et al. (2026). Deep learning perturbation models can outperform baselines on calibrated metrics. Nature Biotechnology. doi:10.1038/s41587-026-03307-w.
[3] Istrate, A.-M., et al. (2025). rbio1—training scientific reasoning LLMs with biological world models as soft verifiers. bioRxiv (version 1). doi:10.1101/2025.08.18.670981.
[4] Lambert, N. (2025). Reasoning and Inference-Time Scaling. In Reinforcement Learning from Human Feedback, Chapter 7 (web edition). Accessed October 20, 2026.
[5] Brixi, G., et al. (2026). Genome modelling and design across all domains of life with Evo 2. Nature, 652, 1349–1361. doi:10.1038/s41586-026-10176-5.
[6] Nebius. (2026). See how Basecamp Research scales with Nebius AI Cloud. LinkedIn post.
[7] Ogier du Terrail, J., et al. (2026). BOTANIC-0: a series of foundation models for plant genomic data. bioRxiv (version 2). doi:10.64898/2026.02.23.706817.
[8] Barozet, A., et al. (2026). BOTANIC-1: a series of long-context plant genomic foundation models in the agentic era. bioRxiv (version 3). doi:10.64898/2026.09.04.749355.
[9] DeepSeek-AI, et al. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.
[10] Patel, D., et al. (2025, January 31). DeepSeek Debates: Chinese Leadership On Cost, True Training Cost, Closed Model Margin Impacts. SemiAnalysis.
[11] Videau, M., et al. (2024). Meta Lingua: A minimal PyTorch LLM training library. GitHub [software].
[12] Astral. (n.d.). uv. Documentation. Accessed October 20, 2026.
[13] OpenAI. (2026, March 19). OpenAI to acquire Astral. OpenAI.
[14] Gross, S. (2023). PEP 703—Making the Global Interpreter Lock Optional in CPython. Python Enhancement Proposals.
[15] SkyPilot. (n.d.). SkyPilot. Project website. Accessed October 20, 2026.
[16] PyTorch Contributors. (n.d.). Distributed communication package—torch.distributed. PyTorch 2.14 documentation. Accessed October 20, 2026.
[17] NVIDIA. (n.d.). CUTLASS: CUDA Templates and Python DSLs for High-Performance Linear Algebra. GitHub [software]. Accessed October 20, 2026.
[18] GPU MODE. (n.d.). GPU MODE. Community website. Accessed October 20, 2026.
[19] Ben Allal, L., et al. (2025, October 30). The Smol Training Playbook: The Secrets to Building World-Class LLMs. Hugging Face.
[20] MLflow. (n.d.). MLflow. Project website. Accessed October 20, 2026.
[21] Slack. (n.d.). Slack. Product website. Accessed October 20, 2026.
[22] Snakemake Contributors. (n.d.). Snakemake. Documentation. Accessed October 20, 2026.
[23] tmux Contributors. (n.d.). tmux. GitHub [software]. Accessed October 20, 2026.
[24] SchedMD. (n.d.). Slurm Workload Manager: Overview. Documentation. Accessed October 20, 2026.
[25] Living Models. (n.d.). Botanic1 report. Hugging Face Space. Accessed October 20, 2026.
[26] Living Models, Strouk, L., & Véran, C. (2026, September 8). Introducing BOTANIC-1, a New State-of-the-Art Plant Genomic Language Model. Living Models, Substack.






