Table of Contents

When Sakana AI’s EGGROLL dropped the bombshell that “LLMs can be pretrained without backpropagation”, the industry’s reaction was mixed. The idea was intriguing, but shaking every parameter one by one was far too slow to be practical. Roughly a year later, in early 2026, a follow-up called Dust has reportedly achieved 1,000~10,000× efficiency over EGGROLL. This article breaks down the core mechanisms that make zero-order optimizers work at transformer scale, and how far you can actually trust them in production.
Why zero-order optimizers are back in the spotlight
The hidden cost of backpropagation
If you’ve worked in deep learning, you’ve felt this in your bones: “half of GPU memory exists for the backward graph.” Storing forward activations for a single step of a 7B model already scales linearly with sequence length, and without gradient checkpointing, activations balloon 4~5×. The bottleneck I hit most often working on a 70B model was exactly this — even on 8× H100, batch size 1 with a sequence length of 2,048 was near the limit.
Zero-order optimizers eliminate the backward graph entirely. Instead of computing gradients directly, they perturb parameters slightly, measure which way the loss moves, and update accordingly. The Gaussian smoothing technique that uses f(x+εu) − f(x−εu) as a finite-difference approximation of the directional derivative is a statistical classic that has been used for over 150 years — and applying it to LLMs was the starting point for both EGGROLL and Dust.
Why EGGROLL was 1,000× slower
EGGROLL perturbed each parameter independently and ran bidirectional forwards. For a 7B model, you effectively had to shard the forward pass across the full parameter count, which made communication and synchronization costs absurdly high. The estimator is theoretically unbiased, but the consensus was that it was close to impossible in practice, and later attempts reduced activation memory but left wall-clock time essentially unchanged. That’s where Dust’s central question comes in: “Do we really need to shake every parameter?”
Dust’s structural approach — destroying sample complexity
Block-wise perturbation
Dust’s biggest contribution is the idea: “Don’t perturb all parameters — sample in a low-dimensional subspace.” Instead of adding and subtracting ε across all 7 billion parameters, you only perturb along a k-dimensional vector (typically k = tens to hundreds) built from a random projection. Drawing on the same principle as Johnson–Lindenstrauss dimension reduction, most of the actual loss movement can be captured within this low-dimensional subspace.
Concretely, in PyTorch the pattern looks like this. Sample a Gaussian noise vector u once, then — instead of cloning the model parameters θ into two copies — apply (θ + σu) and (θ − σu) in-place alternately. You only need to run forward twice, so activation memory is fine with forward-only, and there is no backward pass at all. For memory profiling, the canonical approach is to enable record_shapes=True in torch.profiler and verify directly that activation size drops by more than half.
Asymmetric estimator combined with Adam
Earlier-generation zero-order methods all used symmetric two-sided estimation. Dust instead uses a variant that combines a one-sided estimator with momentum. The estimator takes the form g_t = (f(θ_t + σ_t u_t) − f(θ_t)) / σ_t · u_t, requiring only a single forward pass, and then layers Adam-style first/second moment estimates on top to suppress variance. One-sided means wall-clock time is halved, and Adam coupling shrinks variance, so the net effect is a several-hundred- to several-thousand-fold reduction in sample complexity.
The theoretical requirement E[g] = ∇f only holds as σ → 0, which remains a hard limit. But pushing σ too small makes noise explode, so the practical standard is to decay it as σ_t = σ_0 / (1 + t)^α. In PyTorch, you can implement this directly by passing a closure of the same shape to torch.optim.lr_scheduler.LambdaLR.
EGGROLL vs Dust comparison
| Item | EGGROLL | Dust |
|---|---|---|
| Perturbation dimension | Full parameters | Low-dimensional projection (k≪d) |
| Forwards per step | 2× number of parameter blocks | 1 or 2 (one-sided option) |
| Activation memory | Forward only | Forward only |
| Wall-clock vs BP | ~1,000× slower | 1×~10× (scale-dependent) |
| Estimation variance | Very high | Suppressed via Adam coupling |
| 7B model practicality | Demo level | Within reach for fine-tuning |
How far can you trust this paradigm in practice?
The real appeal of forward-only
What I find most interesting in production isn’t throughput itself but memory efficiency. Since activations aren’t stored, sequence-length limits effectively disappear, and pretraining an 8B model with a 1M-token context is theoretically possible as long as GPU memory allows. Furthermore, because only forward passes are required, you can finally use non-differentiable simulators (physics engines, game environments, real API responses) directly as the loss. The most common trap in practice is to read this as “it just got faster” — the real win is activation memory savings and the ability to learn in black-box environments, not throughput.
Overhyped claims vs genuine practical sweet spots
To be frank, I don’t believe we can run 70B-class pretraining from scratch without BP yet. Published results are concentrated at sub-1B scale and on supervised fine-tuning stages; there is still no comparison between 100B-token-scale from-scratch pretraining and EGGROLL/Dust. The narrative that “backpropagation will be fully replaced” is overhyped. Where zero-order optimizers deliver value right now isn’t teaching an entire LLM from scratch — it’s fine-tuning already-trained models to specialized domains, and black-box RL when you have an environment reward but no gradient access.
What to try right now
- Write a forward-only loop that uses
torch.func‘sfunctional_callto apply perturbations without cloning the model, and use it to establish a baseline. Verify withtorch.cuda.max_memory_allocated()that memory usage drops by more than half compared to BP. - Turn on
record_shapes=Trueintorch.profilerand compare activation memory against BP training at the same token count. Useprof.key_averages().table(sort_by="self_cuda_memory_usage")to confirm directly that backward nodes have disappeared. - Build a small ablation that sweeps perturbation dimension k and the σ decay schedule. Using a
torch.optim.lr_scheduler.LambdaLR-style variant of σ_t = σ_0 / (1+t)^α, measure α ∈ {0.1, 0.3, 0.5} × k ∈ {32, 128, 512} combinations on a 1B model to find the sweet spot. - Compensate for the high variance of the one-sided estimator with an exponential moving average. Don’t multiply the 1st-moment directly by the estimator; instead use m_t = β·m_{t-1} + (1-β)·g_t. Start at β=0.9 and push it up to 0.99 as the noise scale grows.
- Run an A/B test of the forward-only loop on a domain-adaptation task or a black-box RL environment. The fastest validation is to collect your own metrics: whether reward or held-out loss lands within ±2% of BP fine-tuning, and how convergence speed compares per training token.
Frequently asked questions
Can you pretrain an LLM from scratch with a zero-order optimizer?
Theoretically yes, but published experiments focus on sub-1B scale. There is still insufficient evidence that 70B-class from-scratch pretraining runs at a practical speed versus BP, so in production the safer path is to validate first at the fine-tuning or domain-adaptation stage.
If backpropagation is fully replaced, how does GPU infrastructure change?
When the backward graph disappears, activation memory drops sharply and longer sequences fit on a single GPU. On the other hand, perturbation plus two forwards means flops actually go up, so on a throughput basis, chips with wider memory bandwidth and stronger inference performance (B200, inference-only ASICs) may become more attractive than compute-specialized training GPUs like the H100.
Can you train against the loss of a non-differentiable system?
Yes — this is the most solid strength of zero-order optimizers. Because only forwards are required, black-box environments like game simulators, physics engines, and real API responses can be used directly as reward signals. The variance is high, however, so a common pattern is to bundle 8~16 forwards per step in a multi-sample estimator to reduce variance.
Reference source
This article was written after reviewing the following source: MarkTechPost — Reflection AI Introduces Beam: A 501B Open-Weight MoE Model With 23B Active Parameters for Coding and Agentic Workloads
Leave a Reply