TLDR:
In case you are compute-constrained, use autoregressive fashions; in case you are data-constrained, use diffusion fashions.
Motivation
Progress in AI over the previous decade has largely been pushed by scaling compute and knowledge. The recipe from GPT-1 to GPT-5 has appeared simple: practice a bigger mannequin on extra knowledge, and the result’s a extra succesful system.
But a central query stays: will this recipe proceed to carry from GPT-6 to GPT-N?
Many analysts and researchers imagine the reply is not any. For example, Ilya Sutskever, in his NeurIPS 2024 Take a look at-of-Time Award speak, remarked: “Compute is rising—higher algorithms, higher {hardware}, greater clusters—however knowledge is just not rising. We’ve only one web, the fossil gas of AI.”
This concern is echoed by AI forecasters, who’ve analyzed compute and knowledge progress extra systematically and concluded that compute is outpacing knowledge at an accelerating price.
The above Determine, illustrates this stress by overlaying projections from EpochAI’s evaluation. Their examine extrapolates historic tendencies in compute, dataset utilization, and internet-scale knowledge availability. The forecast means that by round 2028, we’ll enter a data-constrained regime: much more compute will likely be accessible than there are coaching tokens to eat.
This paper addresses the problem by asking: how can we commerce off extra compute for much less knowledge? Our central concept is to revisit the foundations of recent generative modeling and evaluate the 2 dominant paradigms for scaling AI.
Broadly, there have been two households of algorithms that formed latest progress in AI:
- Autoregressive fashions, popularized in 2019 within the textual content area with the GPT-2 paper.
- Diffusion fashions, popularized in 2020 within the imaginative and prescient area with the DDPM paper.
Each purpose to maximise the joint probability, however they differ basically in how they factorize this joint distribution.
The success of diffusion in imaginative and prescient and autoregression in language has sparked each pleasure and confusion—particularly as every group has begun experimenting with the opposite’s paradigm.
For instance, the language group has explored diffusion on textual content:
D3PM launched discrete diffusion through random masking, whereas Diffusion-LM utilized steady diffusion by projecting tokens to embeddings earlier than including Gaussian noise. Since then, quite a few works have prolonged this line of analysis.
Conversely, the imaginative and prescient group has experimented with doing autoregressive modeling on photos. Fashions comparable to PARTI and DALLE exemplify this strategy with robust outcomes.
This cross-pollination has led to even larger uncertainty in robotics, the place each diffusion-based and autoregressive approaches are broadly adopted. For example this, OpenAI Deep Analysis has compiled an inventory of robotics works throughout each paradigms, highlighting the shortage of consensus within the area.
This ambiguity raises a basic query: ought to we be coaching diffusion fashions or autoregressive fashions?
Fast Background:
Autoregressive language fashions:
They mannequin knowledge distribution in a left-to-right method
Diffusion language fashions:
For a extra detailed understanding, with cool animations, please seek advice from this video from Jia-Bin Huang – https://www.youtube.com/watch?v=8BTOoc0yDVA
Prior outcomes with Diffusion Language fashions
Since 2021, diffusion language fashions have sparked important curiosity, with many works specializing in enhancing their design and efficiency.
Within the desk above, we spotlight consultant outcomes from a preferred work.
The takeaways are as follows:
- Discrete diffusion performs higher than steady diffusion on textual content.
- Autoregressive fashions nonetheless obtain the strongest outcomes general.
A number of works have additionally explored the scaling habits of diffusion-based language fashions.
Nie et al report that discrete diffusion LLMs require roughly 16× extra compute than autoregressive LLMs to match the identical adverse log-likelihood. Comparable outcomes have been noticed in multimodal domains—as an illustration, UniDisc finds that discrete diffusion wants about 12× extra compute than autoregression for comparable likelihoods.
Nevertheless, these outcomes conflate knowledge and compute as a result of they’re measured in a single-epoch coaching regime. This raises an essential ambiguity: do diffusion fashions really require 16× extra compute, or do they actually require 16× extra knowledge?
On this work, we explicitly disentangle knowledge and compute. Our objective is to check diffusion and autoregressive fashions particularly in data-constrained settings.
Our Motivation
To know why diffusion could behave in another way, let’s revisit its coaching goal.
In diffusion coaching, tokens are randomly masked and the mannequin learns to recuperate them. Importantly, left-to-right masking is a particular case inside this framework.
Seen this fashion, diffusion might be interpreted as a type of implicit knowledge augmentation for autoregressive coaching. As a substitute of solely studying from left-to-right sequences, the mannequin additionally advantages from many various masking methods.
And if diffusion is actually knowledge augmentation, then its advantages needs to be most pronounced when coaching is data-bottlenecked.
This angle explains why prior works have reported weaker outcomes for diffusion: they primarily evaluated in single-epoch settings, the place knowledge is ample. In distinction, our examine focuses on eventualities the place knowledge is proscribed and compute might be traded off extra successfully.
Our Experiments
On this work, we practice tons of of fashions spanning a number of orders of magnitude in mannequin measurement, knowledge amount, and variety of coaching epochs to suit scaling legal guidelines for diffusion fashions within the data-constrained setting. We summarize a few of our key findings under.
Discovering #1:
Diffusion fashions outperform autoregressive fashions when skilled with enough compute (i.e., extra epochs & parameters). Throughout completely different distinctive knowledge scales, we observe:
- At low compute, Autoregressive fashions win.
- After a specific amount of compute, efficiency matches—we name this the essential compute level.
- Past this, diffusion retains enhancing, whereas Autoregressive plateaus or overfits.
Every level within the determine exhibits a mannequin skilled to convergence. The x-axis exhibits the full coaching FLOPs of that time, and the y-axis exhibits the very best validation loss achieved by that mannequin household underneath that coaching compute price range.
Discovering #2:
Autoregressive fashions start to overfit a lot rapidly, whereas diffusion exhibits no indicators of overfitting even after 10x the variety of epochs. Within the above determine, we confirmed that rising compute ultimately favors diffusion. However compute might be scaled in two methods: (i) Growing mannequin measurement (ii) Growing the variety of epochs Within the following plot, we separate these axes.
The coloured star marks the 1-epoch level, the place Autoregressive outperforms diffusion. The star (★) denotes the very best loss achieved by every mannequin.
- Autoregressive hits its greatest across the center, then overfits.
- Diffusion retains enhancing and reaches its greatest loss on the far proper.
Not solely does diffusion profit from extra coaching—it additionally achieves a greater remaining loss than Autoregressive (3.51 vs. 3.71).
Discovering #3:
Diffusion fashions are considerably extra strong to knowledge repetition than autoregressive (AR) fashions.
We present coaching curves of fashions skilled with the identical whole compute, however completely different trade-offs between distinctive knowledge and variety of epochs.
An “epoch” right here means reusing a smaller subset of information extra instances(e.g., 4 Ep is 4 epochs whereas utilizing 25% distinctive knowledge, 2 Ep is 2 epochs with 50% and so forth).
- AR fashions start to overfit as repetition will increase—their validation loss worsens and considerably diverges at larger epoch counts.
- Diffusion fashions stay steady throughout all repetition ranges, displaying no indicators of overfitting or diverging—even at 100 epochs.
Discovering #4:
Diffusion fashions exhibit a a lot larger half-life of information reuse (R_D*) —i.e., the variety of epochs after which returns from repeating knowledge begins to considerably diminish.
We undertake the data-constrained scaling framework launched by Muennighoff et al. of their wonderful NeurIPS paper to suit scaling legal guidelines for diffusion fashions. Whereas Muennighoff et al. discovered R_D* ~ 15 for autoregressive fashions, we discover a considerably larger worth of R_D* ~ 500 for diffusion fashions—highlighting their potential to learn from much more knowledge repetition.
The above Determine research the Decay price of information worth underneath repetition: left exhibits diffusion, center AR, and proper the common decay price for each.
Factors are empirical outcomes (darker colour = larger FLOPs, lighter colour =
decrease FLOPs; every line = mounted compute), we discover that fitted curves (represented as strains) intently match the empirical factors, indicating our scaling legal guidelines are consultant. The decay price of worth for repeated knowledge is decrease for diffusion, reflecting its larger robustness to repeating. On this experiment 100% knowledge fraction means coaching 1 epoch with 100% distinctive knowledge, whereas 50% means 2 epoch epoch with solely utilizing 50% distinctive knowledge and so forth.
Discovering #5:
Muennighoff et al. confirmed that repeating the dataset as much as 4 epochs is sort of as efficient as utilizing recent knowledge for autoregressive fashions.
In distinction, we discover that diffusion fashions might be skilled on repeated knowledge for as much as 100 epochs, whereas having repeated knowledge virtually as efficient as recent knowledge.
Discovering #6:
The compute required for diffusion to outperform AR follows a predictable energy regulation. Above we outlined the essential compute threshold as the quantity of FLOPs the place diffusion matches AR efficiency for a given distinctive dataset measurement.
We discover that we will derive a easy closed-form analytical expression for this threshold, this permits us to foretell when diffusion will surpass AR given any distinctive knowledge measurement. Within the determine we present each the fitted curve and empirical essential threshold factors, which align intently.
Discovering #7:
The info effectivity of diffusion fashions interprets to higher downstream efficiency.
Lastly we consider the best-performing diffusion and AR fashions (skilled underneath the identical knowledge price range) on a variety of language understanding duties.
Throughout most benchmarks, diffusion fashions outperform AR fashions, confirming that diffusion’s decrease validation loss interprets to higher downstream efficiency.
Discovering #8:
Publicity to completely different token orderings helps clarify diffusion’s knowledge effectivity. By including specific knowledge augmentations to AR coaching, we discover that diffusion mannequin’s benefit arises from their publicity to a various set of token orderings.
As seen within the above Determine, rising N constantly lowered validation loss and delayed overfitting. At N = 16, the 100-epoch validation lack of AR fashions approached that of diffusion, suggesting that various orderings are certainly a key driver of diffusion’s knowledge effectivity. These outcomes help our interpretation that diffusion fashions outperform AR fashions in low-data regimes as a result of they’re implicitly skilled on a richer distribution of conditional prediction duties.
Lastly, this evaluation suggests a pure continuum between the 2 paradigms: by controlling job range by means of masking or reordering—we may design hybrid fashions that interpolate between compute effectivity (AR-like) and knowledge effectivity (diffusion-like).
For extra experiments and particulars please seek advice from authentic paper –https://arxiv.org/abs/2507.15857
Conclusion
As the provision of high-quality knowledge plateaus, enhancing knowledge effectivity turns into important for scaling deep studying. On this work, we present that masked diffusion fashions constantly outperform autoregressive (AR) fashions in data-constrained regimes — when coaching entails repeated passes over a restricted dataset. We set up new scaling legal guidelines for diffusion fashions, revealing their potential to extract worth from repeated knowledge far past what AR fashions can obtain.
These outcomes problem the traditional perception that AR fashions are universally superior and spotlight diffusion fashions as a compelling different when knowledge—not compute—is the first bottleneck. Wanting forward, environment friendly use of finite knowledge could outline the following frontier in scaling deep studying fashions. Though the research have been carried out within the context of language fashions, we imagine these findings ought to apply throughout any type of sequence modeling knowledge, comparable to in robotics or healthcare. For practitioners, our takeaway is easy: in case you are compute-constrained, use autoregressive fashions; in case you are data-constrained, use diffusion fashions.
Bibtex:
@article{prabhudesai2025diffusion,
title={Diffusion Beats Autoregressive in Knowledge-Constrained Settings},
writer={Prabhudesai, Mihir and Wu, Mengning and Zadeh, Amir and Fragkiadaki, Katerina and Pathak, Deepak},
journal={arXiv preprint arXiv:2507.15857},
yr={2025}
}
