Giant Language Fashions (LLMs) have achieved state-of-the-art efficiency on a broad vary of Pure Language Processing (NLP) duties, together with doc processing and code era. Autoregressive Language Fashions (ARMs), which generate tokens sequentially conditioned on all earlier tokens, have been the predominant paradigm for LLMs. Whereas these fashions have achieved excessive accuracy throughout a spread of downstream duties, they exhibit low arithmetic depth because of the inherent sequential dependency in next-token prediction. Just lately, Diffusion Language Fashions (DLMs) have emerged as a promising various structure. DLMs generate output tokens in parallel, mitigating the restrictions of sequential decoding. Nonetheless, the efficiency implications of DLMs relative to generally deployed ARMs aren’t absolutely understood. On this work, we current a complete research of the efficiency traits of ARMs and DLMs, combining theoretical evaluation with empirical profiling to characterize the trade-offs between these approaches. We present that though DLMs can obtain larger arithmetic depth than ARMs by leveraging parallelism throughout token positions, they fail to scale successfully with longer contexts. We then discover block-wise decoding for DLMs, which decouples arithmetic depth from sequence size and allows higher scaling to lengthy contexts (much like ARMs). We additionally study batched inference and discover that ARMs exhibit superior throughput as they profit extra from parallelism throughout sequences within the batch. Lastly, we spotlight alternatives for accelerating DLM inference, emphasizing that lowering the variety of sampling steps is vital for open-source DLMs to attain decrease latency relative to ARMs.
- †Seoul Nationwide College
- ‡ College of California, Berkeley
- § ICSI
- ¶ LBNL
- ††College of Texas at Austin
- * Advisory position
