This paper was accepted on the Workshop on Latent & Implicit Considering \u2013 Going Past CoT Reasoning 2026 at ICLR.<\/p>\n

Autoregressive language fashions skilled with next-token prediction generate textual content by sampling one discrete token at a time. Though very scalable, this goal forces the mannequin to commit at each step, stopping it from exploring or reflecting upon a number of believable continuations. Moreover, the compute allocation throughout tokens is uniform; each token is shaped based mostly on a single forward-pass, doubtlessly limiting the mannequin\u2019s expressiveness in instances the place troublesome tokens require inherently extra compute. In direction of addressing these limitations, we introduce latent lookahead, a coaching technique that allows fashions to \u201cassume\u201d earlier than producing: at chosen positions within the sequence, earlier than committing to the subsequent token, the mannequin performs a multi-step lookahead in latent area. Extra exactly, as a substitute of sampling future tokens, we leverage the community\u2019s latent area by recursively feeding its hidden states again into the context for \u03c4 steps, investing extra compute on predicting that token. This produces \u03c4 latent predictions which might be supervised in opposition to the subsequent \u03c4 ground-truth tokens, encouraging the mannequin to \u201clookahead\u201d and refine its prediction. We present that latent lookahead considerably outperforms each autoregressive and non-autoregressive baselines on planning duties comparable to maze fixing, Sudoku, and ProsQA, the place foresight is important.<\/p>\n

** Work executed whereas at Apple<\/li>\n<\/ul>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"
This paper was accepted on the Workshop on Latent & Implicit Considering \u2013 Going Past CoT Reasoning 2026 at ICLR. Autoregressive language fashions skilled with next-token prediction generate textual content by sampling one discrete token at a time. Though very scalable, this goal forces the mannequin to commit at each step, stopping it from exploring […]<\/p>\n","protected":false},"author":2,"featured_media":13106,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[117,1511,8388,359,2401,7101],"class_list":["post-13104","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-future","tag-latent","tag-lookahead","tag-thinking","tag-training","tag-transformers"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/13104","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=13104"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/13104\/revisions"}],"predecessor-version":[{"id":13105,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/13104\/revisions\/13105"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/13106"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=13104"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=13104"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=13104"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}