\n<\/p>\n
The unique Transformer had two halves: an encoder and a decoder, designed for sequence-to-sequence duties like translation. Fashionable LLMs usually use just one half, tailored for his or her particular objective.<\/p>\n
Encoder-only fashions course of your entire enter sequence without delay, with each token capable of attend to each different token, together with tokens that come after it (that is known as bidirectional consideration). They\u2019re well-suited to duties that require understanding an entire enter, like classification, sentence similarity, or extracting solutions from textual content.<\/p>\n
Instance: BERT.<\/strong> BERT is skilled utilizing masked language modeling \u2014 some tokens within the enter are hidden, and the mannequin should predict them utilizing context from each instructions. It\u2019s broadly used for duties like search relevance, textual content classification, and named entity recognition, but it surely isn\u2019t designed to generate free-flowing textual content.<\/p>\n Decoder-only fashions use causal (masked) self-attention, the place every token can solely attend to itself and tokens earlier than it, by no means tokens after it. This makes them naturally suited to textual content technology, since producing textual content word-by-word requires solely understanding what got here earlier than.<\/p>\n Instance: GPT.<\/strong> The GPT household (and the overwhelming majority of recent chat-oriented LLMs, together with LLaMA, Mistral, Gemma, and Qwen) are decoder-only. This structure has grow to be the dominant alternative for general-purpose language fashions as a result of next-token prediction is a versatile coaching goal that scales properly and naturally helps open-ended technology.<\/p>\n These retain each halves: an encoder processes the enter, and a decoder generates output whereas attending each to beforehand generated tokens and to the encoder\u2019s output (by way of cross-attention).<\/p>\n Instance: T5.<\/strong> T5 frames each job \u2014 translation, summarization, query answering \u2014 as a text-to-text downside, utilizing the encoder to course of the enter and the decoder to generate the output. This structure stays in style for duties with a transparent, distinct enter and output, comparable to machine translation.<\/p>\n Structure Consideration Sort Finest Suited For Examples Encoder-only Bidirectional Classification, understanding duties BERT, RoBERTa Decoder-only Causal (masked) Open-ended textual content technology, chat GPT, LLaMA, Mistral, Gemma Encoder-Decoder Bidirectional + Cross-attention Translation, summarization T5, BART<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" Encoder vs. Decoder The unique Transformer had two halves: an encoder and a decoder, designed for sequence-to-sequence duties like translation. Fashionable LLMs usually use just one half, tailored for his or her particular objective. Encoder-Solely Fashions Encoder-only fashions course of your entire enter sequence without delay, with each token capable of attend to each different […]<\/p>\n","protected":false},"author":2,"featured_media":16392,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[2696,1894,1199,3818,634,1797,358,226,9640,8596],"class_list":["post-16390","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-architecture","tag-explained","tag-foundation","tag-jul","tag-language","tag-large","tag-model","tag-modern","tag-subhash","tag-transformer"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16390","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=16390"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16390\/revisions"}],"predecessor-version":[{"id":16391,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/16390\/revisions\/16391"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/16392"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=16390"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=16390"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=16390"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Decoder-Solely Fashions<\/h2>\n
Encoder-Decoder Fashions<\/h2>\n
The place Every Is Used<\/h2>\n