\n<\/p>\n
Aligned representations throughout languages is a desired property in multilingual massive language fashions (mLLMs), as alignment can enhance efficiency in cross-lingual duties. Usually alignment requires fine-tuning a mannequin, which is computationally costly, and sizable language information, which regularly is probably not accessible. An information-efficient different to fine-tuning is mannequin interventions — a way for manipulating mannequin activations to steer era into the specified course. We analyze the impact of a preferred intervention (discovering specialists) on the alignment of cross-lingual representations in mLLMs. We establish the neurons to control for a given language and introspect the embedding area of mLLMs pre- and post-manipulation. We present that modifying the mLLM’s activations modifications its embedding area such that cross-lingual alignment is enhanced. Additional, we present that the modifications to the embedding area translate into improved downstream efficiency on retrieval duties, with as much as 2x enhancements in top-1 accuracy on cross-lingual retrieval.<\/p>\n
Aligned representations throughout languages is a desired property in multilingual massive language fashions (mLLMs), as alignment can enhance efficiency in cross-lingual duties. Usually alignment requires fine-tuning a mannequin, which is computationally costly, and sizable language information, which regularly is probably not accessible. An information-efficient different to fine-tuning is mannequin interventions — a way for manipulating […]<\/p>\n","protected":false},"author":2,"featured_media":4814,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[3493,2693,4201,1600,4202,4203,634,358,266,2512,155,4200],"class_list":["post-4812","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-alignment","tag-analyzing","tag-crosslingual","tag-embedding","tag-induced","tag-interventions","tag-language","tag-model","tag-models","tag-multilingual","tag-spaces","tag-steering"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/4812","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4812"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/4812\/revisions"}],"predecessor-version":[{"id":4813,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/4812\/revisions\/4813"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/4814"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4812"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4812"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4812"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}