\n<\/p>\n
Tokenizer design considerably impacts language mannequin efficiency,
\nbut evaluating tokenizer high quality stays difficult. Whereas textual content compression has emerged as a typical intrinsic metric, latest work questions its reliability as a high quality indicator. We examine whether or not evaluating tokenizers on smaller fashions (350M parameters) reliably predicts their influence at bigger scales (2.7B parameters).
\nBy experiments with established tokenizers from widely-adopted language fashions, we discover that tokenizer selection minimally impacts English duties however yields vital, scale-consistent variations in machine translation efficiency.
\nPrimarily based on these findings, we suggest extra intrinsic metrics that correlate extra strongly with downstream efficiency than textual content compression.
\nWe mix these metrics into an analysis framework that permits extra dependable intrinsic tokenizer comparisons.<\/p>\n
Tokenizer design considerably impacts language mannequin efficiency, but evaluating tokenizer high quality stays difficult. Whereas textual content compression has emerged as a typical intrinsic metric, latest work questions its reliability as a high quality indicator. We examine whether or not evaluating tokenizers on smaller fashions (350M parameters) reliably predicts their influence at bigger scales (2.7B […]<\/p>\n","protected":false},"author":2,"featured_media":3258,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[3086,1279,3088,3085,3087],"class_list":["post-3256","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-compression","tag-evaluating","tag-scales","tag-text","tag-tokenizers"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/3256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3256"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/3256\/revisions"}],"predecessor-version":[{"id":3257,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/3256\/revisions\/3257"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/3258"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3256"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3256"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}