{"id":18457,"date":"2026-09-06T07:30:36","date_gmt":"2026-09-06T07:30:36","guid":{"rendered":"https:\/\/techtrendfeed.com\/?p=18457"},"modified":"2026-09-06T07:30:37","modified_gmt":"2026-09-06T07:30:37","slug":"why-transformers-want-positional-encoding-for-time-sequence-a-visible-information","status":"publish","type":"post","link":"https:\/\/techtrendfeed.com\/?p=18457","title":{"rendered":"Why Transformers Want Positional Encoding For Time Sequence: A Visible Information"},"content":{"rendered":"


\n<\/p>\n

\n

Whereas digging into basis fashions for time sequence, I spotted that I may not likely perceive them with out first understanding transformers. I didn’t wish to use these fashions as black packing containers, so I began tracing the concepts backward, from basis fashions to transformers, and from transformers to self-attention. What made the transition fascinating is that though transformers have been initially constructed for language, the core thought carries naturally to time sequence. The 2 modalities are very completely different, however they share one thing basic: each are sequences, and in each circumstances, order modifications which means<\/strong>.<\/p>\n

In language, canine bites man<\/em> could be very completely different from man bites canine.<\/em><\/p>\n

Time sequence aren’t any completely different. A temperature of 30<\/mn>\u2218<\/mo><\/msup><\/mrow>30^circ<\/annotation><\/semantics><\/math><\/span>3<\/span>0<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> yesterday and 20<\/mn>\u2218<\/mo><\/msup><\/mrow>20^circ<\/annotation><\/semantics><\/math><\/span>2<\/span>0<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> right this moment tells a distinct story from 20<\/mn>\u2218<\/mo><\/msup><\/mrow>20^circ<\/annotation><\/semantics><\/math><\/span>2<\/span>0<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>yesterday and 30<\/mn>\u2218<\/mo><\/msup><\/mrow>30^circ<\/annotation><\/semantics><\/math><\/span>3<\/span>0<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> right this moment. The values stands out as the identical, however their order modifications the which means of the sequence.<\/p>\n

The query is that if self-attention seems to be in any respect observations directly, how does a transformer know which remark got here first, which got here later, or how far aside two observations are?<\/strong><\/p>\n

That query led me to positional encoding.<\/p>\n

What stunned me most was how such a easy mathematical thought may give a Transformer a way of order. The precise strategies have developed significantly since then, however the underlying drawback stays the identical.<\/p>\n

This text is my try to construct that instinct from the bottom up, beginning with a easy time sequence and following the trail from uncooked observations to self-attention and eventually to positional encoding.<\/p>\n

From scalar observations to vector representations<\/h2>\n

Take into account a easy time sequence containing the temperature recorded over 5 weekdays:<\/p>\n

\n
\"\"<\/div>
Instance of time sequence: 5-day temperature historical past<\/figcaption><\/figure>\n

Every remark x<\/mi>t<\/mi><\/msub><\/mrow>x_t<\/annotation><\/semantics><\/math><\/span>x<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> is barely a scalar. A transformer, nevertheless, operates on vectors of dimensionality d<\/mi>m<\/mi>o<\/mi>d<\/mi>e<\/mi>l<\/mi><\/mrow><\/msub><\/mrow>d_{mannequin}<\/annotation><\/semantics><\/math><\/span>d<\/span>m<\/span>o<\/span>d<\/span>e<\/span>l<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>. The scalar observations subsequently must be mapped into that illustration house first.<\/p>\n

A easy method to do that is thru a discovered linear projection <\/strong>i.e. embedding<\/em>:<\/p>\n

e<\/mi>t<\/mi><\/msub>=<\/mo>W<\/mi>e<\/mi><\/msub>x<\/mi>t<\/mi><\/msub>+<\/mo>b<\/mi>e<\/mi><\/msub><\/mrow>e_t = W_e x_t + b_e<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>W<\/span>e<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>x<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span><\/span>b<\/span>e<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

giving us a sequence of vector representations: e<\/mi>1<\/mn><\/msub>,<\/mo>e<\/mi>2<\/mn><\/msub>,<\/mo>e<\/mi>3<\/mn><\/msub>,<\/mo>e<\/mi>4<\/mn><\/msub>,<\/mo>e<\/mi>5<\/mn><\/msub><\/mrow>e_1, e_2, e_3, e_4, e_5<\/annotation><\/semantics><\/math><\/span>e<\/span>1<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>e<\/span>2<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>e<\/span>3<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>e<\/span>4<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>e<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n
\n

An embedding is a deep, summary illustration of the sequence within the type of a multidimensional numerical vector that encodes its options and that the mannequin understands.<\/strong> [1]<\/p>\n<\/blockquote>\n

\n
\"\"<\/div>
Every time sequence token is represented by a discovered embedding<\/figcaption><\/figure>\n

Every e<\/mi>t<\/mi><\/msub><\/mrow>e_t<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> captures details about the noticed worth at that timestep, however at this level, it’s only a illustration of the remark.<\/p>\n

The necessary phrase right here is discovered<\/strong><\/em>. The mannequin will not be given a predefined vector illustration for a temperature equivalent to 18<\/mn>\u2218<\/mo><\/msup><\/mrow>18^circ<\/annotation><\/semantics><\/math><\/span>1<\/span>8<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>. The parameters W<\/mi>e<\/mi><\/msub><\/mrow>W_e<\/annotation><\/semantics><\/math><\/span>W<\/span>e<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> and b<\/mi>e<\/mi><\/msub><\/mrow>b_e<\/annotation><\/semantics><\/math><\/span>b<\/span>e<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> are discovered throughout coaching in order that the ensuing representations grow to be helpful for the duty. <\/p>\n

At this level, e<\/mi>t<\/mi><\/msub><\/mrow>e_t<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> represents what was noticed<\/strong>. It doesn’t but inform the mannequin the place that remark occurred within the sequence.<\/p>\n

How self-attention builds context?<\/h2>\n

Self-attention permits every remark to make use of info from the remainder of the sequence.<\/p>\n

Suppose we wish to replace Friday\u2019s illustration. The mannequin first creates three discovered projections from each e<\/mi>t<\/mi><\/msub><\/mrow>e_t<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>:<\/p>\n
q<\/mi>t<\/mi><\/msub>=<\/mo>W<\/mi>Q<\/mi><\/msub>e<\/mi>t<\/mi><\/msub>,<\/mo>okay<\/mi>t<\/mi><\/msub>=<\/mo>W<\/mi>Okay<\/mi><\/msub>e<\/mi>t<\/mi><\/msub>,<\/mo>v<\/mi>t<\/mi><\/msub>=<\/mo>W<\/mi>V<\/mi><\/msub>e<\/mi>t<\/mi><\/msub><\/mrow>q_t = W_Q e_t,qquad\nk_t = W_K e_t,qquad\nv_t = W_V e_t<\/annotation><\/semantics><\/math><\/span>q<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>W<\/span>Q<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>okay<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>W<\/span>Okay<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>v<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>W<\/span>V<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n
\n
\"\"<\/div>
Question, key, and worth vectors are discovered within the self-attention block<\/figcaption><\/figure>\n

The matrices W<\/mi>Q<\/mi><\/msub><\/mrow>W_Q<\/annotation><\/semantics><\/math><\/span>W<\/span>Q<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>, W<\/mi>Okay<\/mi><\/msub><\/mrow>W_K<\/annotation><\/semantics><\/math><\/span>W<\/span>Okay<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> and W<\/mi>V<\/mi><\/msub><\/mrow>W_V<\/annotation><\/semantics><\/math><\/span>W<\/span>V<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> are additionally discovered throughout coaching. The mannequin will not be advised beforehand what a helpful question, key, or worth ought to appear to be.<\/p>\n

For Friday, its question q<\/mi>5<\/mn><\/msub><\/mrow>q_5<\/annotation><\/semantics><\/math><\/span>q<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> is in contrast with the keys of all observations: okay<\/mi>1<\/mn><\/msub>,<\/mo>okay<\/mi>2<\/mn><\/msub>,<\/mo>okay<\/mi>3<\/mn><\/msub>,<\/mo>okay<\/mi>4<\/mn><\/msub>,<\/mo>okay<\/mi>5<\/mn><\/msub><\/mrow>k_1, k_2, k_3, k_4, k_5<\/annotation><\/semantics><\/math><\/span>okay<\/span>1<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>okay<\/span>2<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>okay<\/span>3<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>okay<\/span>4<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>okay<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>.<\/p>\n
\n
\"\"<\/div>\n<\/figure>\n

Every comparability produces an consideration rating<\/strong>:<\/p>\n

s<\/mi>5<\/mn>,<\/mo>j<\/mi><\/mrow><\/msub>=<\/mo>q<\/mi>5<\/mn>\u22a4<\/mi><\/msubsup>okay<\/mi>j<\/mi><\/msub><\/mrow>d<\/mi>okay<\/mi><\/msub><\/msqrt><\/mfrac><\/mrow>s_{5,j} = frac{q_5^high k_j}{sqrt{d_k}}<\/annotation><\/semantics><\/math><\/span>s<\/span>5<\/span>,<\/span>j<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>d<\/span>okay<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/svg><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>q<\/span>5<\/span><\/span><\/span>\u22a4<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>okay<\/span>j<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

which measures how related remark ‘j’ is when updating Friday\u2019s illustration. The scaling issue d<\/mi>okay<\/mi><\/msub><\/msqrt><\/mrow>sqrt{d_k} <\/annotation><\/semantics><\/math><\/span>d<\/span>okay<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/svg><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>prevents the dot merchandise from rising too massive because the dimensionality of the question and key vectors will increase.<\/p>\n

These scores are handed by a softmax perform to transform them into consideration weights:<\/p>\n

\u03b1<\/mi>5<\/mn>,<\/mo>j<\/mi><\/mrow><\/msub>=<\/mo>exp<\/mi>\u2061<\/mo>(<\/mo>s<\/mi>5<\/mn>,<\/mo>j<\/mi><\/mrow><\/msub>)<\/mo><\/mrow>\u2211<\/mo>j<\/mi>\u2032<\/mo><\/msup><\/munder>exp<\/mi>\u2061<\/mo>(<\/mo>s<\/mi>5<\/mn>,<\/mo>j<\/mi>\u2032<\/mo><\/msup><\/mrow><\/msub>)<\/mo><\/mrow><\/mfrac><\/mrow>alpha_{5,j} = frac{exp(s_{5,j})}{sum_{j’} exp(s_{5,j’})}<\/annotation><\/semantics><\/math><\/span>\u03b1<\/span>5<\/span>,<\/span>j<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>\u2211<\/span>j<\/span>\u2032<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>exp<\/span>(<\/span>s<\/span>5<\/span>,<\/span>j<\/span>\u2032<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span>exp<\/span>(<\/span>s<\/span>5<\/span>,<\/span>j<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

Lastly, these weights are used to mix the worth vectors:<\/p>\n

z<\/mi>5<\/mn><\/msub>=<\/mo>\u2211<\/mo>j<\/mi><\/munder>\u03b1<\/mi>5<\/mn>,<\/mo>j<\/mi><\/mrow><\/msub>v<\/mi>j<\/mi><\/msub>.<\/mi><\/mrow>z_5 = sum_j alpha_{5,j}v_j.<\/annotation><\/semantics><\/math><\/span>z<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>j<\/span><\/span><\/span>\u2211<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span>\u03b1<\/span>5<\/span>,<\/span>j<\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>v<\/span>j<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>.<\/span><\/span><\/span><\/span><\/span><\/div>\n

So <\/em>e<\/mi>5<\/mn><\/msub><\/mrow>e_5<\/annotation><\/semantics><\/math><\/span>e<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> is Friday\u2019s illustration earlier than incorporating info from the remainder of the sequence, whereas<\/em> <\/strong><\/em>z<\/mi>5<\/mn><\/msub><\/mrow>z_5<\/annotation><\/semantics><\/math><\/span>z<\/span>5<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> <\/strong><\/em>is its <\/em>context-aware illustration<\/strong><\/em> after self-attention.<\/em><\/p>\n

Briefly:<\/p>\n

\n

Queries and keys study which observations are related to at least one one other. Values carry the knowledge that’s mixed to type the brand new illustration.<\/p>\n<\/blockquote>\n

What occurs if we shuffle the sequence?<\/h2>\n

Now comes the necessary query.<\/p>\n

Suppose the identical 5 temperature observations are rearranged.<\/p>\n

\n
\"\"<\/div>\n<\/figure>\n

The values themselves haven’t modified; solely the order has. After the discovered projection, we nonetheless have the identical set of worth representations, simply rearranged.<\/p>\n

Self-attention can nonetheless examine every illustration with all of the others. The identical question, key, and worth projections are utilized, and the identical sorts of pairwise relationships can nonetheless be computed.<\/p>\n

What has disappeared is the temporal construction.<\/strong><\/em><\/p>\n

Nothing inside e<\/mi>(<\/mo>27<\/mn>\u2218<\/mo><\/msup>)<\/mo><\/mrow>e(27^circ)<\/annotation><\/semantics><\/math><\/span>e<\/span>(<\/span>2<\/span>7<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span> says that it initially got here from Thursday. Nothing inside e<\/mi>(<\/mo>18<\/mn>\u2218<\/mo><\/msup>)<\/mo><\/mrow>e(18^circ)<\/annotation><\/semantics><\/math><\/span>e<\/span>(<\/span>1<\/span>8<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span> says that it occurred after e<\/mi>(<\/mo>27<\/mn>\u2218<\/mo><\/msup>)<\/mo><\/mrow>e(27^circ)<\/annotation><\/semantics><\/math><\/span>e<\/span>(<\/span>2<\/span>7<\/span>\u2218<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span>. Additionally, if Wednesday and Friday have the identical temperature worth, the discovered projection will map them to the identical embedding vector. With out positional info, the mannequin subsequently has no option to distinguish which embedding got here from Wednesday and which got here from Friday.<\/p>\n

That is the important thing limitation:<\/em><\/p>\n

\n

Self-attention can study which observations are associated, however with out an extra positional sign, it has no built-in option to know the place these observations occurred within the sequence.<\/p>\n<\/blockquote>\n

What ought to positional info inform the mannequin?<\/h2>\n

If self-attention doesn’t know the order of the observations, then the following query is: what sort of positional info could be helpful?<\/strong><\/p>\n

At a minimal, we’d need the mannequin to know:<\/p>\n

    \n
  • \n

    Which place an remark belongs to?<\/strong>
    Place 2 needs to be distinguishable from place 20.<\/p>\n<\/li>\n

  • \n

    Which remark got here earlier than or after one other?<\/strong>
    The mannequin ought to have the ability to distinguish t<\/mi>\u2212<\/mo>1<\/mn><\/mrow>t-1<\/annotation><\/semantics><\/math><\/span>t<\/span>\u2212<\/span><\/span>1<\/span><\/span><\/span><\/span><\/span> from t<\/mi>+<\/mo>1<\/mn><\/mrow>t+1<\/annotation><\/semantics><\/math><\/span>t<\/span>+<\/span><\/span>1<\/span><\/span><\/span><\/span><\/span>.<\/p>\n<\/li>\n
  • \n

    How far aside are two observations?<\/strong>
    In time sequence, the distinction between t<\/mi>\u2212<\/mo>1<\/mn><\/mrow>t-1<\/annotation><\/semantics><\/math><\/span>t<\/span>\u2212<\/span><\/span>1<\/span><\/span><\/span><\/span><\/span>, t<\/mi>\u2212<\/mo>7<\/mn>,<\/mo><\/mrow>t-7,<\/annotation><\/semantics><\/math><\/span>t<\/span>\u2212<\/span><\/span>7<\/span>,<\/span><\/span><\/span><\/span><\/span> and t<\/mi>\u2212<\/mo>30<\/mn><\/mrow>t-30<\/annotation><\/semantics><\/math><\/span>t<\/span>\u2212<\/span><\/span>30<\/span><\/span><\/span><\/span><\/span> may be necessary.<\/p>\n<\/li>\n
  • \n

    That close by positions are associated in a structured method.<\/strong>
    Place 10 and place 11 mustn’t appear to be two fully unrelated identifiers.<\/p>\n<\/li>\n

  • \n

    That the illustration stays helpful over longer sequences.<\/strong>
    Ideally, the positional scheme ought to nonetheless present significant construction because the sequence grows.<\/p>\n<\/li>\n<\/ul>\n

    For time sequence, the third property is very helpful. A mannequin might care about an remark one step in the past due to short-term dependence, or seven steps in the past due to a weekly seasonal sample.<\/p>\n

    \n

    So positional info ought to do greater than merely assign a novel label to every timestep. It ought to give the mannequin a structured illustration of order and relative distance.<\/p>\n<\/blockquote>\n

    How can we signify place?<\/h2>\n

    We now know what info is lacking. The following query is learn how to signify it. <\/p>\n

    A easy option to signify place could be to assign every timestep a quantity:<\/p>\n

    1<\/mn>,<\/mo>2<\/mn>,<\/mo>3<\/mn>,<\/mo>\u2026<\/mo><\/mrow>1, 2, 3, ldots<\/annotation><\/semantics><\/math><\/span>1<\/span>,<\/span>2<\/span>,<\/span>3<\/span>,<\/span>\u2026<\/span><\/span><\/span><\/span><\/span><\/div>\n

    However feeding the uncooked place immediately into the mannequin will not be perfect. The values continue to grow with sequence size, and a single quantity doesn’t give the mannequin a wealthy illustration of positional relationships.<\/p>\n

    One of many authentic Transformer\u2019s options was sinusoidal positional encoding<\/strong><\/em>, the place every place is represented utilizing sine and cosine capabilities at completely different frequencies.<\/p>\n

    Why sine and cosine?<\/h3>\n

    Begin with the best two-dimensional instance:<\/p>\n

    p<\/mi>t<\/mi><\/msub>=<\/mo>[<\/mo>sin<\/mi>\u2061<\/mo>(<\/mo>t<\/mi>)<\/mo>\u00a0<\/mtext>cos<\/mi>\u2061<\/mo>(<\/mo>t<\/mi>)<\/mo><\/mrow><\/mstyle><\/mtd><\/mtr><\/mtable>]<\/mo><\/mrow><\/mrow>p_t =\nstart{bmatrix}\nsin(t) \ncos(t)\nfinish{bmatrix}<\/annotation><\/semantics><\/math><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>[<\/span><\/span>sin<\/span>(<\/span>t<\/span>)<\/span>\u00a0<\/span>cos<\/span>(<\/span>t<\/span>)<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>]<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

    As t<\/mi><\/mrow>t<\/annotation><\/semantics><\/math><\/span>t<\/span><\/span><\/span><\/span><\/span> modifications, the positional vector strikes easily round a circle that permits close by positions to have completely different however nonetheless associated representations. <\/p>\n
    \n
    \"\"<\/div>\n<\/figure>\n

    Extra importantly, shifting ahead by the identical variety of steps produces the identical type of change within the positional illustration. For instance, an offset of seven positions has the identical mathematical relationship whether or not we transfer from place 3 to 10 or from place 20 to 27. That’s helpful for time sequence as a result of relative distance typically issues:<\/p>\n

    t<\/mi>\u2212<\/mo>1<\/mn>,<\/mo>t<\/mi>\u2212<\/mo>7<\/mn>,<\/mo>t<\/mi>\u2212<\/mo>30<\/mn><\/mrow>t-1,qquad t-7,qquad t-30<\/annotation><\/semantics><\/math><\/span>t<\/span>\u2212<\/span><\/span>1<\/span>,<\/span>t<\/span>\u2212<\/span><\/span>7<\/span>,<\/span>t<\/span>\u2212<\/span><\/span>30<\/span><\/span><\/span><\/span><\/span><\/div>\n

    can signify very completely different temporal relationships.<\/p>\n

    The total sinusoidal positional encoding extends this concept throughout many dimensions:<\/p>\n

    P<\/mi>E<\/mi>(<\/mo>t<\/mi>,<\/mo>2<\/mn>i<\/mi>)<\/mo>=<\/mo>sin<\/mi>\u2061<\/mo>(<\/mo>t<\/mi>10000<\/mn>2<\/mn>i<\/mi>\/<\/mi>d<\/mi>mannequin<\/mtext><\/msub><\/mrow><\/msup><\/mfrac>)<\/mo><\/mrow><\/mrow>PE(t,2i)=\nsinleft(\nfrac{t}{10000^{2i\/d_{textual content{mannequin}}}}\nproper)<\/annotation><\/semantics><\/math><\/span>P<\/span>E<\/span>(<\/span>t<\/span>,<\/span>2<\/span>i<\/span>)<\/span>=<\/span><\/span>sin<\/span>(<\/span><\/span>1000<\/span>0<\/span>2<\/span>i<\/span>\/<\/span>d<\/span>mannequin<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n
    P<\/mi>E<\/mi>(<\/mo>t<\/mi>,<\/mo>2<\/mn>i<\/mi>+<\/mo>1<\/mn>)<\/mo>=<\/mo>cos<\/mi>\u2061<\/mo>(<\/mo>t<\/mi>10000<\/mn>2<\/mn>i<\/mi>\/<\/mi>d<\/mi>mannequin<\/mtext><\/msub><\/mrow><\/msup><\/mfrac>)<\/mo><\/mrow><\/mrow>PE(t,2i+1)=\ncosleft(\nfrac{t}{10000^{2i\/d_{textual content{mannequin}}}}\nproper)<\/annotation><\/semantics><\/math><\/span>P<\/span>E<\/span>(<\/span>t<\/span>,<\/span>2<\/span>i<\/span>+<\/span><\/span>1<\/span>)<\/span>=<\/span><\/span>cos<\/span>(<\/span><\/span>1000<\/span>0<\/span>2<\/span>i<\/span>\/<\/span>d<\/span>mannequin<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>)<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

    Totally different dimensions use completely different frequencies. Some change shortly throughout close by positions, whereas others change far more slowly.<\/p>\n

    \n
    \"\"<\/div>\n<\/figure>\n

    One helpful method to consider that is as many clocks operating at completely different speeds. Collectively, their readings give each place a structured positional signature.<\/p>\n

    So as a substitute of giving timestep (t) solely a quantity, we give it a vector:<\/p>\n

    p<\/mi>t<\/mi><\/msub>\u2208<\/mo>R<\/mi>d<\/mi>mannequin<\/mtext><\/msub><\/msup><\/mrow>p_t in mathbb{R}^{d_{textual content{mannequin}}}<\/annotation><\/semantics><\/math><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u2208<\/span><\/span>R<\/span>d<\/span>mannequin<\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

    that accommodates details about its place and its relationship to different positions.<\/p>\n

    Combining worth and place<\/h2>\n

    After including positional info, every timestep is represented as:<\/p>\n

    h<\/mi>t<\/mi><\/msub>=<\/mo>e<\/mi>t<\/mi><\/msub>+<\/mo>p<\/mi>t<\/mi><\/msub><\/mrow>h_t = e_t + p_t<\/annotation><\/semantics><\/math><\/span>h<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>=<\/span><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>+<\/span><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

    the place e<\/mi>t<\/mi><\/msub><\/mrow>e_t<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> represents the noticed worth and p<\/mi>t<\/mi><\/msub><\/mrow>p_t<\/annotation><\/semantics><\/math><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> represents its place within the sequence.<\/p>\n

    Self-attention now builds its queries and keys from this mixed illustration:<\/p>\n

    Q<\/mi>=<\/mo>H<\/mi>W<\/mi>Q<\/mi><\/msub>,<\/mo>Okay<\/mi>=<\/mo>H<\/mi>W<\/mi>Okay<\/mi><\/msub><\/mrow>Q = HW_Q,qquad Okay = HW_K<\/annotation><\/semantics><\/math><\/span>Q<\/span>=<\/span><\/span>H<\/span>W<\/span>Q<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span>Okay<\/span>=<\/span><\/span>H<\/span>W<\/span>Okay<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/div>\n

    So when the mannequin computes an consideration rating,<\/p>\n

    rating<\/mtext>(<\/mo>i<\/mi>,<\/mo>j<\/mi>)<\/mo>=<\/mo>q<\/mi>i<\/mi>\u22a4<\/mi><\/msubsup>okay<\/mi>j<\/mi><\/msub>,<\/mo><\/mrow>textual content{rating}(i,j)=q_i^high k_j,<\/annotation><\/semantics><\/math><\/span>rating<\/span><\/span>(<\/span>i<\/span>,<\/span>j<\/span>)<\/span>=<\/span><\/span>q<\/span>i<\/span><\/span><\/span>\u22a4<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>okay<\/span>j<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>,<\/span><\/span><\/span><\/span><\/span><\/div>\n

    the comparability is now not based mostly solely on the noticed values. The question and key vectors have been created from representations that already comprise positional info.<\/p>\n

    In consequence, the mannequin can study relationships that rely on each:<\/p>\n

      \n
    • \n

      what was noticed<\/strong>, and<\/p>\n<\/li>\n

    • \n

      the place the observations occurred within the sequence<\/strong>.<\/p>\n<\/li>\n<\/ul>\n

      For a time sequence, this implies the mannequin can probably study that an remark one timestep in the past needs to be handled in another way from one seven timesteps in the past, even when their values are comparable.<\/p>\n

      Positional encoding subsequently doesn’t inform the mannequin explicitly which lags are necessary. It offers self-attention the knowledge wanted to study which positional relationships matter for the duty<\/strong>.<\/p>\n

      Abstract<\/h2>\n

      We began with a easy drawback: self-attention can examine each remark with each different remark, however by itself it doesn’t know the order wherein these observations occurred.<\/p>\n

      The answer is to counterpoint every worth illustration e<\/mi>t<\/mi><\/msub><\/mrow>e_t<\/annotation><\/semantics><\/math><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b with a positional illustration p<\/mi>t<\/mi><\/msub><\/mrow>p_t<\/annotation><\/semantics><\/math><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b: <\/p>\n
      h<\/mi>t<\/mi><\/msub>\u200b<\/mtext>=<\/mo>e<\/mi>t<\/mi><\/msub>\u200b<\/mtext>+<\/mo>p<\/mi>t<\/mi><\/msub>\u200b<\/mtext><\/mrow>h_t\u200b = e_t \u200b+ p_t\u200b<\/annotation><\/semantics><\/math><\/span>h<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span>=<\/span><\/span>e<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span>+<\/span><\/span>p<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/div>\n

      Self-attention then builds its queries, keys, and values from h<\/mi>t<\/mi><\/msub><\/mrow>h_t<\/annotation><\/semantics><\/math><\/span>h<\/span>t<\/span><\/span><\/span><\/span>\u200b<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u200b fairly than from the worth illustration alone. This implies the eye mechanism can study relationships based mostly not solely on what was noticed<\/strong>, but in addition on the place that remark occurred within the sequence<\/strong>.<\/p>\n

      For usually sampled time sequence, this makes relationships equivalent to t\u22121, t\u22127, or t\u221224 accessible to the mannequin. A lag of 1 step might seize short-term dependence, whereas a lag of seven or twenty-four steps might correspond to a seasonal sample.<\/p>\n

      Positional encoding, nevertheless, represents sequence place, not essentially real-world time. If observations are irregularly spaced, being one place aside doesn’t at all times imply being one hour or at some point aside. That is the place richer temporal encodings and time options grow to be necessary.<\/p>\n

      Sinusoidal positional encoding is just one option to inject order right into a Transformer. Different approaches embrace discovered positional embeddings<\/strong>, the place the place vectors themselves are discovered throughout coaching, and relative positional encodings<\/strong>, which focus extra immediately on the space between two observations fairly than their absolute positions.<\/p>\n

      For time sequence, the issue can grow to be even richer. Sequence place is probably not sufficient: the mannequin may must know the precise timestamp, calendar results, periodicity, or irregular gaps between observations.<\/p>\n

      So the broader query will not be merely:<\/p>\n

      How will we inform a Transformer that that is place 7?<\/em><\/p>\n

      however fairly:<\/p>\n

      What notion of time does the mannequin really need for the duty?<\/em><\/p>\n

      \u00b7\u00b7\u00b7<\/p>\n

      And this brings me again to the place this exploration began: time-series basis fashions. Trendy architectures might use extra subtle approaches equivalent to rotary or discovered positional representations fairly than the unique sinusoidal formulation. However understanding the easy sinusoidal building offers us the muse for understanding why these strategies exist within the first place.<\/p>\n

      We now have now constructed the muse of why positional info is required, how it’s represented, and the way self-attention makes use of it. With this instinct in place, extra superior positional encoding strategies ought to really feel a lot simpler to know once we encounter them in future.<\/p>\n

      \u00b7\u00b7\u00b7<\/p>\n

      Observe: The figures on this article have been conceptually designed by the writer and generated and refined with the help of an AI image-generation device.<\/p>\n

      References<\/h2>\n

      [1] Peixeiro, Marco. Time Sequence Forecasting Utilizing Basis Fashions: Learn how to Construct Excessive Accuracy Predictive Fashions<\/em>. Manning, 2025.<\/p>\n

      [2] Davidson, Graeme, and Lei Ma. Time Sequence with PyTorch: Trendy Deep Studying Toolkit for Actual-World Forecasting Challenges<\/em>. Packt Publishing, 2026.<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":"

      Whereas digging into basis fashions for time sequence, I spotted that I may not likely perceive them with out first understanding transformers. I didn’t wish to use these fashions as black packing containers, so I began tracing the concepts backward, from basis fashions to transformers, and from transformers to self-attention. What made the transition fascinating […]<\/p>\n","protected":false},"author":2,"featured_media":18459,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[10452,78,10451,2302,956,7101,1555],"class_list":["post-18457","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-encoding","tag-guide","tag-positional","tag-series","tag-time","tag-transformers","tag-visual"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/18457","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=18457"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/18457\/revisions"}],"predecessor-version":[{"id":18458,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/18457\/revisions\/18458"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/18459"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=18457"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=18457"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=18457"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}