1Score

query of “it” · key of each word

  1. animal25.6
  2. tired15.2
  3. so3.2
  4. it8.8

2Scale

divide by √64 = 8

  1. animal3.2
  2. tired1.9
  3. so0.4
  4. it1.1

3Softmax

positive, sums to 100%

  1. animal69%
  2. tired19%
  3. so4%
  4. it8%

4Mix

output = Σ weight × value

  1. animal×0.69
  2. tired×0.19
  3. so×0.04
  4. it×0.08

One head, one word. Illustrative numbers; 64 is the key size per head in the original paper. Every word does this for every other word at the same time.