Attention Is All You Need

Attention Is All You Need introduces the Transformer, a neural network for turning an input sequence into an output sequence, such as translating an English sentence into German. Its central change is to let positions exchange information through attention instead of passing it step by step through a recurrent network. This makes more of the training computation parallel while retaining strong translation results. Attention itself predates this paper; the contribution is the particular architecture built around it.^[1]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin are the eight authors in the original printed order. Their contribution footnote specifies equal contributions and random listing order. The chronology marks the first arXiv submission on June 12, 2017; the paper also appeared in the 2017 NIPS proceedings. The explanation and scores below refer specifically to arXiv v1.^[1]

How the architecture works

An encoder turns the input tokens—words or smaller text units—into contextual vectors. A decoder uses those vectors and the output produced so far to predict the next output token. The base Transformer stacks six encoder layers and six decoder layers, with 512-dimensional representations. Each layer combines attention with a feed-forward network applied separately at each position; residual connections and layer normalization surround the sublayers. “Attention is all you need” therefore does not mean the network contains only attention operations.^[1]

In self-attention, each position supplies a query, key and value, obtained through learned projections. A query is compared with the keys to determine how much weight to give each value. The model divides query–key dot products by the square root of the key dimension, then applies softmax to obtain normalized weights and forms a weighted sum of the values. The base model runs eight attention heads in parallel, each with 64-dimensional keys and values, so different heads can combine different information rather than forcing it into one weighted average.^[1]

The encoder can attend across the input sentence. Decoder self-attention masks future output positions, preventing a prediction from using the answer it is supposed to predict. A separate encoder–decoder attention sublayer lets output positions consult the encoded input. Training can calculate many token positions together because the target sequence is available, but generating a new translation still proceeds one token at a time.^[1]

Without recurrence or convolution, the model needs another way to represent order. It adds fixed sine and cosine positional encodings, at different frequencies, to token embeddings at the bottoms of both stacks. These provide position information alongside token identity; they are distinct from the learned attention weights.^[1]

What the experiments measured

The translation experiments use WMT 2014 data: about 4.5 million English–German sentence pairs and 36 million English–French sentences. English–German uses a shared vocabulary of about 37,000 byte-pair tokens; English–French uses 32,000 word-piece tokens. Batches group similar sequence lengths and contain approximately 25,000 source and 25,000 target tokens. Training uses Adam with 4,000 learning-rate warmup steps, followed by inverse-square-root decay, plus dropout and label smoothing.^[1]

Table 2 reports the following newstest2014 results. BLEU measures translation overlap with reference translations; these scores compare performance on the specified tests, rather than measuring general intelligence or alignment.^[1]

v1 model	English–German BLEU	English–French BLEU
Base	27.3	38.1
Big	28.4	41.0

Both models train on eight NVIDIA P100 GPUs. Base trains for 100,000 steps, about 12 hours; big trains for 300,000 steps, about 3.5 days. The big configuration has wider, 1,024-dimensional representations and 16 attention heads. Its English–German score exceeds the best listed ensemble, 26.36, by just over two BLEU. For English–French, 41.0 exceeds the listed single models but remains below the best listed ensemble, 41.29. The English–French big run uses dropout 0.1 rather than the big configuration’s usual 0.3.^[1]

These are single-model results obtained by averaging the last five checkpoints for base and twenty for big. Decoding uses beam search of width four with length penalty 0.6, selected on development data. The paper estimates training computation from elapsed time, GPU count and assumed sustained floating-point throughput: 3.3 × 10¹⁸ operations for base and 2.3 × 10¹⁹ for big. Its cost comparison is an estimate against systems reported in the literature, rather than a controlled rerun of every competitor on identical hardware.^[1]

Tradeoffs and scope

The component experiments use newstest2013 development data, without checkpoint averaging. Single-head attention scores 24.9 BLEU versus the base model’s 25.8; increasing the number of heads to 32 also reduces quality, to 25.4. This supports a useful role for multiple heads under the tested configurations, rather than a rule that more heads always help.^[1]

A separate four-layer model reaches 91.3 F1 on English constituency parsing—assigning a sentence a grammatical phrase structure—with roughly 40,000 training sentences and 92.7 with semi-supervised corpora of about 17 million sentences; the table’s generative grammar comparator scores 93.3.^[1]

Self-attention shortens the computational path between distant positions and removes recurrence’s sequential training dependency, but unrestricted attention compares every pair of positions. Its per-layer complexity grows quadratically with sequence length, so the paper’s efficiency argument is conditional on sequence length and representation width. Faster training also does not remove sequential decoding.^[1]

The origins guide places this architecture advance beside a different research problem: deciding what behavior to train and whose interests it should serve. Translation quality and parsing accuracy do not answer that question. Training language models to follow instructions with human feedback examines how feedback can shape trained behavior; the Transformer paper’s distinctive result is that a different way of moving information through a sequence model can improve the measured capability–computation tradeoff.