The transformer

In 2017 Ashish Vaswani and colleagues publish 'Attention Is All You Need.' The transformer is a network design that looks at a whole sequence at once, using a mechanism called attention, instead of reading one word strictly left to right. It trains well on many machines at once.

The contribution is language at scale. The big models behind chatbots and translation are transformers. So are many image generators, which treat pixels with the same trick. The paper does not ship a product. Almost every large model after it inherits the architecture.

A transformer needs the 2012 lesson, huge text from the web, and chips arranged in clusters. It is not a robot and it does not understand in the way a person does. It predicts the next piece of a pattern.

The year is the paper, not the first chatbot. Public chatbots are 2022.