Pytorch implementation of Google AI's 2017 Transformer model
2017 Transformer: Attention Is All You Need https://arxiv.org/abs/1706.03762
Google AI in 2017 proposed a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. As you see, this model architecture becomes the base stone of the following state-of-the-art pre-trained models in Natural Language Processing (NLP), such as GPT, BERT, Transformer-XL,XLnet, RoBERTa.
This repo will walk through the implementation of Transformer. Code is simple, clean and easy to understand. Some of these codes are based on The Annotated Transformer.
Currently this project is working in process, as always, PRs are welcome :)
- python >= 3.6
- pytorch >=1.0.0
- torchtext >=0.2.3
git clone https://github.com/walkacross/transformer-pytorch.git
cd transformer-pytorch
python setup.py develop
Allen Yu ([email protected])
This project following Apache 2.0 License as written in LICENSE file
Copyright 2019 Allen Yu, Quantitative Finance Lab, respective Transformer contributors
Copyright (c) 2018 Alexander Rush : Annotated Transformer