We’ll cover three of the seminal papers in sequence to sequence learning:
- Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks by Alex Graves et al. (https://dl.acm.org/citation.cfm?id=1143891, author’s copy); check out this very helpful visualization
- Sequence to Sequence Learning for Neural Networls by Sutskever et al. (https://arxiv.org/abs/1409.3215)
- Neural Machine Translation by Jointly Learning to Align and Translate by Dzmitry Bahdanau et al. (https://arxiv.org/abs/1409.0473).
- Listen, Attend and Spell by William Chan et al. (https://arxiv.org/abs/1508.01211)
- Attention is all you need (https://arxiv.org/abs/1706.03762)
As well as Attention and Augmented Recurrent Neural Networks by Chris Olah and Shan Carter.
There is way too much to cover in one exercise. Here are a few options:
- Transformer Implementation in TensorFlow for Machine Translation. Recommended exercise, using theses titles 2014-2018 (see Moodle)
- Extensive TensorFlow tutorial on machine translation using attention (Colab).
- Synthesis of handwriting using rnnlib and the Uni Bern IAM On-Line Handwriting Database; see remarks below.
- You can either register for the dataset, or obtain a copy during class.
- If you run into trouble building openblas (as a submodule of rnnlib), you may need to specify the target CPU architecture (eg. Macbook Pro 2017: HASWELL, see example diff file).
- It’s recommended to add the rnnlib directory to your path (eg.
- You will need the python packages
- On a mac, use
brewto install the packages
ncview(more on netcdf here or here)
- It seems that the current
masterbranch is broken for recent netcfd/python2; use this patch to fix the NCF generation.
- The recipe will take a few hours to compute – it’s best to let it run over night!