Literature

We’ll cover three of the seminal papers in sequence to sequence learning:

• Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks by Alex Graves et al. (https://dl.acm.org/citation.cfm?id=1143891, author’s copy); check out this very helpful visualization
• Sequence to Sequence Learning for Neural Networls by Sutskever et al. (https://arxiv.org/abs/1409.3215)
• Neural Machine Translation by Jointly Learning to Align and Translate by Dzmitry Bahdanau et al. (https://arxiv.org/abs/1409.0473).
• Listen, Attend and Spell by William Chan et al. (https://arxiv.org/abs/1508.01211)
• Attention is all you need (https://arxiv.org/abs/1706.03762)

As well as Attention and Augmented Recurrent Neural Networks by Chris Olah and Shan Carter.

Exercise

There is way too much to cover in one exercise. Here are a few options:

• Transformer Implementation in TensorFlow for Machine Translation. Recommended exercise, using theses titles 2014-2018 (see Moodle)
• Extensive TensorFlow tutorial on machine translation using attention (Colab).
• Synthesis of handwriting using rnnlib and the Uni Bern IAM On-Line Handwriting Database; see remarks below.

Remarks

• You can either register for the dataset, or obtain a copy during class.
• If you run into trouble building openblas (as a submodule of rnnlib), you may need to specify the target CPU architecture (eg. Macbook Pro 2017: HASWELL, see example diff file).
• It’s recommended to add the rnnlib directory to your path (eg. export PATH=$(pwd):$PATH).
• You will need the python packages ScientificPython and scipy
• On a mac, use brew to install the packages netcdf, nco and ncview (more on netcdf here or here)
• It seems that the current master branch is broken for recent netcfd/python2; use this patch to fix the NCF generation.
• The recipe will take a few hours to compute – it’s best to let it run over night!