Layer-Normalized LSTM for Hybrid-Hmm and End-To-End ASR
| Autor: | Albert Zeyer, Ralf Schlüter, Mohammad Zeineldeen, Hermann Ney |
|---|---|
| Rok vydání: | 2020 |
| Předmět: |
Hyperparameter
Normalization (statistics) Computer science Speech recognition Word error rate 02 engineering and technology 010501 environmental sciences 01 natural sciences Recurrent neural network End-to-end principle 0202 electrical engineering electronic engineering information engineering 020201 artificial intelligence & image processing Hidden Markov model 0105 earth and related environmental sciences |
| Zdroj: | ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ICASSP |
| DOI: | 10.1109/icassp40776.2020.9053635 |
| Popis: | Training deep neural networks is often challenging in terms of training stability. It often requires careful hyperparameter tuning or a pretraining scheme to converge. Layer normalization (LN) has shown to be a crucial ingredient in training deep encoder-decoder models. We explore various LN long short-term memory (LSTM) recurrent neural networks (RNN) variants by applying LN to different parts of the internal recurrency of LSTMs. There is no previous work that investigates this. We carry out experiments on the Switchboard 300h task for both hybrid and end-to-end ASR models and we show that LN improves the final word error rate (WER), the stability during training, allows to train even deeper models, requires less hyperparameter tuning, and works well even without pre-training. We find that applying LN to both forward and recurrent inputs globally, which we denoted by Global Joined Norm variant, gives a 10% relative improvement in WER. |
| Databáze: | OpenAIRE |
| Externí odkaz: |
|