i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models

Karel Beneš, Santosh Kesiraju and Lukáš Burget

Abstract:

We show an effective way of adding context information to shallow neural language models. We propose to use Subspace Multinomial Model (SMM) for context modeling and we add the extracted i-vectors in a computationally efficient way. By adding this information, we shrink the gap between shallow feed-forward network and an LSTM from 65 to 31 points of perplexity on the Wikitext-2 corpus (in the case of neural 5-gram model). Furthermore, we show that SMM i-vectors are suitable for domain adaptation and a very small amount of adaptation data (e.g. endmost 5% of a Wikipedia article) brings a substantial improvement. Our proposed changes are compatible with most optimization techniques used for shallow feedforward LMs.

---

Cite as: Beneš, K., Kesiraju, S., Burget, L. (2018) i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models. Proc. Interspeech 2018, 3383-3387, DOI: 10.21437/Interspeech.2018-1070.

---

BiBTeX Entry:

@inproceedings{Beneš2018,
author={Karel Beneš and Santosh Kesiraju and Lukáš Burget},
title={i-Vectors in Language Modeling: An Efficient Way of Domain Adaptation for Feed-Forward Models},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={3383--3387},
doi={10.21437/Interspeech.2018-1070},
url={http://dx.doi.org/10.21437/Interspeech.2018-1070} }