Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks

Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi and Sanjeev Khudanpur

Abstract:

Time Delay Neural Networks (TDNNs), also known as one-dimensional Convolutional Neural Networks (1-d CNNs), are an efficient and well-performing neural network architecture for speech recognition. We introduce a factored form of TDNNs (TDNN-F) which is structurally the same as a TDNN whose layers have been compressed via SVD, but is trained from a random start with one of the two factors of each matrix constrained to be semi-orthogonal. This gives substantial improvements over TDNNs and performs about as well as TDNN-LSTM hybrids.

---

Cite as: Povey, D., Cheng, G., Wang, Y., Li, K., Xu, H., Yarmohammadi, M., Khudanpur, S. (2018) Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks. Proc. Interspeech 2018, 3743-3747, DOI: 10.21437/Interspeech.2018-1417.

---

BiBTeX Entry:

@inproceedings{Povey2018,
author={Daniel Povey and Gaofeng Cheng and Yiming Wang and Ke Li and Hainan Xu and Mahsa Yarmohammadi and Sanjeev Khudanpur},
title={Semi-Orthogonal Low-Rank Matrix Factorization for Deep Neural Networks},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={3743--3747},
doi={10.21437/Interspeech.2018-1417},
url={http://dx.doi.org/10.21437/Interspeech.2018-1417} }