Performance Analysis of the 2017 NIST Language Recognition Evaluation
Seyed Omid Sadjadi, Timothee Kheyrkhah, Craig Greenberg, Elliot Singer, Douglas Reynolds, Lisa Mason and Jaime Hernandez-Cordero
Abstract:
The 2017 NIST Language Recognition Evaluation (LRE) was held in the autumn of 2017. Similar to past LREs, the basic task in LRE17 was language detection, with an emphasis on discriminating closely related languages (14 in total) selected from 5 language clusters. LRE17 featured several new aspects including: audio data extracted from online videos; a development set for system training and development use; log-likelihood system output submissions; a normalized cross-entropy performance measure as an alternative metric; and, the release of a baseline system developed using the NIST Speaker and Language Recognition Evaluation (SLRE) toolkit for participant use. A total of 18 teams from 25 academic and industrial organizations participated in the evaluation and submitted 79 valid systems under fixed and open training conditions first introduced in LRE15. In this paper, we report an in-depth analysis of system performance broken down by multiple factors such as data source and gender, as well as a cross-year performance comparison of leading systems from LRE15 and LRE17 to measure progress over the 2-year period. In addition, we present a comparison of primary versus "single best" submissions to understand the effect of fusion on overall performance.
Cite as: Sadjadi, S.O., Kheyrkhah, T., Greenberg, C., Singer, E., Reynolds, D., Mason, L., Hernandez-Cordero, J. (2018) Performance Analysis of the 2017 NIST Language Recognition Evaluation. Proc. Interspeech 2018, 1798-1802, DOI: 10.21437/Interspeech.2018-69.
BiBTeX Entry:
@inproceedings{Sadjadi2018,
author={Seyed Omid Sadjadi and Timothee Kheyrkhah and Craig Greenberg and Elliot Singer and Douglas Reynolds and Lisa Mason and Jaime Hernandez-Cordero},
title={Performance Analysis of the 2017 NIST Language Recognition Evaluation},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={1798--1802},
doi={10.21437/Interspeech.2018-69},
url={http://dx.doi.org/10.21437/Interspeech.2018-69} }