User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning

Máté Ákos Tündik, György Szaszák, Gábor Gosztolya and András Beke

Abstract:

Punctuation of ASR-produced transcripts has received increasing attention in the recent years; RNN-based sequence modelling solutions which exploit textual and/or acoustic features show encouraging performance. Switching the focus from the technical side, qualifying and quantifying the benefits of such punctuation from end-user perspective have not been performed yet exhaustively. The ambition of the current paper is to explore to what extent automatic punctuation can improve human readability and understandability. The paper presents a user-centric evaluation of a real-time closed captioning system enhanced by a lightweight RNN-based punctuation module. Subjective tests involve both normal hearing and deaf or hard-of-hearing (DHH) subjects. Results confirm that automatic punctuation itself significantly increases understandability, even if several other factors interplay in subjective impression. The perceived improvement is even more pronounced in the DHH group. A statistical analysis is carried out to identify objectively measurable factors which are well reflected by subjective scores.

---

Cite as: Tündik, M.Á., Szaszák, G., Gosztolya, G., Beke, A. (2018) User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning. Proc. Interspeech 2018, 2628-2632, DOI: 10.21437/Interspeech.2018-1352.

---

BiBTeX Entry:

@inproceedings{Tündik2018,
author={Máté Ákos Tündik and György Szaszák and Gábor Gosztolya and András Beke},
title={User-centric Evaluation of Automatic Punctuation in ASR Closed Captioning},
year=2018,
booktitle={Proc. Interspeech 2018},
pages={2628--2632},
doi={10.21437/Interspeech.2018-1352},
url={http://dx.doi.org/10.21437/Interspeech.2018-1352} }