Title: AUTOMATIC EVALUATION METHODS OF A SPEECH TRANSLATION SYSTEM'S CAPABILITY

Authors: Fumiaki Sugaya, Keiji Yasuda, Toshiyuki Takezawa, Seiichi Yamamoto

Abstract: The main goal of the present paper is to propose automatic schemes of the translation paired comparison method, which was proposed by the authors to precisely evaluate a speech translation system's capability. In this method, the outputs of the speech translation system are subjectively compared with results of native Japanese taking the Test of English for International Communication (TOEIC), which is used as a measure of one's speech translation capability. Experiments are conducted on TDMT, which is a language translation subsystem of the Japanese-to-English speech translation system ATRMATRIX developed at ATR Interpreting Telecommunications Research Laboratories. The winning rate of TDMT shows a good correlation with the TOEIC scores of the examinees. A regression analysis on the subjective results shows that the translation capability of TDMT matches a Japanese person scoring around 700 on the TOEIC. The automatic evaluation methods use DP-based similarity, which is calculated by DP distances between a translation output and multiple translation answers. The answers are collected by two methods: paraphrasing and query from a parallel corpus. In both types of collection, the similarity shows the same good correlation with the TOEIC scores of the examinees as the subjective winning rate. A regression analysis using the similarity shows that the system's matched point is around 750. In this paper, we also show effects of paraphrased data.

a01fs118.ps a01fs118.pdf