NIST 2006 Speaker Recognition Evaluation - DET Plot Definitions
Version v5
Date of release: Mon May 21 13:50:23 EDT 2007
A test condition is defined to be one of the following 15 tests:
- 10sec4w-10sec4w
- 1conv4w-10sec4w
- 1conv4w-1conv2w
- 1conv4w-1conv4w (This test condition is also known as the core test condition)
- 1conv4w-1convmic
- 3conv2w-1conv2w
- 3conv2w-1conv4w
- 3conv4w-10sec4w
- 3conv4w-1conv2w
- 3conv4w-1conv4w
- 3conv4w-1convmic
- 8conv4w-10sec4w
- 8conv4w-1conv2w
- 8conv4w-1conv4w
- 8conv4w-1convmic
English trials - data (test segments and model training) that are limited to English
Common test condition - test conditions involved only English trials
- Shows the overall performance for each of the 15 test conditions
- Target restrictions: none
- Non-target restrictions: none
- Each plot contains one DET curve showing the overall performance for a given test condition
- Shows the overall performance for a given test condition by gender
- Excludes summed channel test conditions
- Target restrictions: 1) male trials, 2) female trials
- Non-target restrictions: 1) male trials, 2) female trials
- Each plot contains two DET curves: 1) male - blue, 2) female - red
- Shows the performance for a given test condition when all data (test segments and model training) are limited to English
- Target restrictions: English trials
- Non-target restrictions: English trials
- Shows performance for a given test condition when all data (test segments and model training) are limited to English by gender
- Excludes summed channel test conditions
- Target restrictions: 1) English male trials, 2) English female trials
- Non-target restrictions: 1) English male trials, 2) English female trials
- Each plot contains two DET curves: 1) male - blue, 2) female - red
- Similar to Plot 3 but certain test conditions (see below) are grouped into one graph
- Compares performance on the English trials over the following conditions:
- (1) 8conv4w-1conv4w
- (2) 8conv4w-10sec4w
- (3) 3conv4w-1conv4w
- (4) 3conv4w-10sec4w
- (5) 1conv4w-1conv4w
- (6) 1conv4w-10sec4w
- (7) 10sec4w-10sec4w
- Each plot contains seven DET curves:
- (1) black thick solid
- (2) black thick dashed
- (3) red thick solid
- (4) red thick dashed
- (5) blue thick solid
- (6) blue thick dashed
- (7) green thick dashed
- Similar to Plot 3 but certain test conditions (see below) are grouped into one graph
- Compares performance on the English trials over the following conditions:
- (1) 3conv4w-1conv4w
- (2) 3conv4w-1conv2w
- (3) 3conv2w-1conv4w
- (4) 3conv2w-1conv2w
- (5) 1conv4w-1conv4w
- (6) 1conv4w-1conv2w
- Each plot contains six DET curves:
- (1) black thick solid
- (2) black thick dashed
- (3) red thick solid
- (4) red thick dashed
- (5) blue thick solid
- (6) blue thick dashed
- Shows performance for the core test condition (1conv4w-1conv4w) when
- Target restrictions:
- (1) trained on English and tested on English
- (2) trained on English and tested on non-English
- (3) trained on non-English and tested on English
- (4) trained on non-English and tested on non-English
- Non-target restrictions: none
- Each plot contains four DET curves:
- (1) English train, English test - black thick solid
- (2) English train, non-English test - black thick dashed
- (3) non-English train, English test - red thick dashed
- (4) non-English train, non-English test - red thick solid
- Shows performance for the core test condition (1conv4w-1conv4w) when
- Target restrictions: none
- Non-target restrictions:
- (1) trained on English and tested on English
- (2) trained on English and tested on non-English
- (3) trained on non-English and tested on English
- (4) trained on non-English and tested on non-English
- Each plot contains four DET curves:
- (1) English train, English test - black thick solid
- (2) English train, non-English test - black thick dashed
- (3) non-English train, English test - red thick dashed
- (4) non-English train, non-English test - red thick solid
- Shows performance for the core test (1conv4w-1conv4w) when
- Target restrictions:
- (1) trained on English and tested on English
- (2) trained on English and tested on non-English
- (3) trained on non-English and tested on English
- (4) trained on non-English and tested on non-English
- Non-target restrictions:
- (1) trained on English and tested on English
- (2) trained on English and tested on non-English
- (3) trained on non-English and tested on English
- (4) trained on non-English and tested on non-English
- Each plot contains four DET curves:
- (1) English train, English test - black thick solid
- (2) English train, non-English test - black thick dashed
- (3) non-English train, English test - red thick dashed
- (4) non-English train, non-English test - red thick solid
- Compare performance by microphone channel
- Target restrictions: English trials that have a telephone and microphone version for the 1conv4w training condition (1conv4w-1convmic/1conv4w-1conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Non-target restrictions: English trials that have a telephone and microphone version for the 1conv4w training condition (1conv4w-1convmic/1conv4w-1conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Each plot contains nine DET curves:
- (1) red thick dashed
- (2) blue thick dashed
- (3) black thick dashed
- (4) magenta thick solid
- (5) cyan thick solid
- (6) black thick solid
- (7) green thick solid
- (8) red thick solid
- (9) black thick solid
- Compare performance by microphone channel
- Target restrictions: English trials that have a telephone and microphone version for the 3conv4w training condition (3conv4w-3convmic/3conv4w-3conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Non-target restrictions: English trials that have a telephone and microphone version for the 3conv4w training condition (3conv4w-3convmic/3conv4w-3conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Each plot contains nine DET curves:
- (1) red thick dashed
- (2) blue thick dashed
- (3) black thick dashed
- (4) magenta thick solid
- (5) cyan thick solid
- (6) black thick solid
- (7) green thick solid
- (8) red thick solid
- (9) black thick solid
- Compare performance by microphone channel
- Target restrictions: English trials that have a telephone and microphone version for the 8conv4w training condition (8conv4w-8convmic/8conv4w-8conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Non-target restrictions: English trials that have a telephone and microphone version for the 8conv4w training condition (8conv4w-8convmic/8conv4w-8conv4w) and are limited to:
- (1) microphone channel 1
- (2) microphone channel 2
- (3) microphone channel 3
- (4) microphone channel 4
- (5) microphone channel 5
- (6) microphone channel 6
- (7) microphone channel 7
- (8) microphone channel 8
- (9) telephone channel
- Each plot contains nine DET curves:
- (1) red thick dashed
- (2) blue thick dashed
- (3) black thick dashed
- (4) magenta thick solid
- (5) cyan thick solid
- (6) black thick solid
- (7) green thick solid
- (8) red thick solid
- (9) black thick solid
- Compare performance by transmission type
- Target restrictions:
- (1) English trials, trained on cellular, tested on cellular
- (2) English trials, trained on cellular, tested on cordless
- (3) English trials, trained on cellular, tested on landline
- (4) English trials, trained on cordless, tested on cellular
- (5) English trials, trained on cordless, tested on cordless
- (6) English trials, trained on cordless, tested on landline
- (7) English trials, trained on landline, tested on cellular
- (8) English trials, trained on landline, tested on cordless
- (9) English trials, trained on landline, tested on landline
- Non-target restrictions:
- (1) English trials, trained on cellular, tested on cellular
- (2) English trials, trained on cellular, tested on cordless
- (3) English trials, trained on cellular, tested on landline
- (4) English trials, trained on cordless, tested on cellular
- (5) English trials, trained on cordless, tested on cordless
- (6) English trials, trained on cordless, tested on landline
- (7) English trials, trained on landline, tested on cellular
- (8) English trials, trained on landline, tested on cordless
- (9) English trials, trained on landline, tested on landline
- Each plot contains nine det curves:
- (1) black thick solid
- (2) black thick dashed
- (3) black thin solid
- (4) red thick solid
- (5) red thick dashed
- (6) red thin solid
- (7) blue thick solid
- (8) blue thick dashed
- (9) blue thin solid
- Similar to DET 13 except that we combined landline and cordless as one restriction for the targets and the non-targets have no restrictions
- Target restrictions:
- (1) English trials, trained on landline/cordless, tested on landline/cordless
- (2) English trials, trained on landline/cordless, tested on cellular
- (3) English trials, trained on cellular, tested on cellular
- (4) English trials, trained on cellular, tested on landline/cordless
- Non-target restrictions: none
- Each plot contains four det curves:
- (1) red thick solid
- (2) red thick dashed
- (3) black thick solid
- (4) black thick dashed
- Compare performance by microphone type
- Target restrictions:
- (1) English trials, trained on ear-bud, tested on ear-bud
- (2) English trials, trained on ear-bud, tested on hand-held
- (3) English trials, trained on ear-bud, tested on head-mounted
- (4) English trials, trained on ear-bud, tested on speaker-phone
- (5) English trials, trained on hand-held, tested on ear-bud
- (6) English trials, trained on hand-held, tested on hand-held
- (7) English trials, trained on hand-held, tested on head-mounted
- (8) English trials, trained on hand-held, tested on speaker-phone
- (9) English trials, trained on head-mounted, tested on ear-bud
- (10) English trials, trained on head-mounted, tested on hand-held
- (11) English trials, trained on head-mounted, tested on head-mounted
- (12) English trials, trained on head-mounted, tested on speaker-phone
- (13) English trials, trained on speaker-phone, tested on ear-bud
- (14) English trials, trained on speaker-phone, tested on hand-held
- (15) English trials, trained on speaker-phone, tested on head-mounted
- (16) English trials, trained on speaker-phone, tested on speaker-phone
- Each plot contains 16 det curves:
- (1) black thick solid
- (2) black thick dashed
- (3) black thin solid
- (4) black thin dashed
- (5) blue thick solid
- (6) blue thick dashed
- (7) blue thin solid
- (8) blue thin dashed
- (9) red thick solid
- (10) red thick dashed
- (11) red thin solid
- (12) red thin dashed
- (13) cyan thick solid
- (14) cyan thick dashed
- (15) cyan thin solid
- (16) cyan thin dashed
- Target restrictions:
- (1) English trials, handsets used on test and train data are same
- (2) English trials, handsets on train and test data are different
- Non-target restrictions: none
- Each plot contains 2 det curves:
- (1) black thick solid
- (2) red thick solid
- Compare performance for the core test by time between train and test for the target English trials. All English language non-target trials are used in each DET. The DETs plotted limit target trials to English with:
- (1) time between trained and tested <= 5 days
- (2) time between trained and tested > 5 days
- Each plot contains 2 det curves:
- (1) black thick solid
- (2) blue thick solid
- Show performance of mothballed system for the core test condition
- Each plot contains 3 det curves:
- (1) black thick solid - 2006 primary system
- (2) blue thick solid - 2005 primary system on 2006 data
- (3) blue thick dashed - 2005 primary system on 2005 data
- Review the performance of best systems over 2004-2006 for the core test condition
- Each plot contains 3 det curves:
- (1) 2006 best system - black thick solid
- (2) 2005 best system - blue thick solid
- (3) 2004 best system - red thick solid
- Review the performance of best systems over 2004-2006 for 8conv4w, 3conv4w, and 3conv2w training conditions
- Each plot contains 3 det curves:
- (1) 2006 best system - black thick solid
- (2) 2005 best system - blue thick solid
- (3) 2004 best system - red thick solid
- Compare performance of cross channel data by collection location (ICSI, LDC)
- Target restrictions: 1) ICSI, 2) LDC
- Non-target restrictions: 1) ICSI, 2) LDC
- Each plot contains two DET curves: 1) ICSI - red, 2) LDC - blue
- Compare performance of cross channel data over 2005-2006
- Each plot contains two DET curves: 1) 2005 - red dashed, 2) 2006 - black solid