Refine
Document Type
Language
- English (5)
Has Fulltext
- no (5)
Is part of the Bibliography
- no (5)
Keywords
- incrementality (2)
- dialogue systems (1)
- evaluation (1)
- naturallanguage understanding (1)
- speech recognition (1)
- spoken dialogue systems (1)
- spontaneous speech (1)
Institute
Ideally, a spoken dialogue system should react without much delay to a user’s utterance. Such a system would already select an object, for instance, before the user has finished her utterance about moving this particular object to a particular place. A prerequisite for such a prompt reaction is that semantic representations are built up on the fly and passed on to other modules. Few approaches to incremental semantics construction exist, and, to our knowledge, none of those has been systematically tested on a spontaneous speech corpus. In this paper, we develop measures to test empirically on transcribed spontaneous speech to what extent we can create semantic interpretation on the fly with an incremental semantic chunker that builds a frame semantics.
In incremental spoken dialogue systems, par- tial hypotheses about what was said are re- quired even while the utterance is still ongo- ing. We define measures for evaluating the quality of incremental ASR components with respect to the relative correctness of the par- tial hypotheses compared to hypotheses that can optimize over the complete input, the tim- ingof hypothesisformationrelative to the por- tion ofthe inputthey areabout, andhypothesis stability, defined as the number of times they are revised. We show that simple incremen- tal post-processing can improve stability dra- matically, at the cost of timeliness (from 90% of edits of hypotheses being spurious down to 10% at a lag of 320ms). The measures are not independent,and we show how system de- signers can find a desired operating point for their ASR. To our knowledge, we are the first to suggest and examine a variety of measures for assessing incremental ASR and improve performance on this basis.
The potential of using ASR n-best lists for dialogue systems has often been recognised (if less often realised): it is often the case that even when the top-ranked hypothesis is erroneous, a bet- ter one can be found at a lower rank. In this paper, we describe metrics for evaluating whether the same potential carries over to incremental dialogue systems, where ASR output is consumed and reacted upon while speech is still ongoing. We show that even small N can provide an advantage for semantic process- ing, at a cost of a computational overhead.
In this paper we do two things: a) we discuss in general terms the task of incre mental reference resolution (IRR), in particular resolution of exophoric reference, and specify metrics for measuring the performance of dialogue system components tackling this task, and b) we present a simple Bayesian filtering model of IRR that performs reasonably well just using words directly (no structure information and no hand-coded semantics): it picks the right referent out of 12 for around 50 % of real world dialogue utterances in our test corpus. It is also able to learn to interpret not only words but also hesitations, just as humans have shown to do in similar situations, namely as markers of references tohard-to-describe entities.
We define the task of incremental or 0-lag utterance segmentation, that is, the task of segmenting an ongoing speech recognition stream into utterance units, and present first results. We use a combination of hidden event language model, features from an incremental parser, and acoustic / prosodic features to train classifiers on real-world conversational data (from the Switchboard corpus). The best classifiers reach an F-score of around 56%, improving over baseline and related work.