Refine
Document Type
Language
- English (6)
Has Fulltext
- no (6)
Is part of the Bibliography
- no (6)
Keywords
- dialogue systems (1)
- evaluation (1)
- incremental processing (1)
- incrementality (1)
- incrementality, (1)
- naturallanguage understanding (1)
- speech recognition (1)
- spoken dialogue systems (1)
- timing (1)
Institute
Begutachtungsstatus
- peer-reviewed (2)
Incremental spoken dialogue systems, which process user input as it unfolds, pose additionalengineering challenges compared to more standard non-incremental systems: Their processingcomponents must be able to accept partial, and possibly subsequently revised input, and mustproduce output that is at the same time as accurate as possible and delivered with as little delay aspossible. In this article, we define metrics that measure how well a given processor meets thesechallenges, and we identify types of gold standards for evaluation. We exemplify these metrics inthe evaluation of several incremental processors that we have developed. We also present genericmeans to optimise some of the measures, if certain trade-offs are accepted. We believe that thiswork will help enable principled comparison of components for incremental dialogue systems andportability of results.
We present INPROTK, a toolkit for building incremental spoken dia-logue systems. Incremental spoken dialogue systems (systems that may react whilethe user’s utterance is ongoing) are a fairly recent research topic and allow for ex-citing new features. Even though toolkits exist that help in building conventionaldialogue systems, INPROTK offers both a tested architecture for building incre-mental SDSs as well as many of the building blocks necessary when building suchsystems. With INPROTK a researcher can avoid many of the technical difficulties,which hopefully further fosters research in this area.
The potential of using ASR n-best lists for dialogue systems has often been recognised (if less often realised): it is often the case that even when the top-ranked hypothesis is erroneous, a bet- ter one can be found at a lower rank. In this paper, we describe metrics for evaluating whether the same potential carries over to incremental dialogue systems, where ASR output is consumed and reacted upon while speech is still ongoing. We show that even small N can provide an advantage for semantic process- ing, at a cost of a computational overhead.
We present a model of semantic processing of spoken language that (a) is robust against ill-formed input, such as can be expected from automatic speech recognisers, (b) respects both syntactic and pragmatic constraints in the computation of most likely interpretations, (c) uses a principled, expressive semantic representation formalism (RMRS) with a well-defined model theory, and (d) works continuously (producing meaning representations on a word-by-word basis, rather than only for full utterances) and incrementally (computing only the additional contribution by the new word, rather than re-computing for the whole utterance-so-far). We show that the joint satisfaction of syntactic and pragmatic constraints improves the performance of the NLU component (around 10 % absolute, over a syntax-only baseline).
We describe work done at three sites on designing conversational agents capable of incremental processing. We focus on the middleware layer in these systems, which takes care of passing around and maintaining incremental information between the modules of such agents. All implementations are based on the abstract model of incremental dialogue processing proposed by Schlangen and Skantze (2009), and the paper shows what different instantiations of the model can look like given specific requirements and application areas.
When dialogue systems, through theuse of incremental processing, arenot bounded anymore by strict, non-overlapping turn-taking, a whole range ofadditional interactional devices becomesavailable. We explore the use of one suchdevice, trial intonation. We elaborateour approach to dialogue managementin incremental systems, based on theInformation-State-Update approach, anddiscuss an implementation in a micro-domain that lends itself to the use ofimmediate feedback, trial intonations andexpansions. In an overhearer evaluation,the incremental system was judged as sig-nificantly more human-like and reactivethan a non-incremental version.