A black-box approach for response quality evaluation of conversational agent systems

Goh, Ong Sing and Ardil, C. and Wong, W. and Fung, C.C. (2007) A black-box approach for response quality evaluation of conversational agent systems. International Journal of Computational Intelligence, 3 (3). pp. 195-203. ISSN 1304-2386

Preview

PDF A_Black-box_Approach_for_Response_Quality_evaluation_of_the_conversation_agent_system.pdf - Published Version Download (604kB)

Abstract

The evaluation of conversational agents or chatterbots question answering systems is a major research area that needs much attention. Before the rise of domain-oriented conversational agents based on natural language understanding and reasoning, evaluation is never a problem as information retrieval-based metrics are readily available for use. However, when chatterbots began to become more domain specific, evaluation becomes a real issue. This is especially true when understanding and reasoning is required to cater for a wider variety of questions and at the same time to achieve high quality responses. This paper discusses the inappropriateness of the existing measures for response quality evaluation and the call for new standard measures and related considerations are brought forward. As a short-term solution for evaluating response quality of conversational agents, and to demonstrate the challenges in evaluating systems of different nature, this research proposes a blackbox approach using observation, classification scheme and a scoring mechanism to assess and rank three example systems, AnswerBus,START and AINI.

Item Type:	Article
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions:	Faculty of Information and Communication Technology > Department of Industrial Computing
Depositing User:	Assoc. Prof. Dr. Ong Sing Goh
Date Deposited:	11 Nov 2014 14:25
Last Modified:	28 May 2015 04:24
URI:	http://eprints.utem.edu.my/id/eprint/12389
Statistic Details:	View Download Statistic

Actions (login required)

View Item