Evaluation of a Conversation Management Toolkit for Multi Agent Programming
David Lillis, Howell R. Jordan and Rem W. Collier
In Proceedings of the 10th International Workshop on Programming Multi-Agent Systems (ProMAS 2012), Valencia, Spain, 2012.
Abstract
The Agent Conversation Reasoning Engine (ACRE) is in- tended to aid agent developers with the management of conversations to improve the management and reliability of agent communication. To evaluate its effectiveness, a problem was presented to two groups of undergraduate students, one of which was required to create a solution using the features of ACRE and one without. This paper describes the requirements that the evaluation scenario was intended to meet and how these motivated the design of the problem that was presented to the subjects. The solutions were analysed using a combination of simple objective metrics and subjective analysis, which indicated a number of benefits of using ACRE. In particular, sub- jective analysis suggested that ACRE by defaults prevents some common problems arising that would limit the reliability and extensibility of conversation-handling code.