Evaluation criteria

The winner of the competition will be decided by a board of judges after a careful evaluation of both paper presentation and demonstration of the tool.

Given that the focus of this year competition is on translators and translation processes, competitors should try to synthesize in the presentation the added value of their problem translation in the following two directions:

• what advantages does planning bring in solving problems in the application domain?
• what is the added value for the planning community in having the chance of using P&S technologies in the application domain?

For the demonstration session, both Software Engineering (SE) issues and P&S issues will be taken in account. Examples of SE related criteria are

• Portability, meant as a measurement of the "difficulty of using the tool out of the laptop of the competitor".
• Robustness, meant as a measurement of how much the value quality of the translation is input-dependent.
• Usability, meant as a measurement of how much is the tool usable either by AI experts or target domain experts
• Spread of use of the translator. (How many people so far have used the tool? To do what? How much is the input language spread?)
• Perceived added value to the application area. This is meant to be an evaluation of the impact of the tool.
• Flexibility. How would be easy to use the tool for domains out of those foreseen by the authors? How demanding is to extend the set of problems that it is possible to translate?

Examples of more specific P&S criteria are:

• Originality. Is the approach original (with respect to previous proposals by other authors)?
• Comprehensiveness of the translation process. Is the sub-set of the input language/problems that can be translated well defined? Is the chosen subset enough for targeting significant problems in the application community?
• The challenges involved in the translation process. The more different the input/output languages are, the more valuable should the tool be.
• Connection with the P&S technology. Is the tool a comprehensive KE tool? Is there any available useful translation of output plan or schedule back into the application domain? Is there any feature available to interact with the planner/scheduler?
• Ingenuity of the translator.
• Availability of solvers to input the translated domain model and performances of the planner and/or schedule (when available) with the translated domains

The above criteria are meant only as general guidelines to drive the evaluation process. Given the high variability of competing tools, not all the above listed criteria will be necessarily taken into account for every tool, and further criteria might be considered whenever needed (in these cases the use or not use of different criteria will be explained).

Given the practical infeasibility of giving an objective measure of each criteria, in practice the judges will express their point of view about them. The final result will be in any case justified by the sum of the evaluations taken for each criteria.

Only the winner and winner's performances will be publicly discussed at the end of the competition (there will be no public ranking); however, the evaluations of the remaining participants will be available on the participant's request, and privately discussed.

The board of judges:

Simone Fratini - ISTC-CNR, Italy
Piergiorgio Bertoli - Fondazione Bruno Kessler, Italy
Adi Botea - NICTA and The Australian National University, Australia

In conjunction with the International Conference on Automated Planning and Scheduling (ICAPS-09)