International Competition on Knowledge Engineering for Planning and Scheduling
Thessaloniki, Greece, September 20, 2009


Welcome | Organizers | Call | Criteria | Programme | Results | Repository

Evaluation criteria

The winner of the competition will be decided by a board of judges after a careful evaluation of both paper presentation and demonstration of the tool.

Given that the focus of this year competition is on translators and translation processes, competitors should try to synthesize in the presentation the added value of their problem translation in the following two directions:

  • what advantages does planning bring in solving problems in the application domain?
  • what is the added value for the planning community in having the chance of using P&S technologies in the application domain?

For the demonstration session, both Software Engineering (SE) issues and P&S issues will be taken in account. Examples of SE related criteria are

  • Portability, meant as a measurement of the "difficulty of using the tool out of the laptop of the competitor".
  • Robustness, meant as a measurement of how much the value quality of the translation is input-dependent.
  • Usability, meant as a measurement of how much is the tool usable either by AI experts or target domain experts
  • Spread of use of the translator. (How many people so far have used the tool? To do what? How much is the input language spread?)
  • Perceived added value to the application area. This is meant to be an evaluation of the impact of the tool.
  • Flexibility. How would be easy to use the tool for domains out of those foreseen by the authors? How demanding is to extend the set of problems that it is possible to translate?

Examples of more specific P&S criteria are:

  • Originality. Is the approach original (with respect to previous proposals by other authors)?
  • Comprehensiveness of the translation process. Is the sub-set of the input language/problems that can be translated well defined? Is the chosen subset enough for targeting significant problems in the application community?
  • The challenges involved in the translation process. The more different the input/output languages are, the more valuable should the tool be.
  • Connection with the P&S technology. Is the tool a comprehensive KE tool? Is there any available useful translation of output plan or schedule back into the application domain? Is there any feature available to interact with the planner/scheduler?
  • Ingenuity of the translator.
  • Availability of solvers to input the translated domain model and performances of the planner and/or schedule (when available) with the translated domains

The above criteria are meant only as general guidelines to drive the evaluation process. Given the high variability of competing tools, not all the above listed criteria will be necessarily taken into account for every tool, and further criteria might be considered whenever needed (in these cases the use or not use of different criteria will be explained).

Given the practical infeasibility of giving an objective measure of each criteria, in practice the judges will express their point of view about them. The final result will be in any case justified by the sum of the evaluations taken for each criteria.

Only the winner and winner's performances will be publicly discussed at the end of the competition (there will be no public ranking); however, the evaluations of the remaining participants will be available on the participant's request, and privately discussed.

The board of judges:

Simone Fratini - ISTC-CNR, Italy
Piergiorgio Bertoli - Fondazione Bruno Kessler, Italy
Adi Botea - NICTA and The Australian National University, Australia

In conjunction with the International Conference on Automated Planning and Scheduling (ICAPS-09)

Organizers

Roman Barták, Charles University, Czech Republic
contact email: bartak (AT) ktiml.mff.cuni.cz

Simone Fratini, ISTC-CNR, Italy
contact email: simone.fratini (AT) istc.cnr.it

Lee McCluskey, University of Huddersfield, UK
contact email: lee (AT) hud.ac.uk

Programme Committte

Sara Bernardini
London Knowledge Lab, UK
Amedeo Cesta
ISTC-CNR, Italy
Stephen Cresswell
University of Huddersfield, UK
Stefan Edelkamp
University of Bremen, Germany
Susana Fernandez Arregui
Universidad Carlos III de Madrid, Spain
Jeremy Frank
NASA, USA
Antonio Garrido
Universitat Politecnica de Valencia, Spain
Robert Goldman
SIFT, USA
Arturo Gonzalez Ferrer
Universidad de Granada, Spain
Peter Jarvis
NASA, USA
Ugur Kuter
University of Maryland, USA
Clayton T. Morrison
The University of Arizona, USA
Julie Porteous
University of Teeside, UK
Tiago S. Vaquero
University of Sao Paulo, Brazil
Dimitris Vrakas
Aristotle University of Thessaloniki, Greece

Board of judges

Simone Fratini
ISTC-CNR, Italy
Piergiorgio Bertoli
Fondazione Bruno Kessler, Italy
Adi Botea
NICTA/ANU, Australia

Dates:

Expression of interest: April 30, 2009
Submission deadline:
June 18, 2009
Notification of acceptance: July 13,2009
Camera-ready copy due: August 3, 2009
Workshop (full day): September 20, 2009