Uni-Logo

IPC 2020

The 2020 IPC for Hierarchical Planning

ICAPS 2020 has hosted the 2020 IPC for Hierarchical Planning - as proposed in the following paper. It was the first IPC solely dedicated to Hierarchical Planning and HTN Planning in particular.

The 2020 IPC for Hierarchical Planning featured two tracks:

  • total-order, recursive
  • partial-order, recursive
  • total-order, non-recursive -- cancelled due to low interest

All tracks share common restrictions on the input:

  • All preconditions and effects contain only literals, negated literals, conjunctions, and universal quantifiers (forall),
  • all actions have unit-cost, and
  • methods may contain preconditions (but no other state constraints).

Input Language

The competition used domains and problems formulated in HDDL. Its syntax and semantics are explained in a recent paper. In addition we have made some minor changes and guarantees for the IPC. You can find details in the following document. For convenience, we provide an automated translation of HDDL into (J)SHOP2 syntax. Note that there are domains which can be expressed in HDDL, but for which there is no equivalent SHOP2 model. The issue lies in the restrictions SHOP2 puts on the allowed ordering. If you want to participate and need any other input format, please contact us as soon as possible s.t. we can assist you as best as possible.

A set of sample problems is available at the status website of the IPC.

  • totally-ordered as HDDL
  • partially-ordered as HDDL
  • totally-ordered as SHOP2
  • partially-ordered as SHOP2

The rover domain is non-recursive, while transport is recursive. If you have any issues with or questions about these sample, please contact us!

The SHOP2-variants of the instances were generated automatically. The code for this translation is part of the pandaPI planning system, more specifically its parser. Its code is available on github.

Output Format

The output format supported by our plan validator is descried in this pdf. The validator's code is available on github.

Mauricio Cecilio Magnaguagno kindly created a tool for visualising the output format.

Evaluation and Technical Specification

To evaluate planners, we used the agile metric used in the last IPC, i.e. a planner is regarded better wrt that metric if it finds any solution to the problem faster. The score of a planner on a solved task is 1 if the task was solved within 1 second and 0 if the task was not solved within the resource limits. If the task was solved in t seconds (1 \leq t \leq T) with a time limit of T seconds then its score is min{1, 1 - log(t)/log(T)}. The score of a planner is the sum of its scores for all tasks.

All tracks share the following techincal specifications:

  • 1 CPU core
  • 8 GB memory limit
  • 30 min time limit
  • If an invalid plan is returned, all tasks in the domain are counted as unsolved.
  • If that happens in more than one domain, the entry is disqualified.

For the specification of the tracks we mean with

  • total-order: A domain is totally ordered iff the subtasks in all methods and in the initial task network form a sequence, i.e. the declared ordering arranges the tasks in a sequence.
  • partial-order: A domain is totally ordered iff the subtasks in all methods and in the initial task network may have any order that can be specified in HDDL.
  • non-recursive: A domain is non-recursive if there is a total order on its lifted tasks (primitive and abstract) such that for every method that decomposes an abstract task A, the tasks in the methods' task network occur after A in the total order.
  • recursive: A domain that is not non-recursive
  • STRIPS actions: We consider an action to be STRIPS, if both its preconditions and effects are lists of literals. I.e. preconditions and effects do not contain quantifiers, disjunctions, or conditional effects. However negative preconditions are allowed.
  • Method precondition: A list of literals that may be associated with a method. A method may only be applied if its method precondition holds. For totally-ordered HTN planning problems, the precondition has to hold in the state in which the first action (that originates via further decompositions from the method's subtasks) is executed. For partially-ordered HTN planning problems, the precondition has to hold in some state before the first action (that originates via further decompositions from the methods' subtasks) is executed, but after the state resulting from the last necessarily preceding tasks for the tasks contained in the method.

Planner Submission

As in other IPC parts, the competitors must submit the source code of their planners that will be run by the organizers on the actual competition domains/problems, unknown to the competitors until this time. This way no fine-tuning of the planners will be possible.

Bug Fix Policy

In some cases, we will offer the opportunity to fix bugs that arise during the evaluation period, but any changes after the submission deadline will be strictly limited to bugfixes only. We will use a diff tool to check that patches don't contain new features or parameter tuning, and will reject patches that don't look like minimal changes to fix bugs. It is your responsibility to provide patches that are easy to verify with a diff tool. We reserve the right to reject changes for which the only-bugfixes rule is unnecessarily hard to check (e.g. because you reformatted the whole code).

IPC Proceedings

All competitors must submit a 2-page short paper describing their planners. After the competition we encourage the participants to analyze the results of their planner and submit an extended version of their paper (max. 4 pages). An important requirement for IPC 2020 competitors is to give the organizers the right to post their paper and the source code of their planners on the IPC 2020 website.

Results

The results of the IPC are now available. You can download the presentation slides here. The video presentation of the results can be watched here. In the slides shown at ICAPS 2020, the partial order track did not feature the domain Barman-BDI. We have thus fixed the slides. We have further also added coverage tables to the slides for informational purposes. You can download the fixed presentation slides here.

Summary

Partial Order Track

  • Winner: SIADEX (Juan Fernández Olivares, Ignacio Vellido Expósito)
  • Runner-Up: pyHiPOP (Charles Lesire, Alexandre Albore)
Further participant:
  • PDDL4J-PO

Total Order Track

  • Winner: HyperTensioN (Mauricio Cecilio Magnaguagno, Felipe Meneguzzi, Lavindra de Silva)
  • Runner-Up: LiloTane (Dominik Schreiber)
Further participants (sorted alphabetically)
  • PDDL4J-PO
  • PDDL4J-TO
  • pyHiPOP
  • SIADEX

Executable Planners

The following table describes the planners that participated in the IPC.
Nr.Participant Git Repo Singularity
1LiloTane ANU Gitlab container
2SIADEX ANU Gitlab container
4HyperTensioN ANU Gitlab container
5PDDL4J-TO ANU Gitlab container
6PDDL4J-PO ANU Gitlab container
7pyHiPOP ANU Gitlab container
You can build each planner by running singularity build ../image.sif Singularity in the cloned folder. In order to run a planner that is packaged as a singularity container, you need to create a folder in which the planner is executed. This can be your current folder, but be aware that the planner may write files into this folder. If you run the containers on a cluster, it is advisable to create one such folder per execution of the planner - else the written files may interfere. Assure that this folder has the absolute path PATH. You can then run the planner with singularity run -H PATH -C image.sif domain.hddl problem.hddl planner.out TIME MEM SEED, where TIME is the time limit in seconds, MEM is the memory limit in MB, and SEED is a seed. Note that the planners may exceed time and memory limits as these are only informational to the planner. If you need to enforce limits, you need to do this yourself.

Domains

The domains and instances used in the IPC are available at github. We have received further domain submissions which have not been used for various reasons in the IPC. These domains are also available in a separate git on github.

Datasets

For both tracks of the IPC, you can download the full and a reduced dataset. Note that all experiments were executed 10 times with a different seed. Per run, we defined a base seed. For every run, the actually used seed was the base seed plus the instance number. The full dataset contains the log files for all planners on every instance. This is a lot of data (partial-order track: 148MB, total-order track: 6,7GB). Every log contains the full output of the planner and the output of the subsequent verification of the found plan. In some cases, the verification took longer than the maximum time-limit of the job on our compute cluster (primaliry very long plans in the Towers domain). All such plans were valid. The planners are identified by a number which is shown in the table of planner above. The file names of the full logs comprise several information. The most notable one is the last number before the long sequence of dashs: it is the instance number (see below). The full data sets can be found here: In addition, we offer a reduced dataset. This dataset contains one line per run of a planner in which it found a solution (not necessarily a correct one). Each line contains 6 entries:
  1. A string describing the planner. The number contained in this string corresponds to the number of the planner.
  2. The instance number (see below).
  3. The domain number. This number has no deeper meaning, it is just a unique number given to all instances of one domain.
  4. Runtime in seconds. This already includes the reduction of 0.12 seconds given to all planners due to singularity startup time.
  5. Peak memory usage in kilobytes. This already includes the reduction by 20000 kb given to all planners due to the memory used by singularity.
  6. Length of the found plans in number of actions.
The reduced data sets can be found here: In the logs and in the reduced dataset, every instances is referred to with a numerical code. For technical reasons, the instance numbers are separate between the partial-order and the total-order tracks and partially overlap! After the experiments were run, the structure of the domain repository was changed to allow for the usage of LAB. As such, the file names of the instances in the log files may not be correct any more. The following two files contain a mapping of instance numbers to the instance files. The paths refer to the github repository.

Plans

In addition to the logs, we also offer the plans that were generated by the planners in the IPC as a resource. This can e.g. be used for research into plan verification. We are still in the process of gathering these plans and translating them into a uniform format (casing, handling of special characters). We will also include plans generated by other planners on the competition domains. So far, we have gathered all plans that were generated by the IPC planners during the evaluation runs and plans generated by the planner PANDA. The current version of the plans is available on github. All duplicate plans have already been removed. We plan to add further plans to the repository in the future.

Timetable

Call for interest September 30th
Announcement of Tracks October 18th
Demo domains and problems are available (including a (J)SHOP2 translator) October 22nd
Output-format specification end of January
Initial submission of planners June 9th
Deadline for domain submissions June 9th
Final submission of planners August 9th
Submission of 2-page planner abstracts August 30th
Presentation of Results Community Meeting, ICAPS 2020,
October 29th, 8PM CEST
Submission of 4-page planner abstracts January, 31st, 2021
Submission of domain description papers January, 31st, 2021

Organisers

Note that we also have a mailing list for everyone who is interested in hierarchical planning (including non-standard HTN planning), so this list is independent of this IPC endeavor. If you want to join this moderated, SPAM-free list, please write an email to Pascal Bercher.

Note that ICAPS regularly hosts HPlan, the workshop on Hierarchical Planning: hplan2020.hierarchical-task.net (you can replace the year accordingly, the earliest year was 2018).