Program Committee Roles Description and Guidelines

Table of contents

  1. Topic Chair
  2. Area Chair
  3. Reviewer

Topic Chairs (TC)

ICAPS 2025 does not have “tracks” but “topics”, following up the changes made in ICAPS 2024 when all submissions were directed to a single track, and the notion of “primary keyword” to classify papers was put forward. Topics are a way to cater to the research interests of the planning community and a way to give visibility to closely related research questions which can be both well-established or new and emerging. While some of the topics have been chosen by the ICAPS 2025 PC team, we expect the community to put forward topics. For more details, please see an overview of our paper classification system.

The main roles of Topic Chairs (TCs) are:

  • Before the submission deadline, to nominate and invite candidates for Area Chair (AC) and Reviewer roles, who the TC considers to be experts in the conference topic they are overseeing.
  • During the assignment phase, we expect TCs to help the Program Chairs resolve the status of papers flagged as eligible for desk rejection and decide whether they can continue into Round 1. The conditions for a paper to be considered for desk rejection can be found here
  • During the rest of the conference workflow, we expect TCs to oversee the activities of the ACs and assist them to reach a recommendation, moderate discussions and reviews, and coach them to resolve any issues.

We expect TCs to be assigned to about 5-20 submissions each, depending on actual submission intake.

Area Chair (AC)

The role of “Area Chair” is roughly equivalent to that of “Meta-Reviewer” and resp. “SPC” in previous editions of AAAI and IJCAI. We use the nomenclature put forward by OpenReview out of expediency. ICAPS 2025 ACs though, are expected to have a more involved role than meta-reviewers and SPCs in other conferences. We want to experiment with the notion that a small conference like ICAPS can be run in a similar way as a quick-turnaround journal like IEEE Robotics and Automation Letters, and ACs take the role of junior editors, actively engaging with the reviewers and being required to provide substantive meta-reviews. We think that empowering and increasing the responsibility of this figure makes the proposition more interesting and attractive.

Role Description

Area Chairs (ACs) are expected to perform the following functions during the reviewing and discussion phases of the conference:

  • Read the conditions for acceptance, and contact the ICAPS-25 organising committee to resolve any doubts or questions.
  • Place bids on papers that the AC is interested in and fall within their field of expertise. Avoid any obvious conflicts of interest.
  • Lead the process of desk rejection.
  • Report if they have a Conflict of Interest (see below) with one or more papers in the assignment. This may affect yourself or the Reviewers assigned to the paper. ACs will know the identity of the Reviewers
  • The AC should read the paper carefully, taking notes where appropriate. These notes can evolve into the AC own review, if the AC is so inclined.
  • Engage with the Reviewers during the review and discussion period to resolve relevant issues (see below).
  • Write a meta-review summarising the views of the Program Chairs assigned to the paper, and provide a recommendation justified on the basis of the content of the reviews, the discussion, and their own views on the paper.

Guidelines (Phase 1 and Phase 2)

Our guidelines for ICAPS-25 follow from the excellent advice by Dr. Pin Sym Foong at the National University Singapore:

  1. Read the assigned paper at least twice. Take notes and organise them into a review. See the guidelines provided to the Reviewers.

  2. When the reviews are complete for a paper, do the meta-review. Decide whether to make your review available to the Reviewers.

  3. Read all the reviews.

  4. Keep your own notes or review (Step 2) in front of you.

  5. In a separate document, copy and paste each point that each reviewer made for or against the paper, making sure to note which reviewer it came from. For instance, R2: “this paper needs more data”.

  6. When you are done with that, take an overall view of the comments, and start to group them into themes. Name the themes, e.g. “Lack of convincing data” or “Key strengths”

  7. Now start writing the meta-review. The main task for an Area Chair is to summarise the reviewer’s comments into one main document, balancing the differing opinions and adding your own where necessary. When opinions conflict, look at the PC’s expertise in the area to disambiguate, or look at the paper to verify which position seems more justifiable. We note that your own opinion may be in conflict with that of the reviewers. This conflict needs to be addressed via discussion on the OpenReview system. Challenge the Reviewers opinions and comments that are in conflict with yours, or you find are not substantiated by the facts of the paper or do not follow logically from these, in a respectful and factual manner. We suggest that your meta-review contains at least the following three sections: “Strengths”, “For Improvement” and “Recommendation”.

  8. Make a recommendation on whether to accept or reject the paper. Justify your recommendations based on the ICAPS-25 conditions for acceptance. In other words, look at the conditions for acceptance and see if it matches the points you see in front of you.

  9. Edit your meta-review. The authors will read it if the paper is rejected in Phase 1, and it is meant to provide a road-map for them to improve their paper, so it is good to take a kind, but objective tone.

Guidelines specific to Phase 2

If a paper is to progress onto Phase 2, the new reviewer will not have access to the discussion or meta-review until they have completed their own. Therefore, for Phase 2, you should go over the checklist above, introducing the new information and comments offered by the new reviewer.

Importantly, if you recommend the paper for acceptance to online proceedings only, you must provide private comments justifying your recommendation.

Issues to address during Discussion

ACs should drive the discussion actively by keeping a “bird’s eye view.” You may have your own personal view of the paper, but you should remain open and uncommitted in your language when sharing these views. It may not be possible to bring the reviewers to a “consensus” on their views of the paper because of inherent ambiguity in value judgements and the psychological dynamics of online discussion. You, as the AC, is the one decides the recommendation to decide the outcome. The Reviewers need to inform your final decision, and be responsive to your queries.

Some specific advice follows to address specific situations.

Determining when reviews are not substantive

Reviews may not be substantive, and you need to be ready to direct the reviewer to improve the quality of their feedback. While, statistically speaking, this is a problem which has been historically less frequent in ICAPS, it has happened, happens and will keep happening. Adjudicating what is not a “substantive” review is not easy, so we would like to share some examples (which are all inspired from real cases of ICAPS reviews and related venues):

  • The review rates the paper highly in technical aspects but the final score is negative due to alleged poor scholarship or unclear positioning.
    • Tactic: chances are that the reviewer has had trouble following the technical parts of the paper in detail. Check the related work section (there should be one) and for the papers that are singled out as most relevant, contrast the content of the papers with the comments made by the authors. In any case, you should for one be familiar with those cited works and have a clear idea of any important missing ones.
  • The review uses a lot of space to enumerate typos and minor use of English issues, but comparatively little in technical or scholarly aspects:
    • Tactic: It may be the case that the paper is very good and requires little feedback other than encouragement and editorial support. If the reviewer final recommendation is rejection, consider whether the feedback provided could be addressed editorially as part of the camera-ready preparation process. If so, remind the reviewer gently that the reviewing panel is not just an editorial one, and that the technical merits of the paper trump any perceived or real issues with how the paper is written. Negative reviews need to lean heavily on technical aspects rather than minutiae.
  • Short reviews are acceptable for papers that are clearly going to be rejected (e.g. partially written in languages other than English, making outlandish claims without any evidence or contradicting existing well-known results). Other than this, check the following:
    • Tactic for Positive Reviews: it is likely that the reviewer does not feel confident enough to make an assessment but still has a go at it, or maybe it has been delegated. Apply firmly but kindly some pressure on the reviewer so that the positive aspects are substantiated and made specific, clearly articulating why the paper should be accepted.
    • Tactic for Negative Reviews: in this case it is necessary to ask directly the reviewer to put examples of each negative aspect in the paper. For instance, if related work is noted as missing, then ask which are those works and why they are so important, if it is expressed that experiments are incomplete then the reviewer needs to precisely define what is missing, and identify what metrics and baselines must be used, giving bibliographic references if possible. Claims of lack of novelty need to be articulated in the context of the paper. Do not allow reviewers to get away with a review that does not provide specific statements for the authors to reply to.

Dealing with inappropriate reviews

All reviewers must abide by the ICAPS code of conduct, and the reviews cannot be the vehicle for anyone to display unlawful (as per the code of conduct) behaviours. If the review contains offensive remarks and the like take immediate action, getting in touch with your Senior Area Chair and write to the authors requesting them to rectify and change their behaviour.

Sometimes reviews may come with attempts at using humour to get the authors to reflect on a topic, or stern language to provide them with clear directions. Reach out to the reviewer if these comments sound likely to be misinterpreted or overbearing instead of being pedagogical, and help them find a different voice to convey their feedback. Finding the right tone to convey negative feedback is tricky, but essential for that feedback to be perceived by the authors as constructive and consequently taken on board.

Reviewer

Besides what it says on the tin, the role of reviewers in ICAPS-25 is to ensure that authors whose submissions have passed our desk rejection protocols receive high-quality, constructive feedback and criticism on their work. This means that we expect you to have something meaningful to say regardless of the paper being “strong” or “weak”. We also require you not to focus on editorial feedback at a low level: if the paper has typos or questionable grammar, point that out at a general level. Tools for correcting and enhancing written English are very advanced as of the time of writing this, and it is safe to assume that authors know about and how to use them effectively. Clearly, if the paper is written in a language other than English or is gibberish, raise this immediately with the AC.

Guidelines for Reviewing

You should review papers along the following dimensions:

  • Scholarship
    • How does the paper engage with the literature in the topic?
    • Is the paper carefully selecting what works to discuss?
    • Do the authors establish a logical relation between the cited works and the contributions?
  • Novelty
    • Is the work being plagiarised or overlaps with existing literature on the topic?
    • Does it contain innovative approaches to definitions, formulations, algorithms or experimental designs?
    • Does it provide insights that are new to you (and potentially the wider ICAPS community)?
  • Clarity
    • Is the paper structured in such a way that it is trivial to determine what the contributions are?
    • Is the nature of these contributions stated in a precise manner?
    • Are the authors able to articulate their ideas in a precise and coherent way?
  • Quality
    • Are the definitions in the paper consistent?
    • If it contains theorems, can you follow the argument proving their validity if one is given? And if no proof is provided, can you convince yourself of their correctness from the definitions, axioms and other assumptions stated in the paper?
    • For algorithms, is the algorithm specified with enough detail to understand its behaviour? If it is presented along an empirical analysis, are the tests conducted designed to support claims and stress the proposed algorithm?
    • If there are theoretical guarantees provided, are the arguments structured in a way that follows logically from the algorithm description and other definitions and results in the paper? Are the experiments designed to illustrate the predictions put forward by the theoretical results of the paper?
  • Technical Readiness (Tools papers only)
    • In addition to the four dimensions above, tool paper submissions provide an artefact, either a downloadable piece of software or web-base application that provides evidence of the content of the paper.
    • Artefact evaluation needs to be conducted in a manner consistent with the EAPLS guidelines, and reviewers are expected to state how papers track towards the levels of achievement associated with three EAPLS artefact badges.

Below we provide more specific directions and advice on how to evaluate your assigned submissions.

Scholarship

Scholarship is important but for conference papers it may need to be sacrificed on the altar of paper length limits. Conference papers are the natural vehicle to disseminate your own ideas and work rather than that of others’, unless the paper being submitted is a “position paper” or a “survey paper”. If no meaningful discussion of third-party work is necessary requiring folks to cite papers is just adding noise to metrics like the h-index. Otherwise, you should explicitly articulate that necessity in your review avoiding appeal to authority arguments.

Some authors may manage to add some detailed comparative analysis with other authors’ work, either on the Related Work section of the paper, or perhaps inline in other sections to provide context to the discussion. Acknowledge those efforts but remain critical. Careful reviewing of these comments will be time-consuming if you are not already familiar with the cited works. If you feel uncertain that the authors are representing accurately the contributions of those other works, signal that in your review or in the private comments to the AC.

If the paper is not citing your own work, or that of close associates, and you strongly feel the authors should have, take a step back for a minute and check with the AC to see what is their opinion. You can do this via the private comments, or by directly messaging the AC via the OpenReview user interface. You can add your thoughts to your review on multiple occasions along the reviewing period. Similarly, if you think the authors are mis-representing your work, ask for an opinion from someone who was not involved in the original publication. Reach out for support and second opinions to your colleagues.

Novelty

Novelty cannot be judged in absolute terms, but only by degrees. Innovation can also happen at many levels, which you may or not be able to appreciate, and it is always easier to judge in retrospective. So why bother? No matter how subjective, the perception of novelty (or lack thereof) is an important qualitative assessment of the submitted manuscript, and provides useful insight to the AC (who may or not appreciate novelty in a submitted work in the same way).

Be open and accepting of what is not “standard”, as that is a deeply problematic notion when it comes to evaluate the merit of scientific research, a creative endeavour where playfulness and transgression of “norms” is as important as effectively using mathematics and computer science to articulate and verify theories and algorithms. This intellectual tolerance needs to be balanced with the exercise of critical thinking and healthy scepticism.

If a manuscript causes reaction along the lines of “this can’t work or won’t make a difference”, then the matter does not belong to the realm of novelty, but that of quality, which we address below.

Clarity of Presentation

Clarity in scientific communications is perhaps best conveyed in the negative. The following are to our mind, clear instances of “lack of clarity”:

  • The abstract is very long yet leaves out important parts of the paper other than related work.
  • There is no proper introduction written, describing the problem and state-of-the-art in an informal (or formal) way.
  • The authors do not offer a clear guide to read the paper, signposting key results, etc. Note that this may or not be signalled with a specific heading.
  • The paper relies on complex definitions which are not given mathematically or only pointers to other papers or books are given. This may be due to space limitations, but if it is truly important, why isn’t the notion devoted careful discussion and connection with other parts of the paper?
  • The authors overuse figures of speech and metaphors, or use them inline in definitions, theorems, or proofs.
  • Algorithms invoke non-trivial or undefined procedures, or use flow control other than conditionals and loops. For instance, if the authors say “choose x s.t. P(x)” one should expect that both the domain of x and the background necessary for interpreting P(x) are explained in the text.
  • Notation changes meaning throughout the paper without warning, or in ways that greatly detract from the reviewing experience. For instance, one can use the letter “i” to denote indices of different sets, but it would be problematic that “i” is used to denote both indices and elements of sets.
  • The paper relies exclusively on text without any figures or diagrams, or if it uses them, it does not explain them unless using some formal graphical language which is clearly identified and cited.

Quality of the Contribution

Assessing quality is essentially to watch out for unsound procedures and fallacious arguments. These may be due to error, as in following from applying some mathematical operator incorrectly, presenting a procedure that does not terminate when it is clear it is assumed it should, or misunderstanding some result or concept in some mathematical theory such as Set or Graph Theory. More subtle and challenging to identify are logical fallacies, arguments that are logically unsound as they lack well-grounded premises. They are challenging because often papers are written to convince the reader, and there is a fine line separating persuasive and misleading writing. Note that the authors may be misleading themselves, rather than using a tactic to get a particular outcome of the reviewing process.

Some common fallacies to be found in papers on Computer Science and Artificial Intelligence are:

  • Argument to moderation: striking a trade-off between two positions depicted as ’extreme’ is not always correct.
  • Ambiguous middle term: arriving at the conclusion “balloons are dangerous”, from two premises: “weapons are dangerous” and “balloons are round”. Note that the premises do not connect “balloon” and “dangerous”. They just appear contiguously.
  • Fallacy of composition: a property of part of a whole must necessarily be true for the whole.
  • Appeal to authority: when papers, theorems or algorithms are cited without explaining clearly what elements in those works, results or procedures support the conclusion reached by the authors.
  • If-by-whiskey: putting forward an argument that supports both sides in an issue using terms that are emotionally sensitive and ambiguous.
  • Mind projection fallacy: this comes in two modes. A positive one, in which it is assumed that a statement about an object describes an inherent property of an object, rather than a subjective judgement. And a negative one, where the lack of knowledge about a phenomenon means that the phenomenon is not or cannot be understood.
  • Proof by assertion: a statement is repeatedly restated regardless of contradiction.
  • Retrospective determinism: assuming that because an event happens under some circumstance, that circumstance must have made the event inevitable.
  • Begging the question: assuming the conclusion and other forms of circular reasoning.
  • No true Scotsman: rendering a generalisation true by excluding counterexamples.
  • False analogy: an argument by analogy in which the analogy is poorly suited.
  • Cherry-picking: using individual cases or data that confirm a particular position, ignoring related data that may contradict the position.
  • Hasty generalisation: justifying a broad conclusion on a small or unrepresentative sample.

Importantly, it is itself unsound to put forward a review that “pins” one of the above phrases to a paper without clearly articulating the reasons for doing so. That would be an example of “begging the question”. Reviewers must articulate their concerns in a constructive tone and using diplomatic terms in their review and/or as questions to the authors. Always keep in mind that authors may have struggled to find the right words, leaving out a key sentence or two while editing the paper by accident, etc. Be generous but not credulous.

Discussion Phase

We would like you to adhere to the following guidelines during the discussion phase:

  • Be gracious with fellow members of the reviewing panel if they get some technical detail wrong. Deal with disagreements in a respectful and moderate manner, as you will probably have an easier time to influence people. If other members of the panel are being unreasonable or unresponsive, raise this privately with the Area Chair or the Topic Chair.
  • Be accepting of feedback on your review. You may be the one getting some detail wrong, or misunderstanding a statement made by the authors or other members of the reviewing panel.
  • Besides writing your review you need to respond to any queries or feedback by the AC. If the AC is sceptical of your assessment and asks you to do some further work, you must comply and respond in a professional and timely fashion.

Resources

  • This list of fallacies is a useful resource to keep at hand while reading other people’s papers and writing reviews.