Reviewing a CS conference paper Reviewing a conference paper is a non-trivial task. Often, reviewers have to read and review more than one paper, and usually under a tight time deadline. Regardless, there is a right way to review a paper, and many wrong things. This document contains my thought on the "right way", and points out several mistakes that many (or even most) reviewers make. The following is a list of principles to use; I'll elaborate on each of these below: 1. Review the paper 2. Review to accept papers 3. Don't demand too much 4. Write a review 5. Note little things, but don't make them your review 6. Things to avoid Much of this is subjective and is based on my reviewing experiences (from both sides) in Computer Graphics and Geometric Modeling. Things may differ in your area, and you may have a different opinion. Feel free to send me email about this, although please apply the criteria below when composing your message. Stephen Mann April, 2009 =========== 1. Review the paper This may sound obvious, but it's harder to do than it sounds. The grossest violation of this principle are the reviews that say "the authors should have done this instead." But your review should not be about what should have been done; rather it should be a critique of what they authors actually did. If you feel the authors should have done something else, accept the paper and discuss it with them at the conference. But reviewing the paper is hard for an important reason: usually, the authors are too close to their work, and thus have difficulties stating precisely what they did, why it's of interest, and why it's important. "Reviewing the paper" means reading to a level that you understand what the authors did, why it's interesting, and why it's important. As part of your review, you should note these things. And you should accept or reject the paper based on whether you think the *contribution* is significant enough. If you think the paper is poorly written, or the contribution is poorly described, state that, but do not make it your basis for rejecting the paper. This rule is usually violated because reviewers are overloaded and under time pressure. A poorly stated result may be hard to tease out of the paper, but if you're not going to take the time to do so, then you shouldn't be reviewing the paper, and if you don't have the time to do so, you need to reduce your reviewing load. [NOTE TO AUTHORS: to help ensure that reviewers can determine what you did, etc., spell it out. Mention it in the abstract; state it in the introduction; and restate it in the conclusions, where you should link back to the body of text to support your statements.] 2. Review to accept papers. When you read a paper, try to find reasons to accept the paper. If nothing else, if you're following the first principle (Read the paper), you should spot what is good about the paper and highlight that in your review. If you don't like the approach, fine, but try to decide what about the authors paper makes it acceptable for publication. Yes, not all papers are worth publishing, but almost all papers have an idea that the author is promoting and you should review to accept that idea. Sometimes the idea is bad/wrong/already-been-done. And that's fine - the paper can't be accepted. But read the paper looking for a reason to accept it, and don't reject it unless that reason doesn't exist. And sometimes an idea is clearly half-done. The temptation is to reject the paper with the recommendation that it be resubmitted when the research is complete. But often it's the idea itself that is the research contribution. And if it's a good idea, then consider accepting the paper on that basis. This becomes particularly important when you realize that a lot of research is done by graduate students, and papers submitted on their work may be all that ever gets done on it. By rejecting a great idea because it wasn't perfectly polished, the idea may never get published despite being worthy of publication, since that student's work is done. Related to this is when you write your review, write with the mind set "how to improve this paper" rather than "here's a list of things that are wrong with this paper." 3. Don't demand too much The paper is a conference submission, and there are page limits. Don't write a review saying "the authors should include the following", where "the following" would push the paper well past the page limits. If there is something so critical that it MUST be included, suggest something to remove/reduce so that the authors can kept to the page limits. Likewise, don't demand any additional work that can't be done in the time between acceptance notification and the final submission deadline. While analysis can sometimes be redone, it's unlikely that another experiment can be run or significant code can be written. 4. Write a review Review forms have check boxes, and there is a temptation is rely on the check boxes with minimal comments. But your written comments are really the important part of your review, and you should write comments that help both the program committee and the authors. In particular, be sure to cover the following in your written comments; some of this material will be in response to questions on the review form, but regardless it should all be covered somewhere in your review: 1. Outline the paper. 2. Highlight the contribution of the paper, both what the authors perceive it to be and what you perceive it to be, as well as how significant it is. 3. State your recommendation and why. 4. State ways to improve the paper, but don't ask too much (see both the previous and next sections). The first forces you to reread the paper, which helps in writing your review. The second is the hard part: you need to figure out what the authors think is their contribution. The 3rd is one that the program committee will focus on. Of these four, the 2nd and 3rd are the important ones. 5. Note little things, but don't make them your review "The authors should include the following references." "The grammar needs to be improved." "The figures are poor quality." No paper is perfect. There will be details that are wrong, often of the above variety, but sometimes of a bit more substance ("the authors give the wrong formula for X"). These are not reasons to reject a paper (although if you can NOT read a paper because the grammar is terrible, you have no choice but to reject it for that reason). Again, focus on the contribution and base your recommendation on the contribution and not the writing details. You should note the small things, but ideally place them at the end of your review in a separate section as "details to improve". 6. Things to avoid Here's a list of miscellaneous things to watch out for in your reviews. A. Do not say "the authors should add additional references on X" without actually listing those references. If you're enough of an expert to make the judgement, then you should be enough of an expert to explicitly list those references and state why they should be added. In particular, since there is a page limit, the bibliography should be focused on the most relevant work, and not be a complete survey of the topic. So if you think a paper should be cited, give a strong reason as to why, since potentially the authors know of the reference and decided not to list it for their own reasons - your argument to include the reference should be strong enough to convince someone who has already decide to not include it. And NEVER, NEVER, NEVER reject a paper because it omitted references! Don't even HINT at rejecting a paper for this reason. This may sound obvious, but if you decide that a paper should not be accepted and in summarizing your reasons you mention the missing references, you have just hinted that you rejected the paper for missing references. (As a more general rule, never reject a paper for something that can be fixed in 5 minutes.) B. Usually you get to rank the paper on a scale like 1 to 5 as to whether or not the paper should be accepted. Around 2/3 your rankings should be 1 or 5, around 1/3 should be 2 or 4, and you should rarely, rarely, rarely give a rating of 3, which should be considered a reject anyway. If you can't give a strong recommendation, then you likely didn't understand the paper well enough to review it. I have heard the statement "I never give a 1 because I don't want to hurt the authors feelings." That just makes "2" the new "1", and you won't have spared anyones feelings. If you don't want to hurt the authors feelings, then do a good job of understanding the paper and base your decision on what the authors did, and write your review as "how to improve the paper" rather than "bash the paper". C. Some people will try to tell you that for some conferences, the conference papers are the same quality as a journal paper. This is wrong for several reasons: there is usually an explicit page limit, and there is no chance for resubmission (resubmitting the paper to a future conference is different than resubmitting a journal paper). The result is a lower quality paper than a journal paper. This doesn't mean conference papers are terrible, nor does it mean they are worthless. Making a distinction between the two is important, however, since you review a journal submission with different standards/criteria/etc than you review a conference submission. In particular, a journal paper needs to be more complete than a conference paper: there needs to be a better literature review; there needs to be a more complete result; there needs to be a more in-depth analysis of the result. Understanding the difference will help improve your converence paper reviews. D. Don't be insulting, be positive. Other review guidelines usually state the former; I've never seen an insulting review, but I guess it happens. More of a problem is being positive: the authors put a lot of effort into writing the paper and will be sensitive to (and even insulted by) criticism. So phrase things positively. In general, write your entire review in a tone of having accepted a paper, even when you're not recommending acceptance. This will help change what you subconciously write as condemning criticism to helpful comments.