Flammie A Pirinen on github pages
This is not an official guideline for reviewing or anything but my personal notes, please see ACL’s guidelines and last minute reviewing advice of ACL 2017 for serious advice for your reviews.
As a senior science making human, I spend as much time writing reviews in academic conferences as I do receiving them. As we all know, there are plenty of problems in the whole process still in 2020, but I try my best not to add to the specific problem of bad or useless or even rage-quit-worth reviews. Here are some guidelines I have found useful as a writer of reviews with some insights also from the point-of-view of the receiver of the reviews. I try to always include real-world examples, most of them are either written or received by me, but some are also found in the wild from twitter or similar contexts.
One of the things that is common throughout this text is that in my opinion, the scientific peer review fills only few purposes:
By and far, the first is the main reason of the review process for me, most of the articles I read are usually suitable for presentation… eventually though sometimes you do need to ask for change of venue or few months more of work for the benefit of the content really. This is one of the reasons that I have found it super useful to formulate all my feedback in a way as I would tell my students or colleagues how to improve their articles, I do not say: this is rubbish, I don’t get why you did that, I always write: it would be interesting for this conference visitors to know if… or it would really help explaining the … here if you added examples and these numbers.
It is my pet peeve enough that I have written other rants on academic english, but ignoring that, it is sometimes necessary to help non-native English speakers to improve the English. The only really useful way to talk about spelling, grammar etc. errors is to explain things very pedagogically, you are not improving the article by saying You’re English is too bad, ask a *native speaker, *there are 7 grammatical errors in this article (these are all from feedback I received! The one that counted the errors didn’t even give one example or pointer of what the errors were!) that’s really just insulting without any positive effect, like, you were not “hired” to programme committee to give the writers homework. Write the line number, the sentence fragment and say why it’s hard to follow: I cannot figure out what the pronoun it refers and I suspect the verb here should’ve been singular for the reading you intended?, the missing commas here make me think that …. I also use a lot this may be just my pet peeve in grammar but, …, if you are a younger reviewer you might think it undermines your authority but it actually makes it more likely for the author to consider the feedback.
When I have been complaining this on twitter, many have commented that the reviewers have so much to do that they cannot have time to go through all the grammar mistakes in detail, and that is fair, it is one of the big problems of the current model, but if you don’t have the time to deal with grammar errors properly, then don’t mention grammar or English quality at all, it will make the rest of the review more valuable for the authors. You probably wouldn’t deal with errors in pseudo-code or maths that way either: “there was some logic errors in the codes of the article but I leave finding them as homework to the authors”?
Some examples from twitter:
Every conference should offer proofreading for papers without a confident anglophone author. Raise your hand if you've given up on a relevant paper because of consistent subject-verb mismatch in the abstract. 🙋♀️ https://t.co/NoNgxgDMXP
— Naomi'sAFRAID (@nsaphra) July 14, 2020
Writing a good review often includes revealing that you had expectations from the article that were not fulfilled, and while it may not be fault of the authors of the article, making them understand that potential readers feel this way, is a good improvement to the article. What I often write in reviews and like receiving as feedback says this clearly: When I saw the title I thought I’d find a solution to my problem, but I didn’t find it in the article, You said in the introduction that there is a method to handle frugglypuffs, but I cannot handle mine with the methods in this article. and the like.