The Elements of Productive Peer Review

One of the most important academic practices to learn is that of peer review — both producing peer reviews of other people’s work, and learning how to read peer reviews productively. And yet, we’re rarely taught how to write generative peer reviews, and even more rarely are we taught how to read peer reviews to improve our work. So, here’s a first stab.

Preparing Peer Reviews

At the outset, I should say that many journals have peer review expectations, and these are embedded in forms that they ask peer reviewers to fill out at the time they submit their peer reviews to the editors. At the most basic, they ask reviewers to evaluate the submission in terms of its ability to be published (e.g. publish without revisions, publish with revisions, revise and resubmit, do not publish). Beyond that, though, most peer reviews are rather free form and are left up to peer reviewers to do with it what they will. That being said, useful peer reviews tend to share some qualities:

Make sure that throughout your review to flag what the author needs to address and what she or he may choose to address. More than anything else, this can really help make the difference between a useful review and one that’s not so useful…

Start with a summary of the author’s argument, as you understand it. This is a pretty good index to the author as to whether or not he or she is conveying their argument, and what, if any, interpretive problems are happening on the reviewer’s end. If the reviewers just aren’t getting it, then there’s a big problem. Or if the reviewers are missing a nuance in the argument, that’s important to see too. This summary needn’t be long, but it should paraphrase the thesis and then cover the argumentation (i.e. the argument is X, which is followed in the paper by discussions of A, B and C). If there are logical inconsistencies in the argumentation, this is the place to point that out. If there are argumentative problems, this section can be much longer — upwards of two pages. But if it stretches beyond that, it needs to be broken into subsections to help the author interpret what to do and where specific problems lie.

The middle section of a peer review usually falls into the realm of free association, which is where the reviewer can spend some time discussing the relative merits and shortfalls of the paper, the paper’s linkage to other existing scholarship, and the overall consistency of the paper. Throughout, it’s important to flag which comments need to be addressed by the author and which are not so critical: it’s fine to go on for a paragraph or two about a pet peeve or some flight of fancy, but if it has little bearing on how the author should be rewriting his or her paper, make sure to flag that (e.g. ‘The author might be interested in…’ rather than ‘The author needs to address…’).

In the middle section of the peer review, it’s worth taking the time to discuss each of the sections of the paper (as the author has it broken down), including a brief description of the section along with some evaluative language (does it work or doesn’t it?). If there are parts of the paper that just don’t work or parts of the paper that really do work, this is the place to point them out. Praising things about the paper is just as important as saying critical things: if one reviewer really likes a section and others do not, it’s helpful to see why that’s the case.

Connecting the article to existing literature that hasn’t been discussed by the author also usually falls into this middle section. Again, it’s important to flag what needs to be addressed and what might be addressed. If an author is making vague references to a body of theoretical literature and would really do well to spell out her or his assumptions more directly, point that out; if an author has a huge blind spot in his or her discussion of relevant literature, point that out too. But if there’s some tangentially related literature that you know but the author doesn’t — and it won’t really have any bearing on the article anyway — make sure to mention that citing that work isn’t consequential, but might be interesting to the author.

The last section of peer reviews usually focuses much more on very specific things that need to be addressed by the author: are there specific awkward or vague sentences, missing citations, bibliographic errors, etc.? This isn’t to say that a peer review’s job is to find syntactical or grammatical problems, but if there are writing issues that interfere with the ability of the paper to be read, these problems need to be pointed out to the author.

The final paragraph of the review should succinctly restate your overall assessment of the paper and outline the major things (if any) that need attention on the part of the author.

It may seem paradoxical, but a really good review can actually be a bad review for an author. I’ve received reviews that say things along the lines of ‘This paper is ready to be published; the author shouldn’t change anything.’ (Usually only after already having passed through peer review multiple times, mind you…) But if the other reviewers are highly critical of parts of the paper, such a blanket positive peer review doesn’t help much. Instead, working through the positives of the paper is more helpful for the author, so that she or he can see why you like sections or the paper as much as you do — if you like things that others do not, it’s vital to see why that is. And, if there’s an editor involved, they’re more likely to be swayed by articulate, negative appraisals than a blanket and vague positive one.

Interpreting Peer Reviews

If you’re an author and have peer reviews that look roughly like what I’ve outline here, there should be very little problem in interpreting them. But, more likely, you’ll have peer reviews that don’t strictly (or even closely) follow this format. The best way to tackle reviews is to read through them and identify those things that reviewers think need to be changed (and thereby isolating them from less pertinent critiques). Often, the best way to do this is to see if there is convergence between readers: are people having similar problems with the paper? If they are, that’s fairly easy to see and address. If they’re having wildly different kinds of problems, it’s worth writing down the criticisms and seeing what the problem lies: is it that they really understand your argument differently? Are they coming from very different interpretive traditions? Sometimes you can address these problems; sometimes, it’s just the luck of the draw. And the important thing here is to isolate what you actually need to address and what you don’t. In not addressing some concerns, it’s important to point out why — sometimes in the body of the paper itself (which may be a way to get your argument more precise).

If you’re preparing to resubmit a paper after peer review, editors will often ask that you enumerate the changes that you’ve made to the text — and which changes you haven’t made and explanations as to why. If you have a list of requested changes, this is a fairly easy document to prepare, as you can list which changes you’ve made, how the criticism has been addressed, and where the change appears in the paper. Doing this can be helpful both for the editor and for peer reviewers, who often want to see that you’ve made the changes that they’ve suggested.