Since I started reviewing papers, I have always had this notion that my reviews are too harsh – I seem to always get the bad papers. No, dear reader, your papers are fine: I’m talking about all the other ones. So – am I being that guy? I don’t think so, and here’s why.
Assuming that the distribution of paper quality to reviewers is uniform, a fair reviewer should rate papers as their acceptance rate. So in an area where the conferences accept between 15 and 30 percent of the submissions, we shouldn’t be surprised to give four bad reviews in a row, even though I tend to feel like the bastard reviewer from hell.
But assuming this uniformity is probably wrong. In fact, if I were thinking as a primary reviewer, I would maximize reviewing fairness if I spent the scarce reviewer resources wisely. This means that if I think a paper is bad (from the proverbial 1-minute look), I won’t want to send it to a “great” reviewer – this reviewer time is better spent carefully reading the papers I think are good. So if you’re a grad student, you should expect to get worse papers than average, at least until you become known in the field.
Now, I have not received formal training in the mechanics of academia, so I fully expect the quality of my reviews to be worse than average. If this holds in general, it means that papers that tend to be worse (or simply to look worse, which is probably an important distinction) will naturally tend to get reviews of a lower quality. This is self-reinforcing, and leads to all sorts of strife – since authors of bad papers that are rejected could then, in part, be justified in claiming that it was due to bad reviews. I certainly remember sending horrible papers and being upset at the time by less-than-stellar reviews. Nowadays, I’m glad those reviewers didn’t track me down and burned down my home.
With that in mind, the best I can do is to be as suspicious of my own reviews as possible. The way I see it, this has two advantages. First, it will naturally offset my bias from the expectation that the paper will be below the accept threshold. Second, and more selfishly, I can hopefully make an impresion with the primary reviewer, which cannot hurt. So the system is not great (bad papers get bad reviewers, which is not terribly fair), but it’s not really that bad – if the agents are rational, there is a natural pressure to increase the quality of both submissions and reviews. Obviously, being aware of the metagame at all is also bad, so just pretend you haven’t read any of this and keep writing the reviews.
Yes, this is my way of rationalizing the results of the reviews ahead of time – we’ll see what happens tomorrow.