There’s a funny bit of asymmetry in the way we review papers for academic conferences and journals (at least the ones I’m aware of). When a paper is accepted and published, the reviewers get to know who the authors are, but no information comes from the other end. Why don’t we fix this? I think we should.
There are good reasons to keep author and reviewer unaware of one another’s identity during the review. But the author anonymity is gone when the paper makes it, so why not just publish all the good stuff created along the way? In my dream journal, the reviews for all accepted papers would be available, with full attribution.
.. but the papers will be too big! No one cares about paper journals anyway: I just want an electronic version of it. Dump the .txt files somewhere in the DVD for all I care; I just want to be able to look at them. Every conference I can think of has electronic proceedings, and some are even moving to electronic-only, which is great. There really is no publishing cost associated with it.
I often hear claims that reviews are typically of worse quality than submissions because there’s less value attached to them. If the reviews are made part of the final accepted submission with full attribution, life is much fairer for both the good reviewer. It makes it much easier for people to recognize them, and I know they’re out there. (I just recently got the best review I’ve seen – there was so much detail it added almost a full page to the paper) I sometimes read the acknowledgements and I swear I can hear the authors going “can you please give my reviewer some candy? He was so nice!”. And, obviously, full-disclosure can also expose those guys (mildly NSFW language).
I would love to be learn how to review a paper from a public archive of decent reviews. We all have good papers to learn from, but no reviews at all. That is sad: we’re robbing ourselves of cheap, good education!
.. but it will overburden the reviewer! Well, yes. I would still rather have few good reviews than many bad ones. And if you really think that, it means you would write your reviews differently if people knew who you were, which is not a particularly defensible stance.
.. but who reviews the reviewers? The community! With electronic publishing like the arXiv, we really should start getting used to it. Publishing in journals will become less important in the future, and having a culture of continually evaluating papers after they’re published can only help in the long term. Since the final information will actually be available for the community, this also makes it easier to spot hidden-information shenanigans like mutual back-scratching.
So, PC members and conference chairs of the world, help us all out: stop the asymmetry!
(and yes, Vis was good, if you have to ask.)