Noah Wardrip-Fruin in Grand Text Auto (was ist das für ein cooler Titel für ein Blog) über sein Experiment zu Blog-Based Peer Review: Four Surprises. [via liblicense-l] Fettdruck wie immer durch mich:
In most cases, when I get back the traditional, blind peer review comments on my papers and book proposals and conference submissions, I don’t know who to believe. Most issues are only raised by one reviewer. I find myself wondering, “Is this a general issue that I need to fix, or just something that rubbed one particular person the wrong way?” I try to look back at the piece with fresh eyes, using myself as a check on the review, or sometimes seek the advice of someone else involved in the process (e.g., the papers chair of the conference). But with this blog-based review it’s been a quite different experience. […] faced with just his comment, in anonymous form, I might have made only a small change. However, once they started the conversation rolling, others agreed with their points and expanded beyond a focus on The Sims — and people also engaged me as I started thinking aloud about how to fix things — and the results made it clear that the larger discussion of process intensity was problematic, not just my treatment of one example. In other words, the blog-based review form not only brings in more voices (which may identify more potential issues), and not only provides some “review of the reviews” (with reviewers weighing in on the issues raised by others), but is also, crucially, a conversation.
I’m convinced that the ability to engage with one’s reviewers conversationally, and have them engage with each other in this way, is one of the key strengths of this approach. It should be noted, however, that the conversational nature of the blog-based review process also had a less-positive side, which produced my second surprise. [und jetzt kommt der Twitter-Faktor ins Spiel: :-)] But the flow of blog conversation is mercilessly driven by time. While it is possible to try to pick up threads of conversation after they have been quiet for a few days, the results are generally much less successful than when one responds within a day or, better yet, an hour. I hadn’t anticipated or planned for this.
Das Kommentieren eines ganzen Buches fällt naturgemäß schwerer als sich über bestimmte Abschnitte auszulassen, weswegen die Diskussion anscheinend nicht imemr zielführend oder „ganzheitlich“ war. Bei Zeitschriftenartikeln würde dieser Nachteil aufgrund der Übersichtlichkeit und Kürze vermutlich wegfallen.
I worried the blog-based review form might be worse than useless if its impact was to turn authors (myself included) away from major, systemic issues with manuscripts and toward the section-specific comments of blog visitors with little sense of the book’s project. My concerns in this area became particularly acute as the review went on. A growing body of comments seemed to be written without an understanding of how particular elements fit into the book’s wider frame.
The blog-based reviewers offered almost no remarks comparing chapters to one another — perhaps because they experienced the manuscript more as sections than chapters. Still, as I will discuss below, they also offered much more detailed section-specific commentary, much of it quite useful, than it would be possible to expect from press-solicited anonymous reviews.
Der (scheinbare) Gegensatz anonyme Peer-Reviewer vs. (meist namentlich bekannte) Blog-Reviewer läßt sich vielleicht durch die schiere Menge erklären (es berührt ein fundamentales Problem, das ich kürzlich noch mit dem Chief Editor eines Open Access Journals diskutierte: Die Schwierigkeit, geeignete Referees, d.h. Peer-Reviewer, zu finden – gerade im Zeitalter von Interdisziplinarität):
[…] the number of manuscripts that require review dictates that only a few reviewers should consider each manuscript. Otherwise, the burden of manuscript review would impede the completion of other work. When only a few reviewers look at each manuscript, each will have some areas of relevant expertise. But for many manuscripts, especially interdisciplinary ones, there will be many topics discussed for which none of the reviewers possess particular expertise. There’s no real way around this.
Noah zieht ein interessantes Resümme:
[…] it is Grand Text Auto’s reputation that matters. It makes sense to do a blog-based review because we have, in blogs, already-existing online communities that attract university-based experts, industry-based experts, and interested members of the public. The way we use blogs also already encourages discussion and questioning. Of course, widely read blogs won’t want to be completely taken over by manuscript review, but I can imagine them hosting two or three a year, selected for their level of interest or because they are written by one of the blog’s authors. […] I think there is a hunger, on both sides, to connect the kinds of inquiry and expertise that exist inside universities and outside of them. Blogs are one of our most promising connection points — and blog-based peer review offers one simple way for the two groups to contribute to common work.