Current academic practice
Our approach to peer reviewing is informed by Whitworth and Friedman’s (2009a) criticism of current academic publishing as a form of competitive economics in which “scarcity reflects demand, so high journal rejection rates become quality indicators”. This self-reinforcing system where journals that reject more attract more results in a situation where “avoiding faults becomes more important than new ideas. Wrongly accepting a paper with a fault gives reputation consequences, while wrongly rejecting a useful paper leaves no evidence”.
Whitworth and Friedman (2009b) propose an alternative evaluation system:
1. higher rating discrimination: a many-point scale, not just accept-reject
2. more submissions to be rated: rate all
3. more people to rate: community involvement
4. different ways of rating: formal review vs. informal use ratings.
JOURNAL OF PEER PRODUCTION review process
Prospective authors and reviewers are invited to carefully read this section.
1. PAPER PROPOSAL
1.1. Non-special issues and Varia
Authors send a proposal to the (not publicly archived) editorial team mailing list: jopp [dash] editorial [at] lists [dot] ourproject [dot] org. The editorial team will assess the suitability of the proposal for the journal.
1.2. Special issues
Authors submit proposals directly to the special issue editors.
2. SUBMISSION AND REVIEW
Once authors have completed a full submission, they send it to the editor who assigns it to three anonymous reviewers. The reviewers assess the paper following our suggested review categories (see Appendix A). Once the reviewers have provided any necessary recommendations for improvement, these reviews are sent to the author for possible revision and the author decides whether to follow these recommendations.
3.REVISION AND SIGNALS
The revised paper is sent back to reviewers for final evaluation following signaling categories (see Appendix B). The editor forwards the signals to the author, who then decides whether to publish the paper or not.
4. PUBLICATION
The paper is published alongside the signals.
Signals will remain anonymous to ensure frank and fearless signaling.
IMPORTANT – PLEASE NOTE:
– All reviews will be released alongside published papers. Reviewers may opt whether to remain anonymous or not.
– All initial draft submissions will be released alongside published papers unless the author provides compelling reasons (such as privacy of author or subjects) why this should not be the case.
Since draft submissions will be published, authors are encouraged to use a respectful tone when dealing with other researchers’ work in their drafts.
Outcomes and benefits
It is up to the author to decide whether they are happy with publishing a paper with the given signals. Letting authors decide whether to publish enables the journal to release a wider variety of submissions more rapidly whilst protecting its scientific reputation.
Signaled papers will be published alongside reviewers’ initial reports and recommendations. This will help to promote review quality, as well as ensure that reviewers receive public appreciation for their work.
Appendix A: Suggested categories for peer review
1. Is the subject matter relevant?
2. Is the treatment of the subject matter intellectually interesting? Are there citations or bodies of literature you think are essential to which the author has not referred?
3. Are there any noticeable problems with the author’s means of validating assumptions or making judgments?
4. Is the article well written?
5. Are there portions of the article that you recommend be shortened, excised or expanded?
Appendix B: Suggested categories for signaling
Objective categories
Activist
Article proposes a critique of a policy or practice with specific action proposals or suggestions: yes / no
Academic
Article follows conventions of academic research article — e.g. position in literature, cited sources, and claimed contribution: yes / no
Prospective
Article is based on developments that have not yet occurred: yes / no
Formalised
Article is based on formal logic or mathematical technique: yes / no
Language quality
Standard of English expression in article is excellent: yes / no
Subjective categories
Scope of debate
Article addresses an issue which is widely known and debated: yes / no
Comprehensiveness
Most related sources are mentioned in article [this is an invitation to careful selection rather than a demonstration of prowess in citation collection — i.e. apt and representative choices made in source citations]: yes / no
Logical flow
Ideas are well organised in article: yes / no
Originality
The argument presented in article is new: yes / no
Review impact
The article has been significantly changed as a result of the review process: yes / no
Commendation
Reviewers indicate their appreciation of the article in the form of a 50 word statement.
References
Whitworth, B. and Friedman, R. (2009a) “Reinventing academic publishing online. Part I: Rigor, relevance and practice”, First Monday, 14(8-3), August.
Whitworth, B. and Friedman, R. (2009b) “Reinventing academic publishing online. Part II: A socio-technical vision”, First Monday, 14(9-7), September.