Reviewer: Alex Halavais

Paragraph comments:

p.2 should be p.1, yes?

p.2 para. 1: The paragraph seems to accept that disputes are
“prevalent” on Wikipedia. This leaves the reader asking, “when
compared with what.” But more broadly that sentence is confusing; it’s
not clear if the forms of power are the cause or the result of
conflicts in the community. Finally, it seems as though two forms of
critique are being suggested in this paragraph: the critical stance of
the author, as he will define it, and critiques leveled tactically by
members of the community. Left a bit confused by this introduction.

p.2 para. 2: The shift from 3rd person “one” in the last paragraph to
you/I here is jarring. I like the tongue in cheek, but it might not be
widely accessible (?).

p.5 para. 1: Someone (I believe from Wikipedia) recently suggested
that 13% of contributors on the site are women.

p.5 para. 2: I find this line of argument uncompelling. While people
*may* engage in structured argumentation, relying on evidence, etc., I
suspect this isn’t what you find most often in discussions on
Wikipedia. People are not good at assessing evidence or the structure
of argument.

p.6 para. 1: It will require a bit of gymnastics to perform a
meta-analysis of people’s analytical (critical) approaches. I wonder
if it’s not best to leave that to those who are engaged in the
discussions more directly? Or is this meta-analysis the purview of the
sociologist?(!)

p.6 para. 2: Some awkwardness in “To characterise…”

p.6 para. 2: You presume perhaps too much of the reader’s
understanding of the workings of Wikipedia (which have changed over
time). I guess depending on the “who” and “when” of the audience of
this piece, it might be useful to ground this more concretely.
Especially considering that many open source projects *do* have rank:
what would you call a Mozilla “module owner” for example? I’m just
wondering if this is so new a structure of marking merit?

p.7 para 2: I think you unfairly conflate Weberian bureaucracy with
“bureaucracy” as it is used in everyday speech. You can analyze
criticism on Wikipedia through its contributors everyday language, but
then you have to adopt that language. When they claim that Wikipedia
is not bureaucratic, they clearly do not intend that word in its
precise definition, but in the everyday sense of being negatively
bound up in red tape–that is, burdened by dysfunction. There are
certainly problems with such a definition, but just because they may
use this word does not mean they are using it in the same sense. In
fact, most dictionaries include this negative sense of an organization
“impeded by process” or “needlessly complex.”

p.13 para 3: “such as their edit counts” This seems to me to be
significantly different from external validation. It suggests, e.g.,
that a sockpuppet might actually have a great deal of authority in the
closed world of the site, even if it has no authority outside of it…

p.15 sub. 5: This section feels a bit grafted on. If there is a
clearer connection between the flows of authority on the site and
appropriate researcher ethics, it needs to be made clearer.

p.16 para. 3: “experienced editors expect their words and actions to
be be evaluated and criticised by their peers.” The question then
becomes who they see as their peers. The chief concern, ethically,
when investigating any community is whether and to what degree
participants expect their words to be decontextualized and analyzed
elsewhere, but those who may not share some of the cultural norms of
the community. If other “public” venues are any indication, a certain
percentage of Wikipedians would *not* sign an informed consent form.

Now, that doesn’t mean that they cannot be studied. It just means that
you have to show that your work provides them (and the world) with
more benefit than potential harm (as you say below).

p. 17, para. 1: It almost sounds here as if you are suggesting that
any exercise of social power/authority == injustice… Perhaps that is
not what you mean?

p. 17, para. 2: It seems that the researcher here is a power-neutral
actor? One of the issues may be that a judge is vested with a
significant amount of social power. You may be able to criticize, but
she can then jail. The power relationship (recognizing that it is
multi-dimensional) seems reversed in the case of most university
researchers and most Wikipedians.

The case made about concrete cases is interesting, since it feels as
though the article could benefit from such–though that may be the
unrepentant empiricist talking. Most pointedly, you go through
significant circumlocutions to avoid Jimmy Wales name, which seems a
bit silly if this is your central argument.

Reviewer: Mayo Fuster Morell

1) Is the subject matter relevant?

Yes, very relevant.

2) Is the treatment of the subject matter intellectually interesting?

Yes, however, I think the paper would benefit for a bit more work in making his argument more explicit from the beginning and structuring the flow of sections.

Are there citations of bodies of literature you think are essential to which the author has not referred?

The literature of peer production and Wikipedia research is quiet complete; however, the paper might benefit by pointing also more literature on ethical/critical/committed research. But it is not a big problem, it would just make it more rich.

3) Are there any noticeable problems with the author’s mean of validating assumptions or making judgements?

The author has not point to the methods used to sustain some of its observations and arguments. Introducing the base of his writing made make the text more transparent.

There are a bit of jumps in the writing styles. In most parts it is quiet clear if he is referring to previous works or if he is suggesting or stating something, but in (few) other parts it is unclear if the author is incorporating opinions or hypothesis to be explored with further research or features resulting for a systematic or systematized observation.

4) Is the article well written?

It is well writing but the structured remains a bit unclear. It remains a sense of separate parts which is not clear what links them, which might be the more rich contribution of the author.

5) Are there portions of the article that you recommend to be shortened, excised or expanded?

5. Are there portions of the article that you recommend be shortened, excised or expanded?

No, but to make more clear the core argument(s) from the beginning and a structure and flow of the writing more integrated. For that, perhaps some parts (which might result secondary) would be good to put in note and not in the core text.

REVIEWING COMMENTS FOR THE AUTHOR:

* To me it was difficult to follow the abstract. Particularly, it would be beneficial to improve the text flow in connecting the first three sentences. At the end of reading of it, I didn’t have a clear idea about what is the paper about.

* At the end of reading the article I did not have a clear view of what put all its parts together. I think that making the structure of the article more explicit at the beginning of the text (in the introduction) and creating more continuous passages between its parts might facilitate the reader understanding of the argument. For example, in the introduction there are introduced type of critiques, but in the conclusions there are not reference to them, so it came the question of why they were introduced. I see that there are interesting link-adages between the several aspects introduced by the author, but it remains not clear enough what links them (which might be the more fruity and tasty part of the author contribution in my view). So I would suggest to try to make those more explicit.

* I particularly liked the conclusion and gave me the taste that, it is a pity what is refered at the conclusion is not advanced some how from the beginning. So the reader can be reading the text having in mind the questions in the conclusions.

* Pag. 5. I would suggest to check the result of the United Nations University Survey, as far as I remember the percentage of women participation was 12-13%.

* Pag. 7: I would suggest to check if Jimmy Wales still block users, after he resigned from his special status at the English Wikipedia.

* The author said in Pag. 7: “Though these actions are undemocratic”. There are several conceptions of democracy, without suggesting to enter into that long debate, I think it might make more robust your argument, if you make explicit which is your normative approuch of democracy from which you judge the democratic quality of features of Wikipedia.

* In note 19, as you are referring to a specific expression in text, you might want to provide with a more specific reference, such as text page.

* In note 20, you might want to provide a more completed source, as with the information you provided I couldn’t find the source. Additionally, I am not totally convinced from what you provide to sustain your argument that there is a consensus among wikipedians concerning not conceiving Wikipedia as a bureaucracy. Would be more likely that most wikipedians think that as part of Wikipedia there is at least some bureaucracy?

* Pag. 7. I would make stronger the argument that 1500 people are determining what is “encyclopedic”. From how it is presented here it seems that you are saying that what is encyclopedic in Wikipedia is only defined by the possibility to block pages by administrators. This is an element, but what is encyclopedic in Wikipedia is also defined by for other questions (such as policy on content), no?.

* I couldn’t follow the two last paragraphs of section 2. To me it was difficult to see the flow between them.

* It might make your paper more transparent if you point to the methodology used in some of its passage when you present Wikipedia features (for example explaining the conflict dynamic: i.e. did you conclude that from virtual ethnographic observation, in which period did take place, in which parts or articles of Wikipedia did you analyzed the conflict dynamics?) .

* The reference in note 31 is uncompleted.

* In regards to section 3., you start by pointing to conflict as characteristic of Wikipedia “the main costs of anonymous crowd-sourcing in mass projects such as Wikipedia is many-to-many fighting”. Then by presenting several reasons why conflict might emerge in Wikipedia (nature of participants, controversial topics, institutional issues, identity of participants). Perhaps you can make more robust your point of in which point the conflict become too problematic or unproductive. Conflict is part of the social life (it is difficult to think of social relations which do not involve some level of conflict) and conflict can also be positive, as an pre-stage for some positive changes for example. In any case, there are several theories about conflict, there is not need that you enter into them too much, but perhaps making more explicit at what point you consider conflict is becoming overwhelming in Wikipedia might make your writing less reading as “conflict can only be bad”, otherwise it seems you are saying conflict just should not be there at any level.

* Pag. 14: “Wikipedia undoubtedly attracts a fair number of incorrigible lunatics and provocateurs who should quite property be kicked out”. I don’t know if the tone of this sentence is in line with the overall writing. Perhaps you can refer to this phenomenon as trolling behavior which has been study and defined with accuracy, also inside Wikipedia.

* Pag. 15: You might want to be more precise on the dip of recruitment numbers. Or all the article is only referring to the English encyclopedia? If this is the case, you might want to refer at the beginning of the article that your analysis refers only to the English Wikipedia. How do you arrived to the conclusion that the decreases of edits account in English encyclopedia can be attributed to the increase of disciplinary control?. This “might” be the case, but in order to verify it might be requere to interview the people who left Wikipedia or to provide any other type of source from Wikipedia live which state that conclusion, otherwise it seems an hypothesis to be verified with further research. An hypothesis that you can point to and leave as a possibility, but in my view not something that can be stated (even if I personally share your intuition).

* In the following sentence to the note 57 in text, there is a typography error: there are “to be be”.

* In the conclusion, you point that “Wikipedians do not participate in the project to share personal stories and experiences, find emotional support, experiment with identity, or play: (…) Wikipedia is a working environment; it is also imbued with a strong psedudo-legal culture etc”. Even if Wikipedia is a working environment, these does not mean that the motivations of Wikipedians are not to play (“get fun”) or build personal relationships, at least the element of getting fun and make friend is among the motivations that emerge from the surveys to contribute to peer production (see Weber or Benkler reference to these question).

* I consider the author does a valuable contribution pointing that “The ethics of not doing harm to subjects needs to be balanced to the ethics of potentially not addressing injustice unearthed by research”. I would suggest the author to build more upon this point and refer to it from the beginning (at least if it is as central as I understood it is for the author).

* The link between the need to also apply the ethic of research to “injustice unearthed” and the question of “Should online tribal bureaucratic be held accountable for their actions? etc” is a bit unclear. Even if I can intuitively see what the author try to say and I consider that both points seems valuable.

PERSONAL COMMENTS ON THE TOPIC:

* Starting from the first sentence “To understand how Wikipedia research can be “critical”… ” and in general from the approach of the text, it came me the question: Is not research (or should be) critical by “nature”?. Research always should be critical, non-critical research lose its meaning of “search”. To me (but this is a very personal comment, more than a comment as a reviewer) it would make more sense to build the argument from: the risks that research loose its critical character or in what aspects the research develop in Wikipedia has loose or is weak in regards to its critical nature, than to assume that there can be critical and non-critical research.

Reviewer: Matthew Allen, Curtin

1) Is the subject matter relevant?

Yes, clearly – Wikipedia is a key component of peer production in contemporary world and the paper’s critique demonstrates analysis of it suitable for publication in CCPP.

2) Is the treatment of the subject matter intellectually interesting? Are there citations of bodies of literature you think are essential to which the author has not referred?

Yes. No. To expand: the approach is astutely critical and theoretical, with a very clear understanding of the way the sprawling body of literature in sociology might unlock certain views on the role and substance of Wikipedia. It’s refreshing to read an article that takes its theory seriously. The focus on authority is especially important as a corrective to the ‘pop’ complacency with the power dynamics involved in Produsage. By making clear that authority matters, in terms of production, we also find out how authority matters in social, political and economic life.  Perhaps some analysis of the failure of some internet scholars to critically assess this question (in relation to blogging, social media etc) might have been productive? Work on blogging (eg bruns) might, for example, provide an example of this failing which then strengthens the author’s stance. Furthermore, it would situate the article within the broader field of internet research – at the moment it feels (but is not) like an outsider, to some extent, looking in (which I know is definitely not the case). It’s a small point, however.

3) Are there any noticeable problems with the author’s mean of validating assumptions or making judgements?

None, aside from the small point above.

4) Is the article well written?

Yes, though perhaps with a little too self-consciously styled introduction. I rather like the didactic style but might suggest the author review it and consider whether it will attract (rather than turn off) the majority of readers – in other words, will it be a disincentive for those to continue to reading? As an example of what I like, the phrase ‘In short, online projects should be understood as expressing the values of the dominated fraction of the dominant social group that is, people endowed with intellectual rather than economic assets’ stands out – very nicely put, without some of the colloquial address of the pages preceding it. Pithy style works.

5) Are there portions of the article that you recommend to be shortened, excised or expanded?

Slight revision of the introduction, perhaps (see 1); possibly greater discussion of POV?

Overall – this is a fine paper and should be published!