Reviewer: Anonymous

1) Is the subject matter relevant?

The paper touches a relevant issue, the one of evaluating openness in open hardware initiatives, and it does so with a thought analysis of the open-o-meter framework in relation to three case studies.

2) Is the treatment of the subject matter intellectually interesting? Are there citations or bodies of literature you think are essential to which the author has not referred?

The author is well place in the discussion on netarchical capitalism and the vision of digital commons of the P2P foundation. Although some reference to other critical theorist, e.g. Carlo Vercellone, could make sense for the discussion, in general I consider the literature sufficient and the reading instructive and interesting.

3) Are there any noticeable problems with the author’s means of validating assumptions or making judgments?

The main problem with the article is that the concepts used in the three research questions are not defined in a way that can be operationalized. Reading through the theory, it is possible to identify what the author has been thinking about but this lack of clarity makes the discussion in particular a weak section. In the commented file, I have suggested what needs more definition and I think that the author should revise the discussion after having clarified the dimensions part of the research questions.

4) Is the article well written?

Yes, it was a very good reading.

5) Are there portions of the article that you recommend be shortened, excised or expanded?

See above: the discussion section needs more work after the research questions are better defined. Although embracing a case study methodology, I think that the method section should provide more information on how data have been analyzed (and which kind of data).

Reviewer: Anonymous

1) Is the subject matter relevant?

Yes, the subject is (broadly defined) open source hardware, with also a focus on evaluation of projects and more wider discussion of open source underpinning principles. This is a relevant topic for the journal and it is a topic that connects well to actual development of the field e.g. in relation to IoT, Industry 4.0  etc.

2) Is the treatment of the subject matter intellectually interesting? Are there citations or bodies of literature you think are essential to which the author has not referred?

Overall I think the subject matter is treated in an interesting manner, but the paper is not well organised and several aspects of it need to be clarified. There is a lot of good material in the paper, especially the evaluation and in the later, long discussion (An Emerging direction). However, both the structure of the paper and how the argument flow have serious limits and at this stage the paper does not seem ready for publication. Substantial rewriting will be needed to get the paper publishable.

I do not have specific comments about the literature, most known literature on the general matter of open source is cited and there is specialised literature cited in relation to open hardware.

3) Are there any noticeable problems with the author’s means of validating assumptions or making judgments?

Overall, the manuscript presents a number of problems, and at this stage it is not ready for publication. These are my main comments:

1. the Introduction section is quite “flat”, it does not anticipate what is the purpose of the paper nor what is the relevance of the study within the purpose. It is too descriptive and does not say what is the paper goal.
2. research questions are presented later in the literature review. However it is not good practice to have the questions in the literature section, which should be usually devoted to the review of literature only. A general research question should be anticipated in the introduction (see point above) and then specific questions could find their place in another section.
3. In the section “Assessment of current practices”, the author(s) mention the adoption of a qualitative-quantitative approach. However the approach used seems entirely qualitative as it is based on an evaluation done on documents and practices with then numbers assigned (e.g. 0, 0,5, 1) depending on how the authors have evaluated a project according to a certain principle of the open-o-meter. The use of these numbers e.g. 0 and 1 is not per se quantitative, although it is clear it produces a score at the end.
4. Overall, I think it is not very clear what the authors want to achieve with the paper. The paper has some interesting content (with the section on the 3 projects being the most interesting), but I fail to see why we need this evaluation. Or better it is not clearly explained I think, which leads to the next point (e)
5. The other main problem is with the open-o-meter. This instrument necessitates a better discussion, in the methodology section. Much is left out and a reader which may not be entirely familiar with it struggles to understand the purpose of the open-o-meter. Also it is not clear if the authors are the proposers of the open-o-meter of if they take it elsewhere, assuming it is the second, a better discussion and presentation of it is necessary, against the context of the Open Source principles in general (if possible).
6. Further on the open-o-meter, the author seems to argue that it needs improvements (Section Emerging Direction). The question that a reader may ask then is why the paper does not proposed a revised version (optimised, improved) of the open-o-meter and then does the evaluation of case studies based on the improved version of the open-o-meter?. This would be a much better paper, providing a useful contribution for practice and also demonstrating an improved version of the open-o-meter metric.

4) Is the article well written?

The language of the article is acceptable overall, but a language revision is highly needed.

However, what limits the readability is the structure of the paper and how the argument flows in the text overall. Rewriting will be necessary, especially for a) clarifying better the goals of the paper b) clarifying better what the open-o-meter is c) possibly consider to change the focus and proposed the optimisations of the open-o-meter (discussed in the “Emerging Direction”) as the main focus of the paper.

Note that there are many acronyms in the paper and it easy for a reader to get lost. Double check that all are defined e.g. Sometimes acronyms are introduced before giving the full concepts e.g. in the introduction OSH is given few lines before its “acronymisation” i.e. open source hardware (OSH). I would consider an appendix with all the acronyms.

5) Are there portions of the article that you recommend be shortened, excised or expanded?

The abstract needs expansions and rewriting. It is too short and it does not summarise well the content of the paper. The open-o-meter description needs expanding and better presentation as the reader is left wandering about many aspects of it (as if the open-o-meter is something known to most and does not need full explanation, but this is clearly not the case and most reader won’t know it).

I do not know if it is journal policy guidelines to have the tables in an appendix, however Table 2 needs definitely to be included in the main text, as that is essentially the main finding of the empirical evaluation.

Reviewer: Peter Lyle

1) Is the subject matter relevant?

Yes. The subject matter – further unpacking ideas of what the ‘open’ in open source hardware actually means in practice, and how it is evolving over time is valuable. Looking to open source software and reflecting on differences between software, peer produced knowledge and hardware is important to be understood. Using this as a way to reflect on the use of the existing open-o-meter to understand some experiments in open-source-hardware is useful, although I am not entirely convinced by the case study screening process as is currently written.

2) Is the treatment of the subject matter intellectually interesting? Are there citations or bodies of literature you think are essential to which the author has not referred?

There is mention early of the ambiguity of how open-design is used, with a mention of participatory design, among others. Participatory design has a long tradition of its own, and in many ways could fit well with the principles that the author is trying to convey in terms of future directions, and while I don’t think it is necessary to return to this, I do think it would be worthwhile returning to multiple understanding of open-design.

3) Are there any noticeable problems with the author’s means of validating assumptions or making judgments?

Regarding case selection:

One of the criteria for the selection of cases was some level of maturity – but the authors also state at multiple points that OSH is relatively young. Could it be that this selection of ‘older’ experiments that consider more open principles have been superseded by newer examples? I would recommend the authors acknowledge this limitation.

In addition, I do find the use of wikipedia list as a source is interesting – because of course behind any wikipedia page is the talk page, which is where some of the tensions about what should or not be included plays out in discussions amongst page editors. Some reflection on this would be appreciated, especially given that wikipedia is mentioned early on as an example of commons based peer production

A minor point:

“Apparently, this is related to the nature of the for-profit organization but it seriously affects the project’s openness” – This seems a leap in logic unless I have misunderstood the paragraph so far. Can you envisage, for instance, a for-profit organisation that did have a greater level of openness to collaboration?

4) Is the article well written?

The general structure of the article is good. I did notice a number of sentences would benefit from some adjustment to improve readability, in addition to correcting a number of spelling errors, so would stress the need for careful editing.

Below I include a number of comments about how the document could readability could be improved:

The tables are images – without a high resolution, so the reproduction of them is not the best. I would recommend these be either recreated, or higher quality versions found. I’m also unsure what permissions might be required (if any) to include the table from someone else’s research.

Statements like “there is evidence that openness (transparency, replicability and accessibility) affects the level of contribution (Balka et al., 2014)”, in order to be more useful in the article, should be more explicit in terms of what the effect is – currently it’s not clear (without reviewing the sourced literature) as to whether this work serves to promote or not the level of openness/contribution.

I found the explanation of ‘OS Grid’ by Openstructures a bit unclear to follow.

The reference to footnote 8 is just a copy/paste of the about page of the open structures site. This can be summarised or presented differently – at least to make clear that this is not the views of the authors, but rather the views of the site itself (their own marketing).

When the list of cases is first introduced – I think it would be helpful to mention in a couple of words ‘what’ they actually tend towards producing (i.e. furniture, houses, etc), such that when you say for instance “As the object of design is considerably smaller comparing to other OSH projects” the context is more clearly understood.

5) Are there portions of the article that you recommend be shortened, excised or expanded?

I think the article accessibility could be improved with more explicit reference back to the research questions in the conclusion.

I am interested to hear more reflection and introspection about the values held by the author and how they relate or conflict with the open-o-meter – such as in the case of the commercial license possibility. I suspect I agree with the authors on their position, but I would like to better understand the rationale for their position, and think it will help with the overall context of this paper.