This string has been defined as a Wikidata lexeme via https://www.wikidata.org/wiki/Lexeme:L593933.

see https://hypothes.is/a/pmbmbsftEeuOoNNu8o1SRQ

Europe PMC

W

W

something amiss here

See also "Wikipedia for multilingual COVID-19 vaccine education at scale".

In such a skewed distribution, the mean does not have much of a meaning. The peak does, though, and the interpretation of it makes sense.

The DOI given for Fehr and Perlman should be "10.1007/978-1-4939-2438-7_1" , as correctly stated in https://zenodo.org/record/3901741/files/top_20_wiki_cited_doi_annotated_europmc.csv .

WikiProject Medicine has been mentioned a number of times already, so introducing thee acronym here is odd.

as was the case during

perhaps plural is intended here?

typo in "WikiHypelink"

perhaps say something about the time lag between the two - spikes in pageviews often trigger spikes in editing, which then trigger activity around references

This is "WikiProject COVID-19": https://en.wikipedia.org/wiki/Wikipedia:WikiProject_COVID-19

add comma

striving for a policy is an odd framing here, especially if that policy (MEDRS) already exists

something missing

This number is not obvious from Fig. 2D., which would suggest something like 160. What is missing?

should be plural

should be plural

WHO is listed once by acronym, once by name

grammar

In the figure legend, "# Citation" should be "# Citations"

should be "as a function of"; also for other figures

drop

It is confusing that Fig. S3 is mentioned before Fig. S2.

give precise p value

add space

grammar

looks like Table 3 is meant here

check talk pages; consider that, especially in the early phases of the pandemic, peer-reviewed research was often not available on some specific issues, so the information would either have to be based on other sources (e.g. media reports) or not included. Such things are discussed on the talk pages of individual Wikipedia articles and of the relevant WikiProjects.

add dash

perhaps the wrong word if it is a policy

A simple explanation for this is that Wikipedia has a lot of content on topics for which preprints are (or at least have traditionally been) more popular than in medicine and biology.

this is weird grammar

typo

the percentages given actually do not refer to the preprints, just the red slice does - this should be clarified.

typo

arXiv has been around since 1991, and preprints had been around before arXiv

Note that the open access percentage in the overall Wikipedia corpus is below that of the COVID-19 corpus is not surprising because the overall corpus tends to include more references from before 2020, and the percentage of open access is growing over time.

that's probably not the best fit here

typo

citation needed

metadata about the supplementary files is scarce

in the legend for Fig. S1C, "May" should be upper case

there seems to be some figure confusion here, along with a superfluous parenthesis

consider putting this in italics, similar to the journal names

delete

Not sure what that degree does here - the Wikipedia article title does not contain it: https://en.wikipedia.org/wiki/Anthony_Fauci .

Here, the comma is missing. It is present in the CSV used for visualization, which contributes to minor formatting problems.

add comma

I think what is meant is https://en.wikipedia.org/wiki/Category:WikiProject_COVID-19_articles . A link, citation or footnote would be useful.

there are actually several such templates, but I assume the one referred to here is the navigation template https://en.wikipedia.org/wiki/Template:COVID-19_pandemic . A link, citation or footnote would be useful.

Please archive the repositories on Zenodo as well, e.g. as described at https://guides.github.com/activities/citable-code/ .

Nothing wrong with citing this:

Sobel, Jonathan, Benjakob, Omer, & Aviram, Rona. (2021). A meta analysis of Wikipedia's coronavirus sources during the COVID-19 pandemic (Version 0.1) [Data set]. Zenodo. http://doi.org/10.5281/zenodo.3901741

Alternatively, simply hyperlinking would be useful: http://doi.org/10.5281/zenodo.3901741 .

This functionality has high potential for being reused by others (including for replication, e.g. for running the numbers until May 2021), and perhaps even be expanded on. Kudos to the authors for creating and sharing such a resource! Would be good to share a quick tutorial too (say, as a Jupyter notebook or RMarkdown file), with a complete workflow, from the MediaWiki API calls to the timeline and network visualization.

The date of publication is often ambiguous (e.g. "online first" versus "PDF-only" versus full-text versus paper version, or actual versus scheduled publication date) or even incorrect. How should this be taken into account?

a scale of years is not useful at the beginning of a pandemic

I suggest to replace the hashtag notation in the equation with something other of a less informal nature.

At this point, it is not clear whether "references" in equ. 1 refers to "non-DOI" references or "total of DOI references and non-DOI references" ==> this info comes too late - see "Ranging from 1 to 0" in the "Scientific Score" section

Consider using a logarithmic axis for the citations.

A key thing here is probably the combination of decentralized activity with some central coordination (including through shared values, principles and workflows).

In https://arxiv.org/abs/2006.08899 , this results in what they called "coherence of collaboration".

add "the"

should be plural

should be plural

add comma

It is annoying to the reader that BioRxiv renders tables as bitmaps.

equating citation count with academic quality is problematic

What were the sources for these? Note that Wikidata also provides such regexes, e.g. at https://www.wikidata.org/w/index.php?title=Property:P957&oldid=1429624446#P1793 .

should be plural

add comma

This still belongs to the "retrieved" clause. You probably did not retrieve "the authors, publishers [etc.]" but metadata about them, which may have unique and/ or ambiguous components.

This sentence is too long - please chop it up.

add comma

add link/ ref

perhaps also mention that anyone can edit Wikipedia

unclear how this was done. This is also relevant to Table 3.

This way of defining the latency in the parenthesis is a bit odd. Perhaps cite ref. 8 here already, which is currently introduced later.

refer to the visualization, as per the previous comment at https://hypothes.is/a/-hG2Gsf1Eeu-0ecpwAEkmA

mention Fig. S1A, as it helps with understanding this. Alternatively, use a table

EuroPMC or Europe PMC

This is rather confusing to parse.

the distribution of Altmetrics scores in the Wikipedia COVID-19 corpus

clarify how that number came about: explain the steps more clearly

add link/ citation

Better: "The DOIs from this set were"

add by when, to clarify the time horizon of the current study.

Others have looked into search strings more comprehensively, e.g. https://doi.org/10.2196/23449

the database is called "Europe PMC" or occasionally "PMC Europe". The R package EuroPMC accesses data from Europe PMC.

Also, adding a link to or a reference about Europe PMC may be useful

typo

add link and/ or citation

Cite the software properly:

• create an official release with a version number
• archive a copy of the release on Zenodo
• cite the DOI of the Zenodo archive, along with the version number

Here, it would be useful to add a link: https://en.wikipedia.org/wiki/Wikipedia:WikiProject_COVID-19 .

Disclosure: I am actively involved.

this should be plural

such standards have not been introduced to the manuscript yet, so this statement comes a bit out of the blue.

citation needed. The relationship between news and Wikipedia editing has been studied before, including in the context of the current pandemic - see e.g. https://arxiv.org/abs/2006.08899 and references therein.

The key policy here is https://en.wikipedia.org/wiki/WP:MEDRS . It is referred to in several indirect ways throughout the manuscript but gets only one explicit mention, (towards the end) and no link or reference.

this is problematic - see comments below

Perhaps clarify that the term is used in this text as essentially interchangeable with "COVID-19-related".

add comma

add comma

the motivation for looking at references may not be obvious to some readers, e.g. since most news articles do not provide references.

perhaps refer back to it as that “key tool for global public health promotion”, so as to make the sentence flow better.

perhaps better: "emerging scientific topics like" - otherwise, it's not clear what the "like" refers to

perhaps better say "preprint" here

this is ambiguous

Perhaps this acronym does not need to be explained in this context, but it is nevertheless good practice to explain them upon first usage.

Not sure which Wikipedia or Wikipedias are referred to here, or where that number comes from. The cited source is more specific:

"Wikipedia had 155,805 medical articles across 255 natural languages at the end of 2013. A further 31 languages did not contain any medical articles per our methodology. Of the more than 155,000 articles, 29,072 (18.66%) were in English."

add "pandemic"

typo

The PDF also has a set of keywords - not sure where to find these in the HTML version.

They were "COVID-19 | Wikipedia | Infodemic | sources", and I think the latter is a bit too ambiguous to be useful.

I hope so too, and I might use the tool chain myself, so I would like to encourage the authors to share it more completely.

This popup does not provide sufficient resolution to actually access the information contained in the file in any useful manner.

these two strings would seem straightforward to map to each other, which would also enhance Fig. 2D

add "the"

drop the s

add link

don't forget the shared values and open infrastructure

Wikipedia's

There is also https://en.wikipedia.org/wiki/Wikipedia's_response_to_the_2019%E2%80%9320_coronavirus_pandemic , which may well end up incorporating some of the materials from the present manuscript.

If the workflows described here in the paper were shared more comprehensively, it should be relatively straightforward to rerun the analysis to include times after the period considered here.

decrease

superfluous

This blurs the meaning of the term "open access", which does not include temporary lifts of paywalls.

add dash

perhaps clarify that most of those have never been "locked" in any way.

"prominence" is perhaps a better term in this context.

The URL in this link is broken.

Brush.

This is the first concrete mention of MEDRS, even though it was insinuated, alluded to or otherwise invoked on several occasions above.

The policy should thus be introduced earlier, and with a link: http://en.wikipedia.org/wiki/WP:MEDRS .

better: "course" or similar

superfluous

add dash

with

Not clear what the revision ID refers to. Also, CSV has some formatting issues.

It's not clear how the prominence of these six (and their number) was determined - bot from the enlarged part in Fig. 4 and from the inset of the full graph, other numbers could well be a reasonable choice.

citation needed

The banner was about a message from the Wikimedia Foundation.

There is a process by which Wikipedia articles pertaining to current news can be linked from the Wikipedia homepage, and this process came into play here.

link is missing

typo

the weight column in the CSV is not explained

This needs brushing. While correct in terms of the definition of "scientificness" given in equ. 1, this phrase leaves too much room for misinterpretation.

typo

typo

typo

Apparently, Hypothesis cannot annotate the In [8]: text, so I am using this import command to comment on the cell numbering.

For sharing executed Jupyter notebooks, it is important to do at least one of the following:

• keep a full version history of the states of each cell (e.g. via ProvBook)
• Before sharing, clear all outputs, restart the kernel and rerun all cells

Neither was done here, so it is theoretically possible that some previous state of one of the cells could influence the calculations shown in the notebook.

I do not think this problem occurred here, but such occurrences are hard to review, so it is best to follow the practices outlined above.

Compared to the version I ran, the sauter_diameter dataframe had a different order.

It would probably be advisable to avoid that by defining the order in some way.

Would be useful to link to the hashtag https://twitter.com/hashtag/LecturesForFuture .

Better use absolute dates, not relative ones - easier to maintain.

Time for an update

Given that the Case studies were broken down by country, I would have expected at least the first paragraph of this section to bind the Case studies together from a policy diffusion/ policy learning perspective (e.g. in terms of more or less strenuous conditions during outbreaks or 'peacetime'), before zooming back in to highlight certain aspects from the Case studies.

Odd way of referring to a statement not in the previous sentence but in the sentence before that.

delete the "s"

Not sure what that dash is doing here.

I assume this stands for "Standard Operating Procedures", but it was not defined as such.

In order to adapt to new contexts, policy diffusion often triggers such semantic drift of key concepts.

Would be great to see that linked to the policy learning framework.

Perhaps some comments as to how the international guidelines cover local training and capacity building would be a useful addition.

Good example of policy diffusion

Clear example of equity being neglected

Are there templates available for "appropriately drafted MTAs"?

Reference to policy diffusion

Given that the article is framed as being about policy diffusion and using a policy learning framework, I would have expected more details here.

This seems to exist already, but still I could not find it.

Does this exist already? I could not find it.

Would be great to connect these efforts with others who work on this from the data end, e.g. RDA as mentioned above.

Also, the presentation at http://www.gfbr.global/wp-content/uploads/2018/12/PG4-Alpha-Ahmadou-Diallo.pptx states

This data will be made available to the public and to scientific and humanitarian health communities to disseminate knowledge about the disease, support the expansion of research in West Africa, and improve patient care and future response to an outbreak.

but the notion of public access is not clearly articulated in the present article.

This is a nice conference report, but are there any slides or recordings available from the event as well?

Does it have a name and online presence? The details provided here go beyond what's given in reference 13, but some more detail would still be useful, e.g. to connect the initiative to efforts directed at data management and curation more generally, for instance in the framework of the Research Data Alliance, https://www.rd-alliance.org/ .

Very useful initiative!

Odd word order

The sentence is non-grammatical.

Some more details would be useful here, perhaps in a dedicated Methods section.

What about at least mentioning the other three?

ORCID is missing

I followed this remotely via Twitter (don't know whether other remote channels were available). My notes on this sit at https://github.com/Daniel-Mietchen/events/issues/508 .

I would have expected the Background section to set the stage for the case studies, not to sum up their results.

Given that this document cites a number of non-persistent web resources, I have archived a copy of https://wellcomeopenresearch.org/articles/4-170/v1 at http://web.archive.org/web/20191224000829/https://wellcomeopenresearch.org/articles/4-170/v1 using the "Save outlinks" mode.

Probably a good idea to do this routinely for all articles in the journal.

Archived at http://web.archive.org/web/20191224000537/https://apps.who.int/iris/bitstream/handle/10665/205944/WHO_HIS_SDS_2016.2_eng.pdf

Not sure what the purpose of quoting with session ID is here, as these are usually non-persistent and not usable by anyone other than the website operator.

It could be made clearer that the following is quoted directly from the CIOMS document, and specifically the Governance section of the Commentary on Guideline 11.

It could be made clearer that this refers to the document cited in Footnote 1.

Archived at http://web.archive.org/web/20191223234626/https://www.who.int/blueprint/what/research-development/guidance_for_managing_ethical_issues.pdf

This URL seems very unstable, so I archived the file at http://web.archive.org/web/20191223233751/https://cioms.ch/wp-content/uploads/2017/01/WEB-CIOMS-EthicalGuidelines.pdf .

In general, it is good practice to provide not just links but also an archived version when citing a URL.

Of course, it would be even better if policies themselves were FAIR (Findable, Accessible, Interoperable and Reusable), as discussed, for instance, in https://github.com/Daniel-Mietchen/events/blob/master/PIDapalooza-2018.md .

https://tools.wmflabs.org/scholia/author/Q18211072

As far as I can tell, the two are (nowadays at least) separate entities, so why are they listed under the same affiliation?

That ORCID profile has this paper as its only entry. Is this correct?

That ORCID profile has this paper as its only entry. Is this correct?

Would be nice if that identifier would lead somewhere useful. A web search for it yielded https://doi.org/10.12688/wellcomeopenres.15442.1 , which is also included in the collection "GFBR: The ethics of data sharing and biobanking in health research" available via https://wellcomeopenresearch.org/collections/gfbr18 .

This should be "Eretmapodites" for both species

This is referenced in the Wikidata entry about WDR60 at https://www.wikidata.org/w/index.php?title=Q21124736&oldid=619612847#P682 , which states that WDR60 (Q21124736) is involved in the biological process (P682) of embryonic skeletal system morphogenesis (Q14886895).

I'm missing something about making policies themselves more FAIR, which we had included in an earlier version of these draft principles.

A link would be useful here, ideally to a persistent version of the policy.

These keywords should be linked to persistent identifiers (rule number 5).

Test

I left a number of annotations here as part of my review of the paper.

The review is accessible via https://doi.org/10.21956/wellcomeopenres.13272.r25804 , and the notes I took on the process can be found at https://github.com/Daniel-Mietchen/ideas/issues/494 .

superfluous

That's not a long-term solution - what if someone comes along a few years down the line, or 20?

looks like a missing "that"

i.e. tsv

not in the copyright sense