On 2014 May 27, Chris Hafner-Eaton commented:

This study should be viewed as one "data pull" of 29 systematic reviews. In order to support the conclusions, the study must be replicated many times. It will be through repeated true positives (the sensitivity) with minimal false positives/maximizing the specificity or true negatives that we will come closER (although never declarative) to saying that Google Scholar "could be use alone for systematic reviews." As others have noted, PubMed doesn't capture all and yet it is entirely possible to pick up too much erroneous material--particularly in the grey literature and for certain review topics such as Comparative Effectiveness Reserach. However, one must always weigh the costs of being wrong versus being late with the results!

On 2014 May 21, Francesc Roig commented:

From my point of view, according to the methodology of the study of Gehanno et col, and according to the results presented, they cannot affirm that "If the authors of the 29 systematic reviews had used only GS, no reference would have been missed". The only conclusion we could maintain would be something like "all references in the 29 systematic reviews selected were accessible through GS", but not that these references would be retrieved in a search with the objective to conduct the systematic reviews. As far as the study doesn’t compare search results in both engines (as other studies posted here actually do), it seems clear that you cannot maintain that results would be the same and then, you cannot maintain that using GS for the systematic reviews would produce the same results.

On 2014 Apr 01, Stephen E. O. Ogbonmwan commented:

It is scholarly wise to use more than one data base in searching for articles for review publications. Google Scholar is good but it is not better than a combination of two or three or all the other search engines together as each database has different criteria for inclusions and exclusion of articles. An article not in data base A may be found in data base B hence it is wise to use more than one data base for search purposes. The recall potential of the data in Google Scholar is not 100% like the other data bases which is another good reason to use a combination of data bases.

On 2014 Mar 31, M Felix Freshwater commented:

The AMSTAR standard http://amstar.ca/Amstar_Checklist.php is 2 electronic sources. According to the Cochrane Handbook http://handbook.cochrane.org/ 6.2.1.3 Database overlap

Of the 4,800 journals indexed in EMBASE, 1,800 are not indexed in MEDLINE. Similarly, of the 5,200 journals indexed in MEDLINE, 1,800 are not indexed in EMBASE. o www.info.embase.com/embase_suite/about/brochures/embase_fs.pdf

On 2014 Mar 28, Farhad Shokraneh commented:

I think the conclusion of the paper is not supported by the data. This study just shows that "If you KNOW that a paper ALREADY exist, you can find it in Google Scholar". So the authors have NOT used Google Scholar for searching for systematic review but just for re-finding the list of the papers have ALREADY found by other resources and included in the systematic reviews. The conclusion could be right just when the authors use GS for systematic searching and finding the relevant studies among search results. Also, the next time I think 'Google' gives the same coverage!

On 2013 Dec 27, Wichor Bramer commented:

More recently another article was published that revisited the conclusions from Gehanno: Bramer WM, 2013.

In this article we conclude that, though the coverage of Google Scholar is near 100%, the retrieval is far from that. Because Google Scholar is only able to show the first 1000 hits, we investigated whether the authors of 21 reviews that used google scholar would have identified every included reference if they had used only google scholar.

Only 72% of all references would have been found, which is almost equal to the recall of PubMed (68%).

Therefor Google Scholar should not be used alone in searching for Systematic Reviews.

Not even with improved precision, because the factual precision (the number of hits found in the first 1000, divided by 1000) is not as problematic as when precision is calculated based on the (estimated) total number of hits, which very often is exeptionally high when searching Google Scholar.

On 2013 Nov 15, Gaétan Kerdelhué commented:

Two later studies confirmed a high recall of Google Scholar but argued it could not be used in realistic settings for systematic reviews. See Giustini D, 2013 and Boeker M, 2013.

On 2013 Nov 15, Gaétan Kerdelhué commented:

Two later studies confirmed a high recall of Google Scholar but argued it could not be used in realistic settings for systematic reviews. See Giustini D, 2013 and Boeker M, 2013.

On 2013 Dec 27, Wichor Bramer commented:

More recently another article was published that revisited the conclusions from Gehanno: Bramer WM, 2013.

In this article we conclude that, though the coverage of Google Scholar is near 100%, the retrieval is far from that. Because Google Scholar is only able to show the first 1000 hits, we investigated whether the authors of 21 reviews that used google scholar would have identified every included reference if they had used only google scholar.

Only 72% of all references would have been found, which is almost equal to the recall of PubMed (68%).

Therefor Google Scholar should not be used alone in searching for Systematic Reviews.

Not even with improved precision, because the factual precision (the number of hits found in the first 1000, divided by 1000) is not as problematic as when precision is calculated based on the (estimated) total number of hits, which very often is exeptionally high when searching Google Scholar.

On 2014 Mar 28, Farhad Shokraneh commented:

I think the conclusion of the paper is not supported by the data. This study just shows that "If you KNOW that a paper ALREADY exist, you can find it in Google Scholar". So the authors have NOT used Google Scholar for searching for systematic review but just for re-finding the list of the papers have ALREADY found by other resources and included in the systematic reviews. The conclusion could be right just when the authors use GS for systematic searching and finding the relevant studies among search results. Also, the next time I think 'Google' gives the same coverage!

On 2014 Mar 31, M Felix Freshwater commented:

The AMSTAR standard http://amstar.ca/Amstar_Checklist.php is 2 electronic sources. According to the Cochrane Handbook http://handbook.cochrane.org/ 6.2.1.3 Database overlap

Of the 4,800 journals indexed in EMBASE, 1,800 are not indexed in MEDLINE. Similarly, of the 5,200 journals indexed in MEDLINE, 1,800 are not indexed in EMBASE. o www.info.embase.com/embase_suite/about/brochures/embase_fs.pdf

On 2014 May 21, Francesc Roig commented:

From my point of view, according to the methodology of the study of Gehanno et col, and according to the results presented, they cannot affirm that "If the authors of the 29 systematic reviews had used only GS, no reference would have been missed". The only conclusion we could maintain would be something like "all references in the 29 systematic reviews selected were accessible through GS", but not that these references would be retrieved in a search with the objective to conduct the systematic reviews. As far as the study doesn’t compare search results in both engines (as other studies posted here actually do), it seems clear that you cannot maintain that results would be the same and then, you cannot maintain that using GS for the systematic reviews would produce the same results.

On 2014 May 27, Chris Hafner-Eaton commented:

This study should be viewed as one "data pull" of 29 systematic reviews. In order to support the conclusions, the study must be replicated many times. It will be through repeated true positives (the sensitivity) with minimal false positives/maximizing the specificity or true negatives that we will come closER (although never declarative) to saying that Google Scholar "could be use alone for systematic reviews." As others have noted, PubMed doesn't capture all and yet it is entirely possible to pick up too much erroneous material--particularly in the grey literature and for certain review topics such as Comparative Effectiveness Reserach. However, one must always weigh the costs of being wrong versus being late with the results!