On 2021-10-11 18:46:13, user Andrew T Levin wrote:
Comment #2: Methodological Issues
-
Given the stated purpose of this study, it is remarkable that the manuscript never specifically defines the term “community-dwelling population.” In practice, the study analyzes the incidence of COVID-19 fatalities that have occurred outside of nursing homes, but even that distinction is not very precise. For example, the spectrum of U.S. nursing homes encompasses board & care homes, assisted care facilities, and skilled nursing facilities. About two-thirds of U.S. nursing home residents rely on Medicaid to cover that cost. By contrast, higher-income individuals can afford to receive home health care or choose to live in “retirement communities” with on-site medical staff. In effect, the distinction of whether someone is “community-dwelling” or a “nursing home resident” is linked to a complex set of socioeconomic characteristics as well as to various aspects of their individual health. Making international comparisons along these lines is even more fraught with difficulty, because the size and composition of the nursing home population inevitably reflects differences in social norms as well as socioeconomic factors, access to healthcare, and the extent of public assistance. Indeed, such comparisons may be practically meaningless when considering developing countries such as the Dominican Republic and India, where nursing home care may only be an option for a very small fraction of the population.
-
Search Procedure. This manuscript uses an arbitrary search cutoff date of 31 March 2021, which excludes some landmark seroprevalence studies that have been disseminated since then. For example, Sullivan et al. (2021) analyzed seroprevalence of the U.S. population over the second half of 2020 using a large representative sample that included 1154 adults ages 65+, and hence that study would clearly satisfy the stated eligibilitry criteria for this meta-analysis.[11] Moreover, the study carefully adjusts for assay characteristics and seroreversion and estimates that as of 31 October 2020, the IFR for U.S. adults ages 65+ was 7·1% (CI: 5·0¬-10·4%). Those results can be also be used in conjunction with data on nursing home deaths to obtain the corresponding IFR estimate of 4·7% for community-dwelling adults ages 65+.
-
Minimum Size Threshold. This analysis excludes seroprevalence results from any studies involving fewer than 1000 adults ages 70+, and hence it is remarkable that the manuscript neither provides any rationale for imposing such a constraint nor provides citations to any existing works that might motivate it. Indeed, this approach is inconsistent with basic principles of statistical analysis, e.g., making inferences based on all available information and avoiding arbitrary selection criteria that could induce bias in the results. Consequently, meta-analysis should downweight studies with relatively lower precision rather than simply discarding those studies. Moreover, it is incoherent to specify an eligibility criterion based solely on sample size, because the precision of seroprevalence estimates also hinges on the level of prevalence. A small sample may be adequate in a context of relatively high prevalence, whereas a much larger sample may be needed to obtain precise inferences in a context of very low prevalence. The national study of Hungary was included in this meta-analysis because that study included 1454 adults ages 70+. However, only nine of those individuals were seropositive. Consequently, the test-adjusted seroprevalence for this cohort of older adults is not statistically distinguishable from zero, and hence the confidence interval of the age-specific IFR is not even well-defined.[12] By contrast, the regional study of Geneva was excluded from this meta-analysis because it only included 369 individuals ages 65+. But that sample was large enough to facilitate inferences about seroprevalence (6·8%; CI: 3·8¬¬ 10·5%) and corresponding inferences regarding IFR for that age cohort (5·6%: CI: 4·3 7·4%).[13, 14] Finally, setting the sample size threshold at 1000 is clearly an arbitrary choice. Since seroprevalence studies can be readily identified using the SeroTracker tool, this meta-analysis should be extended using a lower threshold of 250 adults ages 65+ that would encompass the national studies of Netherlands and Sweden as well as a substantial number of regional studies.
-
Sample Selection. In characterizing which seroprevalence studies have been included in <br />
the meta-analysis, this manuscript specifies the key criterion as “aimed to generate samples reflecting the general population.” However, this criterion is extraordinarily vague and judgmental (as evident from subjective words like “aimed” and “reflecting”). <br />
(a) United Kingdom. The inadequacy of this approach to sample selection is evident from the fact that the meta-analysis places equal weight on four U.K. seroprevalence studies, even though only two of those studies (UK BioBank and REACT-2) utilized samples designed to be representative of the general population.[15, 16] By contrast, the other two studies used convenience samples that were not designed or even re-weighted to be broadly representative, and hence those two studies should have been excluded from this meta-analysis. First, Hughes et al. (2020) studied a panel of primary and secondary patients at a large Scottish health board, with the stated objective of assessing viral transmission patterns.[17] The paper never suggested that this panel could be interpreted as representative of the wider population; indeed, some of these patients may have been receiving care related to COVID-19. Second, in one of its weekly surveillance reports, Public Health England (2020) reported seroprevalence results for a panel of patients ages 65+ who had a routine blood test at the Royal College of General Practioners Research and Surveillance Centre.[18] Evidently, this panel was not aimed to reflect the general population and may well have included patients recovering from COVID or experiencing COVID-like symptoms. <br />
(b) United States. One of the two U.S. seroprevalence studies used a sampling design that is intended to be broadly representative, whereas the other U.S. study used a convenience sample of patients at kidney dialysis centers. Unfortunately, as a consequence of gross disparities in healthcare access, higher-income individuals typically utilize in-home dialysis machines, whereas low-income individuals must travel multiple times per week to a dialysis center, often using public transit. Consequently, the prevalence of COVID-19 infections among such patients has crucial public health implications but should not be interpreted as representative of the general population.<br />
(c) Canada. Among the three Canadian seroprevalence studies, two use representative sampling designs (Ontario and Canada-ABC), whereas the third study conducted by Canadian Blood Services (CBS) uses a convenience sample of blood donors. In its public announcement of those results, CBS specifically warned that “caution should be exercised in extrapolating findings to all healthy adult Canadians, because blood donors self-select to be blood donors, in some areas access to a donation clinic may be limited, and there are fewer elderly donors who donate blood compared to the general population.” [19] That caution was specifically cited as the reason for excluding this study from a previous meta-analysis.[5] Indeed, given the scarcity of elderly blood donors, there is an even stronger rationale for excluding that study from the analysis here. Indeed, this meta-analysis should have specifically excluded all convenience samples, whether from blood donors, commercial lab tests, or medical patients. Dodd et al. (2020) analyzed a large panel of U.S. blood donors and found that the proportion of first-time donors jumped in June 2020 following the introduction of COVID-19 antibody testing, consistent with the hypothesis of “donors with higher rates of prior exposure donating to obtain antibody test results,” and concluded that “blood donors are not representative of the general population.”[20] Bajema et al. (2021) found seroprevalence of 4·94% using commercial lab residual sera from residents of Atlanta (USA), compared to seroprevalence of 3·2% using a representative sample of the same location.[21, 22] These findings highlight the extent to which convenience samples may be associated with upward bias in seroprevalence and hence downward bias in IFR. It should also be noted that the incidence of COVID-19 infections has a strong association with race and ethnicity, reflecting disparities in employment, residential arrrangements, and various other factors. Such patterns have been evident in numerous countries (not just the USA), and hence the manuscript should follow a consistent approach in addressing this issue.
-
Open-Ended Age Brackets. This manuscript proceeds on the assumption that open-ended age brackets for older adults are essentially equivalent regardless of whether the bracket is 60+, 65+, or 70+. But this assumption is inconsistent with the consistent findings of preceding studies, namely, the IFR for COVID-19 increases continuously with age rather than jumping discretely at any specific age threshold. Indeed, the measured IFR for any particular age bracket is a convolution of the age distribution of the population, the age-specific pattern of prevalence, and the fact that IFR increases exponentially with age. The complexity of this convolution underscores the pitfalls of comparing IFRs for open-ended age brackets of older adults. Ontario serves as a useful case study for illustrating these issues. The Ontario Public Health seroprevalence study reported results for three broad age brackets: 0-19, 20-59, and 60+ years. However, this manuscript assesses IFR for ages 70+ using results obtained via private correspondence. However, that assessment may be very imprecise, because COVID-19 prevalence was very low in the general population and may well have been even lower among the oldest community-dwelling adults. By contrast, the Ontario study is very informative for characterizing the cohort of individuals ages 60-69 years. In particular, there were 9 positives among 804 specimens for that cohort; the test-adjusted prevalence of about 1% indicates that about 17000 Ontario residents ages 60-69 had been infected by mid-June 2020. As of 30 June 2020, that age group had 240 COVID-19 deaths—none of which occurred in nursing homes. Consequently, the IFR for community-dwelling Ontario adults ages 60-69 was 1·4% -- identical to the predicted IFR t the midpoint of this age interval from the metaregression of Levin et al. (2020).[5]
-
Adjusting for Assay Characteristics. Seroprevalence studies have generally been conducted using antibody assays with imperfect specificity and sensitivity, and these characteristics exhibit substantial variation across assays. Moreover, the implications of these characteristics depend on the actual level of prevalence, e.g., adjusting for specificity is crucial in a context of relatively low prevalence.[23] Consequently, all three of the preceding meta-analyses consistently used seroprevalence estimates and confidence intervals that had been adjusted for test sensitivity and specificity using the Gladen-Rogan formula and/or Bayesian methods.[5, 8, 9] By contrast, this meta-analysis simply uses raw seropositive data from those studies that did not report test-adjusted seroprevalence.
-
Low Prevalence. The shortcomings of this manuscript’s approach are particularly evident in assessing IFRs for locations with relatively low prevalence. For example, as shown in manuscript Table 1, the seroprevalence study of Hungary used the Abbott Architect IgG assay to analyze 1454 specimens and obtained 9 positive results, i.e., raw seropositivity of 0·6%. According to the manufacturer’s data submitted to the U.S. Food and Drug Administration, this assay has sensitivity of 100% and specificity of 99·6%.[24] Consequently, the Gladen-Rogan formula indicates that the test-adjusted prevalence is only 0·2%, i.e., only one-third of the observed seropositive results were likely to be true positives. Moreover, this test-adjusted estimate has a 95% confidence interval of 0 to 0·4%, i.e., the prevalence is not statistically distinguishable from zero, and hence its IFR does not have a well-defined confidence interval. Indeed, that was precisely the reason why this cohort was not included in the meta-analysis of Levin et al. (2020).
-
Unmeasured Antibodies. This manuscript follows a completely unorthodox approach in adjusting seroprevalence for unmeasured antibodies: “When only one or two types of antibodies (among IgG, IgM, IgA) were used in the seroprevalence study, seroprevalence was corrected upwards (and inferred IFR downwards) by 10% for each non-measured antibody.” (p.8) This approach is particularly objectionable when applied to test-adjusted seroprevalence results, since those estimates have already been adjusted to reflect sensitivity and specificity. Moreover, such an approach has never been used by any other epidemiologist or statistician, in the context of the COVID-19 pandemic or for any other purpose, and hence should not be applied in a meta-analysis without providing any compelling rationale for doing so.
-
Seroreversion. The manucript “explores” the issue of seroreversion using proportionality factors based on the timing of each seroprevalence study relative to the preceding peak of COVID-19 deaths. However, the manuscript provides no rationale for following this approach instead of the rigorous Bayesian methodology utilized in a preceding meta-analysis.[9] Moreover, the manuscript makes no reference to the findings of longitudinal studies of the evolution of antibodies in confirmed positive individuals, which have concluded that the degree of seroreversion is substantial for some assays and negligible for others.[25, 26]
-
Measurement of Fatalities. Data on COVID-19 fatalities should be obtained directly from official government sources, not from media reports, web aggregators, or Wikipedia. For example, the European Center for Disease Control has an online COVID-19 database with daily data on reported cases and fatalities for nearly every country in the world. Moreover, whenever possible, fatalities should be measured using official tabulations of case data (based on actual date of death) rather than real-time reports that may be relatively incomplete and subject to substantial revision over time. These issues are particularly relevant for assessing fatalities in nursing homes: If a patient tested positive for COVID-19 and died soon thereafter, investigation would be needed to determine whether the death resulted from COVID-19 or unrelated causes. To illustrate these issues, consider the manuscript’s estimate of IFR based on the U.S. national seroprevalence study of Kalish et al. (2021). As shown in table 1 and appendix table 2 of this manuscript, the U.S. CDC case database (accessed in February 2021) indicates a total of 103862 deaths for adults ages 70+ as of 04 July 2020. To determine the corresponding fatalities in U.S. nursing homes, however, the manuscript relies on a news summary dated 26 June 2020 that reported a total of 52428 nursing home deaths in 41 U.S. states.[27] Using that real-time report, manuscript infers a somewhat higher national total of 57291 nursing home deaths and hence 46571 deaths outside of nursing homes. By contrast, the U.S. CMS case database (accessed in August 2021) indicates 38239 deaths in U.S. nursing homes as of 05 July 2020.[28] Evidently, there were 65623 fatalities outside of nursing homes, implying a correspondingly higher IFR of 3·6% for U.S. community-dwelling adults ages 70+.
-
Developing Countries. The use of confirmed COVID-19 fatalities can be highly misleading in assessing IFRs of developing countries, where testing capacity has been much more limited than in Europe or North America. Consequently, in developing country locations, the measure of fatalities should include both confirmed and suspected COVID-19 cases, or alternatively, a measure of excess deaths relative to preceding years. Indeed, several recent studies of India have concluded that confirmed COVID-19 fatalities understate the true death toll by an order of magnitude.[29-31]
-
Younger Age Groups. The manuscript states that “the studies considered here offered a <br />
prime opportunity to assess IFR also in younger age strata” (p.9) even though such analysis <br />
had not been proposed in the protocol. Nevertheless, this secondary analysis is at odds with the key eligibility criterion of this meta-analysis, namely, seroprevalence studies with at least 1000 participants ages 70+. Indeed, imposing that eligibility criterion leads to the exclusion of numerous other seroprevalence studies that would be highly informative for analyzing IFRs of younger adults, with an unknown degree of bias associated with that exclusion.
-
Self-Citations. A meta-analysis is intended to serve as an objective synthesis of information extracted from existing studies. Consequently, methodological decisions and substantive claims should not be based solely on citations of the authors’ own prior work. For example, in discussing the preceding meta-analysis of Levin et al. (2020), the manuscript asserts that “almost all included studies came from hard-hit locations, where IFR may be substantially higher”, with a sole citation to Ioannidis (2021a). However, that assertion is clearly false: The meta-analysis of Levin et al. (2020) included locations such as Australia, New Zealand, Ontario, and Salt Lake City that experienced very few infections during the first wave of the pandemic. Similarly, the manuscript asserts that “selection bias for studies with higher seroprevalence and/or higher death counts may explain why their estimates for middle-aged and elderly are substantially higher than ours” (p.14), with a sole citation to Ioannidis (2021b).
References Cited Here:<br />
1. Ferguson N, Laydon D, Nedjati-Gilani G, et al. Report 9: Impact of non-pharmaceutical interventions (NPIs) to reduce COVID-19 mortality and healthcare demand2020.<br />
2. Mizumoto K, Kagaya K, Zarebski A, Chowell G. Estimating the asymptomatic proportion of coronavirus disease 2019 (COVID-19) cases on board the Diamond Princess cruise ship, Yokohama, Japan, 2020. eurosurveillance. 2020;25(10). doi:10.2807/1560-7917.ES.2020.25.10.2000180<br />
3. Salje H, Tran Kiem C, Lefrancq N, et al. Estimating the burden of SARS-CoV-2 in France. Science. 2020;369(6500):208-11. doi:10.1126/science.abc3517<br />
4. Verity R, Okell LC, Dorigatti I, et al. Estimates of the severity of coronavirus disease 2019: a model-based analysis. lancet infectious diseases. 2020;20(6):669-77. doi:10.1016/S1473-3099(20)30243-7<br />
5. Levin AT, Hanage WP, Owusu-Boaitey N, Cochran KB, Walsh SP, Meyerowitz-Katz G. Assessing the age specificity of infection fatality rates for COVID-19: systematic review, meta-analysis, and public policy implications. European Journal of Epidemiology. 2020;35(12):1123-38. doi:10.1007/s10654-020-00698-1<br />
6. Williamson EJ, Walker AJ, Bhaskaran K, et al. Factors associated with COVID-19-related death using OpenSAFELY. Nature. 2020. doi:10.1038/s41586-020-2521-4<br />
7. Mak JKL, Kuja-Halkola R, Wang Y, Hägg S, Jylhävä J. Frailty and comorbidity in predicting community COVID-19 mortality in the U.K. Biobank: The effect of sampling. Journal of the American Geriatrics Society. 2021;69(5):1128-39. doi:https://doi.org/10.1111/jgs...<br />
8. O’Driscoll M, Ribeiro Dos Santos G, Wang L, et al. Age-specific mortality and immunity patterns of SARS-CoV-2. Nature. 2021;590(7844):140-5. doi:10.1038/s41586-020-2918-0<br />
9. Brazeau N, Verity R, Jenks S, al. e. COVID-19 Infection Fatality Ratio: Estimates from Seroprevalence. 2020. doi:https://doi.org/10.25561/83545.<br />
10. Arora RK, Joseph A, Van Wyk J, et al. SeroTracker: a global SARS-CoV-2 seroprevalence dashboard. The Lancet Infectious Diseases. 2020. doi:10.1016/s1473-3099(20)30631-9<br />
11. Sullivan PS, Siegler AJ, Shioda K, et al. Severe Acute Respiratory Syndrome Coronavirus 2 Cumulative Incidence, United States, August 2020–December 2020. Clinical Infectious Diseases. 2021. doi:10.1093/cid/ciab626<br />
12. Merkely B, Szabo AJ, Kosztin A, et al. Novel coronavirus epidemic in the Hungarian population, a cross-sectional nationwide survey to support the exit policy in Hungary. Geroscience. 2020;42(4):1063-74. doi:10.1007/s11357-020-00226-9<br />
13. Perez-Saez J, Lauer SA, Kaiser L, et al. Serology-informed estimates of SARS-CoV-2 infection fatality risk in Geneva, Switzerland. The Lancet Infectious Diseases. doi:10.1016/S1473-3099(20)30584-3<br />
14. Stringhini S, Wisniak A, Piumatti G, et al. Seroprevalence of anti-SARS-CoV-2 IgG antibodies in Geneva, Switzerland (SEROCoV-POP): a population-based study. The Lancet. 2020;396(10247):313-9. doi:10.1016/s0140-6736(20)31304-0<br />
15. United Kingdom BioBank. UK Biobank SARS-CoV-2 Serology Study Weekly Report - 21 July 2020. 2020.<br />
16. Ward H, Atchison CJ, Whitaker M, et al. Antibody prevalence for SARS-CoV-2 in England following first peak of the pandemic: REACT2 study in 100,000 adults. medRxiv. 2020:2020.08.12.20173690. doi:10.1101/2020.08.12.20173690<br />
17. Hughes EC, Amat JAR, Haney J, et al. Severe Acute Respiratory Syndrome Coronavirus 2 Serosurveillance in a Patient Population Reveals Differences in Virus Exposure and Antibody-Mediated Immunity According to Host Demography and Healthcare Setting. The Journal of Infectious Diseases. 2020;223(6):971-80. doi:10.1093/infdis/jiaa788<br />
18. U.K. Public Health England. Weekly Coronavirus Disease 2019 (COVID-19) Surveillance Report, Week 32. 2020. <br />
19. Canadian Blood Services and COVID-19 Immunity Task Force. Final Results of Initial Canadian SARS-Cov-2 Seroprevalence Study Announced. 2020. <br />
20. Dodd RY, Xu M, Stramer SL. Change in Donor Characteristics and Antibodies to SARS-CoV-2 in Donated Blood in the US, June-August 2020. JAMA. 2020;324(16):1677-9. doi:10.1001/jama.2020.18598<br />
21. Bajema KL, Dahlgren FS, Lim TW, et al. Comparison of Estimated Severe Acute Respiratory Syndrome Coronavirus 2 Seroprevalence Through Commercial Laboratory Residual Sera Testing and a Community Survey. Clinical Infectious Diseases. 2020. doi:10.1093/cid/ciaa1804<br />
22. Boyce RM, Shook-Sa BE, Aiello AE. A Tale of 2 Studies: Study Design and Our Understanding of Severe Acute Respiratory Syndrome Coronavirus 2 Seroprevalence. Clinical Infectious Diseases. 2020. doi:10.1093/cid/ciaa1868<br />
23. Gelman A, Carpenter B. Bayesian analysis of tests with unknown specificity and sensitivity. Journal of the Royal Statistical Society: Series C (Applied Statistics). 2020;n/a(n/a). doi:10.1111/rssc.12435<br />
24. U.S. Food and Drug Administration. EUA authorized serology test performance. 2020.<br />
25. Dan JM, Mateus J, Kato Y, et al. Immunological memory to SARS-CoV-2 assessed for up to 8 months after infection. Science. 2021;371(6529):eabf4063. doi:10.1126/science.abf4063<br />
26. Muecksch F, Wise H, Batchelor B, et al. Longitudinal Serological Analysis and Neutralizing Antibody Levels in Coronavirus Disease 2019 Convalescent Patients. The Journal of Infectious Diseases. 2020;223(3):389-98. doi:10.1093/infdis/jiaa659<br />
27. Kaiser Family Foundation. This Week in Coronavirus: June 18 to June 25. 2020. <br />
28. U.S. Center for Medicare & Medicaid Services (CMS). COVID-19 Nursing Home Data. 2021. <br />
29. Anand A, Sandefur J, Subramanian A. Three New Estimates of India’s All-Cause Excess Mortality during the COVID-19 Pandemic. Center for Global Development. 2021. <br />
30. Deshmukh Y, Suraweera W, Tumbe C, et al. Excess mortality in India from June 2020 to June 2021 during the COVID pandemic: death registration, health facility deaths, and survey data. medRxiv. 2021:2021.07.20.21260872. doi:10.1101/2021.07.20.21260872<br />
31. Shewade HD, Parameswaran GG, Mazumder A, Gupta M. Adjusting Reported COVID-19 Deaths for the Prevailing Routine Death Surveillance in India. Frontiers in Public Health. 2021;9(1045). doi:10.3389/fpubh.2021.641991