 Methodology
 Open Access
 Published:
Metaanalysis with zeroevent studies: a comparative study with application to COVID19 data
Military Medical Research volume 8, Article number: 41 (2021)
Abstract
Background
Metaanalysis is a statistical method to synthesize evidence from a number of independent studies, including those from clinical studies with binary outcomes. In practice, when there are zero events in one or both groups, it may cause statistical problems in the subsequent analysis.
Methods
In this paper, by considering the relative risk as the effect size, we conduct a comparative study that consists of four continuity correction methods and another stateoftheart method without the continuity correction, namely the generalized linear mixed models (GLMMs). To further advance the literature, we also introduce a new method of the continuity correction for estimating the relative risk.
Results
From the simulation studies, the new method performs well in terms of mean squared error when there are few studies. In contrast, the generalized linear mixed model performs the best when the number of studies is large. In addition, by reanalyzing recent coronavirus disease 2019 (COVID19) data, it is evident that the doublezeroevent studies impact the estimate of the mean effect size.
Conclusions
We recommend the new method to handle the zeroevent studies when there are few studies in a metaanalysis, or instead use the GLMM when the number of studies is large. The doublezeroevent studies may be informative, and so we suggest not excluding them.
Background
Metaanalysis is a statistical method to synthesize evidence from a number of independent studies that addressed the same scientific questions [1, 2]. In clinical studies, experimental data are commonly composed of binary outcomes, and consequently, metaanalyses of binary data have attracted increasing attention in evidencebased medicine [3, 4]. For each study, an effect size is reported to quantify the treatment effect by comparing the event probabilities between the treatment group and the control group, including the odds ratio (OR), the relative risk (RR), and the risk difference (RD). In metaanalysis, when the studyspecific effect size is estimated based on a twobytwo contingency table, the zeroevent problem in one or both groups frequently occurs, which may cause an unexpected calculation complication in the statistical inference of the effect size. If the study involves a zero event in one group, we refer to it as a singlezeroevent study; and if the study involves zero events in both groups, we refer to it as a doublezeroevent study [5]. Vandermeer et al. [6] and Kuss [7] applied random sampling techniques and found that 30% of metaanalyses from the 500 sampled Cochrane reviews included one or more singlezeroevent studies, while 34% of the reviews involved at least one metaanalysis with a doublezeroevent study.
As a recent example, Chu et al. [8] conducted several metaanalyses to evaluate the effectiveness of physical distancing, face masks, and eye protection on the spread of three coronaviruses, which caused severe acute respiratory syndrome (SARS), Middle East respiratory syndrome (MERS) or coronavirus disease 2019, also known as COVID19 [9, 10]. Specifically, they considered RR as the effect size and applied the randomeffects model to pool the observed effect sizes with an inversevariance weight assigned to each study [11, 12]. As a result, for their metaanalysis on physical distancing, they concluded that the risk of infection will be significantly decreased with a further physical distance. We note, however, that there are 8 singlezeroevent studies and 7 doublezeroevent studies among a total of 32 studies. In particular for the 7 studies on COVID19 data, 4 of them are singlezeroevent studies and 2 of them are doublezeroevent studies. To escape the zeroevent problem, Chu et al. [8] excluded the doublezeroevent studies from their metaanalyses, which, however, may introduce an estimation bias to the overall effect size [7]. More recently, Xu et al. [13] revisited 442 metaanalyses with or without the doublezeroevent studies, and then by a comparative study, they concluded that the doublezeroevent studies do contain valuable information and should not be excluded from the metaanalysis.
Inspired by the aforementioned examples, we provide a selective review on the existing methods for metaanalysis that can handle the zeroevent studies. For ease of presentation, we will mainly focus on the randomeffects model with RR as the effect size, whereas the same comparison also applies to OR and RD. For more details on metaanalysis of OR and RD with the zeroevent studies, one may refer to [7] and the references therein, in which the author discussed the methods applicable to all the three effect sizes as well as some methods only applicable to one of them. For a given study, we let n_{1} be the number of samples in the treatment group with X_{1} being the number of events, and n_{2} be the number of samples in the control group with X_{2} being the number of events. Let also X_{1} follow a binomial distribution with parameters n_{1} and p_{1}>0, and X_{2} follow a binomial distribution with parameters n_{2} and p_{2}>0. We further assume that X_{1} and X_{2} are independent of each other. Then to estimate RR=p_{1}/p_{2}, the maximum likelihood estimator is known as
Note that \(\widehat {\text {RR}}\) is often rightskewed. To derive the statistical inference on RR, researchers frequently apply the log scale so that the resulting estimator can be more normally distributed. Specifically by Agresti [14], the approximate variance of \(\text {ln}\left (\widehat {\text {RR}}\right)\) is
By (1) and (2), when there are zero events in one or both groups, the classic method for estimating RR suffers from the zeroevent problem and will no longer be applicable.
To have a valid estimate of RR, originated from Haldane [15], one often recommends to add 0.5 to the counts of events and nonevents if some count is zero [16, 17]. This method is referred to a correction method and has been extensively used in metaanalysis to deal with the zeroevent studies. For further developments on the continuity correction, one may refer to Sweeting et al. [18], Carter et al. [19], and the references therein. On the other side, there are also statistical models without the continuity correction to handle metaanalysis with the zeroevent studies, such as the generalized linear mixed models [4, 20, 21].
The remainder of this paper is organized as follows. In “Methods with the continuity correction” section, we first review the randomeffects model and the existing methods with the continuity correction, and then propose a new method of the continuity correction for estimating RR. In “The generalized linear mixed models” section, we review the generalized linear mixed models for metaanalysis. In “Simulation studies” section, we conduct simulation studies to evaluate the performance of the reviewed methods and our new method. In “Application to COVID19 data” section, we apply all the well performed methods to a recent metaanalysis on COVID19 data for further evaluation of their performance. We then conclude the paper in “Discussion” and “Conclusions” sections with some interesting findings, and provide the supplementary materials in the Appendix.
Methods
Methods with the continuity correction
Suppose that there are k studies in the metaanalysis, and y_{i} for \(i=1, \dots,k\) are the observed effect sizes for each study. By DerSimonian and Laird [22], the randomeffects model can be expressed as
where θ is the mean effect size, ζ_{i} are the deviations of each study from θ, and ε_{i} are the sampling errors. We further assume that ζ_{i} are independent and identically distributed random variables from N(0,τ^{2}),ε_{i} are independent random errors from \(N(0,\sigma _{i}^{2})\), and that they are independent of each other. In addition, τ^{2} is referred to as the betweenstudy variance, and \(\sigma _{i}^{2}\) are referred to as the withinstudy variances.
For the randomeffects model in (3), by the inversevariance method the mean effect size θ can be estimated by
where \(w^{*}_{i} = 1/\left (\sigma _{i}^{2}+ \tau ^{2}\right)\) are the weights assigned to each individual study [23]. In metaanalysis, the withinstudy variances \(\sigma _{i}^{2}\) are routinely estimated by the variances of the observed effect sizes, denoted by var(y_{i}). While for the betweenstudy variance, DerSimonian and Laird [22] proposed the method of moments estimator as
where \(Q = \sum _{i} w_{i} \left (y_{i}  \sum _{i} w_{i} y_{i} / \sum _{i} w_{i}\right)^{2}\) is known as the Q statistic, and \(C = \sum _{i} w_{i}  \sum _{i} w_{i}^{2} / \sum _{i} w_{i}\) with \(w_{i} = 1/\sigma _{i}^{2}\) for \(i=1,\dots,k\).
We note, however, that the randomeffects model may suffer from the zeroevent problem. Taking RR as an example, if we apply the randomeffects model for metaanalysis, then the effect sizes y_{i} will be the observed ln(RR) values. Now for estimating ln(RR), if we plug in \(\widehat {\text {RR}}\) from formula (1) directly, then ln\(\left (\widehat {\text {RR}}\right)\) will not be well defined when the studies involve the zero events, and so is for the variance estimate of \(\text {ln}\left (\widehat {\text {RR}}\right)\) in formula (2). Consequently, without a valid estimate of the effect size and of its withinstudy variance, the randomeffects model cannot be applied to estimate the mean effect size by the inversevariance method. This shows that a correction on \(\widehat {\text {RR}}\) is often desired in metaanalysis with some studies involving zero events.
Existing methods with the continuity correction
Let c_{1}>0 and c_{2}>0 be two values for the continuity correction. To overcome the zeroevent problem, one common approach is to estimate p_{1} by (X_{1}+c_{1})/(n_{1}+2c_{1}) and estimate p_{2} by (X_{2}+c_{2})/(n_{2}+2c_{2}). Plugging them into (1) and (2), we have
Accordingly, the 95% confidence interval (CI) of RR is
For the values of c_{1} and c_{2} in (6), there are mainly three suggestions in the literature that are widely used for the randomeffects metaanalysis.

(i)
When c_{1}=c_{2}=0.5, it yields the Haldane estimator [15] as
$$ \begin{aligned} \widetilde{\text{RR}}_{\text{Haldane}} = \left\{ \begin{array}{ll} \frac{X_{1}+0.5}{n_{1}+1}\cdot \frac{n_{2}+1}{X_{2}+0.5} & ~~~~~~~~ X_{1}= 0~\text{or}~n_{1}, X_{2}=0~\text{or}~n_{2}, \\ \frac{X_{1}n_{2}}{n_{1}X_{2}} & ~~~~~~~~ \text{otherwise} \end{array} \right. \end{aligned} $$(8) 
(ii)
When c_{1}=n_{1}/(n_{1}+n_{2}) and c_{2}=n_{2}/(n_{1}+n_{2}), it yields the TACC estimator [18] as
$$ \begin{aligned} \widetilde{\text{RR}}_{\text{TACC}} = \left\{ \begin{array}{ll} \frac{X_{1}+c_{1}}{n_{1}+2c_{1}}\cdot \frac{n_{2}+2c_{2}}{X_{2}+c_{2}} & ~~~~~~~~ X_{1}= 0~\text{or}~n_{1}, X_{2}=0~\text{or}~n_{2}, \\ \frac{X_{1}n_{2}}{X_{2}n_{1}} & ~~~~~~~~ \text{otherwise} \end{array} \right. \end{aligned} $$(9)For the balanced case when n_{1}=n_{2}, the TACC estimator is equivalent to the Haldane estimator. Also to implement this estimator, one may apply metabin in the R package “meta” with the setting incr=“TACC” [24].

(iii)
When c_{1}=c_{2}=1, it yields the Carter estimator [19] as
$$\begin{array}{@{}rcl@{}} \widetilde{\text{RR}}_{\text{Carter}} = \frac{X_{1}+1}{n_{1}+2}\cdot \frac{n_{2}+2}{X_{2}+1} \end{array} $$(10)
Besides the continuity correction methods in family (6), another alternative is to estimate p_{1} by (X_{1}+c_{1})/(n_{1}+c_{1}) and estimate p_{2} by (X_{2}+c_{2})/(n_{2}+c_{2}). Then with c_{1}=c_{2}=0.5, it yields the Pettigrew estimator [25] as
and the 95% CI of RR as
Moreover, to avoid a zero standard error, Hartung and Knapp [26] suggested not to correct X_{1} and X_{2} when X_{1}=n_{1} and X_{2}=n_{2}.
A hybrid method with the continuity correction
Note that the existing methods are all constructed to first estimate p_{1} and p_{2}, and then take their ratio as an estimate of RR=p_{1}/p_{2}. Nevertheless, noting that p_{2} is in the denominator rather than in the numerator, inverting an optimal estimate for p_{2} may not necessarily yield an optimal estimate for 1/p_{2}. In this section, we propose a hybrid method that is to estimate p_{1} and 1/p_{2} directly, and then take their product to estimate RR.
For the estimation of p_{1}, we show in Appendix 1 that the mean squared error (MSE) of (X_{1}+c_{1})/(n_{1}+2c_{1}) is smaller than the MSE of (X_{1}+c_{1})/(n_{1}+c_{1}) in most settings. We thus consider to apply (X_{1}+c_{1})/(n_{1}+2c_{1}) to estimate p_{1} in RR. While to estimate the reciprocal of p_{2}, one may consider (n_{2}+2c_{2})/(X_{2}+c_{2}) as in (6). Or instead, another option can be to consider (n_{2}+c_{2})/(X_{2}+c_{2}) as originated in (??), see also [27] and [28] for more discussion. And if we take the latter one, then a hybrid estimator of RR can be constructed as
For the optimal values of c_{1} and c_{2} in (11), our simulation studies in Appendices 2 and 3 show that c_{1}=0.5 and c_{2}=0.5 are among the best options. In view of this, our new hybrid estimator is taken as follows:
whereas the 95% CI of RR is given as
Comparison of the continuity correction methods
In this section, we conduct a numerical study to compare the finite sample performance of the existing and new methods. For ease of presentation, we refer to the confidence intervals associated with (8), (9), (10), (??) and (12) as the Haldane interval, the TACC interval, the Carter interval, the Pettigrew interval, and the hybrid interval, respectively.
To generate the data, we let p_{2}=0.05, 0.15, 0.85 or 0.95, and p_{1}=p_{2}×RR with RR ranging from 0.2 to min{5,1/p_{2}}. We also consider different combinations of the sample sizes. For the sake of brevity, only the results for balanced samples with n_{1}=n_{2}=10 or 50 are presented, whereas the results for the unbalanced samples are postponed to Appendix 4. Recall that the Haldane and TACC intervals are the same when n_{1}=n_{2}, and we thus present the results for the Haldane interval only. With N=100,000 repetitions for each setting, we generate random numbers from the binomial distributions with parameters (p_{1},n_{1}) and (p_{2},n_{2}) to yield the estimates of RR and their CIs. We then compute the frequencies of the true RR falling in the CIs as the coverage probability estimates. Moreover, the expected lengths of the CIs on the log scale are computed by \(N^{1}\sum _{s=1}^{N}\left (\text {ln(UL}_{\text {s}})  \text {ln(LL}_{\text {s}})\right)\), where UL_{s} and LL_{s} are the upper and lower limits of the sth CI.
For p_{2}=0.05 or 0.15, the top four panels of Figs. 1 and 2 show that the Haldane interval is the most conservative interval in most settings, and it provides the longest expected lengths compared to the other three intervals. The Carter interval may have downward spikes in the left or right tail, although it leads to the shortest expected lengths. We also note that the simulation results of the Pettgrew interval and the hybrid interval are nearly the same. Their coverage probabilities and expected lengths are intermediate between those of the other two intervals in most settings.
From the bottom four panels of Figs. 1 and 2 with p_{2}=0.85 or 0.95, it is evident that the Haldane interval has a satisfactory performance in most settings with the coverage probabilities around the nominal level. In contrast, the Carter interval fails to provide enough large coverage probabilities in most settings, so does the Pettgrew interval when n_{1} and n_{2} are small. Note also that the coverage probabilities of the hybrid interval are comparable to the Haldane interval as long as p_{2} is not extremely large. Moreover, the hybrid interval yields shorter expected lengths than the Haldane interval.
To sum up, when p_{2} is small, the Pettgrew interval and the hybrid interval are less conservative than the Haldane interval in most settings. While for large p_{2}, the Haldane interval and the hybrid interval perform better than the Pettgrew interval in terms of coverage probability. In addition, the expected lengths of the hybrid interval are always shorter than the Haldane interval. This shows that the hybrid interval can serve as a good alternative for the interval estimation of RR.
The generalized linear mixed models
The generalized linear mixed models (GLMMs) are extensions of the generalized linear model, which include both the fixed and random effects as linear predictors [14]. Different types of the GLMMs have been proposed in the literature including a few reviews and comparison studies [4, 29]. Among the existing models, the bivariate GLMM has been well recognized and being recommended for estimating RR in metaanalysis [20].
Let p_{i1} and p_{i2} be the event probabilities in the treatment and control groups of the ith study, respectively. The bivariate GLMM is represented as
where g(·) is the link function, Ω_{1} and Ω_{2} are the fixed effects, and the random effects are given by
The mean effect size based on model (14) was defined as
where E(p_{1}) and E(p_{2}) are the mean event probabilities in the control and treatment groups, g^{−1}(·) is the inverse function of the link, and ϕ(·) is the probability density function of the standard normal distribution [30]. For the logit link, Zeger et al. [31] proposed an approximate formula \(E\left (p_{j}\right)\approx \text {expit}\left (\Omega _{j} /\sqrt {1+C^{2}\tau _{j}^{2}}\right)\) with \(C = 16\sqrt {3}/(15\pi)\). For the probit link, \(E\left (p_{j}\right)=\Phi \left (\Omega _{j} /\sqrt {1+\tau _{j}^{2}}\right)\), where j=1 or 2, and Φ(·) is the cumulative distribution function of the standard normal distribution. While for the other links, there does not exist a closed form of formula (15) and so a numerical approximation is often needed [32].
For the parameter estimation in model (14), Jackson et al. [4] provided a detailed introduction for the implementation based on the R package “lme4” in their model 6. Alternatively, one may also apply the function meta.biv in the R package “altmeta” maintained by Lin and Chu [33], in which the 95% CI of RR can be derived by the bootstrap resampling method.
Results
Simulation studies
In this section, we compare the performance of the reviewed methods on handling metaanalysis with the zeroevent studies, including the continuity correction methods and the generalized linear mixed models. Among the existing continuity correction methods, we note that the Haldane and TACC estimators are comparable and among the best when estimating the mean effect size, in contrast to the other two methods including the Carter and Pettigrew estimators. Hence, for the sake of brevity, we only present the results of the Haldane and TACC estimators in the main text but provide the simulation results for all four methods in Appendix 5. Besides the Haldane and TACC estimators, we also consider the newly introduced hybrid estimator and the GLMM with the logit link for further comparison.
To conduct the metaanalysis, we consider k=3, 6 and 12 as three different numbers of studies. Also by (3), we let θ=ln(RR) be the mean effect size that ranges from ln(0.2) to ln(5), and then generate the random effects ζ_{i} from N(0,τ^{2}) with τ^{2}= 0.25 or 1. Next, we randomly generate n_{i2} from the lognormal distribution based on the assumption that \({\ln }(n_{i2}) \overset {\text {ind}}{\sim } N(3.35, 1.00)\) [34]. It is also assumed by [34] that the ratios between n_{i1} and n_{i2} follow the uniform distribution with values from 0.84 to 2.04. In addition, we generate the event probabilities of the control group p_{i2} from the uniform distribution with values from 0.01 to min{0.99,1/exp(θ)}. Then accordingly, the event probabilities of the treatment group are given by p_{i1}=exp(θ+ζ_{i})p_{i2}, where exp(θ+ζ_{i})p_{i2}≥1 will be discarded. Finally, we generate X_{i1} and X_{i2} from the binomial distributions with parameters (n_{i1},p_{i1}) and (n_{i2},p_{i2}), respectively. Note that the data will be regenerated if the number of events or nonevents in one group are both zero. Finally, with N=10,000 repetitions for each setting, we compute the mean squared errors (MSEs) between the estimated RR and the true RR to evaluate the accuracy of the methods.
From the top two panels of Fig. 3, it is evident that the three continuity correction methods perform much better than the GLMM in nearly all settings when k is small. Moreover, the hybrid estimator is consistently better than the Haldane and TACC estimators. The middle two panels show that, when k is moderate, the three continuity correction methods still perform better than the GLMM in most settings. Finally, the bottom two panels indicate that the GLMM performs the best in most settings when k is large. To conclude, the accuracy of the different methods depends on the number of studies. In particular, for metaanalysis with few studies, the randomeffects model with the hybrid estimator is more reliable for handling the zeroevent studies than the other methods; and for metaanalysis with large studies, we recommend the GLMM to handle the randomeffects metaanalysis.
Application to COVID19 data
As mentioned earlier, Chu et al. [8] conducted a systematic review that revealed the connections of physical distancing, face masks, and eye protection with the transmission of SARS, MERS, and COVID19. It is noteworthy that their analytical results have attracted more and more attention. As an evidence, their paper has received a total of 1236 citations in Google Scholar as of 16 March 2021. In this section, we propose to reanalyze COVID19 data and compare the performance of the different methods with or without the doublezeroevent studies, including the Haldane estimator, the TACC estimator, the hybrid estimator, and the GLMMs.
Note that the treatment group represents a further physical distance and the control group represents a shorter physical distance. As shown in the top panel of Fig. 4, [8] applied the randomeffects model with the Haldane estimator and removed the doublezeroevent studies from their metaanalysis. The overall effect size of 0.15 with the 95% CI being [0.03,0.73] indicates that the infection risk will be significantly reduced with a further physical distance. The middle panel of Fig. 4 reports that the randomeffects model with the TACC estimator yields the overall effect size of 0.12 with the 95% CI being [0.03,0.50]. Moreover, the bottom panel of Fig. 4 shows that the randomeffects model with the hybrid estimator yields the overall effect size of 0.13 with the 95% CI being [0.03,0.72]. Note also that the studyspecific CIs here are always narrower than the CIs in the top panel, which coincides with the simulation results that the expected lengths of the CI associated with the hybrid estimator are shorter than the Haldane estimator. In addition, the GLMM in (14) does not provide the estimates of the studyspecific effect sizes, so the results are listed as follows. By the bootstrap resampling with 1000 replicates, the GLMM with the logit link yields the overall effect size of 0.20 with the 95% bootstrap CI being [0.05,0.55]. Also, the GLMM with the probit link yields the overall effect size of 0.18 with the 95% CI being [0.04,0.55].
To reanalyze COVID19 data, we now include the doublezeroevent studies. The top panel of Fig. 5 shows that the randomeffects model with the Haldane estimator yields the overall effect size of 0.22 with 95% CI being [0.06,0.82]. The middle panel of Fig. 5 presents that the randomeffects model with the TACC estimator provides the overall effect size of 0.18 with the 95% CI being [0.06,0.57]. While for the hybrid estimator, it is shown by the bottom panel that the overall effect size is 0.21 with 95% CI being [0.05, 0.81]. At last, the GLMM with the logit link provides the overall effect size of 0.29 with the 95% CI being [0.10,0.64], and the GLMM with the probit link provides the overall effect size of 0.28 with the 95% CI being [0.10,0.56].
Discussion
To handle the zeroevent studies in metaanalysis of binary data, researchers often apply the randomeffects model with the continuity correction, or instead, the GLMMs. From the simulation results, we note that the performance of the different methods depends on the number of studies. For metaanalysis with few studies, the randomeffects model with the continuity correction is able to perform better than the GLMM, especially the hybrid continuity correction. We also note that the hybrid continuity correction can yield a reliable confidence interval for a single RR. Although the continuity correction does show some advantages, it should be used with caution since an arbitrary correction may lead to a bias or even reverse the result of a metaanalysis, especially when the numbers of samples in the two groups are fairly unbalanced [7, 13]. When the number of studies is large, the GLMM is preferable to the randomeffects model with the continuity correction. In other words, the performance of the GLMM relies on a sufficient number of studies [35]. Also as shown in Ju et al. [34], the GLMM also requires enough total events in the two groups, e.g., larger than 10.
Besides the randomeffects model we have compared, it is noteworthy that there are also other models for metaanalysis that can handle the zeroevent studies including, for example, the betabinomial model [36–38]. Most metaanalyses with rare events have a small degree of heterogeneity, and so the commoneffect model may be more suitable than the randomeffects model [39]. In addition, Li and Rice [40] showed that the fixedeffects model can also provide an accurate CI for metaanalysis of OR with the zeroevent studies. Apart from that, it is also noteworthy that the fixedeffects model can serve as a convincing model for metaanalysis with few studies [12, 41–43]. As a future work, it can be interesting to investigate the best model for metaanalysis with few studies which include the zeroevent studies as well.
For the doublezeroevent studies in metaanalysis, we have shown by reanalyzing COVID19 data that they do impact the estimate of the mean effect size, and so they may not be uninformative. As noted by Friedrich et al. [44], including the doublezeroevent studies moves the mean effect size estimate toward the direction of the null hypothesis. If one arbitrarily excluded the informative doublezeroevent studies, there would be a risk of overstating the treatment effect such that the conclusion would be less reliable. As recommended by the literature [7, 13] and the references therein, we suggest including the doublezeroevent studies in metaanalysis.
Apart from model comparison, the selection of effect sizes has attracted more and more attention in the literature. In particular, there is a recent debate on the choice of RR or OR in clinical epidemiology, in which a number of important properties of RR or OR together with their pros and cons were discussed including, for example, portability and collapsibility [45–47]. In view of this, we have also analyzed COVID19 data with OR being the effect size and present the results in Appendix 6 with R code in Appendix 7. To handle the zeroevent studies, we apply four methods that have been reviewed in this paper, including Haldane’s continuity correction, TACC, the GLMM, and the empirical continuity correction proposed by Sweeting et al. [18]. For more techniques on metaanalysis of OR with the zeroevent studies, one may refer to [4, 7, 18, 29, 34] and the references therein.
Conclusions
In this paper, we revisited the existing methods that are widely used to handle the zeroevent problem in metaanalysis of binary data, in particular with RR as the effect size which is also known as the risk ratio. For the methods with the continuity correction, we reviewed four existing estimators of RR and also introduced a new hybrid estimator with their applications to the randomeffects model. Apart from those, the GLMM was also included which is a stateoftheart method without the continuity correction. By a comparative study and also a real data analysis on COVID19 data, we found that the randomeffects model with the hybrid estimator can serve as a more reliable method for handling the zeroevent studies when there are few studies in a metaanalysis, and recommend using the GLMM when the number of studies is large. This paper also provides a useful addition to Chu et al. [8], and meanwhile, it also calls for further observational studies in this field.
Availability of data and materials
Not applicable.
Abbreviations
 OR:

Odds ratio
 RR:

Relative risk
 RD:

Risk difference
 SARS:

Severe acute respiratory syndrome
 MERS:

Middle East respiratory syndrome
 COVID19:

Coronavirus disease 2019
 CI:

Confidence interval
 MSE:

Mean squared error
 GLMMs:

Generalized linear mixed models
 TACC:

Treatment arm continuity correction
References
Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to MetaAnalysis. Chichester, UK: John Wiley & Son; 2011.
Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better?Mil Med Res. 2020; 7:7.
Davey J, Turner RM, Clarke MJ, Higgins JPT. Characteristics of metaanalyses and their component studies in the cochrane database of systematic reviews: a crosssectional, descriptive analysis. BMC Med Res Methodol. 2011; 11:160.
Jackson D, Law M, Stijnen T, Viechtbauer W, White IR. A comparison of seven random–effects models for metaanalyses that estimate the summary odds ratio. Stat Med. 2018; 37(7):1059–85.
Ren Y, Lin L, Lian Q, Zou H, Chu H. Realworld performance of metaanalysis methods for doublezeroevent studies with dichotomous outcomes using the cochrane database of systematic reviews. J Gen Intern Med. 2019; 34(6):960–8.
Vandermeer B, Bialy L, Hooton N, Hartling L, Klassen TP, Johnston BC, Wiebe N. Metaanalyses of safety data: a comparison of exact versus asymptotic methods. Stat Methods Med Res. 2009; 18(4):421–32.
Kuss O. Statistical methods for metaanalyses including information from studies without any events–add nothing to nothing and succeed nevertheless. Stat Med. 2015; 34(7):1097–116.
Chu DK, Akl EA, Duda S, Solo K, Yaacoub S, Schünemann HJ, study authors CSURGES. Physical distancing, face masks, and eye protection to prevent persontoperson transmission of SARSCoV2 and COVID19: a systematic review and metaanalysis. Lancet. 2020; 395(10242):1973–87.
Jin YH, Cai L, Cheng ZS, Cheng H, Deng T, Fan YP, et al. A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019ncov) infected pneumonia (standard version). Mil Med Res. 2020; 7:4.
Jin YH, Zhan QY, Peng ZY, Ren XQ, Yin XT, Cai L, et al. Chemoprophylaxis, diagnosis, treatments, and discharge management of COVID19: An evidencebased clinical practice guideline (updated version). Mil Med Res. 2020; 7:41.
Borenstein M, Hedges LV, Higgins JP, Rothstein HR. A basic introduction to fixedeffect and randomeffects models for meta–analysis. Res Synth Methods. 2010; 1(2):97–111.
Lin E, Tong T, Chen Y, Wang Y. Fixedeffects model: the most convincing model for metaanalysis with few studies. Preprint at https://arxiv.org/abs/2002.04211. 2020.
Xu C, Li L, Lin L, Chu H, Thabane L, Zou K, Sun X. Exclusion of studies with no events in both arms in metaanalysis impacted the conclusions. J Clin Epidemiol. 2020; 123:91–9.
Agresti A. Categorical Data Analysis, 2nd Edition. Hoboken: John Wiley & Son; 2003.
Haldane JB. The estimation and significance of the logarithm of a ratio of frequencies. Ann Hum Genet. 1956; 20(4):309–11.
Schwarzer G. meta: An r package for metaanalysis. R News. 2007; 7:40–5.
Weber F, Knapp G, Ickstadt K, Kundt G, Glass Ä. Zero–cell corrections in random–effects meta–analyses. Res Synth Methods. 2020; 11(6):913–9.
Sweeting MJ, Sutton AJ, Lambert PC. What to add to nothing? use and avoidance of continuity corrections in meta–analysis of sparse data. Stat Med. 2004; 23(9):1351–75.
Carter RE, Lin Y, Lipsitz SR, Newcombe RG, Hermayer KL. Relative risk estimated from the ratio of two median unbiased estimates. J Royal Stat Soc: Ser C Appl Stat. 2010; 59(4):657–71.
Chu H, Nie L, Chen Y, Huang Y, Sun W. Bivariate random effects models for metaanalysis of comparative studies with binary outcomes: methods for the absolute risk difference and relative risk. Stat Methods Med Res. 2012; 21(6):621–33.
Chen Y, Hong C, Ning Y, Su X. Meta–analysis of studies with bivariate binary outcomes: a marginal beta–binomial model approach. Stat Med. 2016; 35(1):21–40.
DerSimonian R, Laird N. Metaanalysis in clinical trials. Control Clin Trials. 1986; 7(3):177–88.
Laird NM, Mosteller F. Some statistical methods for combining experimental results. Int J Technol Assess Heal Care. 1990; 6(1):5–30.
Balduzzi S, Rücker G, Schwarzer G. How to perform a metaanalysis with R: a practical tutorial. Evid Based Ment Health. 2019; 22(4):153–60.
Pettigrew HM, Gart JJ, Thomas DG. The bias and higher cumulants of the logarithm of a binomial variate. Biometrika. 1986; 73(2):425–35.
Hartung J, Knapp G. A refined method for the meta–analysis of controlled clinical trials with binary outcome. Stat Med. 2001; 20(24):3875–89.
Fattorini L. Applying the HorvitzThompson criterion in complex designs: a computerintensive perspective for estimating inclusion probabilities. Biometrika. 2006; 93(2):269–78.
Seber GAF. Statistical Models for Proportions and Probabilities. Heidelberg: Springer; 2013.
Bakbergenuly I, Kulinskaya E. Metaanalysis of binary outcomes via generalized linear mixed models: a simulation study. BMC Med Res Methodol. 2018; 18(1):70.
McCullagh P. Sampling bias and logistic models. J R Stat Soc Ser B Stat Methodol. 2008; 70(4):643–77.
Zeger SL, Liang KY, Albert PS. Models for longitudinal data: a generalized estimating equation approach. Biometrics. 1988; 44(4):1049–60.
Lin L, Chu H. Metaanalysis of proportions using generalized linear mixed models. Epidemiology. 2020; 31(5):713–7.
Lin L, Chu H. altmeta: Alternative MetaAnalysis Methods. 2020. https://CRAN.Rproject.org/package=altmeta.
Ju J, Lin L, Chu H, Cheng LL, Xu C. Laplace approximation, penalized quasilikelihood, and adaptive gausshermite quadrature for generalized linear mixed models: towards metaanalysis of binary outcome with sparse data. BMC Med Res Methodol. 2020; 20(1):152.
Gronsbell J, Hong C, Nie L, Lu Y, Tian L. Exact inference for the random–effect model for meta–analyses with rare events. Stat Med. 2020; 39(3):252–64.
Sarmanov O. Generalized normal correlation and twodimensional fréchet classes. Sov Math Dokl. 1966; 7:596–9.
Chen Y, Luo S, Chu H, Su X, Nie L. An empirical Bayes method for multivariate metaanalysis with an application in clinical trials. Commun Stat Theory Methods. 2014; 43(16):3536–51.
Luo S, Chen Y, Su X, Chu H. mmeta: an R package for multivariate metaanalysis. J Stat Softw. 2014; 56(11):11.
Jia P, Lin L, Kwong JSW, Xu C. Many metaanalyses of rare events in the cochrane database of systematic reviews were underpowered. J Clin Epidemiol. 2021; 131:113–22.
Li QK, Rice K. Improved inference for fixed–effects meta–analysis of 2×2 tables. Res Synth Methods. 2020; 11(3):387–96.
Bender R, Friede T, Koch A, Kuss O, Schlattmann P, Schwarzer G, Skipka G. Methods for evidence synthesis in the case of very few studies. Res Synth Methods. 2018; 9(3):382–92.
Rice K, Higgins JP, Lumley T. A re–evaluation of fixed effect(s) meta–analysis. J R Stat Soc Ser A Stat Methodol. 2018; 181(1):205–27.
Yang K, Kwan HY, Yu Z, Tong T. Model selection between the fixedeffects model and the randomeffects model in metaanalysis. Stat Interface. 2020; 13(4):501–10.
Friedrich JO, Adhikari NK, Beyene J. Inclusion of zero total event trials in metaanalyses maintains analytic consistency and incorporates all available data. BMC Med Res Methodol. 2007; 7:5.
Doi SA, FuruyaKanamori L, Xu C, Lin L, Chivese T, Thalib L. Questionable utility of the relative risk in clinical research: a call for change to practice. J Clin Epidemiol. 2020. https://doi.org/10.1016/j.jclinepi.2020.08.019.
Xiao M, Chen Y, Cole SR, MacLehose R, Richardson D, Chu H. Is OR “portable” in metaanalysis? Time to consider bivariate generalized linear mixed model. 2020. Preprint at https://www.medrxiv.org/content/10.1101/2020.11.05.20226811v1.
Doi SA, FuruyaKanamori L, Xu C, Chivese T, Lin L, Musa OA, Hindy G, Thalib L, Harrell Jr FE. The OR is “portable” but not the RR: time to do away with the log link in binomial regression. J Clin Epidemiol. 2021. https://doi.org/10.13140/RG.2.2.31631.10407.
Acknowledgements
The authors sincerely thank the Editor, Associate Editor, and two anonymous reviewers for their insightful comments and suggestions.
Funding
This study was supported by grants awarded to TieJun Tong from the General Research Fund (HKBU12303918), the National Natural Science Foundation of China (1207010822), and the Initiation Grants for Faculty Niche Research Areas (RCIGFNRA/1718/13, RCFNRAIG/2021/SCI/03) of Hong Kong Baptist University.
Author information
Authors and Affiliations
Contributions
TJT, JJW, EXL, and XTZ reviewed the literature and designed the methods. TJT, JJW, and JDS conducted the simulation studies. TJT, JJW, KY, and ZLH conducted the experiments and analyzed the real data. All authors contributed to the manuscript preparation. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Additional file 1
Supplementary information: Appendix
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Wei, JJ., Lin, EX., Shi, JD. et al. Metaanalysis with zeroevent studies: a comparative study with application to COVID19 data. Military Med Res 8, 41 (2021). https://doi.org/10.1186/s40779021003316
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s40779021003316
Keywords
 Continuity correction
 Coronavirus disease 2019 data
 Metaanalysis
 Relative risk
 Zeroevent studies