Meta-Showdown Explorer


What setting describes best the analyzed research environment?

Basic settings

Note: The results of H0 are always displayed and compared to one H1, which is selected here.

Please select all conditions that are plausible for the meta-analyzed research environment. Check at least one for each dimension!

Basic settings

Note: The results of H0 are always displayed and compared to one H1, which is selected here.

Good performance is defined as ...

Fields without a value are not evaluated; all other fields are combined with a logical AND (i.e., all entered conditions must be true to result in a good performance). As p-curve does not provide CIs, it is never positively evaluated if you enter a number there.

Advanced options

For each condition, this app provides 10 demo data sets (the data sets are not simulated on the fly, as this would need too much computing time).

Output options

Typical funnel plots for this condition

<-- Slide through demo data set 1 to 10 to see some other exemplary funnel plots for this condition.
  • Blue triangle is the region of non-significance; dotted black triangle is the funnel of the naive random-effects meta-analysis.
  • The red dot at the bottom shows the true effect size. Blue dots show the naive random-effects estimate, and PET and PEESE estimates, if selected.

Is there an effect or not?

Note: H0 is rejected if the p-value is < .05 and the estimate is in the expected direction.

Under H0

If in reality there is no effect: What is the probability that a method falsely concludes 'There is an effect'?

Under H1

If in reality there is an effect: What is the probability that a method detects it?

RE = random effects meta-analysis, TF = trim-and-fill, PET = precision effect test, PEESE = precision effect estimate with standard errors, PET-PEESE = conditional estimator, 3PSM = three parameter selection model, 4PSM = four parameter selection model, WAAP = weighted average of adequately powered studies, WLS = Weigthed least squares estimator, WAAP-WLS = conditional estimator

Bias-corrected estimates of the true effect

Note: Negative estimates are set to zero.

Under H0

Under H1

RE = random effects meta-analysis, TF = trim-and-fill, PET = precision effect test, PEESE = precision effect estimate with standard errors, PET-PEESE = conditional estimator, 3PSM = three parameter selection model, 4PSM = four parameter selection model, WAAP = weighted average of adequately powered studies, WLS = Weigthed least squares estimator, WAAP-WLS = conditional estimator
Horizontal error bars are 95% quantiles (i.e., 95% of simulated replications were in that range).

Under which conditions does a method perform well?

This app provides all results for the publication:

Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for Bias in Psychology: A Comparison of Meta-Analytic Methods. Advances in Methods and Practices in Psychological Science, 2. doi:10.1177/2515245919847196 Preprint available at https://osf.io/rf3ys/

If you refer to this app, please cite this publication.

The full R code for the app is at Github.

Version history

  • 1.0 (2019/03/01): Version at journal's acceptance
  • 0.2 (2018/02/01): Revised release (with submission of revision of the paper). Based on tagged Github version 0.2.
  • 0.1 (2017/05/18): Initial release (with submission of paper / release of preprint). Based on tagged Github version 0.1.