Evaluation of Heterogeneity and Heterogeneity Interval Estimators in Random-Effects Meta-Analysis of the Standardized Mean Difference in Education and Psychology

Meta-analyses are conducted to synthesize the quantitative results of related studies. The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population. This distribution is often assumed to be normal with a mean and variance. The populat...

Full description

Saved in:
Bibliographic Details
Published in:Psychological methods 2020-06, Vol.25 (3), p.346-364
Main Authors: Boedeker, Peter, Henson, Robin K.
Format: Article
Language:English
Subjects:
Bayesian Analysis
Education
Effect Size (Statistical)
Estimation
Homogeneity of Variance
Humans
Meta Analysis
Meta-Analysis as Topic
Population (Statistics)
Psychology
Sample Size
Simulation
Statistical Probability
Statistics as Topic
Uncertainty
Citations: Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Meta-analyses are conducted to synthesize the quantitative results of related studies. The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population. This distribution is often assumed to be normal with a mean and variance. The population variance, also called heterogeneity, can be estimated numerous ways. Research exists comparing subsets of heterogeneity estimators over limited conditions. Additionally, heterogeneity is a parameter estimated with uncertainty. Various methods exist for heterogeneity interval estimation, and similar to heterogeneity estimators, these evaluations are limited. The current simulation study examined the performance of Bayesian (with 5 prior specifications) and non-Bayesian estimators over conditions found after a review of meta-analyses of the standardized mean difference in education and psychology research. Three simulation conditions were varied: (a) number of effect sizes per meta-analysis, (b) true heterogeneity, and (c) sample size per effect size within each meta-analysis. Estimators were evaluated over average bias and means square error. Methods of interval estimation were then evaluated with the estimators found to operate optimally. Interval estimators were evaluated based on coverage probability, interval width, and coverage of the estimated value. Overall, the Paule and Mandel estimator, in conjunction with the Jackson method of interval estimation, is recommended if no knowledge exists with regards to the expected value of heterogeneity when synthesizing the standardized mean difference effect size. If heterogeneity is expected to be small (e.g., < .075), then REML with the profile likelihood interval estimator is recommended. Sensitivity analysis evaluating differences in substantive conclusions with a suite of heterogeneity estimators, such as Paule and Mandel, REML, and Hedges and Olkin, is recommended. Translational Abstract The random-effects meta-analysis model is based on the assumption that a distribution of true effects exists in the population, often assumed normal. The population variance, also called heterogeneity, and its interval can be estimated numerous ways. We compared 16 estimators of heterogeneity (Bayesian and non-Bayesian) with regard to bias and mean square error and associated heterogeneity interval estimators with regard to coverage and interval width over a broad set of conditions found in educational and psychological meta
ISSN:1082-989X
1939-1463
DOI:10.1037/met0000241