Meta-analysis is a widely used method to synthesize quantitative research. Traditionally, the main purpose of meta-analyses has been to estimate the combined effect size of multiple replications. However, there is more to meta-analysis than merely obtaining an overall effect size estimate. For instance, one can assess whether study-level characteristics have an influence on the effect of interest (e.g., age or ethnicity of respondents, types of measures used for the dependent and independent variables).
The course consists of four sessions. The sessions consist of a mix of lectures and assignments. The assignments apply what we have discussed in the lecture to your own research field. The assignments add up to a final assignment in which you conduct a small meta-analysis in your own research field. The deliverable is a short report (max. 2,000 words) describing the method, presenting the results, and interpreting them (i.e., a draft of a publishable meta-analysis).
The main objectives of this course are:
- To gain knowledge on how to search the literature for relevant studies.
- To know how to obtain the right information from these studies.
- To learn how to perform a meta-analysis and interpret the results.
- To be able to perform additional analyses, such as subgroup analysis, moderator analysis, and publication bias analysis, and interpret the results.
- After completing the course, you will be able to conduct and write up a publishable meta-analysis.
- Although not a main goal of this course, during the course you will most certainly become more critical in reviewing other studies for their methodology and proper reporting thereof.
No prior knowledge about, or experience with meta-analyses is needed for this course. However, you should have basic methodological knowledge, allowing you to evaluate a research design and be able to interpret results of a study in terms of its effect size (e.g., correlation, regression coefficient, Cohen’s d).
During the course, we use Meta-Essentials (a meta-analysis tool freely available from www.meta-essentials.com) for demonstrations and assignments. No prior experience is required to use this tool. You are free to use any other tool if they are already familiar with it (like the meta or metafor packages for R).
In the first session, we will start with an introduction in what meta-analysis actually is (and what it is not). Subsequently, we begin at the very beginning of research synthesis: how to search for relevant studies, minimize publication bias, and retrieve the necessary information from studies.
The second session is about how to derive a combined effect size and the different methodological choices you have to make (weighting of studies, what to do if studies report multiple relevant effect sizes, etc.). We also discuss the potential influence of publication bias on your meta-analysis.
During the third session, we dive into the topic of analyzing differences between effect sizes. We discuss various techniques to explain heterogeneity between studies (subgroup-analysis and moderator analysis).
In the fourth session, we discuss some more advanced meta-analytic methods (network meta-analysis and meta-analytic structural equation modelling, MASEM) and how to report your meta-analysis in a research paper.
Session 1 - Search for relevant studies and retrieve the necessary information
• Define a hypothesis for which you want to synthesize the literature and perform a meta-analysis.
• Select three empirical papers relating to the previously defined hypothesis that you find exemplary in terms of their methodological quality.
• What are ‘relevant studies’?
• How to ‘search’ for relevant studies?
• What is the ‘necessary information’?
Session 2 - Derive a combined effect size
• Formulate your inclusion criteria and search strategy.
• Find at least five studies that meet your inclusion criteria.
• Derive the effect size and variance or standard error of each of these studies.
• Fixed effect vs. Random effects
• Confidence interval vs. Prediction interval
• Publication bias
Session 3 – Analyzing differences between effect sizes
• Find at least five more studies that meet your inclusion criteria.
• Enter the studies you found in Meta-Essentials (or another meta-analysis tool if you are already familiar with it).
• Subgroup analysis
• Moderator analysis (meta-regression)
Session 4 – More advanced methods & Your meta-analysis in a research paper
• Identify potential subgroups and/or moderators of your hypothesis.
• Code these subgroups and/or moderators for your studies.
• Network meta-analysis
• Meta-analytical structural equation modelling (MASEM)
• Reporting guidelines
Henk van Rhee is a PhD candidate at the Department of Technology and Operations Management of the Rotterdam School of Management (RSM), Erasmus University Rotterdam (EUR). His research focusses on work environments; moreover he is one of the developers of Meta-Essentials, a free and easy-to-use tool for meta-analysis. Visit www.meta-essentials.com for more information.