The Meta-analytic approach of reliability generalization

Abstract

Sentences such as «the test reliability is 0.80» are wrong. It is more appropriate to say «the test scores reliability in a given application of it is 0.80». The meta-analytic approach of reliability generalization pretends to show that reliability is an empirical property that varies from one test application to another. This recent meta-analytic approach is helping to make the researchers aware of the importance of reporting reliability estimates obtained from the own data and, of avoiding the malpractice of inducting reliability coefficients from other studies and previous applications of the test. The stages to carry out a reliability generalization study are presented: (a) formulating the problem, (b) searching for the studies, (c) coding studies, (d) statistical analysis and interpretation, and (e) publication. An updated overview of the statistical problems of this approach: (a) to transform versus not to transform the reliability coefficients, (b) to weight versus not to weight the coefficients, (c) how to manage statistical dependency among the coefficients, and (d) which statistical model is the most appropriate (fixed-, random-, and mixed-effects).