A Year in God’s Time Response Shift-Bias and Retrospective Pretest Methodologies in Measuring Spiritual Impact

Abstract

Christ Centered Organizations (CCOs) are increasingly attempting to measure the spiritual impact of their programs. While this presents exciting opportunities for CCOs to move from anecdote to evidence, the subjective nature of many of the metrics used in traditional longitudinal designs presents potential biases that need to be examined and mitigated. Based upon longitudinal and retrospective survey data, as well as follow-up focus groups, from a spiritual community, this paper presents evidence of response shift bias occurring in social and spiritual impact metrics. Response shift bias occurs “when an individual’s internal frame of reference about the construct being measured changes between the pretest and the posttest” (Little et al. 2019, 2). To mitigate this risk, this paper proposes integrating retrospective pretest methodologies in impact evaluations involving spirituality metrics.