Effects of Two Modes of Training Observers in the Systematic Assessment of Patient Functioning in Mental Institutions

Abstract

The comparative effectiveness of two time-limited modes of training observers to code the behavior of clients in residential treatment programs on the Time-Sample Behavioral Checklist (TSBC) was evaluated. In addition, the susceptibility of training procedures to consensual observer drift and the predictability of TSBC mastery from trainee characteristics were also examined. Two equated groups of undergraduate students (N = 15 each) participated in full-time training on the TSBC and another instrument for 27 days, followed by criterion testing in vivo and on videotapes. One group was trained by experienced personnel using procedures known to be effective but potentially subject to consensual observer drift. The other group was trained using a previously untested set of written and videotape procedures that do not rely on experienced personnel. Comparative effectiveness an observer drift were evaluated by multivariate and univariate ANOVAs on mastery scores reflecting both pattern agreement and level differences between each trainee and criterion codings. The new, more efficient training procedures were found to be as effective as the original training procedures in the degree of mastery achieved by trainees. Original training procedures were found to be resistant to consensual observer drift, with such a phenomena appearing in only one instance of 156 opportunities. The high degree of mastery achieved by the original training procedures. No meaningful predictions of coding mastery were found, with only one trainee procedures. No meaningful predictions of coding mastery criteria. The results document procedures that are both efficient and resistant to invaldity for training observers in the use of multidimensional observational systems, as well as providing guidelines for the development of standardized procedures.