The purpose of this small pilot study was three-fold: (a) to begin development of a coding scheme for supervisor and therapist skill acquisition, (b) to preliminarily investigate a pilot train-the-trainer paradigm for skill development, and (c) to evaluate self-reported versus observed indicators of skill mastery in that pilot program. Participants included four supervisor–therapist dyads (N = 8) working with public mental health sector youth. Master trainers taught cognitive-behavioral therapy techniques to supervisors, who in turn trained therapists on these techniques. Supervisor and therapist skill acquisition and supervisor use of teaching strategies were repeatedly assessed through coding of scripted role-plays with a multiple-baseline across participants and behaviors design. The coding system, the Practice Element Train the Trainer – Supervisor/Therapist Versions of the Therapy Process Observational Coding System for Child Psychotherapy, was developed and evaluated though the course of the investigation. The coding scheme demonstrated excellent reliability (ICCs [1,2] = 0.81–0.91) across 168 video recordings. As calculated through within-subject effect sizes, supervisor and therapist participants, respectively, evidenced skill improvements related to teaching and performing therapy techniques. Self-reported indicators of skill mastery were inflated in comparison to observed skill mastery. Findings lend initial support for further developing an evaluative approach for a train-the-trainer effort focused on disseminating evidence-based practices.
Results
Excluding the tapes utilized for PETT-S TPOCS group development (n = 2) and training (n = 3), 43 supervisor problem-solving tapes (total N = 48) were coded using the model ICC(1,2) described above. The Teaching Content and Teaching Style scales for the problem-solving module demonstrated good (M = 0.70, SD = 0.31) and excellent reliabilities (M = 0.79, SD = 0.16), respectively. Similarly, excluding the supervisor exposure tapes used for PETT-S TPOCS group development (n = 2) and training (n = 7), 39 supervisor exposure tapes (total N = 48) were coded using the one-way random effects ICC(1,2) model. The Teaching Content and Teaching Style scales demonstrated excellent reliabilities (M = 0.81, SD = 0.23 and M = 0.93, SD = 0.06, respectively) for the exposure module. Withstanding the therapist problem-solving tapes used for PETT-T TPOCS group development (n = 2) and training (n = 3), and the therapist exposure tapes used for PETT-T TPOCS group development (n = 2) and training (n = 7), 31 therapist problem-solving (total N = 36) and 27 therapist exposure (total N = 36) tapes were coded with the ICC model described above. Both the problem-solving (M = 0.87, SD = 0.12) and exposure (M = 0.87, SD = 0.10) modules of the Technique Content scale evidenced reliabilities in the excellent range. Findings support our first hypothesis, which predicted good (0.60–0.74) to excellent (0.75–1.00) interrater reliability coefficients ( Cicchetti, 1994) for all TPOCS-S/T TPOCS scales.
Examining supervisors' BL to ST and BL to TT effect sizes (see Table 1) allowed investigation of our second hypothesis, which predicted Teaching Content score increases. Problem-solving content effect sizes ranged from 1.6 to 20.7 (M = 7.0, SD = 9.1) for BL to ST, and 2.2 to 38.5 (M = 11.7, SD = 17.9) for BL to TT transitions. For exposure content, effect sizes ranged from 3.2 to 8.4 (M = 5.7, SD = 2.2) for BL to ST, and 5.4 to 15.1 (M = 8.3, SD = 4.8) for BL to TT transitions. These results suggest positive training effects for both techniques with regard to content. Inspection of our third hypothesis involved inspecting supervisors' Teaching Style scale score effect sizes (see Table 1). For problem-solving teaching style, effect sizes ranged from 0.5 to 5.2 (M = 2.5, SD = 2.0) for BL to ST, and 0.6 to 7.1 (M = 3.3, SD = 2.7) for BL to TT transitions. Exposure teaching style effect sizes ranged from −0.3 to 2.2 (M = 1.2, SD = 1.1) for BL to ST, and 0.8 to 2.5 (M = 1.6, SD = 0.7) for BL to TT transitions. These findings hint at positive increases for teaching style post-training. Participant-level effect sizes across all observed scores were 2.7, 3.4, 3.2, and 9.3, for supervisors one through four, respectively.
Examining therapists' BL to TT effect sizes (see Table 2) allowed investigation for our fourth hypothesis, which predicted Technique Content score increases relative to baseline. Effect sizes ranged from 8.7 to 20.6 (M = 14.1, SD = 5.1) for problem-solving, and 3.1 to 41.3 (M = 13.7, SD = 18.5) for exposure. Participant-level effect sizes across both techniques were 25.0, 13.4, 9.1, and 8.0, for therapists one through four, respectively.
The fifth hypothesis predicted that supervisors and therapists would overestimate their performances for teaching and delivering techniques. Mean level observed and self-reported performances by phase for all supervisors are displayed in Table 1. Within each BL, ST, and TT phase, four types of observed versus self-report comparisons were possible for each of the four supervisors: (a) problem-solving teaching content, (b) problem-solving teaching style, (c) exposure teaching content, and (d) exposure teaching style (i.e., four supervisors multiplied by four types of comparisons per supervisor equals a total of 16 possible comparisons per phase). As seen in Table 1, over-estimation occurred for 16/16 comparisons in BL phases, 15/16 comparisons for ST phases, and 14/16 comparisons for TT phases. Table 2 displays all mean level observed and self-reported performances by phase for all therapists. Within each BL and TT phase, two types of observed versus self-report comparisons were calculated for each of the four therapists: (e) problem-solving content and (f) exposure content (i.e., four therapists multiplied by two types of comparisons per therapist equals a total of eight possible comparisons per phase). As seen in Table 2, over-estimation occurred for 8/8 comparisons in BL phases and 6/8 comparisons for TT phases.
