Original Research

Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters

Mwamba M. Mwenge, Caleb J. Figge, Kristina Metz, Jeremy C. Kane, Brandon A. Kohrt, Gloria A. Pedersen, Izukanji Sikazwe, Stephanie Skavenski Van Wyk, Saphira M. Mulemba, Laura K. Murray
Journal of Public Health in Africa | Vol 13, No 3 | a449 | DOI: https://doi.org/10.4081/jphia.2022.2201 | © 2024 Mwamba M. Mwenge, Caleb J. Figge, Kristina Metz, Jeremy C. Kane, Brandon A. Kohrt, Gloria A. Pedersen, Izukanji Sikazwe, Stephanie Skavenski Van Wyk, Saphira M. Mulemba, Laura K. Murray | This work is licensed under CC Attribution 4.0
Submitted: 11 April 2024 | Published: 07 September 2022

About the author(s)

Mwamba M. Mwenge, Centre for Infectious Disease Research in Zambia, Lusaka, Zambia
Caleb J. Figge, Johns Hopkins Bloomberg School of Public Health, Baltimore, United States
Kristina Metz, Johns Hopkins Bloomberg School of Public Health, Baltimore, United States
Jeremy C. Kane, Columbia University Mailman School of Public Health, New York, United States
Brandon A. Kohrt, George Washington University School of Medicine, Washington, DC, United States
Gloria A. Pedersen, George Washington University School of Medicine, Washington, DC, United States
Izukanji Sikazwe, Centre for Infectious Disease Research in Zambia, Lusaka, Zambia
Stephanie Skavenski Van Wyk, Johns Hopkins Bloomberg School of Public Health, Baltimore, United States
Saphira M. Mulemba, Centre for Infectious Disease Research in Zambia, Lusaka, Zambia
Laura K. Murray, Johns Hopkins Bloomberg School of Public Health, Baltimore, United States

Full Text:

PDF (425KB)

Abstract

Background. As evidence supports task-shifting approaches to reduce the global mental health treatment gap, counselor competency evaluation measures are critical to ensure evidence-based therapies are administered with quality and fidelity. Objective. This article describes a training technique for evaluating lay counselors’ competency for mental health lay practitioners without rating scale experience.

Methods: Mental health practitioners were trained to give the Enhancing Assessment of Common Therapeutic Factors (ENACT) test to assess counselor proficiency in delivering the Common Elements Treatment Approach (CETA) in-person and over the phone using standardized video and audio recordings. A two-day in-person training was followed by a one-day remote training session. Training includes a review of item scales through didactic instructions, active learning by witnessing and scoring role-plays, peer interactions, and trainer observation and feedback. The trainees rated video and audio recordings, and ICC values were calculated.

Results:  The training technique presented in this research helped achieve high counselor competency scores among lay providers with no prior experience using rating scales. ICC rated both trainings satisfactory to exceptional (ICC: .71 - .89).

Conclusions.  Raters with no past experience with rating scales can achieve high consistency when rating counselor competency through training. Effective rater training should include didactic learning, practical learning with trainer observation and feedback, and video and audio recordings to assess consistency.


Keywords

training; raters; inter-rater reliability; counselor competency

Metrics

Total abstract views: 493
Total article views: 28

 

Crossref Citations

1. Assessment Scales in Cerebral Palsy: A Comprehensive Review of Tools and Applications
Chaitanya Kumar Javvaji, Jayant D Vagha, Revat J Meshram, Amar Taksande
Cureus  year: 2023  
doi: 10.7759/cureus.47939