Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters


Submitted: 7 April 2022
Accepted: 15 May 2022
Published: 21 September 2022
Abstract Views: 232
PDF: 66
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Authors

Background. As evidence supports task-shifting approaches to reduce the global mental health treatment gap, counselor competency evaluation measures are critical to ensure evidence-based therapies are administered with quality and fidelity. Objective. This article describes a training technique for evaluating lay counselors’ competency for mental health lay practitioners without rating scale experience.
Methods: Mental health practitioners were trained to give the Enhancing Assessment of Common Therapeutic Factors (ENACT) test to assess counselor proficiency in delivering the Common Elements Treatment Approach (CETA) in-person and over the phone using standardized video and audio recordings. A two-day in-person training was followed by a one-day remote training session. Training includes a review of item scales through didactic instructions, active learning by witnessing and scoring role-plays, peer interactions, and trainer observation and feedback. The trainees rated video and audio recordings, and ICC values were calculated.
Results: The training technique presented in this research helped achieve high counselor competency scores among lay providers with no prior experience using rating scales. ICC rated both trainings satisfactory to exceptional (ICC: .71 - .89).
Conclusions. Raters with no past experience with rating scales can achieve high consistency when rating counselor competency through training. Effective rater training should include didactic learning, practical learning with trainer observation and feedback, and video and audio recordings to assess consistency.


Santomauro DF, Herrera AM, Shadid J, et al. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. Lancet. 2020; 398. Available from: https://doi.org/10.1016/S0140-6736(21)02143-7

Rhem J, Shield KD. Global Burden of Disease and the Impact of Mental and Addictive Disorders. Curr. Psychiatry Rep. 2019; 21(10). Available from: https://doi.org/10.1007/s11920-019-0997-0

Wainberg ML, Scorza P, Shultz JM, et al. Challenges and Opportunities in Global Mental Health: a Research-to-Practice Perspective. Curr Psychiatry Rep. 2017; 19(5): 28-34. Available from: doi:10.1007/s11920-017-0780-z

Lund C, Alem A, Schneider M, et al. Generating evidence to narrow the treatment gap for mental health disorders in sub-saharan Africa: rationale, overview and methods of AFFIRM. Epidemiol Psychiatri Sci. 2015; 24: 233-240. Available from: Doi:10.1017/S2045796015000281

Branson A, Myles P, Mahdi M, Shafran R. The relationship between competence and patient outcome with low intensity cognitive behavioural interventions. Behav cogn psychother. 2018; 46(1): 101-114. Available from : https://doi.org/10.1017/S1352465817000522

Liness S, Beale S, Lea S, et al. Multi professional IAPT CBT Training: Clinical Competence and Patient Outcomes. Behav and Cogn Psychother. 2019. Available from: DOI: 10.1017/S1352465819000201

Brown RC, Southern-Gerow M, McLeod BD, et al. The Global Therapist Competence Scale for Youth Psychosocial Treatment: Development and Initial Validation. J. Clin. Psychol. 2019; 74(4): 649-664. Available from: Doi:10.1002/jclp.22537

McLeod BD, Southam-Gerow MA, Rodriguez A, et al. Development and Initial Psychometrics for a Therapist Competence Instrument for CBT for Youth Anxiety. J Clin Child Adolesc Psychol. 2018; 47(1): 47-60. Available from: doi:10.1080/15374416.2016.1253018

Muse K, McManus F. A systematic review of methods for assessing competence in cognitive-behavioural therapy. Clin. Psychol. Rev. 2013; 33(3): 484-499. Available from: Doi: 10.1016/j.cpr.2013.01.010

Bjaastad JF, Haugland BSM, Fjermestad KW, et al. Competence and Adherence Scale for Cognitive Behavioral Therapy (CAS-CBT) for Anxiety Disorders in Youth: Psychometric Properties. Psychol. Assess. [Advance online publication]. 2015. Available from: http://dx.doi.org/10.1037/pas0000230

Ottman K, Kohrt BA, Pedersen G, Schafer A. Use of Role-plays to Assess Therapist Competency and its Association with Client Outcomes in Psychological Interventions: A Scoping Review and Competency Research Agenda. 2019. Available from: https://www.sciencedirect.com/science/article/pii/SOOO5796719302177

Kohrt BA, Jordans MJD, Rai S, et al. Therapist competence in global mental health: Development of the Enhancing Assessment of Common Therapeutic factors (ENACT) rating scale. Behav Res and Ther. 2015; 69: 11-21

Glanzman AM, Mazzone ES, Young SD, et al. Evaluator Training and Reliability for SMA Global Nusinersen Trials. J. Neuromuscul Dis. 2018; 5: 159-166. Available from: DOI 10.3233/JND-180301

Schmidt LK, Andersen K, Nielsen AS, Moyers TB. Lessons learned from measuring fidelity with the Motivational Interviewing Treatment Integrity code (MITI 4). J. Subst. abuse treat. 2019; 97: 59-67

Kohrt BA, Schafer A, Willhoite A, et al. Ensuring Quality in Psychological Support (WHO EQUIP): developing a competent global workforce. World Psychiatry. 2020; 19: 115-6 doi: 10.1002/wps.20704

Murray LK, Dorsey S, Haroz E, et al. A common elements treatment approach for adult mental health problems in low- and middle-income countries. Cogn. Behav. Pract. 2014; 21: 111-123. Available from: https://doi.org/10.1016/j.cbpra.2013.06.005

Murray LK, Kane JC, Glass N, et al. Effectiveness of the Common Elements Treatment Approach (CETA) in reducing intimate partner violence and hazardous alcohol use in Zambia (VATU): A randomized controlled trial. PLoS Med. 2020; 17(4). Available from: https://doi.org/10.1371/journal.pmed.1003056

United Nations, Department of Economic and Social Affairs, Population Division. World Population Prospects. 2019. Volume 2: Demographic Profiles (ST/ESA/SER.A/427)

Munakampe NM. Strengthening Mental Health Systems in Zambia. Int. J. Ment. Health Syst. 2020; 14(28). Available from: https://doi.org/10.1186/s13033-020-00360-z

Kohrt BA, Mutamba BB, Luitel NP, et al. How competent are non-specialists trained to integrate mental health services in primary care? Global perspectives from Uganda, Liberia, and Nepal. Int. Rev. Psychiatry. 2018; 30(6): 182-98. Doi: 10.1080/09540261.2019.1566116

Kohrt BA, Ramaiya MK, Rai S, et al. Development of a scoring system for non-specialist ratings of clinical competence in global mental health: a qualitative process evaluation of the Enhancing Assessment of Common Therapeutic Factors (ENACT) scale. GMH. 2015; 2(23): 1-16. Available from: Doi:10.1017/gmh.2015.21

Koo TK, Li MY. a Guideline of Selecting and Reporting Intra-class Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016; 15: 155-163

Sadler ME, Yamamoto RT, Khurana L, Dallabrida SM. the impact of rater training on clinical outcomes assessment data: a literature review. Int. J. Clin. Trials. 2017; 4(3): xxx-xx

Shweta, Bajpai RC, Chaturvedi HK. Evaluation of Inter-Rater Agreement and Inter-Rater Reliability for Observational Data: An Overview of Concepts and Methods. J. Indian Acad. Appl. Psychol. 2015; 41(3) Special issue: 20-27

West MD, Daniel DG, Opler M, et al. Consensus Recommendations on Rater Training and Certification. Innov Clin Neurosci. 2014; 11(11): 10-13

Rohan KJ, Rough JN, Evans M, et al. A protocol for the Hamilton Rating Scale for Depression: Item Scoring Rules, Rater Training, and Outcome Accuracy with Data on its Application in a Clinical Trial. J. Affect. Disord. 2016; 200: 111-118. Available from: Doi:10.1016/j.jad.2016.01.051

Kobak K, Williams JBW, Engelhardt N, Lipsitz J. Rater Training in Multicenter Clinical Trials. J. Clin. Psychopharmacol. 2004. Available from: doi:10.1097/01.jcp.0000116651.91923.54 PubMed

Opler MGA, Yavorsky C, Daniel DG. Positive and Negative Syndrome Scale (PANSS) Training: Challenges, Solutions, and Future Directions. Innov Clin Neurosci. 2017; 14(11): 77-81

Asan O, Montague E. Using video-based observation research methods in primary care health encounters to evaluate complex interactions. Informatics in primary care. 2014; 21: 161-170. Available from: Doi:10.14236/jhi.v21i4.72

Gulgin H, Hoogenboom B. The Functional Movement Screening (FMS): An inter-rater reliability study between raters of varied experience. Int. J. Sports Phys. Ther. 2014; 9(1): 1-7

Sajatovic M, Gaur R, Tatsuoka C, et al. Rater training for a multi-site, international clinical trial: What mood symptoms may be most difficult to rate? Psycho pharmacol Bull. 2011; 44(3): 5-14

Supporting Agencies

USAID

Mwenge, M. M., Figge, C. J., Metz, K., Kane, J. C., Kohrt, B. A., Pedersen, G. A., Sikazwe, I., Van Wyk, S. S., Mulemba, S. M., & Murray, L. K. (2022). Improving inter-rater reliability of the enhancing assessment of common therapeutic factors (ENACT) measure through training of raters. Journal of Public Health in Africa, 13(3). https://doi.org/10.4081/jphia.2022.2201

Downloads

Download data is not yet available.

Citations