Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Email this article to a Colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Articles & Searches Download to citation manager Citing Articles via HighWire Citing Articles via Google Scholar Articles by Bond, G. R. Search for Related Content PubMed Citation Articles by Bond, G. R. Homeless Persons Veterans Housing and Vocational Support Related Articles

Modest Implementation Efforts, Modest Fidelity, and Modest Outcomes

Introduction

TOP Introduction Acknowledgments and disclosures References

 
It is terrific news that the Department of Veterans Affairs (VA) is disseminating a program model that arguably has the strongest evidence base of all the psychosocial practices for individuals with severe mental illness (1). It is not surprising that Dr. Rosenheck leads this effort; his contributions to implementing innovative practices in the VA system are enormous. However, I have concerns about the authors' conclusion that they have documented that "a sustained training program can be used to implement IPS in systems that have had little past experience with this approach [and the IPS program] was associated with improved employment outcomes."

This project falls short on three critical and interrelated facets. First, as the authors note, the training model for individual placement and support (IPS) was "modestly intensive." After a one-day training and weekly teleconferences in the first three months, the "sustained" training apparently consisted of one site-specific and one systemwide teleconference per month, which does not seem sufficient to ensure adequate implementation. In fairness, the training exceeded the intensity that my colleagues and I naively employed in a series of dissemination projects during the 1980s (2). As did others during this era, we had mixed results. We discovered that initial training typically had little impact (especially with staff turnover) and that written materials and occasional phone calls were inadequate for overcoming the many barriers to implementation. One provisional conclusion from recent dissemination studies is the importance of ongoing face-to-face contact by a trainer-consultant, because site visits usually lead to more accurate assessments and personalized interventions. In contrast to the VA project, the National Evidence-Based Practices Project used a far more intensive consultation approach, with up to two years of monthly face-to-face expert consultant contact. Even with this intensive consultant model, high fidelity was not uniformly achieved (McHugo GJ, Drake RE, Whitley R, et al, unpublished manuscript, 2007).

A second issue concerns the measurement of program fidelity. It is puzzling that the authors did not use the well-validated IPS-Fidelity Scale (3) but rather developed an abbreviated version for which there apparently are no norms, little psychometric data, and no second raters. It is impossible to judge the validity of the authors' ad hoc claim that 78% of programs had "acceptable" fidelity. My impression is that they are overly generous in this judgment. Clues that fidelity may have been problematic are details such as restriction of site implementation to a single employment specialist (not a vocational unit), sites' lack of "a structurally unified treatment team," reliance mostly on office-based client contacts, and continued use of compensated work therapy (CWT). It is difficult to implement IPS when a competing sheltered work program remains in place.

Third, the employment findings for this study were extremely modest. As the authors document, the absolute rates of competitive employment for IPS in this study were much smaller than those found in the literature, as were the differences between the IPS and control groups in this study. Although findings were statistically significant, the clinical significance of these differences is questionable.

Although I recognize this article's timely contributions to understanding the implementation process, my overall conclusion would be different. If we accept marginal improvements as what we can expect when we introduce evidence-based practices "in the real world," we do a disservice to the field. We set the bar too low. Readers may be misled to conclude that, under usual conditions, IPS may increase days of competitive work per month by perhaps only 9% (the improvement found for individuals with severe mental illness in this study). In addition, the site with lowest fidelity (an outlier) had the second-best employment outcomes. The discussion section speculates that other factors might compensate for low fidelity. Thus proponents of CWT and other traditional approaches might reason that their programs need not pay attention to evidence-based principles.

My interpretation of what this study shows is that modest implementation efforts lead to modest fidelity, which in turn leads to modest outcomes. We can do better.

Acknowledgments and disclosures

TOP Introduction Acknowledgments and disclosures References

 
The author reports no competing interests.

Footnotes

 
Dr. Bond is with the Department of Psychology, Indiana University-Purdue University Indianapolis, 402 North Blackford St., Indianapolis, IN 46202 (e-mail: gbond{at}iupui.edu).

References

TOP Introduction Acknowledgments and disclosures References

 

1. Rosenheck RA, Mares AS: Implementation of supported employment for homeless veterans with psychiatric or addiction disorders: two-year outcomes. Psychiatric Services 58:325–333,2007[Abstract/Free Full Text]
2. Bond GR: Variations in an assertive outreach model. New Directions for Mental Health Services 52:65–80,1991[Medline]
3. Bond GR, Becker DR, Drake RE, et al: A fidelity scale for the Individual Placement and Support model of supported employment. Rehabilitation Counseling Bulletin 40:265–284,1997

This article has been cited by other articles:

				G. R. Bond, R. E. Drake, G. J. McHugo, C. A. Rapp, and R. Whitley Strategies for Improving Fidelity in the National Evidence-Based Practices Project Research on Social Work Practice, September 1, 2009; 19(5): 569 - 581. [Abstract] [PDF]

				R. E. Glasgow Critical Measurement Issues in Translational Research Research on Social Work Practice, September 1, 2009; 19(5): 560 - 568. [Abstract] [PDF]

				R. E. Drake, G. R. Bond, and S. M. Essock Implementing Evidence-Based Practices for People With Schizophrenia Schizophr Bull, July 1, 2009; 35(4): 704 - 713. [Abstract] [Full Text] [PDF]

				M. Rinaldi, R. Perkins, E. Glynn, T. Montibeller, M. Clenaghan, and J. Rutherford Individual placement and support: from research to practice Adv. Psychiatr. Treat., January 1, 2008; 14(1): 50 - 60. [Abstract] [Full Text] [PDF]

Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Email this article to a Colleague Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to My Articles & Searches Download to citation manager Citing Articles via HighWire Citing Articles via Google Scholar Articles by Bond, G. R. Search for Related Content PubMed Citation Articles by Bond, G. R. Homeless Persons Veterans Housing and Vocational Support Related Articles

Get information about faster international access.

Privacy Policy

Copyright © 2007 American Psychiatric Association. All rights reserved.

Home | Search | Current Issue | Past Issues | Subscribe | All APPI Journals | Help | Contact Us