Psychiatr Serv 50:1257, October 1999
© 1999 American Psychiatric Association
How Well Are We Evaluating System Change?
Walter Leginski, Ph.D.,
Frances Randolph, Dr.PH., Center for Mental Health Services, Rockville, Maryland and
Debra J. Rog, Ph.D., Vanderbilt Institute for Public Policy Studies, Nashville, Tennessee
Natural experiments occur all the time in community psychiatry. They are caused by changes in funding, policies, leadership, and law. If we are lucky, we learn about them in advance and try to rigorously study how they affect service utilization, client satisfaction, or symptomatology. Otherwise we can use data archives or post hoc study methods, as does the study by Kamis-Gould and associates in this issue.
The opportunity these field experiments offer is moderated by inherent limitations in study designs. Essentially, we are working with simple pre-post designs, taking measurements after the intervention occurs and comparing them with a baseline. There is nothing wrong with this—it is the heartbeat of deductive science. Virtually every advance in sophisticated designs and statistical procedures is built around improving our faith in concluding that pre-post change can be credited to the intervention, not to something extraneous.
But simple evaluation designs often cannot rule out extraneous causes. By attending to the underlying "theory of change," we can move beyond evaluations as description and make them opportunities to produce generalizable knowledge.
Of greatest help is the use of "logic models." This heuristic device disciplines us to spell out clearly what we think is happening in the field experiment: its environment, what resources apply, how the intervention is expected to operate, and the outcomes it will affect. Natural experiments are inherently rich in competing causes. Logic models also require us to articulate other factors that might influence outcomes. By identifying these variables, we anticipate their measurement. If rental-market conditions influence consumer placement in independent housing, measure them. If service quality reduces utilization, quantify it. Careful conceptualization and relevant data capture are our best analytic defenses.
It is equally critical to monitor the intervention itself. It is surprising how often implementation of a practice can deviate from its critical components, threatening "model fidelity." Did less than the therapeutic dose of a medication get taken? Did "community integration" for a discharged inpatient mean assignment to a residence as intensively structured as the hospital? Without monitoring, or measuring, what interventions actually happened, we may conclude something about the central intervention that was not truly operating.
By rigorously exploiting the evaluation opportunities of natural experiments, we contribute to generalizable findings, leave others with a documented conceptual framework that helps them anticipate similar circumstances, and advance the improvement of our services.
Get information about faster international access.
Privacy Policy
Copyright © 1999
American Psychiatric Association.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
