Psychiatr Serv 53:879-881, July 2002
© 2002 American Psychiatric Association
Medical Illness and Severe Mental Illness: Articles and Brief Reports |
Reliable Self-Report of Health Service Use by Individuals With Serious Mental Illness
Richard W. Goldberg, Ph.D.,
Diana C. Seybolt, Ph.D. and
Anthony Lehman, M.D., M.S.P.H.
|
Abstract
|
The purpose of this study was to determine whether individuals with serious mental illness could consistently report their use of medical services. Twenty-nine individuals with schizophrenia completed a test-retest protocol using a modified version of the Medical Expenditure Panel Survey; there was a one-week interval between the two administrations of the instrument. Consistency of reporting was generally strong and was strongest for items indicating whether a service had been used in the past six months, which suggests that people with schizophrenia can provide reliable information about their use of health services. Continued research in this area will increase understanding of the types and quality of medical care received by persons with schizophrenia.
|
Introduction
|
Medical morbidity and mortality rates are higher among persons who have serious mental illness than in the general population (1). In addition, recent research suggests that psychiatric populations have different patterns of health service use than nonpsychiatric populations (2,3). Because data on service use are commonly self-reported (4,5), research is needed to evaluate the reliability with which individuals who have serious mental illness report their use of health care services.
Previous research has suggested that reliable self-report among psychiatric populations, particularly self-assessment of symptoms, may be compromised by the cognitive deficits and lack of insight associated with serious mental illness (6). Whether this observation extends to the self-report of behavioral events such as use of services has been tested less frequently. We reviewed the literature and found only two studies that examined this question with a psychiatric population (7,8), and only one of them assessed test-retest reliability (7). Reporting test-retest correlation coefficients ranging from .87 to .89, Calsyn and colleagues (7) concluded that homeless individuals with mental illness can provide reliable information about their use of mental health services. However, no study has provided detailed information about the extent to which individuals who have serious mental illness reliably report their use of medical services with instruments developed for nonpsychiatric populations.
Using a modified version of a well-known national health services survey, we conducted this study to determine whether individuals with schizophrenia could consistently report their use of medical services. On the basis of the findings of Calsyn and colleagues (7), we hypothesized that this patient population would report service use consistently.
|
Methods
|
The participants in this study were part of a larger study of quality of care in schizophrenia. A modified version of the household component of the Medical Expenditure Panel Survey (MEPS), a scientifically valid and widely used survey of medical service use and expenditures developed by the Agency for Healthcare Research and Quality (AHRQ) (4), was administered to 171 outpatients who had been diagnosed as having schizophrenia. Our review of the literature and MEPS-related documentation from the AHRQ found that no test-retest results were available for the national survey.
Twenty-nine participants completed a one-week test-retest protocol using the modified MEPS. The instrument—read aloud by the interviewer—included items about medical services received in outpatient settings (for example, "During the past six months, did you go to a doctor's office, outpatient clinic, health center, or some other outpatient setting for any kind of medical care?"), hospital emergency departments (for example, "During the past six months, did you go to a hospital emergency room for medical care?"), and inpatient hospital settings (for example, "During the past five years, approximately how many times were you a patient in the hospital because of a medical condition or problem?"). There were also items related to dental, vision, and hearing services.
For dichotomous variables, we measured agreement across administrations by using a simple (unweighted) kappa statistic. On the basis of the standard error, a 95 percent confidence interval was also calculated as a measure of the precision of the estimate of agreement. Because low-prevalence phenomena can influence the kappa statistic, percent agreement scores were also calculated. For ordinal variables with three or more levels, a weighted kappa statistic was used to allow for maximum weighting of the more discrepant cases of disagreement. To increase the cell counts and improve statistical power, some response levels were combined. In accordance with the method of Landis and Koch (9), estimated kappa values below .40 were considered poor to fair; .41 to .60, moderate; .61 to .80, substantial; and .81 to 1.00, almost perfect.
|
Results
|
Twenty-seven (93 percent) of the study participants were men; 25 (86 percent) were African American, and four (14 percent) were Caucasian. The mean±SD age of the patients was 40±7.4 years (range, 26 to 56 years). Efforts were made to administer the two interviews as closely as possible to seven days apart. The mean period between administrations was 5±2 days.
Response distributions across 17 items for each administration of the survey, kappa values and related confidence intervals, a strength-of-consistency classification based on the criteria of Landis and Koch, and total agreement percentages are listed in Table 1. Consistency across items was generally strong: of the 16 variables for which kappa values could be classified, three were classified as almost perfect, six as substantial, six as moderate, and one as fair.
View this table:
[in this window]
[in a new window]
|
Table 1. Reliability of self-reported use of health services in a sample of 29 persons with schizophrenia who participated in a test-retest protocol
|
Consistency was strongest for items that required respondents to report whether they had used a particular service in the past six months—for example, for six-month use of an emergency department, kappa=.91; for six-month use of outpatient medical services, kappa=.86. Kappa values and percent agreement scores were generally somewhat lower for items that sought information about the number of visits—for example, for the number of outpatient visits in the past six months, kappa=.50; for the number of visits to an emergency department in the past five years, kappa=.30. Respondents were moderately to substantially consistent in reporting receipt of medications (kappa=.45) and use of dental services (.61 and .62). Substantial agreement was found for receipt of a vision test (.79) and receipt of a hearing test (.63).
|
Discussion and conclusions
|
The purpose of this study was to determine whether individuals with schizophrenia could consistently report their use of medical services. Overall, the kappa values and percent agreement scores indicated strong consistency. Respondents were most consistent about whether they had used a particular service and were somewhat less consistent in reporting their actual level of service use. These results both support and extend the findings of Calsyn and colleagues (7). Although previous research has shown that the cognitive deficits and lack of insight associated with schizophrenia may compromise the reliability of self-assessments of symptoms (6), our results suggest that people with schizophrenia can reliably report behavioral events such as use of medical services.
It is not known, however, how a longer interval between the two administrations of the survey would have affected consistency scores. Nor do we know how our results compare with those from other samples of individuals who responded to the MEPS, as we did not find reports of test-retest reliability for the instrument in our review of the literature and related MEPS documentation. Also, our findings do not address the validity of the study participants' reports. Although the results of recent research have suggested that the validity of self-report among individuals with serious mental illness may vary by the type of service use reported (7,8,10), the services assessed in those studies were primarily mental health services.
Thus future research also needs to assess the validity of self-reported use of medical services among individuals with serious mental illness. Once the reliability and validity of such information can be established, researchers can use national surveys of medical service utilization, such as the MEPS, to better understand the types and quality of medical care received by people with schizophrenia. Such information will also enable services researchers and policy makers to learn more about the integration of psychiatric and medical services for this vulnerable population.
|
Acknowledgments
|
Support for this project was provided by grant MH-57707 from the National Institute of Mental Health and from the Mental Illness Research, Education, and Clinical Center of the Veterans Affairs Capitol Health Care Network (VISN 5).
|
Footnotes
|
The authors are affiliated with the School of Medicine at the University of Maryland at Baltimore. Send correspondence to Dr. Goldberg at the Center for Mental Health Services Research, University of Maryland at Baltimore, 685 West Baltimore Street, MSTF Building, Room 300, Baltimore, Maryland 21201 (e-mail, rgoldber{at}psych.umaryland.edu). This paper is one of several in this issue focusing on the health and health care of persons with severe mental illness.
|
References
|
- Felker B, Yazel JJ, Short D: Mortality and medical comorbidity among psychiatric patients: a review. Psychiatric Services 12:1356-1363, 1996
- Berren MR, Santiago JM, Zent MR, et al: Health care utilization by persons with severe and persistent mental illness. Psychiatric Services 50:559-561, 1999[Abstract/Free Full Text]
- Steiner JL, Hoff RA, Moffett C, et al: Preventive health care for mentally ill women. Psychiatric Services 49:696-698, 1998[Abstract/Free Full Text]
- Cohen J: Design and Methods of the Medical Expenditure Panel Survey Household Component. MEPS methodology report no 1, AHCPR publication 97-0026. Rockville, Md, Agency for Health Care Policy and Research, 1997
- Narrow WE, Regier DA, Rae DS, et al: Use of services by persons with mental and addictive disorders: findings from the National Institute of Mental Health Epidemiologic Catchment Area Program. Archives of General Psychiatry 50:95-107, 1993[Abstract/Free Full Text]
- Atkinson M, Zibin S, Chuang H: Characterizing quality of life among patients with chronic mental illness: a critical examination of the self-report methodology. American Journal of Psychiatry 154:99-105, 1997[Abstract]
- Calsyn RJ, Allen G, Morse GA, et al: Can you trust self-report data provided by homeless mentally ill individuals? Evaluation Review 17:353-366, 1993
- Calsyn RJ, Morse GA, Klinkenberg WD, et al: Reliability and validity of self-report data of homeless mentally ill individuals. Evaluation and Program Planning 20:47-54, 1997[CrossRef]
- Landis JR, Koch GG: The measurement of observer agreement for categorical data. Biometrics 33:159-174, 1977[CrossRef][Medline]
- Kashner TM, Suppes T, Rush AJ, et al: Measuring use of outpatient care among mentally ill individuals: a comparison of self-reports and provider records. Evaluation and Program Planning 22:31-39, 1999
This article has been cited by other articles:
|
|
|
|
 
M. Horvitz-Lennon, R. G. Frank, W. Thompson, S. H. Baik, M. Alegria, R. A. Rosenheck, and S.-L. T. Normand
Investigation of Racial and Ethnic Disparities in Service Utilization Among Homeless Adults With Severe Mental Illnesses
Psychiatr Serv,
August 1, 2009;
60(8):
1032 - 1038.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
P. McCrone, H. Killaspy, P. Bebbington, S. Johnson, F. Nolan, S. Pilling, and M. King
The REACT Study: Cost-Effectiveness Analysis of Assertive Community Treatment in North London
Psychiatr Serv,
July 1, 2009;
60(7):
908 - 913.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
D. W. Bradford, M. M. Kim, L. E. Braxton, C. E. Marx, M. Butterfield, and E. B. Elbogen
Access to Medical Care Among Persons With Psychotic and Major Affective Disorders
Psychiatr Serv,
August 1, 2008;
59(8):
847 - 852.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
L. S. Matza, R. Buchanan, S. Purdon, J. Brewster-Jordan, Y. Zhao, and D. A. Revicki
Measuring Changes in Functional Status Among Patients With Schizophrenia: The Link With Cognitive Impairment
Schizophr Bull,
October 1, 2006;
32(4):
666 - 678.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
M. SLADE, P. McCRONE, E. KUIPERS, M. LEESE, S. CAHILL, A. PARABIAGHI, S. PRIEBE, and G. THORNICROFT
Use of standardised outcome measures in adult mental health services: Randomised controlled trial
The British Journal of Psychiatry,
October 1, 2006;
189(4):
330 - 336.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
R. W. Goldberg, S. Himelhoch, J. Kreyenbuhl, F. B. Dickerson, A. Hackman, L. Juan Fang, C. H. Brown, K. A. Wohlheiter, and L. B. Dixon
Predictors of HIV and Hepatitis Testing and Related Service Utilization Among Individuals With Serious Mental Illness
Psychosomatics,
December 1, 2005;
46(6):
573 - 577.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
D. H. LAM, P. McCRONE, K. WRIGHT, and N. KERR
Cost-effectiveness of relapse-prevention cognitive therapy for bipolar disorder: 30-month study
The British Journal of Psychiatry,
June 1, 2005;
186(6):
500 - 506.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
A. B. Rothbard, S. Metraux, and M. B. Blank
Cost of Care for Medicaid Recipients With Serious Mental Illness and HIV Infection or AIDS
Psychiatr Serv,
September 1, 2003;
54(9):
1240 - 1246.
[Abstract]
[Full Text]
[PDF]
|
|
|
Get information about faster international access.
Privacy Policy
Copyright © 2002
American Psychiatric Association.
All rights reserved.
Home
| Search
| Current Issue
| Past Issues
| Subscribe
| All APPI Journals
| Help
| Contact Us
|
TOP
Abstract
Introduction
