Quantitative Analysis question. Attacahed
tadei
Early Detection of High-Risk Claims at the Workers’ Compensation Board of
British Columbia Ernest Urbanovich • Ella E. Young • Martin L. Puterman • Sidney O. Fattedad
The Workers’ Compensation Board of British Columbia, PO Box 5350, Station Terminal, Vancouver, British Columbia, Canada V6B 5L5
Marsh Canada Limited, 1300-510 Burrard Street, Vancouver, British Columbia, Canada V6C 3J2 Faculty of Commerce and Business Administration, University of British Columbia, 2053 Main Mall,
Vancouver, British Columbia, Canada V6T 1Z2 The Workers’ Compensation Board of British Columbia, PO Box 5350, Station Terminal, Vancouver,
British Columbia, Canada V6B 5L5 [email protected] • [email protected] • [email protected] • [email protected]
This paper was refereed.
We developed a combined decision-analysis and logistic-regression approach for identifying high-risk claims at the Workers’ Compensation Board of British Columbia (WCB). The early detection of such claims and subsequent intervention is likely to reduce their eventual cost and to speed up worker rehabilitation. High-risk claims are extremely costly to the WCB; for the approximately 321,000 short-term disability claims with injury dates between 1989 and 1992, high-risk claims accounted for $1.2 billion (64 percent) of the total payment of $1.8 billion, even though they constituted only 4.2 percent of the claims. We developed separate logistic regression models for each injury type. We found that the age of worker and number of workdays lost were predictive of high-risk status. We used decision analysis to develop a classification rule that has high out-of-sample predictive power. The WCB has incorporated these results in a claims-profiling scorecard, which identifies claims needing early intervention. We estimate that our method saves the WCB $4.7 million annually. (Decision analysis: applications. Financial institutions: insurance.)
The Workers’ Compensation Board of BritishColumbia (WCB) is a statutory agency responsi- ble for the occupational health and safety, rehabilita- tion, and compensation interests of British Columbia’s workers and employers. Created in 1917, the WCB’s main objective is to help workers and employers to create and maintain safe workplaces and to ensure injured workers secure income and safe return to work. The WCB obtains the funds to make compen- sation payments and meet its other financial obli- gations from assessments levied on employers. In return, employers receive protection from lawsuits arising from work-related injuries and diseases. For waiving the right to sue, injured workers receive the
right to benefits on a no-fault basis. In 2002, the WCB served more than 165,000 employers who employed about 1.8 million workers in British Columbia and spent over $1 billion (all figures in Canadian dollars) on compensation and rehabilitation. The main objective of our study was to develop a
systematic approach to identifying short-term disabil- ity (STD) claims that pose a potentially high financial risk to the WCB. We refer to these claims as high- risk claims. We anticipated that by detecting high-risk claims early and subsequently intervening, the WCB would both improve claims management and reduce future costs associated with these claims. To dis- criminate between high-risk and low-risk claims, we
0092-2102/03/3304/0015$05.00 1526-551X electronic ISSN
Interfaces © 2003 INFORMS Vol. 33, No. 4, July–August 2003, pp. 15–26
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
combined decision analysis with logistic regression. Logistic-regression models use claim characteristics to assign a probability that a claim becomes high risk. Claims with sufficiently high probabilities are can- didates for intensive claim-management intervention. We used decision-analytic methods to determine a threshold or cutoff point above which a claim is clas- sified as high risk. Logistic regression, although not as widely used as
linear regression, has been used in a variety of busi- ness and medical applications in which the goal was to find models that separate data into two groups. For example, Wiginton (1980) used logistic regres- sion in credit scoring. The model allowed classifica- tion of credit applicants into high- and low-credit-risk groups. In a similar context, Johnson (1998) developed a logistic regression model to determine whether or not college students should be given credit for future purchases at a campus department store. Thompson (1985) used logistic regression to study outcomes in a community mental health program, while Lemeshow et al. (1988) determined factors affecting the proba- bility of patients surviving to hospital discharge after admission to an intensive care unit. Recently, WCB managers have become increasingly
aware that they could use quantitative tools to iden- tify high-risk claims. Two unpublished internal stud- ies motivated our research. Jessup and Gallie (1996) identified age of worker, gender, and nature of injury as factors that can be used for profiling high-risk claims, but they did not develop a model to quan- tify the risk associated with them. Fattedad and Charron (1999) studied the WCB’s inventory of high- risk claims and concluded that a claim that exceeds 70 workdays lost is likely to become high risk.
Claim Processing at the WCB A request for compensation from an injured worker for an injury or illness sustained at work is referred to as a claim. Whenever a claim corresponding to an injury or illness resulting from a person’s employ- ment causes temporary absence from work, the claim is referred to as a short-term disability claim or sim- ply an STD claim. For STD claims, the WCB pro- vides wage-loss payments for the workdays lost (STD days paid) and pays for the cost of hospitalization,
treatment, prescription drugs, and necessary medical appliances. The vast majority (96 percent) of injured workers with STD claims recover from their injuries and return to work within a few months. Whenever a worker fails to recover completely from
a work-related injury or illness and is left with a per- manent partial disability or permanent total disabil- ity, the WCB ends the STD status of the claim and begins paying permanent disability benefits. Perma- nent disability benefits are called long-term disability or LTD benefits. Once an STD claim receives LTD ben- efits, it is no longer classified as an STD claim but as an LTD claim. If a worker incurs a permanent total disability, the WCB awards a periodic payment equal to 75 percent of his or her estimated average earn- ings for life. A worker who incurs permanent par- tial disability returns to work after his or her medical condition becomes fairly stable. In this case, the com- pensation is a periodic payment of 75 percent of the estimated loss of average earnings resulting from the impairment and is payable for life. The estimated loss of earnings is positively cor-
related with the percentage of disability—a quantity determined by the WCB’s health-care professionals based on the physical condition of the worker. Gen- erally, the lower the percentage of disability the less the estimated loss of earnings. To reduce the cost of LTD claims, claim managers and rehabilita- tion specialists focus on reducing the percentage of disability through early detection and intensive inter- vention; Head (1995) gives more details. For example, a lower-back-strain claim that may have resulted in a 25 percent permanent disability without early inter- vention may be reduced to a 10 percent disability through intensive early treatment and rehabilitative efforts.
Converted and Nonconverted STD Claims We refer to the transition of STD claims into LTD claims as conversion. STD claims that become LTD claims are called converted STD claims. Con- versely, STD claims that result in the injured workers’ returning to work are called nonconverted STD claims. We assumed it would be beneficial to detect potential
16 Interfaces
Vol. 33, No. 4, July–August 2003
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
LTD claims early in their life cycles. To this end, we developed logistic regression models to detect STD claims that have a high likelihood of converting. The sample we used to investigate the conversion
of STD claims consisted of all 320,973 STD claims that had injury dates between January 1, 1989 and December 31, 1992. We chose this time period to reduce any bias that could have resulted from includ- ing claims with more recent injury dates because roughly eight percent of STD claims that convert require over five years to do so. To assess the financial impact, we captured all pay-
ments these claims received up to July 1999. Our investigation revealed that although converted STD claims represented only 4.2 percent of all STD claims, they accounted for 64.3 percent ($1,173 million) of the total payments of $1,824 million. On the other hand, nonconverted STD claims represented 95.8 percent of all claims in the sample but incurred only 35.7 percent of the costs. Thus, from a financial point of view, we viewed converted STD claims as high-risk claims, whereas we categorized nonconverted STD claims as low-risk claims. In addition, the $86,000 average cost of converted claims is 41 times greater than the aver- age cost of $2,100 for nonconverted claims. These costs vary by injury type.
Number of Converted Number of STD Claims STD Claims and the Average Cost Average Cost and the Corresponding Corresponding Percentage Conversion of Nonconverted of Converted
Nature of Injury Percentage of All Claims (%) of All Converted Claims (%) Rate (%) Claims ($) Claims ($)
Sprains, strains 159,100 (49.6) 5,059 (37.4) 3�2 2�395 106�066 Contusion, crushing, bruise (soft tissue) 50,717 (15.8) 1,395 (10.3) 2�8 1�587 99�666 Cut, laceration, puncture (open wound) 44,234 (13.8) 1,846 (13.7) 4�2 1�073 40�827 Fracture 16,163 (5.0) 2,557 (18.9) 15�8 4�047 80�982 Scratches, abrasions (superficial wound) 11,596 (3.6) 62 (0.5) 0�5 487 94�801 Tenosynovitis, synovitis, tendonitis 11,096 (3.5) 641 (4.7) 5�8 2�604 88�857 Burn or scald (heat or hot substances) 7,095 (2.2) 141 (1.0) 2�0 967 62�216 Bursitis (epicondylitis or tennis elbow) 4,884 (1.5) 383 (2.8) 7�8 3�752 72�271 Carpal tunnel syndrome 1,706 (0.5) 197 (1.5) 11�5 6�017 69�016 Amputation, enucleation 699 (0.2) 582 (4.3) 83�3 3�311 41�220 Other injury types 13,683 (4.3) 649 (4.8) 4�7 2�496 106�958
All injury types 320,973 (100) 13,512 (100) 4�2 2�101 86�223
Table 1: We show the distribution of short-term disability (STD) claims and converted STD claims by nature of injury, the corresponding conversion rates, and the average costs of converted (to long-term disability benefits) and nonconverted claims. The conversion rate is the proportion of STD claims that become long-term disability claims. (All costs are in 1991 Canadian dollars.)
Classification of STD Claims by Injury Type The WCB uses the nature of injury (NOI) to classify injuries or illnesses in terms of their principal physi- cal characteristics (Table 1). Ten of the most frequently observed injury types made up 95.7 percent of all claims in our sample. The conversion rate (CR) is the proportion of STD
claims that convert. Although the overall CR is 4.2 percent, it varies from 0.5 percent for scratches and abrasions to 83.3 percent for amputations or enu- cleations. The CR is a key indicator of the severity of the injury type. The WCB now uses it to compare the extent of conversion within different subsets of claims. Within each injury-type group, the average cost of
converted STD claims is 12 to 195 times higher than the average cost of nonconverted STD claims.
Data Sources and Collection We extracted data for this study from the WCB data warehouse—a single integrated source of information developed in the late 1990s by combining data from many different department and service databases.
Interfaces Vol. 33, No. 4, July–August 2003 17
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
It is updated monthly. We used Crystal Reports to access and extract data from the data warehouse and SPSS to perform statistical analyses. The informa- tion extracted included injury date, age of the injured worker on the date of injury, gender of the injured worker, payment dates, monthly payments, number of STD days paid, nature of injury, and injured body part.
Logistic Regression Regression models relate a response variable to one or more predictor variables. In logistic regression, the response variable is binary, while in the linear regres- sion model, it is continuous and unbounded. Asso- ciated with each claim in our study are a binary response variable Y and p predictor variables, which we denote by the vector x = �x1�x2� � � � � xp�. The con- dition Y = 1 represents a converted STD claim, and Y = 0 represents a nonconverted STD claim. Denote by ��x�= P�Y = 1 �x� the conditional probability that Y = 1 for a predictor variable vector with values x. The probability that the response variable equals 0 is P�Y = 0 �x� = 1−��x�. The odds favoring Y = 1 over Y = 0 are given by O�Y = 1 �x�=��x�/�1−��x� . The logit transformation �L� expresses the odds of conversion on the natural logarithm scale as L�x� = ln�O�Y = 1 �x� . Logistic regression models represent the logit L as
a linear function of the predictor variables as follows:
L�x�= �0+�1x1+�2x2+· · ·+�pxp� where �0��1� � � � ��p are the parameters of the model. For application, we convert logits to probabilities with the equation ��x�= eL�x�/�1+ eL�x� . We estimated the parameters of the logistic
regression using maximum likelihood (Hosmer and Lemeshow 1989, McCullagh and Nelder 1989). Most statistical software packages include maximum likeli- hood estimation and model diagnostic measures. The diagnostics we used include the Wald test to assess the significance of model parameters, the deviance and Hosmer-Lemeshow tests to assess overall signif- icance, and residual and influence analyses to deter- mine the effects of individual data points on the parameter estimates. An additional decision-focused
criterion for evaluating model quality is the correct classification rate, that is, the proportion of observa- tions that are classified correctly using model esti- mates of ��x�. The implementation of this measure requires specification of a cutoff point or a value of the P�Y = 1 �x� above which an observation is classi- fied as 1 based on its predictor values and the esti- mated parameters. In most applications, analysts use a default value of 0.5, but this choice ignores the consequences of incorrect decisions (McCullagh and Nelder 1989, Neter et al. 1996, Ryan 1996).
Logistic Regression Model Development The predictor variables we considered for analysis were nature of injury, industry of worker, gender, age of worker, STD days paid, and injured body part. Sev- eral of these variables were categorical with multi- ple levels. An aggregate model including all of these variables together with first-order interactions would have required hundreds of predictor variables. We felt that such a large model would have been diffi- cult to interpret, analyze, and apply. Further, it would have been difficult to reliably estimate parameters for this combined model given the large data set. Instead, we chose to stratify data on the basis of the nature of injury and develop separate models for the 10 most frequent injury types. We did consider, however, using a finer stratification based on cross-classifying injured body part and nature of injury. The cross- classification would have resulted in over 200 models to assess and analyze, for which we did not have time. Moreover, over 120 of these categories had too few claims for reliable estimates of the parameters of the logistic-regression models. Further, at this stage of analysis, we chose not
to incorporate all of the potential predictors in our model because of unavailability of data. Gender of claimant might have proved to be a significant pre- dictor, but initial examination of the data set showed that 68 percent of the cases were missing this infor- mation. Moreover, restricting the analysis to cases that included gender of claimant would have biased the analysis because only the most severe claims were coded for gender. On the other hand, all claims from
18 Interfaces
Vol. 33, No. 4, July–August 2003
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
our sample had information regarding the injured worker’s age and the number of STD days paid. Con- sequently, after stratifying the claims on the basis of nature of injury, we used two quantitative variables to predict the conversion of STD claims; the claimant’s age (AGE), and the number of STD days paid (DAYS). The resulting model had the form:
L�AGE�DAYS�= �0+�1AGE+�2DAYS�
An aggregate model with different slopes for AGE and DAYS within each injury category would require 30 parameters. We decided it would be easier to esti- mate, apply, and justify separate models for each injury type (Table 2). All models include DAYS and AGE as significant
(at the 0.10 level) predictors except those for scratches and abrasions, burns, and carpal tunnel syndrome for which only DAYS is significant. We chose not to develop a regression model for amputations and enu- cleations because of the extremely high conversion rate (83.3 percent) for injuries of this type; we classi- fied all claims within this category as high risk.
Model Intercept DAYS AGE Nature of Injury
Sprain or strain −5�7373 0.0197 0.0311 Sprains, strains Contusion −5�5536 0.0247 0.0236 Contusion, crushing,
bruise (soft tissue) Laceration −4�8297 0.0473 0.0124 Cut, laceration, puncture
(open wound) Fracture −3�6756 0.0212 0.0088 Fracture Abrasion −6�1125 0.0528 — Scratches, abrasions
(superficial wound) Joint −4�6126 0.0172 0.0224 Tenosynovitis, synovitis,
inflammation tendonitis Burn −5�1627 0.0455 — Burn or scald (heat or
hot substances) Bursitis −3�7105 0.0138 0.0103 Bursitis (epicondylitis
or tennis elbow) Carpal tunnel −2�9705 0.0109 — Carpal tunnel syndrome Amputation N/A N/A N/A Amputation, enucleation Other −4�1689 0.0153 0.0150 Other injury types
Table 2: We show model descriptions and estimated parameters of the logistic regression models. The abrasion, burn, and carpal tunnel models have only DAYS paid as predictor. The amputation model is not based on logistic regression; the conversion rate for this model is so high that all claims within this category are classified as high risk.
Using the estimates of the parameters of the mod- els, we calculate estimates of the probability of con- version for each claim (Figure 1) by substituting the parameter estimates into the equation
��x�= e �0+�1AGE+�2DAYS
1+ e�0+�1AGE+�2DAYS �
Quantifying this relationship improved the WCB’s claims management methodology.
Claim Classification Our goal was to link the logistic regression models to the decision to classify a claim as high risk. We sought a cutoff point on the probability of conversion that would allow the WCB to classify a claim as either high risk or low risk. Any claim with an estimated probability of conversion equal to or exceeding the cutoff point is classified as a high-risk claim, while any claim that has an estimated probability of conver- sion below the cutoff value is classified as a low-risk claim. Prior to this study, the WCB used a cutoff point of 85
STD days paid for all claims regardless of the nature of injury and the age of injured worker, even though Fattedad and Charron (1999) noted that 70 STD days would have been preferable. It classified any claim with over 85 STD days paid as high risk. The major drawback of this approach was its very low accuracy in identifying converted claims. By using the 85 STD days paid cutoff point within our sample, we classified only 7,538 (55.8 percent) of the 13,512 converted claims correctly. On the other hand, using the same cutoff point, we correctly classified 295,409 (96.1 percent) of the nonconverted claims. Even though the overall clas- sification error was less than six percent (we misclassi- fied only 18,026 claims using this rule), the high error rate (44.2 percent) in the most costly case (classifying a high-risk claim as low risk) clearly indicated the need for an improved approach.
Decision Analytic Approach The managerial decision problem is to classify claims as high risk or low risk based on their probability of conversion. A simple decision tree (Figure 2) repre- sents the problem a claim manager faces once a cutoff
Interfaces Vol. 33, No. 4, July–August 2003 19
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
5
2 5
4 5
6 5
8 5
1 0 5
1 2 5
1 4 5
1 6 5
1 8 5
2 0 5
2 2 5
2 4 5
2 6 5
2 8 5
3 0 5
3 2 5
3 4 5
3 6 5
3 8 5
Number of STD days paid
P ro
b a b ili
ty o
f C
o n v e rs
io n
Age=20
Age=30
Age=40
Age=50
Age=60
Figure 1: We show the estimated probabilities of conversion for claimants with sprains or strains as a function of the number of STD days paid and age of claimant. The greater the number of STD days paid and the older the injured worker, the higher the probability of conversion.
CConv|HR + CInterv
CNonconv + CInterv
CConv|LR
CNonconv
Classify as high risk and intervene
Classify as low risk and do not intervene
Claim converts
Claim does not convert
Claim converts
Claim does not convert
Figure 2: We use a decision tree to classify claims as high risk or low risk. CInterv represents the average per- claim cost of early intervention. It includes the costs for extra claim management and care. CNonconv represents the average cost of claims that do not convert (to long-term disability benefits). CConv�HR represents the average cost of claims classified as high risk that subsequently convert, while CConv�LR represents the average cost of claims classified as low risk that convert. The difference (CConv�LR – CConv�HR) represents that component of the average cost that early intervention may prevent; it exceeds CInterv.
20 Interfaces
Vol. 33, No. 4, July–August 2003
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
point for separating high-risk and low-risk claims has been specified. The managerial objective is to minimize the
expected total cost of claims to the WCB. If the esti- mated probability of conversion for a claim is higher or equal to a cutoff probability P , the claim man- ager should classify it as high risk and intervene. If the estimated probability for the claim is less than P , the claim manager should classify it as low risk and not intervene. The decision analyst’s role is to choose the cutoff point P from a continuum of possible val- ues that affect the probabilities of classifying the two claim types correctly. The analysts’ decision problem can be represented by a decision tree in which choos- ing a cutoff point precedes the claim-classification decision. The quantities CInterv, CNonconv, CConv�HR, and CConv�LR
represent the average costs associated with each deci- sion and outcome and vary among injury types. The quantity CInterv represents the average per-claim cost of early intervention. It includes the costs for extra claim management and care. The quantity CNonconv represents the average cost of claims that do not convert. The quantity CConv�HR represents the aver- age cost of claims classified as high risk that subse- quently convert. CConv�LR represents the average cost of claims classified as low risk that convert. The differ- ence �CConv�LR−CConv�HR� represents that component of the average cost that early intervention may prevent. Our data shows that it exceeds CInterv. These costs may vary with claim characteristics other than injury type, but we did not pursue this issue. Thus, a claim man- ager wishes to avoid two errors; intervening when not necessary and incurring an extra cost of CInterv, and not intervening when necessary and incurring a cost of �CConv�LR−CConv�HR�. Let EC(P �x) denote the expected cost for an indi-
vidual claim associated with the decision rule that is based on a cutoff point P for a claim with character- istics x. This cost is given by
EC�P �x� = �CInterv+CConv�HR� ×Pr�claim is classified as high risk
and converts �x�
+ �CInterv+CNonconv� ×Pr�claim is classified as high risk
and does not convert �x� +CConv�LR×Pr�claim is classified as low risk
and converts �x� +CNonconv×Pr�claim is classified as low risk
and does not convert �x�� Because all the probabilities vary with claim char-
acteristics, to determine the expected cost of using cutoff point P for the whole inventory of claims, we must sum this cost over the distribution of claims with different characteristics. The expected total cost ETC(P ) is
ETC�P� = �CInterv+CConv�HR�×NConv-HR�P� +CConv�LR×NConv-LR�P� + �CInterv+CNonconv�×NNonconv-HR�P� +CNonconv×NNonconv-LR�P��
where for each cutoff point P , NConv-HR�P�= number of claims that converted and
were classified as high risk, NConv-LR�P� = number of claims that converted but
were classified as low risk, NNonconv-HR�P�= number of claims that did not con-
vert but were classified as high risk, and NNonconv-LR�P�= number of claims that did not con-
vert and were classified as low risk. Using estimates of CNonconv, CInterv, CConv�HR, and
CConv�LR, we evaluated ETC(P ) for various values of P using historical data and determined the optimal cut- off point by minimizing ETC�P� with respect to P . CNonconv represents the average cost of nonconverted claims (Table 1). On the other hand, estimating the values of CInterv, CConv�HR, and CConv�LR was difficult and required a new way of thinking about claim- management costs. Prior to this study, WCB manage- ment did not explicitly evaluate these costs. To esti- mate the values of CConv�HR and CConv�LR, we used the average costs of converted claims (Table 1) as refer- ence points. Because these costs resulted from using a cutoff point of 85 STD days paid, we denoted them by CConv�85. If claim managers had not intervened early
Interfaces Vol. 33, No. 4, July–August 2003 21
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
Optimal Model CConv�85 ($) CConv�HR ($) CConv�LR ($) Cutoff Point
Sprain or strain 106,000 97,000 118,000 0.036 Contusion 99,700 92,000 112,000 0.031 Laceration 41,000 37,000 46,000 0.057 Fracture 81,000 77,000 88,000 0.094 Abrasion 95,000 92,000 99,000 0.024 Joint inflammation 89,000 82,000 98,000 0.047 Burn 62,000 57,000 70,000 0.028 Bursitis 72,000 67,000 82,000 0.061 Carpal tunnel 69,000 64,000 76,000 0.109 Amputation 41,000 37,000 44,000 N/A Other 107,000 100,000 115,000 0.063
Table 3: We obtained the average costs associated with categories of claims and outcomes and calculated the corresponding optimal cutoff points. (All figures are in 1991 Canadian dollars.) The amputation model is not based on logistic regression.
for any of the converted claims, the WCB would have incurred an average cost per converted claim of CConv�LR that would be greater than CConv�85. Con- versely, if claim managers had intervened early for all converted claims, the WCB would have incurred an average cost per converted claim of CConv�HR that would be lower than CConv�85. To estimate the values of CConv�LR and CConv�HR, we used CConv�85 as a refer- ence point and adjusted it based on expert knowl- edge within the organization regarding various cost components of converted claims (Table 3). We set CInterv equal to a common value of $1,000 for each injury type. We estimated this amount by taking into account the average intervention time and the aver- age cost of other resources, such as the medical exam- ination and treatment required for rehabilitating the injured worker. We used a simple search procedure to determine
the optimal cutoff point. We calculated ETC�P� for various values of P and identified that value of P that minimized ETC�P� (Table 3). To our knowledge, this is the only study that provides a formal method for combining logistic regression with decision analysis.
Model Accuracy To assess the accuracy of the models, we determined their rates of correct classification using the optimal
cutoff points within the sample and with a different data set. We evaluated the percentages of claims cor- rectly classified for the 1989–1992 claims data, which we used in developing our model (Table 4). Most of the models were highly accurate; eight out of 11 have over 80 percent overall accuracy. They were more accurate in predicting nonconverted claims than con- verted claims with six out of the 11 strata having over 90 percent accuracy. They were fairly accurate in pre- dicting converted claims; eight out of 11 models had over 70 percent accuracy. The final step of the model-building process was
cross-validation. We investigated the model’s perfor- mance on a set of data that contained all 77,815 STD claims with injury dates in 1993. We used the logistic regression models to predict the likely outcome (con- verted or nonconverted) of all claims from the cross- validation set. For each model, we used the optimal cutoff point of the 1989–1992 data set (Table 3). The percentages of claims correctly classified in the cross- validation sample are very similar to the percentages of claims correctly classified in the 1989–1992 sam- ple. These results provide strong support for using the logistic regression models for claim classification.
Critical Number of STD Days Paid Claim managers found it easier and more intuitive to make decisions based on claim characteristics rather than cutoff probabilities. Consequently, we translated the optimal cutoff point on the probability scale into critical values of the predictor variables incorporated into the models. Because the logistic regression mod- els explicitly incorporated age and number of STD days paid and implicitly involved nature of injury, for each case (claim with a specific age and injury type), we transformed the cutoff point on the probability scale to a cutoff point on the number-of-STD-days- paid scale for different ages. We refer to this as the critical number of STD days paid. A claim is high risk as soon as the number of STD
days paid equals or exceeds the critical level. For example, the optimal cutoff point of 0.094 for frac- ture claims translates into a critical number of 50 STD days paid for a 40-year-old claimant (not the 85 days the WCB traditionally used for all claims) (Table 5).
22 Interfaces
Vol. 33, No. 4, July–August 2003
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
Accuracy: 1989–1992 Claims Cross-Validation: 1993 Claims
Converted Claims Nonconverted Claims Overall Correctly Converted Claims Nonconverted Claims Overall Correctly Model Correctly Predicted (%) Correctly Predicted (%) Classified (%) Correctly Predicted (%) Correctly Predicted (%) Classified (%)
Sprain or strain 70�8 91�3 90�7 74�1 89�9 89�4 Contusion 74�3 93�0 92�5 77�9 92�2 91�8 Laceration 74�2 93�9 93�1 73�9 93�5 92�6 Fracture 85�4 73�5 75�4 85�1 71�6 73�9 Abrasion 62�9 99�6 99�4 63�6 99�3 99�0 Joint inflammation 71�6 83�8 83�1 79�9 80�4 80�3 Burn 82�3 96�0 95�7 72�0 95�6 95�2 Bursitis 73�4 76�7 76�4 77�1 75�6 75�7 Carpal tunnel 65�5 75�4 74�3 73�7 73�2 73�3 Amputation 100 0 83�3 100 0 83�0 Other 53�9 93�2 91�4 60�9 91�1 89�4
Table 4: To assess the accuracy of the logistic regression models, we determined their rates of correct classifi- cation using the optimal cutoff points within the sample and a different data set. Most of the models were highly accurate; eight out of 11 models have over 80 percent overall accuracy. The percentages of claims correctly classified in the cross-validation sample are very similar to the percentages of claims correctly classified in the 1989–1992 sample. These results provide strong support for using the logistic regression models in making decisions.
In practice, we use a table that provides the critical number of STD days for every age between 18 and 65. For most combinations of age and injury type, the critical level is substantially below 85 days. For all injuries for which we incorporated age as a predic- tor, the value of the critical number of STD days paid decreases as age increases. This means that the WCB should intervene sooner for older claimants. On the other hand, for abrasions, burns, and carpal tunnel
Age
Model 20 25 30 35 40 45 50 55 60
Sprain or strain 93 85 77 69 61 53 45 37 30 Contusion 66 62 57 52 47 42 38 33 28 Laceration 37 36 35 34 32 31 30 28 27 Fracture 58 56 54 52 50 48 46 44 42 Abrasion 46 46 46 46 46 46 46 46 46 Joint inflammation 67 61 54 48 41 35 28 21 15 Burn 35 35 35 35 35 35 35 35 35 Bursitis 56 52 48 45 41 37 34 30 26 Carpal tunnel 79 79 79 79 79 79 79 79 79 Other 76 71 67 62 57 52 47 42 37
Table 5: We show the critical number of STD days paid for some repre- sentative values of the age of the injured worker; we classify a claim as high risk as soon as the number of STD days paid is equal to or exceeds the critical value.
syndrome, for which we did not include age as a pre- dictor, the critical number of STD days does not vary with age.
Estimated Savings To estimate savings from using our decision-analytic approach, we first calculated ETC�P� on the basis of the old practice of using 85 STD days paid as the cutoff point. We then computed ETC�P� using the optimal cutoff points (Table 3) and evaluated the dif- ference between the two costs (Table 6). Under the classification scheme we proposed, many more con- verted claims were classified as high risk in each category. The greatest savings occur for lacerations, sprains
and strains, amputations, contusions, and fractures. The reductions in costs mainly derive from three sources. One is the cost difference �CConv�LR−CConv�HR�; the greater the difference between the cost of con- verted claims classified as low risk and the cost of converted claims classified as high risk, the greater the savings (Table 3). A second factor affecting sav- ings is the difference between our model’s critical number of STD days and the 85 STD days previously used, that is, the greater the difference the greater the
Interfaces Vol. 33, No. 4, July–August 2003 23
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
Cutoff Point = 85 STD Days (Old Practice) Decision-Analytic Approach (New Practice) ETC�P � ETC�P � Savings
Model NConv-HR NConv-LR NNonconv-HR NNonconv-LR ($ million) NConv-HR NConv-LR NNonconv-HR NNonconv-LR ($ million) ($ million)
Sprain or strain 3�128 1�931 7�284 146�757 910�62 3�580 1�479 13�396 140�645 907�69 2�93 Contusion 800 595 1�157 48�165 220�47 1�036 359 3�437 45�885 218�27 2�20 Laceration 620 1�226 304 42�084 125�74 1�369 477 2�586 39�802 122�03 3�71 Fracture 1�783 774 1�496 12�110 263�74 2�184 373 3�602 10�004 261�84 1�90 Abrasion 25 37 19 11�515 11�62 39 23 48 11�486 11�57 0�05 Joint inflammation 353 288 616 9�839 85�36 459 182 1�699 8�756 84�85 0�51 Burn 54 87 48 6�906 16�00 116 25 281 6�673 15�48 0�52 Bursitis 199 184 377 4�124 45�89 281 102 1�050 3�451 45�41 0�48 Carpal tunnel 119 78 338 1�171 23�08 129 68 371 1�138 23�00 0�08 Amputation 189 393 5 112 24�87 582 0 117 0 22�62 2�25 Other 268 381 408 12�626 103�82 350 299 882 12�152 103�15 0�67
Total Savings 15�30
Table 6: To estimate savings from using the decision-analytic approach, we first calculated the expected total cost, ETC(P ), based on the old practice of using 85 STD days paid as the cutoff point. We then computed the ETC(P ) using the optimal cutoff points (Table 3) and computed the difference between the two costs (all figures are in 1991 Canadian dollars). Under the classification scheme we proposed, many more converted claims were classified as high risk in each category.
savings (Table 5). A third factor is the conversion rate, that is, the greater the conversion rate, the higher the savings. We estimated that the WCB would save
$15.3 million over four years by implementing the pro- posed method, or approximately $3.8 million annually (1991 value). This represents about $4.7 million in September 2002 dollars (adjusted using the British Columbia consumer price index).
Implementation The compensation services division of the WCB administers STD claims and is responsible for deter- mining a worker’s rights to compensation and the amount of benefits awarded. To implement our results, the compensation services division developed a claim-profiling scorecard. The scorecard includes several quantitative and qualitative measures that are indicators of whether a claim is likely to become high risk. These measures were categorized as primary or secondary. The only primary measure is the age- and injury-type-dependent critical number of STD days paid (Table 5). Secondary measures include severity of injury, expected recovery time of over 12 weeks,
prior injury or pre-existing condition in the area of the body where the injury occurred, and whether or not the claimant had previous claims with the WCB. Several of the secondary measures may not be
known in the early stages of claim processing. Con- sequently, if the number of STD days paid on a claim exceeds the age- and injury-dependent critical value, the WCB classifies the claim as high risk regardless of the status of the secondary measures. In a pilot study, one of the WCB’s area offices used
this system for more than four months at the begin- ning of 2000. Because the pilot study showed that under the new system the office processed claims in a more cost-effective way than it had previously, the WCB now uses it in all its offices and has incorporated it into the compensation services division’s computer- based claims-management system to automate the process. The new scorecard system has greatly improved
the WCB’s processing of STD claims. It helps the WCB to use its adjudication staff more efficiently. First-level adjudicators working in the call center con- tinue to deal with simple, straightforward claims, but they send claims classified as high-risk to the next
24 Interfaces
Vol. 33, No. 4, July–August 2003
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
adjudication level (claim managers) as soon as they identify them.
Conclusion Early identification of potentially high-risk claims is crucial for any workers’ compensation board because it can then improve claims management, provide early preventive intervention that is likely to reduce future costs, and return claimants to work sooner. The implementation of this new method has consid-
erably improved the practice of claims management at the WCB. An immediate and unanticipated benefit is that the WCB has fewer claims in the queue waiting to be processed because it identifies high-risk claims earlier. The biggest advantage for the WCB of this approach is reduced claims costs because of its appro- priate early intervention; about $4.7 million per year. Further, we anticipate that more injured workers will return to work sooner than in the past. This conser- vative estimate does not include the increased contri- bution of workers returning to the workforce sooner and does not account for the possibility that managed claims do not eventually convert. As a result of this study, the WCB increasingly accepts and adopts statis- tical and operations research methods for improving all aspects of its operations. In the next step of our study, we will focus on deter-
mining whether we should incorporate other predic- tors into the logistic regression models to improve their prediction accuracy. Predictors that we might consider are gender, injured body part, and worker’s industry. Model recalibration will be ongoing.
Acknowledgments This research was partially supported by Natural Sciences and Engineering Research Council of Canada and the Mathematics of Information Technology and Complex Systems national centers of excellence programs and was carried out in collaboration with the Centre for Operations Excellence at the University of British Columbia. This paper was awarded the 2002 Canadian Operations Research Society Practice Prize.
References Fattedad, S., M. Charron. 1999. Claims inventory control at the
Workers’ Compensation Board of British Columbia. Unpub- lished study, British Columbia, Canada.
Head, G. L. 1995. Rehabilitation management. G. L. Head, ed. Essentials of Risk Control, 3rd ed. Vol. 1. Insurance Institute of America, Malvern, PA, 280–282.
Hosmer, D. W., S. Lemeshow. 1989. Applied Logistic Regression. John Wiley and Sons, New York.
Jessup, B., K. Gallie. 1996. Profiling high-risk claims at the Workers’ Compensation Board of British Columbia. Unpublished study, British Columbia, Canada.
Johnson, D. E. 1998. Applied Multivariate Methods for Data Analysis. Duxbury Press, Pacific Grove, CA.
Lemeshow, S., D. Teres, J. S. Avrunin, H. Pastides. 1988. Predict- ing the outcome of intensive care unit patients. J. Amer. Statist. Association 83(402) 348–356.
McCullagh, P., J. A. Nelder. 1989. Generalized Linear Models, 2nd ed. Chapman and Hall, London, U.K.
Neter, J., M. H. Kutner, C. J. Nachtsheim, W. Wasserman. 1996. Applied Linear Statistical Models, 4th ed. Richard D. Irwin, Chicago, IL.
Ryan, T. P. 1996. Modern Regression Methods. Wiley-Interscience, New York.
Thompson, C. M. 1985. Characteristics associated with outcome in a community mental health partial hospitalization program. Community Mental Health J. 21(2) 179–188.
Wiginton, J. C. 1980. A note on the comparison of logit and discrim- inant models of consumer credit behavior. J. Financial Quant. Anal. 15(3) 757–770.
Ralph W. McGinn, P. Eng., President and CEO, Workers’ Compensation Board of British Columbia, P.O. Box 5350, Station Terminal, Vancouver, British Columbia, Canada V6B 5L5, writes: “This letter is to confirm the implementation of the method of early detection of high-risk claims at the Workers’ Compen- sation Board (WCB) of British Columbia, described in the article, ‘Early Detection of High-Risk Claims at the Workers’ Compensation Board of British Columbia.’ “During the past few years the WCB has become
increasingly involved in developing quantitative tools for improving its business decision making. This pro- cess was catalyzed by the implementation of our data warehouse that gave our staff the opportunity to access detailed information on more than 3 million claims. The ‘Early Detection of High-Risk Claims’ project is one of the major achievements for the research staff of our organization, and the ‘critical short-term disability days’ provided by the study are being used in the Compensation Services division in a ‘claim profiling’ scorecard to early detect and manage high-risk claims.
Interfaces Vol. 33, No. 4, July–August 2003 25
URBANOVICH, YOUNG, PUTERMAN, AND FATTEDAD The Workers’ Compensation Board of British Columbia
“The method of early detection of high-risk claims has considerably improved the practice of claims management at the WCB. The immediate impact is that we have fewer claims in the queue (waiting to be processed) since potentially high-risk claims are identified earlier. In the longer term, we expect that the early intervention on high-risk claims will reduce their average cost by $2,000–$3,000 and return
workers to their place of employment sooner. The quantifiable savings for the WCB are difficult to deter- mine at this time, as the effects of using the new method cannot be fully measured since the new WCB claims management system needs a warm-up period. We expect, however, annual savings of millions of dollars as a result of the implementation of the new method.”
26 Interfaces
Vol. 33, No. 4, July–August 2003