For Concept Writer Only

profilePsygirl76
Aegisdottir.pdf

Major Contribution

The Meta-Analysis of Clinical Judgment Project: Fifty-Six Years of Accumulated Research on Clinical Versus Statistical Prediction

Stefanía Ægisdóttir Michael J. White Paul M. Spengler

Alan S. Maugherman Linda A. Anderson

Robert S. Cook Cassandra N. Nichols

Georgios K. Lampropoulos Blain S. Walker Genna Cohen

Jeffrey D. Rush Ball State University

Clinical predictions made by mental health practitioners are compared with those using statistical approaches. Sixty-seven studies were identified from a comprehensive search of 56 years of research; 92 effect sizes were derived from these studies. The overall effect of clinical versus statistical prediction showed a somewhat greater accuracy for statisti- cal methods. The most stringent sample of studies, from which 48 effect sizes were extracted, indicated a 13% increase in accuracy using statistical versus clinical methods. Several variables influenced this overall effect. Clinical and statistical prediction accu- racy varied by type of prediction, the setting in which predictor data were gathered, the type of statistical formula used, and the amount of information available to the clinicians and the formulas. Recommendations are provided about when and under what conditions counseling psychologists might use statistical formulas as well as when they can rely on clinical methods. Implications for clinical judgment research and training are discussed.

A large portion of a counseling psychologist’s work involves deciding what information to collect about clients and, based on that information, predicting future client outcomes. This decision making can occur both at the microlevel, such as moment-to-moment decisions in a counseling session, and at the macrolevel, such as predictions about outcomes such as suicide risk, violence, and response to treatment (Spengler, Strohmer, Dixon, & Shivy, 1995). Because the quality of client care is often determined

THE COUNSELING PSYCHOLOGIST, Vol. 34 No. 3, May 2006 341-382 DOI: 10.1177/0011000005285875 © 2006 by the Society of Counseling Psychology

341

TCP285875.qxd 3/29/2006 12:54 PM Page 341

by the accuracy of these decisions (Dawes, Faust, & Meehl, 1989; Meyer et al., 1998; Spengler, 1998), determining the best means for decision making is important.

Two major approaches to decision making have been identified: the clin- ical and the statistical, which is also called mechanical (Dawes et al., 1989). Clinical prediction refers to any judgment using informal or intuitive processes to combine or integrate client data. Psychologists use the clinical method when their experience, interpersonal sensitivity, or theoretical perspective determines how they recall, synthesize, and interpret a client’s characteris- tics and circumstances.

Such intuitive or “gut-level” inferences are greatly reduced in the statis- tical approach. Predictions are based on empirically established relations between client data and the condition to be predicted (Dawes et al., 1989). A psychologist who declares that his or her clinical impression suggests a client may be suicidal has used the clinical method. By contrast, when using the statistical method, client data are entered into formulas, tables (e.g., actuarial tables), or charts that integrate client information with base rate and other empirical information to predict suicide risk. While the sta- tistical method is potentially 100% reproducible and well specified, the clinical method is neither as easily reproduced nor as clearly specified (Grove, Zald, Lebow, Snitz, & Nelson, 2000).

Meehl (1954) contended that while the clinical method requires specific

342 THE COUNSELING PSYCHOLOGIST / May 2006

Alan S. Maugherman is now in private practice at the Center for Psychological Development, Muncie, Indiana. Linda A. Anderson is now at the University Counseling and Psychological Services, Oregon State University. Robert S. Cook is now in private practice at Lifespan, North Logan, Utah. Cassandra N. Nichols is now at the Counseling and Testing Services, Washington State University. Blain S. Walker is now at the Tripler Army Medical Center, Honolulu, Hawaii. Genna R. Freels is now at the Louisiana Professional Academy, Lafayette, Louisiana. Funding for this project was provided to Paul M. Spengler by Grant MH56461 from the National Institute of Mental Health, Rockville, Maryland; by six grants to Paul M. Spengler from the Internal Grants Program for Faculty, Ball State University, Muncie, Indiana (two summer research grants, two summer graduate research assistant grants, an academic year research grant, and a new faculty research grant); and by three Lyell Bussell summer graduate research assistant grants, Ball State University, Muncie, Indiana. Preliminary findings from the clinical versus statistical prediction meta-analysis were presented at the annual meetings of the American Psychological Association in Washington, D.C., August 2000; Boston, August 1999; San Francisco, August 1998; Toronto, Ontario, Canada, August 1996; and New York City, August 1995, and by invited address at the annual meeting of the Society for Personality Assessment, New Orleans, March 1999. The authors extend special thanks to Kavita Ajmere, Corby Bubp, Jennifer Cleveland, Michelle Dorsey, Julie Eiser, Layla Hunton, Christine Look, Karsten Look, K. Christopher Rachal, Teresa Story, Marcus Whited, and Donald Winsted III for instrumental assistance in obtaining articles, coding studies, and managing data for the project. Correspondence concerning this article should be addressed to Stefanía Ægisdóttir, Department of Counseling Psychology, Teachers College 622, Ball State University, Muncie, IN 47306; e-mail: [email protected].

TCP285875.qxd 3/29/2006 12:54 PM Page 342

training, the statistical method does not. The statistical method requires only inserting data into a formula specifically designed for a particular judgment task. This may not be entirely true. Despite the use of formulas or tables to inte- grate information, the statistical method may require advanced training in the collection of relevant clinical and research-based information. Furthermore, advanced training may enhance a clinician’s perceptions, which in turn may be quantified and used in a statistical model. For example, a clinician may believe a client has the potential for suicide, translate this impression into a number on a rating scale, and then statistically combine this number with other data to predict the client’s risk for suicide (e.g., Westen & Weinberger, 2004).

To determine how counseling psychologists can be most effective in their decision making, knowing when and under what conditions each method is superior is important. The purpose of our meta-analysis is to articulate this knowledge.

THE CLINICAL VERSUS STATISTICAL PREDICTION CONTROVERSY

The search for the most accurate decision-making method is not new. In fact, this question has been debated for more than 60 years (Dawes et al., 1989; Meehl, 1954). The debate began with Meehl’s (1954) book Clinical Versus Statistical Prediction, in which Meehl theoretically analyzed the relation between the clinical and statistical methods of prediction and sum- marized findings from existing literature. Meehl found that in all but 1 of 20 studies, statistical methods were more accurate than or equally accurate as the clinical method. He concluded that clinicians’ time should be spent doing research and therapy, whereas work involving prognostic and classi- fication judgments should be left to statistical methods.

Holt (1958), the most adamant defender of the clinical method, criticized Meehl’s (1954) conclusions. Holt’s critique involved essentially two issues: (a) the identification and assessment of predictive variables and (b) how they should be integrated. Holt believed that Meehl had given insufficient attention to the sophistication with which clinicians identify the criteria they are predicting, what variables to use in their prediction, and the strength of the relationship between predictors and criteria. In Holt’s view, clinicians can identify these variables only through training and experience with com- parable cases. After identifying the relevant variables, they are assessed. Assessment may be as much qualitative as quantitative. Holt’s second criti- cism was that Meehl pitted “naïve clinical integration” of prediction against statistical decision making. A fairer comparison would compare statistical methods with “sophisticated clinical decision making and integration” (Holt,

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 343

TCP285875.qxd 3/29/2006 12:54 PM Page 343

1958) According to Holt, sophisticated clinical decision making is based on sophisticated data. These data are both qualitative and quantitative, have been gathered in a systematic manner, and have known relationships with what is being predicted. Unlike the statistical approach, the clinician remains the prime instrument, combining the data and making predictions that are tailored to each person. Holt presented data suggesting a superiority for sophisticated clinical rather than statistical procedures in predicting suc- cess in clinical training. On the basis of these findings, Holt argued for a combination of clinical and statistical methods (i.e., sophisticated clinical) that would be systematic and controlled and sensitive to individual cases.

Since this time, other narrative and box-score reviews of the literature on the differential accuracy of clinical and statistical methods have been published (e.g., Dawes et al., 1989; Garb, 1994; Grove & Meehl, 1996; Kleinmuntz, 1990; Russell, 1995; Sawyer, 1966; Wiggins, 1981). Narrative reviews are traditional literature reviews; box-score reviews count statistical signifi- cance and summarize studies in a table format. These reviews nearly always supported Meehl’s (1954) conclusion that statistical methods were more accurate than or, at minimum, equally as accurate as clinical prediction methods (for a rare exception, see Russell, 1995). A recent meta-analysis of the clinical versus statistical literature (Grove et al., 2000) also supported earlier findings. Grove et al. (2000) found a consistent advantage (d = .12) for statistical prediction over clinical prediction across various types of nonmental health and mental health predictors and criteria.

Influence of the Statistical Versus Clinical Prediction Controversy

Despite the repeated conclusion that statistical prediction methods are more accurate than clinical procedures, the findings have had little influ- ence on clinical practice (Dawes et al., 1989; Meehl, 1986). Dawes et al. (1989) and Meehl (1986) offered several reasons for this. They suggested that clinicians lack familiarity with the literature on clinical versus statisti- cal prediction, are incredulous about the evidence, or believe that the com- parisons were procedurally biased in favor of statistical prediction methods. They also proposed that certain aspects of education, training, theoretical orientation, and values might influence their reluctance to recognize advan- tages associated with statistical decision methods. Most clinicians highly value interpersonal sensitivity. Because of this, some may believe that the use of predictive formulas dehumanizes their clients. A corollary is that the use of group-based statistics or nomothetic rules is inappropriate for any particular individual. Practitioners are also subject to confirmatory biases such that they recall instances in which their predictions were correct but fail

344 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 344

to recall those instances in which statistical prediction was more accurate. One might add another reason: Some accounts have simply been too broad to convince mental health practitioners. In some instances (e.g., Grove et al., 2000), the literature that has reviewed clinical versus statistical prediction includes research and criteria that range from mental health to medicine to finance.

Use of Statistical Prediction Models

Perhaps as a result of the limited influence of clinical versus statistical comparison studies, few statistical prediction models are available to coun- seling psychologists and psychotherapy practitioners (Meyer et al., 1998). Clinicians working in forensic settings, however, have developed such models. In fact, numerous funded research projects have been conducted to aid in classifying juvenile and adult prison inmates (e.g., Gottfredson & Snyder, 2005; Quinsey, Harris, Rice, & Cormier, 1998; Steadman et al., 2000; Sullivan, Cirincione, Nelson, & Wallis, 2001). One such effort is the Violence Risk Appraisal Guide (VRAG; Quinsey et al., 1998), which is a statistical system for predicting recidivism of imprisoned violent offenders.

The VRAG is based on more than 600 Canadian maximum security inmates who were released either back to the community, to a minimum security hospital, or to a halfway house. After a series of correlation analy- ses of predictor and outcome variables, a set of stepwise regression models was conducted. These analyses reduced the original 50 predictors to 12. These include psychopathy checklist scores (Hare, 1991), elementary school mal- adjustment scores, presence of a personality disorder, age at time of offense, separation from parents at an age younger than 16, failure on prior condi- tional release, nonviolent offense history score (using an instrument), marital status, schizophrenia diagnosis, most serious injury of offender’s victim, alcohol abuse score, and gender of offender’s victim. Each predictor was assigned a specified weight based on the empirical relationship with the outcome variable. Summing the resultant scores yields a probability estimate for an offender’s future violence within the next 7 and 10 years. For instance, scores between +21 and +27 indicate a 76% likelihood for future violence, whereas scores between –21 and –15 suggest a probability of only 8%. The authors have validated this model for different groups of inmates (e.g., arsonists or sex offenders), with promising results (see Quinsey et al., 1998, for more detailed use of this statistical model).

In addition to forensics, statistical prediction formulas have been developed to aid with student selection for undergraduate, graduate, and profes- sional schools. As an example, Swets et al. (2000) described a statistical prediction formula used in selecting candidates at the University of Virginia

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 345

TCP285875.qxd 3/29/2006 12:54 PM Page 345

School of Law. This formula consists of four predictor variables: undergradu- ate grade point average (GPA), mean GPA achieved by students from the appli- cants’ college, scores from the Law School Admissions Test (LSAT), and the mean LSAT score achieved by all students from the applicants’ college. Scores from these predictors are combined into a decision index of which a specific score indicates a threshold for admission. This statistical prediction formula predicts grades for 1st-year students and is used in combination with variables that are harder to quantify to select students (cf. Swets et al., 2000). Harvey- Cook and Taffler (2000) developed a statistical model using biographical data, frequently found on application forms and resumes, to predict success in accounting training in the United Kingdom. This six-variable model was devel- oped on 419 accounting trainees. Retesting it on an independent sample of 243 trainees, Harvey-Cook and Taffler showed that their model could classify 88% of those failing and 33% of those successful in accounting training. The authors concluded that their model delivered better and more cost-effective results than clinical judgment methods currently used for this purpose in the United Kingdom (Harvey-Cook & Taffler, 2000).

Test cutoff scores offer another instance of a statistical procedure that may aid clinical decision making. Indeed, cutoff scores may be more readily avail- able and easily constructed than statistical formulas. As an example, three Minnesota Multiphasic Personality Inventory–2 (MMPI-2) scales have been useful in classifying substance abuse: MacAndrew Alcoholism–Revised (MAC-R), Addiction Potential Scale (APS), and Addiction Acknowledgment Scale (AAS) (Rouse, Butcher, & Miller, 1999; Stein, Graham, Ben-Porath, & McNulty, 1999). Relying on data from 500 women and 333 men seeking outpatient mental health services, Stein et al. (1999) found that cutoff scores on the MAC-R correctly classified 86% of the women and 82% of the men as either substance abusers or nonabusers. In the case of the AAS, cutoff scores could predict 92% of women and 81% of men as either substance abusers or nonabusers. Likewise, cutoff scores with the APS enabled accurate prediction of 84% of women and 79% of men as either abusing or not abusing substances. This method of classification greatly exceeds the base rates for chance classi- fication. For women, the positive predictive power (ability to detect substance abusers) for MAC-R, AAS, and APS was 100%, 79%, and 53%, respec- tively. These values compare with a base rate of 16%. For men, the respec- tive positive predictive power for MAC-R, AAS, and APS was 100%, 68%, and 77%, respectively, which compare with a base rate of 27%.

Purpose of This Meta-Analysis

The current meta-analysis seeks to address several omissions in the liter- ature on clinical versus statistical prediction. Although Grove et al.’s (2000)

346 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 346

important study confirmed prior conclusions about the relative merits of clinical and statistical prediction methods, questions still remain regarding the application of the findings to judgment tasks commonly encountered by mental health practitioners. First, their review combined literature from psy- chology, medicine, forensics, and finance. Consequently, conclusive results are not provided about prediction accuracy for mental health clinical and counseling practitioners relative to statistical methods. Second, even though Grove et al. examined the influence of various study design characteristics (e.g., type of criterion, professional background of clinical judges, judge’s level of experience, and amount of data available to the judges versus the statistical formulas), the influence of these design characteristics on the accuracy of prediction was not investigated when the criteria were psycho- logically related. Instead, Grove et al. investigated the influence of these study design variables on the overall effect, including studies from the diverse professional fields listed earlier. Similarly, despite Grove et al.’s examination of the influence of criterion type on the overall effect of the difference between clinical and statistical prediction accuracy, their criteria breakdown was broad (i.e., educational, financial, forensic, medical, clinical- personality, and other). The breakdown offers little specific information on which counseling psychologists can rely to decide when and under what conditions they should use clinical or statistical methods.

The first aim of this meta-analysis was to synthesize studies that had examined the differential accuracy of clinical and statistical judgments in which the prediction outcome was relevant to counseling psychology. Second, we examined studies in which predictions by mental health professionals were compared with statistical methods. In a typical study comparing these two methods, clinicians first synthesized client data (e.g., interview data, psychological tests, or a combination of interview information and one or more psychological tests) and then made a classification judgment (e.g., diagnosis) or predicted some future outcome (e.g., prognosis). The accuracy of these judgments was compared with a statistical prediction scheme in which the same (sometimes less or more) information was entered into a statistical formula that had been previously designed on the basis of empir- ical relations between the predictors (specific client data) and the criterion (the prediction task of interest). Third, we examined questions generated from the years of debate about the relative merits of clinical and statistical prediction. More specifically, we examined how the differential accuracy between clinical and statistical methods was affected by (a) type of predic- tion, (b) setting from which the data were gathered, (c) type of statistical formula, (d) amount of information provided to the clinician and formula, (e) information provided to the clinician about base rates, (f) clinician access to the statistical formula, (g) clinician expertness, (h) our evaluation

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 347

TCP285875.qxd 3/29/2006 12:54 PM Page 347

of the validity of the criteria for accurate judgment, (i) publication source, (j) number of clinicians performing predictions in a study, (k) number of criterion behaviors predicted in a study, and (l) publication year.

Meta-analyses provide detailed and comprehensive syntheses of the professional literature. As such, they are especially relevant for bridging the gap between the science of counseling psychology and how it is prac- ticed by counseling psychologists (e.g., Chawalisz, 2003; Stricker, 2003; Wampold, 2003). The current meta-analysis addresses how counseling psychologists should best make decisions: when they should use clinical methods, when they would do well to use statistical methods, and when either is acceptable. In addition to relying on empirically supported treat- ment strategies, the counseling psychologist scientist-practitioner may be informed by the current meta-analysis about situations when statistical decision methods lead to more accurate clinical predictions than the clini- cal method.

Spengler et al. (1995), for instance, proposed an elaborated model of the scientist-practitioner, basing their clinical judgment model on Pepinsky and Pepinsky (1954). In this model, strategies were proposed to increase judgment accuracy relying on scientific reasoning. They suggested that to improve judgment accuracy, counseling psychologists (a) should be aware of their values, preferences, and expectations; (b) should use multiple methods of hypothesis testing (both confirming and disconfirming); and (c) should use judgment debiasing techniques (cf. Spengler et al., 1995). We argue that the current meta-analysis will further inform counseling psychologists as scientists not by providing information about the absolute accuracy of clinical judgment (i.e., when it may be most vulnerable to error) but instead by assessing the relative accuracy of clinical versus statistical prediction. Under conditions in which statistical prediction is superior, a successful debiasing method would use prediction methods based on empirical relations between variables (i.e., statistical methods). On the basis of this meta-analysis, we hope to also suggest options for future research and training relevant to decisions typically made by counseling psychologists.

METHOD

Study Selection

This study is part of a large-scale meta-analysis of the clinical judgment (MACJ) literature (Spengler et al., 2005). By using 207 search terms, the MACJ project identified 1,135 published and unpublished studies between

348 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 348

1970 and 1996 that met our criteria for inclusion in meta-analyses of mental health clinical judgment.1 However, because of the extensive historical debate about the relative benefits of statistical versus clinical prediction, we extended our search strategy for the present study back to 1940, thus defining the current study’s search period from 1940 to 1996. After an iterative process, we identified 156 studies that investigated some form of statistical predic- tion or model of clinical prediction for a mental health criterion compared with the accuracy of clinical judgment.

To be included in the meta-analysis, studies had to meet the following criteria: (a) a direct comparison was reported between predictions made by mental health practitioners (i.e., professionals or graduate students) and some statistical formula, (b) a psychological or a mental health prediction was made (e.g., diagnosis, prognosis, or psychological adjustment), (c) the clinicians and the statistical formula had access to the same predictor vari- ables or cues (even though the amount of information might vary), (d) the clinicians and the formula had to make the same predictions, and (e) the studies had to contain data sufficient to calculate effect sizes. By using these selection criteria, 67 studies qualified for inclusion, yielding 92 effect sizes. When Goldberg (1965) and Oskamp (1962) were included, 69 studies pro- duced 173 effect sizes (see below).

Specialized Coding Procedures

The MACJ project used a coding form with 122 categories or character- istics (see Spengler et al., 2005) that were grouped under the following con- ceptual categories: judgment task, judgment outcomes, stimulus material, clinician individual differences, standard for accuracy, method of study, and type of design. An additional coding form was constructed including study design characteristics identified in historical literature and more contempo- rary research as potentially affecting the differential accuracy of clinical and statistical prediction. These design characteristics became the indepen- dent variables. We also noted whether the statistical formulas were cross- validated. In this instance, cross-validated formulas refer to any statistical formulas that have been independently validated on a different sample from which the formula was originally derived. For example, if a score of 10 on an instrument developed to diagnose major depressive disorder correctly identifies 95% of persons with that disorder, to be considered a cross- validated formula (i.e., a score of 10 indicates major depression), that same score (10) had to be able to identify major depressive disorder with com- parable accuracy using another sample of persons with the disorder. Coding disagreements were resolved by discussion among coders until agreement was reached.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 349

TCP285875.qxd 3/29/2006 12:54 PM Page 349

Dependent Measure: Judgment Accuracy

The dependent variable for all analyses was judgment accuracy. For a study to be included, a criterion had to be established as the accurate judgment (e.g., prior diagnosis or arrest records). For instance, Goldberg (1970) compared clinical and statistical judgments of psychotic versus neurotic MMPI profiles to actual psychiatric diagnosis. MMPI profiles from psychiatric patients diagnosed as clearly psychotic or neurotic were presented to clinical psychologists. Their judgment about whether the MMPI profiles belonged to either a psychotic or a neurotic patient was compared with a statistical formula constructed to categorize patients as psychotic if five MMPI scales (the lie, 6 [Pa], 8 [Sc], 3 [Hy], 7 [Pt]) were elevated. These two types of judgments were compared with the prior diagnoses, which were considered the accurate judgment. In another example, Gardner, Lidz, Mulvay, and Shaw (1996) examined clinical and statistical prediction of future violence. Gardner et al. developed three sta- tistical formulas to predict future violence on the basis of clinical (e.g., diagnosis and drug use) and demographic information as well as informa- tion about prior violence. Violence prediction based on these three models was compared with predictions made by clinicians who had access to the same information as the formulas. The accuracy of these judgments was then compared with records of violent behavior (psychiatric, arrest, or commitment records) or from patients’ reports about their violent behav- ior. In this study, available records and patient self-reports about violent behavior served as the criteria for accurate judgment. Thus, specific crite- ria for accurate judgments had to be reported for a study to be included in this meta-analysis.

Effect Size Measure

As Cohen (1988) noted in his widely read book, effect sizes may be likened to the size of real differences between two groups. Estimates of effect size are thus estimates of population differences—they estimate what is really happening and are not distorted by sample size. The purpose of a meta-analysis is to estimate the effect size in a population of studies. In our case, a mean weighted effect size (d+) was used to represent the differ- ence between clinical and statistical prediction accuracy.2 Effect size mea- sured by d+ represents the mean difference between two samples of studies expressed in standard deviation units (g) and corrected for sample size (Johnson, 1993). More specifically, the mean judgment accuracy of statis- tical prediction was subtracted from the mean judgment accuracy of clinical

350 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 350

prediction divided by the pooled standard deviation and then corrected for sample size.

In this study, the effect size (d+) represents the magnitude, not the statis- tical significance, of the relative difference between clinical and statistical prediction accuracy. A negative d+ value indicates superiority of the statis- tical prediction method, whereas a positive d+ indicates superiority of the clinical method. An effect of zero indicates exactly no difference between the two methods. In addition to d+, we reported the 95% confidence interval for the effect size. Confidence interval provides the same information as that extracted from significant tests. It permits one to say with 95% confidence (i.e., α = .05) that the true effect size falls within its boundaries. If the confi- dence interval includes zero, the population effect may be zero; one cannot say with confidence that a meaningful difference exists between the two groups. However, if the confidence interval does not include zero, one can conclude that a reliable difference exists between clinical and statistical prediction (e.g., Johnson, 1993).

The data were reduced to one representative effect size per study in most cases. This prevented bias that would result if a single study was overrepre- sented in the sample (Cooper, 1998; Rosenthal, 1991). For instance, if a study reported more than one statistical or clinical prediction (e.g., brain impair- ment and lateralization of the impairment; Adams, 1974), an average of the reported judgment accuracy statistic was calculated and transformed into one effect size. Also, if a study reported results from both non–cross-validated and cross-validated statistical prediction schemes, only results from the cross- validated statistical formula were used. This was done to prevent bias in favor of the statistical method, given the possibility of inflated correlations (based on spurious relations) between predictor and criterion variables in non–cross- validated statistical formulas (for more discussion of these issues, see Efron & Gong, 1983). Table 1 notes whether the studies used cross- or non–cross- validated statistical formulas.

Even though one average effect size per study was usually calculated, 18 studies produced more than one effect size (see Table 1). These studies included more than one design characteristic (independent variables) that we hypothesized might influence clinical versus statistical prediction accu- racy and reported accuracy statistics for various levels of the independent variable. An example would be a study investigating clinical versus statis- tical prediction under two conditions. In one condition, the clinicians have access to the statistical prediction scheme, whereas in another condition they do not. In our studies, we extracted two effect sizes. That is, the study’s two conditions (with and without access to the statistical formula) were treated as two independent projects. Furthermore, a study was allowed to produce

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 351

TCP285875.qxd 3/29/2006 12:54 PM Page 351

T A

B L

E 1

: S

tu d

ie s

In cl

u d

ed i

n M

et a-

A n

al ys

is

A cc

u ra

cy A

cc u ra

cy S ta

ti st

ic C

it a ti

o n

P re

d ic

ti o n

R ep

o rt

ed C

li n ic

a l

S ta

ti st

ic a l

d+

A da

m s

(1 97

4) 1

B ra

in i

m pa

ir m

en t

H it

r at

e 53

52 .0

2 A

da m

s (1

97 4)

2a B

ra in

i m

pa ir

m en

t H

it r

at e

53 56

–. 06

A le

xa ko

s (1

96 6)

1 A

ca de

m ic

p er

fo rm

an ce

H it

r at

e 39

51 –.

24 A

le xa

ko s

(1 96

6) 2

A ca

de m

ic p

er fo

rm an

ce H

it r

at e

39 52

–. 27

A st

ru p

(1 97

5) b

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

78 74

.0 9

B ar

ro n

(1 95

3) P

sy ch

ot he

ra py

o ut

co m

e H

it r

at e

62 73

–. 23

B lu

m et

ti (

19 72

) L

en gt

h of

p sy

ch ot

he ra

py

H it

r at

e 61

54 .1

5 B

ol to

n et

a l.

( 19

68 )

P ro

gn os

is C

or re

la ti

on .3

5 .4

8 –.

16 C

ar li

n an

d H

ew it

t (1

99 0)

R ea

l vs

. ra

nd om

M M

P I

pr of

il e

H it

r at

e 63

95 –.

73 C

on ra

d an

d S

at te

r (1

95 4)

A ca

de m

ic p

er fo

rm an

ce C

or re

la ti

on .3

6 .4

6 –.

12 C

oo ke

( 19

67 a)

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

77 76

.0 2

C oo

ke (

19 67

b) a

P sy

ch ia

tr ic

d ia

gn os

is C

or re

la ti

on .4

2 .5

1 –.

11 D

an et

( 19

65 )

P ro

gn os

is H

it r

at e

64 70

–. 13

D ev

ri es

a nd

S hn

ei dm

an (

19 67

)a M

at ch

in g

M M

P I

pr of

il es

t o

pe rs

on s

H it

r at

e 75

10 0

–. 81

D ic

ke rs

on (

19 58

) C

om pl

ia nc

e w

it h

co un

se li

ng p

la n

H it

r at

e 57

52 .1

0 D

un ha

m a

nd M

el tz

er (

19 46

)b L

en gt

h of

h os

pi ta

l st

ay H

it r

at e

38 58

–. 40

E ve

ns on

,A lt

m an

, S

le tt

en ,

an d

C ho

( 19

75 )

L en

gt h

of h

os pi

ta l

st ay

H it

r at

e 64

71 –.

14 F

er o

(1 97

5) 1

P ro

gn os

is

C or

re la

ti on

.3 5

.5 7

–. 29

F er

o (1

97 5)

2b P

ro gn

os is

C or

re la

ti on

.3 5

.7 3

–. 57

G ar

dn er

e t

al .

(1 99

6) 1b

O ff

en se

/ v

io le

nc e

H it

r at

e 62

74 –.

25 G

ar dn

er e

t al

. (1

99 6)

2 O

ff en

se o

r vi

ol en

ce H

it r

at e

62 71

–. 20

G ar

dn er

e t

al .

(1 99

6) 3

O ff

en se

o r

vi ol

en ce

H it

r at

e 62

70 –.

17 G

ar dn

er e

t al

. (1

99 6)

3 O

ff en

se o

r vi

ol en

ce H

it r

at e

62 70

–. 17

352

TCP285875.qxd 3/29/2006 12:54 PM Page 352

G au

de tt

e (1

99 2)

a B

ra in

i m

pa ir

m en

t C

or re

la ti

on .3

9 .4

5 –.

07 G

ol db

er g

(1 96

5) ×

65 d

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

62 64

–. 04

G ol

db er

g (1

97 0)

1a P

sy ch

ia tr

ic d

ia gn

os is

C or

re la

ti on

.2 8

.3 1

–. 03

G ol

db er

g (1

97 0)

2 P

sy ch

ia tr

ic d

ia gn

os is

C or

re la

ti on

.2 8

.4 4

–. 18

G ol

db er

g (1

97 0)

3a P

sy ch

ia tr

ic d

ia gn

os is

C or

re la

ti on

.2 8

.4 6

–. 21

G ol

ds te

in ,

D ey

sa ch

, an

d K

le in

ne ch

t (1

97 3)

B ra

in i

m pa

ir m

en t

H it

r at

e 95

75 .5

7 G

re bs

te in

( 19

63 )a

IQ C

or re

la ti

on .6

2 .5

6 .0

8 G

us ta

fs on

, G

re is

t, S

ta us

s, S

ui ci

de a

tt em

pt H

it r

at e

65 80

–. 33

E rd

m an

, an

d L

au gh

re n

(1 97

7) a

H al

bo w

er (

19 55

) P

ro gn

os is

C or

re la

ti on

.6 0

.7 9

–. 21

H al

l (1

98 8)

a O

ff en

se o

r vi

ol en

ce H

it r

at e

55 81

–. 59

H ea

to n

et a

l. (

19 81

)b B

ra in

d am

ag e

H it

r at

e 79

72 .1

4 H

ol la

nd ,

H ol

t, L

ev i,

an d

B ec

ke tt

( 19

83 )

O ff

en se

o r

vi ol

en ce

C or

re la

ti on

.2 1

.2 8

–. 08

H ol

t (1

95 8)

A ca

de m

ic o

r tr

ai ni

ng p

er fo

rm an

ce C

or re

la ti

on .2

5 .1

3 .1

2 H

ov ey

a nd

S ta

uf fa

ch er

( 19

53 )

P er

so na

li ty

c ha

ra ct

er is

ti cs

H it

r at

e 74

63 .2

3 Jo

hn st

on a

nd M

cN ea

l (1

96 7)

L en

gt h

of h

os pi

ta l

st ay

H it

r at

e 72

72 .0

0 K

ap la

n (1

96 2)

1 P

ro gn

os is

H it

r at

e 58

70 –.

25 K

ap la

n (1

96 2)

2 P

ro gn

os is

H it

r at

e 66

70 –.

09 K

el ly

a nd

F is

ke (

19 50

) A

ca de

m ic

o r

tr ai

ni ng

p er

fo rm

an ce

C or

re la

ti on

.2 8

.2 9

–. 01

K le

hr (

19 49

) P

sy ch

ia tr

ic d

ia gn

os is

H it

r at

e 57

64 –.

14 K

le in

m un

tz (

19 67

) A

dj us

tm en

t H

it r

at e

66 71

–. 10

K li

ng er

a nd

R ot

h (1

96 5)

b P

sy ch

ia tr

ic d

ia gn

os is

H it

r at

e 94

71 .6

3 L

ef ko

w it

z (1

97 3)

M ar

it al

a dj

us tm

en t

H it

r at

e 54

55 –.

03 L

el i

an d

F il

sk ov

( 19

81 )1

B ra

in i

m pa

ir m

en t

H it

r at

e 41

62 –.

44 L

el i

an d

F il

sk ov

( 19

81 )2

B ra

in i

m pa

ir m

en t

H it

r at

e 50

62 –.

24 L

el i

an d

F il

sk ov

( 19

84 )1

a B

ra in

i m

pa ir

m en

t H

it r

at e

60 83

–. 52

L el

i an

d F

il sk

ov (

19 84

)2 a

B ra

in i

m pa

ir m

en t

H it

r at

e 71

83 –.

30 L

em er

on d

(1 97

7) S

ui ci

de a

tt em

pt H

it r

at e

53 50

.0 6

(c o n ti

n u ed

)

353

TCP285875.qxd 3/29/2006 12:54 PM Page 353

T A

B L

E 1

( co

n ti

n u

ed )

A cc

u ra

cy A

cc u ra

cy S ta

ti st

ic C

it a ti

o n

P re

d ic

ti o n

R ep

o rt

ed C

li n ic

a l

S ta

ti st

ic a l

d+

L ew

is a

nd M

ac K

in ne

y (1

96 1)

C ar

ee r

sa ti

sf ac

ti on

C or

re la

ti on

.1 2

.2 1

–. 09

L in

ds ey

( 19

65 )

H om

os ex

ua li

ty H

it r

at e

70 57

.0 3

L yl

e an

d Q

ua st

( 19

76 )

B ra

in i

m pa

ir m

en t

H it

r at

e 67

68 –.

02 M

cH ug

h an

d A

po st

ol ak

os (

19 59

)b A

ca de

m ic

f ie

ld H

it r

at e

70 46

.5 0

M ee

hl (

19 59

) P

sy ch

ia tr

ic d

ia gn

os is

H it

r at

e 69

74 –.

11 M

el to

n (1

95 2)

1b A

ca de

m ic

p er

fo rm

an ce

E rr

or r

at e

.4 7

.4 0

–. 26

M el

to n

(1 95

2) 2

A ca

de m

ic p

er fo

rm an

ce E

rr or

r at

e .4

5 .3

5 –.

35 M

ey er

( 19

73 )1

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

65 63

.0 5

M ey

er (

19 73

)2 P

sy ch

ia tr

ic d

ia gn

os is

H it

r at

e 67

63 .0

8 M

il le

r, K

un ce

, an

d G

et si

ng er

( 19

72 )a

A dj

us tm

en t

/ em

pl oy

ab il

it y

C or

re la

ti on

.6 5

.2 8

.4 9

M ox

le y

(1 97

0) 1a

L en

gt h

of p

sy ch

ot he

ra py

s ta

y H

it r

at e

55 67

–. 25

M ox

le y

(1 97

0) 2a

L en

gt h

of p

sy ch

ot he

ra py

s ta

y H

it r

at e

61 67

–. 13

M ox

le y

(1 97

0) 3a

L en

gt h

of p

sy ch

ot he

ra py

s ta

y H

it r

at e

67 67

.0 0

M ox

le y

(1 97

0) 4a

L en

gt h

of p

sy ch

ot he

ra py

s ta

y H

it r

at e

69 67

.0 4

O sk

am p

(1 96

2) ×

16 d

P ri

or h

os pi

ta li

za ti

on

H it

r at

e 72

65 .1

3 O

xm an

, R

os en

be rg

, S

ch nu

rr ,

an d

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

66 80

–. 32

T uc

ke r

(1 98

8) 1a

O xm

an e

t al

. (1

98 8)

2a P

sy ch

ia tr

ic d

ia gn

os is

H it

r at

e 66

62 .0

8 P

er ez

( 19

76 )1

a H

om ic

id al

it y

H it

r at

e 50

83 –.

77 P

er ez

( 19

76 )2

a H

om ic

id al

it y

H it

r at

e 51

83 –.

77 P

er ez

( 19

76 )3

a H

om ic

id al

it y

H it

r at

e 51

83 –.

77 P

er ez

( 19

76 )4

a H

om ic

id al

it y

H it

r at

e 58

83 –.

61 P

op ov

ic s

(1 98

3) IQ

C or

re la

ti on

.2 0

.1 9

.0 1

354

TCP285875.qxd 3/29/2006 12:54 PM Page 354

S ar

bi n

(1 94

2) A

ca de

m ic

p er

fo rm

an ce

C or

re la

ti on

.5 6

.6 0

–. 06

S ha

ff er

, P

er li

n, S

ch m

id t,

an d

S te

ph en

s (1

97 4)

1a S

ui ci

de a

tt em

pt C

or re

la ti

on .4

0 .4

0 .0

0 S

ha ff

er e

t al

. (1

97 4)

2 S

ui ci

de a

tt em

pt C

or re

la ti

on .4

0 .1

8 .2

4 S

ha ff

er e

t al

. (1

97 4)

3b S

ui ci

de a

tt em

pt C

or re

la ti

on .4

0 .1

2 .3

0 S

ha go

ur y

an d

S at

z (1

96 9)

a B

ra in

i m

pa ir

m en

t H

it r

at e

83 87

–. 11

S tr

ic ke

r (1

96 7)

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

70 79

–. 21

S zu

ko a

nd K

le in

m un

tz (

19 81

) L

ie d

et ec

ti on

C or

re la

ti on

.2 3

.5 2

–. 32

T au

lb ee

a nd

S is

so n

(1 95

7) 1

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

64 78

–. 31

T au

lb ee

a nd

S is

so n

(1 95

7) 2

P sy

ch ia

tr ic

d ia

gn os

is H

it r

at e

64 63

.0 2

T ho

m ps

on (

19 52

)b Ju

ve ni

le d

el in

qu en

cy H

it r

at e

65 90

–. 60

W al

te rs

e t

al .

(1 99

1) a

M al

in ge

ri ng

H it

r at

e 59

73 –.

30 W

at le

y (1

96 6)

A ca

de m

ic p

er fo

rm an

ce H

it r

at e

68 75

–. 14

W at

le y

an d

V an

ce (

19 64

)a A

ca de

m ic

p er

fo rm

an ce

H it

r at

e 70

80 –.

25 W

eb b

et a

l. (

19 77

)b O

cc up

at io

na l

ch oi

ce H

it r

at e

35 55

–. 41

W ed

di ng

( 19

83 )1

B ra

in i

m pa

ir m

en t

H it

r at

e 55

63 –.

17 W

ed di

ng (

19 83

)2 B

ra in

i m

pa ir

m en

t H

it r

at e

55 60

–. 11

W ei

nb er

g (1

95 7)

1a P

er so

na li

ty c

ha ra

ct er

is ti

cs C

or re

la ti

on .4

9 .5

4 –.

15 W

ei nb

er g

(1 95

7) 2a

P er

so na

li ty

c ha

ra ct

er is

ti cs

C or

re la

ti on

.4 0

.5 4

–. 06

W er

ne r,

R os

e, Y

es av

ag e

an d

S ee

m an

( 19

84 )a

O ff

en se

/ v

io le

nc e

C or

re la

ti on

.1 4

.4 0

–. 27

W ig

gi ns

a nd

K oh

en (

19 71

)1 a

A ca

de m

ic p

er fo

rm an

ce C

or re

la ti

on .3

3 .6

9 –.

51 W

ig gi

ns a

nd K

oh en

( 19

71 )2

a A

ca de

m ic

p er

fo rm

an ce

C or

re la

ti on

.3 3

.5 0

–. 20

W ir

t (1

95 6)

a P

ro gn

os is

H it

r at

e 54

79 –.

54 W

it tm

an a

nd S

te in

be rg

( 19

44 )b

P ro

gn os

is H

it r

at e

43 78

–. 77

N O

T E

: H

it r

at e

re fe

rs t

o pe

rc en

ta ge

c or

re ct

; co

rr el

at io

n re

fe rs

t o

co rr

el at

io n

w it

h th

e cr

it er

ia ;

er ro

r ra

te d

if fe

re nc

e w

as c

al cu

la te

d by

a t

t es

t th

at w

as t

he n

tr an

sf or

m ed

i nt

o d

+ . A

ll a

cc ur

ac y

st at

is ti

cs a

re r

ep or

te d

ra w

( un

tr an

sf or

m ed

). d

+ : E

ff ec

t si

ze o

f di

ff er

en ce

b et

w ee

n cl

in ic

al a

nd s

ta ti

st ic

al p

re di

ct io

n ac

cu -

ra cy

; ne

ga ti

ve d

+ in

di ca

te s

su pe

ri or

it y

of s

ta ti

st ic

al p

re di

ct io

n. S

tu di

es p

ro du

ci ng

m or

e th

an o

ne e

ff ec

t si

ze h

av e

ea ch

e ff

ec t

id en

ti fi

ed b

y su

pe rs

cr ip

ts 1

t o

4. M

M P

I =

M in

ne so

ta M

ul ti

ph as

ic P

er so

na li

ty I

nv en

to ry

. a.

S tu

di es

t ha

t di

d no

t us

e cr

os s-

va li

da te

d st

at is

ti ca

l fo

rm ul

as .

b. S

tu di

es t

ha t

w er

e di

ag no

se d

as o

ut li

er s.

T he

se s

tu di

es w

er e

ex cl

ud ed

i n

so m

e an

al ys

es .

c. S

tu di

es t

ha t

w er

e on

ly i

nc lu

de d

in t

he o

ve ra

ll e

ff ec

t. T

he a

ve ra

ge e

ff ec

t is

r ep

or te

d. d.

A ve

ra ge

h it

r at

e fo

r G

ol db

er g

(1 96

5) a

nd O

sk am

p (1

96 2)

s tu

di es

.

355

TCP285875.qxd 3/29/2006 12:54 PM Page 355

more than one effect size if it used more than one statistical prediction scheme (e.g., regression formula and test cutoff score).

Despite the potential for bias, the questions addressed by these analyses were essential to understanding what factors influence clinical versus sta- tistical judgment accuracy. Their value thus outweighed the risk of adding bias to the overall results. Both Cooper (1998) and Johnson and Eagly (2000) suggest that this is acceptable under conditions similar to ours.

Nonetheless, two studies (Goldberg, 1965; Oskamp, 1962) produced so many effect sizes that they were treated as separate cases. Oskamp (1962) reported hit rates of 16 different cross-validated statistical formulas com- pared with the judgment accuracy of the average clinician. Goldberg (1965) reported the prediction accuracy of 65 different statistical methods com- pared with the average clinician. To prevent an overrepresentation of data from Goldberg and Oskamp, the overall effect size was reported with and without these two studies. They were not included in our analysis of study design characteristics.

Strategy for Data Analyses

The overall effect size was first calculated for 69 studies, including Goldberg (1965) and Oskamp (1962), producing 173 effect sizes. Second, as noted under “Study Selection,” the overall effect size was also calculated without the Goldberg and Oskamp studies, producing 92 effect sizes. Third, the overall effect size was calculated using only the 49 studies that compared cross-validated statistical formulas with the clinical method. This last proce- dure resulted in 60 effect sizes and eliminated artificially inflated accuracy of statistical methods caused by chance associations among variables (e.g., Dawes et al., 1989; Meehl, 1954). Next, the influence of study design char- acteristics (independent variables) on the overall effect size of these 49 studies was calculated3 (see Results for description and hypothesized effects). Because heterogeneity was not eliminated when study design characteristics were con- sidered, an outlier analysis was performed to reduce heterogeneity4 (Johnson & Eagly, 2000). The 41 studies (48 effect sizes) remaining after this analysis represented the most conservative sample of studies and were used to calcu- late the influence of study design characteristics on the overall effect.

It was necessary to remove 12 outliers (20%) from the 60 effects pro- duced by the use of cross-validated statistical formulas to reach a homoge- nous set of effects, Q (47) = 65.83, p > .05. The resulting set of 41 studies produced 48 effects and shared a common effect size. They are sufficiently homogenous to be characterized as coming from the same population. Outliers, by contrast, have extreme effects. Their extremity may be because of error or may be because the studies have characteristics that are deviant

356 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 356

relative to most studies in the sample (Hunter & Smith, 1990). According to Hedges (1987) and others, removing up to 20% of studies to reach a homogeneous set of data is not uncommon for meta-analyses of psycho- logical topics. This conservative sample of studies, those that used cross- validated statistical formulas and did not have unrepresentatively extreme effect sizes, was used to interpret differences between clinical and statistical prediction accuracy.

RESULTS

Studies investigated the clinical and statistical judgment accuracy of the following 11 prediction tasks: brain impairment, personality, length of hospital stay or treatment, diagnosis, adjustment or prognosis, violence or offense, IQ, academic performance, if an MMPI profile was real or fictional, suicide attempt, and homosexuality. In all studies, the prediction accuracy of mental health professionals and graduate students in the mental health field were compared with a statistical prediction method. The studies, prediction tasks, and accuracy statistics are listed in Table 1.

Overall Effect Size

Figure 1 presents a distribution of the difference between clinical and statistical prediction accuracy transformed into a weighted effect size (d+) in stem-and-leaf diagram format for the most conservatively selected set of effect sizes (n = 48). As may be seen, effect sizes ranged from .57 in favor of the clinical method to –.73 in favor of the statistical method. Visual inspection reveals a slight skew in favor of the statistical method. Similar results were obtained when outlier effects were included, without Goldberg (1965) and Oskamp (1962). For these 92 effect sizes, effects ranged from .63 to –.81 (see Note 3).

On the basis of suggestions from the literature (Dawes et al., 1989; Garb, 1994; Grove et al., 2000; Grove & Meehl, 1996; Kleinmuntz, 1990; Meehl, 1954; Russell, 1995; Sawyer, 1966; Wiggins, 1981), we expected that statis- tical prediction would be, in general, more accurate than clinical prediction. Our hypothesis was confirmed. By using the Grove et al. (2000) approach to interpret the relative difference between clinical and statistical prediction accuracy, Figure 1 reveals that of the 48 effect sizes, 25 effects (52%) favored statistical prediction methods, 18 (38%) reported no difference between the two methods, and 5 (10%) favored the clinical method.

Overall effect sizes (both in the form of d+ and r+) for the four sets of analyses performed are presented in Table 2. Effect sizes (d+) ranged from

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 357

TCP285875.qxd 3/29/2006 12:54 PM Page 357

–.16, without Goldberg (1965) and Oskamp (1962), to –.12 for studies using cross-validated formulas with outlier studies excluded. As Table 2 also shows, the test for homogeneity of effects, Q, was significant (p < .001) in all instances except when outlier effects were removed from the analysis. This indicates that when outlier effects were included, the variability among effect sizes was too great for a reliable interpretation. Therefore, significant heterogeneous effect sizes (reflected by Q values) must be interpreted with caution. By contrast, confidence can be placed in the overall effect size of d+ = –.12 (r+ = –.06) produced by nonoutlying studies with cross-validated formulas. In this case, the effect size is reliable as indicated by a non- significant Q. Additionally, the 95% confidence interval did not cross zero, further supporting the reliability of the effect.

358 THE COUNSELING PSYCHOLOGIST / May 2006

0 0

–.8 –.7

1 –.7 3 0 –.6 0 –.6 0 –.5 0 –.5 0 –.4 1 –.4 4 1 –.3 2 –.3 21 5 –.2 97655 5 –.2 43110 3 –.1 775 8 –.1 44432110 3 –.0 997 4 –.0 3321 6 012223 4 5688 1 2 1 5 2 34 0 0 0 0 0 0 1 7 0

.0

.0

.1

.1

.2

.2

.3

.3

.4

.4

.5

.5

.6

FIGURE 1. Stem-and-Leaf Diagram for Clinical and Statistical Prediction Differences of Transformed Effect Sizes (d++) for the Most Conservative Sample of Studies (N == 48 Effects)

NOTE: A negative d+ indicates superiority of statistical method.

TCP285875.qxd 3/29/2006 12:54 PM Page 358

Because of these two conditions, we can conclude that statistical prediction methods are, in general, more accurate than clinical prediction methods. This is a small effect by conventional standards (Cohen, 1988), but it is consistent and reliable. When consistency and reliability are established in a meta-analysis, confidence can be placed in the effect as a true effect (i.e., statistical is better than clinical prediction). Another perspective on the meaning of this effect is possible through the use of a binomial effect size display analysis (Rosenthal & Rubin, 1982). This analysis permits the level of improvement achieved with one decision-making method versus another to be determined. In this case, clinical decisions are accurate 47% of the time, (.50 – r/2) × 100, whereas statistical decisions are accurate 53% of the time, (.50 + r/2) × 100. As a result, the likelihood of a successful decision can go up 13% when statistical rather than clinical methods are used.

Study Design Characteristics

The overall effect size of the difference between clinical and statistical methods was analyzed as a function of study design characteristics (i.e., independent variables hypothesized to influence the overall effect size). We used only the most conservative sample of studies. The study design char- acteristics were fitted to the effect sizes using a procedure analogous to the analysis of variance (Hedges & Olkin, 1985). The between-class effect (QB) in this procedure is similar to a main effect in an analysis of variance. The 95% confidence interval and a test of the homogeneity of the effect size within each class (QW) were also calculated. The homogeneity of studies within

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 359

TABLE 2: Overall Effect Sizes for Clinical Versus Statistical Prediction

Mean Weighted 95%

Effect Confidence Homogeneity Size (d+) Interval r+ Index Q

With Goldberg (1965) and Oskamp (1962) –.16 –.18 to –.14 –.08 460.54*** (N = 69 studies and 173 effects)

Without Goldberg (1965) and Oskamp (1962) –.15 –.17 to –.13 –.08 470.71*** (N = 67 studies and 92 effects)

Cross-validated studies –.14 –.17 to –.12 –.07 371.82*** (N = 49 studies and 60 effects)

Cross-validated studies without outliers –.12 –.14 to –.09 –.06 65.84** (N = 41 studies and 48 effects)

NOTE: Negative d+ indicates superiority of statistical method. ***p < .001.

TCP285875.qxd 3/29/2006 12:54 PM Page 359

a class (QW) determines the confidence that can be placed in interpreting the effect size (d+). If studies are heterogeneous within a class (e.g., prediction task), nonrandom variance in d+ remains in that class (Johnson, 1993).

Results of these analyses are reported in Table 3. Each study design char- acteristic or independent variable had several levels. For example, studies varied in how much information (predictor cues) they provided to clinicians and the formula on which clinicians rely in making predictions. Some gave the same amount to the clinician and the statistical formula, others gave the clini- cians more, others gave the clinician less, and some did not report how much information was provided to either. Before our analysis, we determined that to have sufficient reliability and representativeness, three or more studies would have to examine a design characteristic level to be included in the analysis. Because only two studies gave the clinician less information than the formula, that condition is not included in the overall test of the design characteristic. Within-class effects (QW) values are included in Table 3 for information only. Seven studies did not report how much information was given to clinicians and the formula. These seven were also excluded from our analyses.

Type of prediction. Since Meehl’s (1954) original publication, the methodology and the nature of the studies comparing clinical and statisti- cal prediction accuracy have been questioned. One particular criticism con- cerned the unrepresentativeness of the judgment tasks evaluated. It was argued that unusual predictions (e.g., predicting future grade point average; Holt, 1970) did not provide a fair evaluation of the accuracy of clinicians’ judgment. In response to this criticism, subsequent studies broadened the scope of the prediction tasks (cf. Rock, Bransford, & Maisto, 1987). We identified 11 prediction tasks and examined six of them (Table 3).

Effect sizes were expected to vary across different prediction tasks. This expectation was confirmed, QB (5) = 12.42, p < .05. Three of the six pre- diction tasks that met our criteria for analysis had an effect size in which the confidence interval did not include zero. These were predictions of adjustment or prognosis (d+ = –.14), offense or violence (d+ = –.17), and academic performance (d+ = –.14). For the other three prediction tasks, the confidence interval included zero, which indicates that any differences between the clinical and statistical methods were unreliable. Data were insufficient to analyze differences in determining personality type, estimat- ing client IQ, determining whether an MMPI profile was fictitious, predict- ing suicide attempt, or determining homosexuality. In no instance was the clinical method consistently more accurate than the statistical method.

Data collection setting. Holt (1958, 1970) stated that in many studies, clinicians were asked to make unusual predictions with limited data. These

360 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 360

T A

B L

E 3

: E

ff ec

t S

iz e

b y

S tu

d y

D es

ig n

C h

ar ac

te ri

st ic

s, C

ro ss

-V al

id at

ed S

tu d

ie s

W it

h ou

t O

u tl

ie rs

( N

== 41

S tu

d ie

s, 48

E ff

ec ts

)

B et

w ee

n -C

la ss

M ea

n W

ei g h te

d 9 5 %

C I

9 5 %

C I

H o m

og en

ei ty

E ff

ec t

(Q B )

n E

ff ec

t S iz

e (d

+ ) L

o w

er H

ig h er

r+ W

it h in

C la

ss (

Q W )

T yp

e of

p re

di ct

io n

12 .4

2* B

ra in

i m

pa ir

m en

t 7

–. 05

–. 22

.1 2

–. 03

6. 33

H os

pi ta

l or

t re

at m

en t

le ng

th 3

–. 02

–. 13

.0 9

–. 01

2. 33

D ia

gn os

is 9

–. 05

–. 12

.0 1

–. 03

7. 39

A dj

us tm

en t

or p

ro gn

os is

11 –.

14 –.

19 –.

10 –.

07 6.

69 F

ut ur

e of

fe ns

e or

v io

le nc

e 4

–. 17

–. 24

–. 11

–. 09

1. 46

A ca

de m

ic p

er fo

rm an

ce 8

–. 14

–. 18

–. 09

–. 07

11 .3

0 IQ

(1 )

(. 01

) (–

.3 6)

(. 00

) (.

00 )

(0 .0

0) P

er so

na li

ty t

yp e

(1 )

(. 23

) (–

.1 8)

(. 64

) (.

11 )

(0 .0

0) R

an do

m M

M P

I pr

of il

e (1

) (–

.7 3)

(– 1.

24 )

(– .2

2) (–

.3 4)

(0 .0

0) S

ui ci

de a

tt em

pt p

re di

ct io

n (2

) (.

16 )

(– .0

6) (.

38 )

(. 07

) (.

63 )

H om

os ex

ua li

ty (1

) (.

27 )

(– .2

4) (.

78 )

(. 13

) (0

.0 0)

D at

a co

ll ec

ti on

s et

ti ng

6. 96

** C

li ni

ci an

s fr

om s

am e

se tt

in g

as d

at a

21 –.

14 –.

17 –.

11 –.

07 28

.0 3

C li

ni ci

an s

no t

fr om

s am

e se

tt in

g as

d at

a 25

–. 06

–. 11

–. 01

–. 03

30 .3

7 N

ot r

ep or

te d

(2 )

(– .0

6) (–

.3 1)

(. 18

) (–

.0 3)

(. 29

) S

ta ti

st ic

al f

or m

ul a

ty pe

11 .9

4* *

L in

ea r

st at

is ti

ca l

fo rm

ul a

27 –.

15 –.

19 –.

12 –.

08 32

.2 9

T es

t cu

to ff

s co

re 8

–. 10

–. 15

–. 06

–. 05

8. 06

L og

ic al

ly c

on st

ru ct

ed r

ul e

11 –.

03 –.

09 .0

3 –.

02 11

.8 7

M od

el o

f cl

in ic

al j

ud gm

en t

(2 )

(– .1

4) (–

.2 3)

(– .0

5) (–

.0 7)

(2 .4

7) A

m ou

nt o

f in

fo rm

at io

n 4.

06 *

S am

e am

ou nt

f or

c li

ni ci

an s

an d

fo rm

ul a

24 –.

06 –.

11 –.

01 –.

03 28

.2 6

C li

ni ci

an s

ha ve

m or

e th

an f

or m

ul a

15 –.

13 –.

16 –.

09 –.

06 25

.7 7*

C li

ni ci

an s

ha ve

l es

s th

an f

or m

ul a

(2 )

(– .1

2) (–

.4 2)

(. 19

) (–

.0 6)

(1 .6

4) N

ot r

ep or

te d

(7 )

(– .1

5) (–

.2 0)

(– .1

0) (–

.0 8)

(3 .8

1)

(c o n ti

n u ed

)

361

TCP285875.qxd 3/29/2006 12:54 PM Page 361

362

T A

B L

E 3

( co

n ti

n u

ed )

B et

w ee

n -C

la ss

M ea

n W

ei g h te

d 9 5 %

C I

9 5 %

C I

H o m

og en

ei ty

E ff

ec t

(Q B )

n E

ff ec

t S iz

e (d

+ ) L

o w

er H

ig h er

r+ W

it h in

C la

ss (

Q W )

In fo

rm at

io n

ab ou

t ba

se r

at es

4. 25

B as

e ra

te p

ro vi

de d

8 –.

02 –.

12 .0

8 –.

01 5.

40 B

as e

ra te

s am

e as

n at

ur al

s et

ti ng

8 –.

11 –.

15 –.

06 –.

05 6.

37 C

li ni

ci an

s do

n ot

k no

w b

as e

ra te

27 –.

13 –.

16 –.

09 –.

06 45

.5 6*

N ot

r ep

or te

d (5

) (–

.1 4)

(– .1

9) (–

.0 9)

(– .0

7) (3

.0 4)

A va

il ab

il it

y of

s ta

ti st

ic al

f or

m ul

a 2.

70 A

va il

ab le

5 –.

14 –.

18 –.

09 –.

07 7.

33 N

ot a

va il

ab le

40 –.

09 –.

12 –.

06 –.

05 53

.3 4

N ot

r ep

or te

d (3

) (–

.1 5)

(– .2

1) (–

.1 0)

(– .0

8) (.

05 )

C li

ni ci

an e

xp er

tn es

s 2.

47 E

xp er

ts i

n th

e pr

ed ic

ti on

t as

k 7

–. 05

–. 14

.0 3

–. 03

14 .9

6* N

on ex

pe rt

s in

t he

p re

di ct

io n

ta sk

41 –.

12 –.

15 –.

10 –.

06 48

.4 1

P ub

li ca

ti on

s ou

rc e

2. 06

A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

ti on

j ou

rn al

19 –.

11 –.

15 –.

07 –.

05 22

.5 3

N on

–A m

er ic

an P

sy ch

ol og

ic al

A ss

oc ia

ti on

j ou

rn al

16 –.

11 –.

15 –.

06 –.

05 21

.8 5

D is

se rt

at io

n 13

–. 15

–. 19

–. 10

–. 07

19 .3

9 C

on fi

de nc

e in

c ri

te ri

on f

or a

cc ur

ac y

.1 2

L ow

12 –.

12 –.

17 –.

08 –.

06 12

.0 3

H ig

h 36

–. 12

–. 14

–. 08

–. 06

53 .6

9

N O

T E

: A

n eg

at iv

e d

+ in

di ca

te s

su pe

ri or

it y

of t

he s

ta ti

st ic

al m

et ho

d. V

al ue

s in

p ar

en th

es es

w er

e no

t in

cl ud

ed i

n th

e ov

er al

l an

al ys

is .

E it

he r

to o

fe w

s tu

di es

ex am

in ed

t he

d es

ig n

ch ar

ac te

ri st

ic l

ev el

, or

t he

s tu

dy d

id n

ot r

ep or

t ho

w t

he c

ha ra

ct er

is ti

c w

as t

re at

ed .

V al

ue s

ar e

in cl

ud ed

f o

r in

fo rm

at io

n on

ly .

C I

= co

nf id

en ce

i nt

er va

l. M

M P

I =

M in

ne so

ta M

ul ti

ph as

ic P

er so

na li

ty I

nv en

to ry

. *p

< .

05 .

** p

< .

01 .

** *p

< .

00 1.

TCP285875.qxd 3/29/2006 12:54 PM Page 362

conditions, in Holt’s opinion, did not adequately represent the clinical approach and invalidated the comparison between clinical and statistical methods. Yet if the data given to clinicians are from the same setting that the clini- cians inhabit, then clinicians should be familiar with the predictors. They should know how to use them to form effective judgments. Based on this rationale, we expected that if the clinicians were drawn from the same set- ting as were the data (e.g., clinicians who worked in the clinic from which the client data were drawn), the difference between clinical and statistical prediction would be less than if the clinicians and the data were not from the same setting.

We found that data collection setting indeed influenced effect size, QB (1) = 6.96, p < .01, although not as we expected. Contrary to our expecta- tions, when clinicians were not from the same setting as the data on which they based their predictions (d+ = –.06), the effect size was smaller than if the data were from their work setting (d+ = –.14). In neither instance did the effect-size confidence interval include zero. Therefore, even though statis- tical methods yield more accurate predictions independent of the familiarity of clinicians with the data, the difference between the clinical and the statis- tical method is smaller when clinicians do not come from the same setting as the data used to make their judgments. Thus—and most unexpected—existing studies indicate that clinicians seem to be more accurate when they are working with less familiar or novel information.

Statistical formula type. Studies have compared several types of statistical formula with the clinical method. We organized the studies on four types of formula: linear statistical models (e.g., regression or discriminant function), logically constructed rules (e.g., Goldberg rule for differential diagnosis of psychosis and neurosis from MMPI), test cutoff scores, and models of clinical judgment (mechanization of clinicians’ judgment processes). Even though no specific type of statistical formula was expected to yield the most accurate results, on the basis of Dawes and Corrigan’s (1974) hypotheses we assumed that simple linear models would be as accurate as more complex models. We also expected, on the basis of narrative and empirical evidence (e.g., Dawes et al., 1989; Grove et al., 2000), that any of the four types of statis- tical method would be more accurate than the clinical method.

Our hypotheses were partially supported. The type of statistical formula used affected overall effect size, QB (3) = 11.94, p < .01. Effect sizes (d

+) ranged from –.15 for linear statistical formulas (i.e., regression and dis- criminant function analysis) to –.03 for logically constructed rules or signs (e.g., patterned MMPI rules). All categories of statistical formulas, except logically constructed rules, yielded effect sizes in which the confidence interval did not include zero. Logically constructed rules did not differ from

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 363

TCP285875.qxd 3/29/2006 12:54 PM Page 363

clinical prediction methods. Hence, statistical methods are more accurate than clinical methods but only when purely statistical models are used. Logical rules are unexpectedly no better and no worse than clinical methods. Data were insufficient to compare models mechanizing clinicians’ judgment processes to the clinical method.

Amount of information. Yet another controversy surrounding clinical versus statistical prediction concerns the amount of information made available to clinicians and the formula. Holt (1970), for instance, noted that in many studies the only data available to clinicians were quantitative (e.g., an MMPI profile). This, Holt contended, placed the clinical method at a serious dis- advantage because “given a chance to show what it can do, it has to have meaningful, qualitative data to work with” (p. 343). In contrast, Dawes et al. (1989) argued that even if clinicians were provided more information than the formulas, regardless of whether the information was qualitative, they will still fail to surpass the statistical method (see also Faust, 1986, 2003). This assumption was supported by Grove et al. (2000). To resolve this con- troversy, we studied the influence of the amount of information provided to the clinicians and the statistical formula. We hypothesized, in accordance with Dawes et al. (1989), Faust (1986, 2003), and Grove et al. (2000), that the differential accuracy between clinical and statistical prediction would not be affected by the amount of informational cues.

Our findings indicate that the amount of information provided to the clin- icians influenced their prediction accuracy, QB (2) = 4.06, p < .05. Increasing the amount of information, however, decreased clinicians’ judgment accu- racy. More information may thus not be better.

Information about base rates. Humans tend to ignore base rate informa- tion when making judgments under ambiguous conditions (Kahneman & Tversky, 1973). Several studies reviewed in this meta-analysis investigated the influence of base rate information on clinicians’ predictions. Because base rate information is usually underutilized, we anticipated that effect sizes would be the same when clinicians knew the base rate for the outcome they were predicting (i.e., base rate provided or base rate the same as the natural setting) and when base rate information was not provided.

This was the case. Whether clinicians had base rate information avail- able when making judgments did not matter, QB (2) = 4.25, p > .05. Even so, the difference between the two methods tended to diminish when clini- cians had base rate information (d+ = –.02). The range of this small effect includes zero, which suggests no difference between the two methods. The statistical method surpassed the clinical method when the clinicians were

364 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 364

not informed about base rate (d+ = –.13) and when the base rate for the prediction was the same as in the clinician’s work setting (d+ = –.11).

Availability of statistical formula. An important question raised in the debate about clinical and statistical prediction is whether clinical prediction can be improved when clinicians are given the statistical formula and its outcome. Because some studies provided clinicians with the statistical for- mula before they made predictions, we could compare clinical and statis- tical methods with and without access to the formula. In their narrative review of the literature, both Sines (1970) and Dawes et al. (1989) stated that even though clinicians had access to and were free to use the statistical formulas, their predictions remained less accurate than statistical predic- tion methods. Therefore, we hypothesized that accessibility to the statisti- cal formula would not influence the differential accuracy of clinical and statistical prediction.

As predicted, clinicians’ access to the statistical formulas did not improve their accuracy. With and without access to statistical formulas, clinical pre- diction remained less accurate than statistical prediction, QB (1) = 2.70, p > .05. In the five studies in which clinicians were provided with the statistical formula, the effect size was –.14, and in the 40 studies in which the clinicians did not have access, the effect size was –.09. In both these instances, the confidence interval did not include zero, providing support for the stability of the effect.

Clinical expertness. Holt (1970) noted that studies often failed to com- pare comparable clinical and statistical prediction methods. For example, the average judge (not the best judge) was compared with the best statistical formulas (for exceptions, see Goldberg, 1965; Oskamp, 1962). Although few studies compared the best judge with the best formula, clinicians iden- tified as experts in the predictions under study may provide a more compet- itive contrast with the statistical formula. Spengler et al. (2005) found in a recent meta-analysis a small but reliable increase in clinical judgment accu- racy with increased experience. They did not, however, assess the impact of expertness as distinct from experience on judgment accuracy. Drawing on the notion that clinicians vary considerably in their ability to make accurate judgments and that the best clinicians can do well (Holt, 1970), we hypoth- esized that experts in the prediction task would be more accurate than non- experts. Likewise, we anticipated that the performance of experts would be similar to or better than that of statistical methods.

Our predictions were not supported, QB (1) = 2.47, p > .05, even though a trend in the hypothesized direction was observed. In seven studies, clinicians

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 365

TCP285875.qxd 3/29/2006 12:54 PM Page 365

were considered experts in the judgment task. These studies yielded an effect size of –.05, although the confidence interval included zero. In the 41 studies in which judges were not considered experts, d+ was larger (–.12) and the confidence interval did not cross zero. Thus, when judgments are made by expert clinicians, the difference between clinical and statistical methods seems to disappear. However, when the clinicians are nonexperts, they are consistently outperformed by statistical formulas.

Publication source. The direction of publication bias is that studies reporting significant results are more often published than studies present- ing nonsignificant results (e.g., Greenwald, 1975; Rosnow & Rosenthal, 1989). Because of competition for publication in major journals such as those published by the American Psychological Association, studies in these jour- nals may have larger effects. From this assumption, we anticipated that the differential accuracy of clinical and statistical prediction would be larger in studies retrieved from American Psychological Association journals than from studies retrieved from other sources. This hypothesis was not sup- ported, QB (2) = 2.06, p > .05.

Confidence in the criterion for accuracy. We assigned a dichotomous rating of high or low to the confidence we placed in the reliability of each study’s criterion for accuracy (see Spengler et al., 2005). If we determined that the criteria had suspected unreliability, we rated our confidence in the accuracy criterion as low (e.g., peer or supervisor ratings of successful aca- demic training; Kelly & Fiske, 1950). A high rating was given if use of a relatively reliable and valid criterion for accuracy was reported (e.g., neu- roradiological or operative verification of brain damage; Heaton, Grant, Anthony, & Lehman, 1981). We hypothesized larger effect sizes among studies with a highly valid and reliable criterion for accuracy owing to the well-known relation between criterion reliability and a ceiling on predictive validity (for further discussion, see Schmidt & Hunter, 1996). This hypoth- esis was not confirmed, QB (1) = .12, p > .05. Furthermore, the effect-size confidence interval did not cross zero. Therefore one may conclude that the greater accuracy of statistical over clinical prediction is not affected by our rating of the quality of the criterion used to determine accuracy.

Other independent variables. Three additional independent variables were identified to clarify the differential accuracy of clinical and statistical prediction. These were the number of prediction tasks performed within a study, the number of clinicians making predictions, and year of publication or completion. While more reliable values were expected for higher numbers of predictions and greater numbers of clinicians, no specific hypotheses

366 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 366

were associated with these variables. Similar to assumptions made by Spengler et al. (2005), we anticipated that as study age becomes more recent, the clinical method would fair better in relation to the statistical. This trend would be because of the increasing attention given through the years to improving the accuracy of clinical judgment (e.g., Arkes, 1981, 1991; Dawes et al., 1989; Faust, 1986; Garb, 1989, 1998; Spengler et al., 1995). Regression analyses (i.e., focused comparison; Rosenthal & Rubin, 1982) were performed to examine the effects of these three independent variables on the overall effect size. None of these continuous variables significantly predicted the effect size (p > .05).

DISCUSSION

Our examination of the differential prediction accuracy of clinical and statistical methods from studies completed over a 56-year span shows that in general, statistical prediction methods are somewhat more accurate than the clinical method. This confirms, in most instances, the independent and parallel meta-analytic findings of Grove et al. (2000); it is also in accord with earlier narrative reviews of clinical and statistical prediction (e.g., Dawes et al., 1989; Grove & Meehl, 1996; Meehl, 1954; Sawyer, 1966). Using the most conservative sample of studies, we found an effect size of –.12 favoring the accuracy of statistical over clinical methods. This overall effect is virtually identical to the effect size (.12) of Grove et al. across a wide range of decision contexts. Note that although the signs of effects from the two studies are different, this difference is only because of the opposite coding method used. These effect sizes reflect a 13% increase in accuracy using statistical rather than clinical prediction techniques.

Should a relatively small effect, such as that which we observed, be dismissed as unimportant? We think not. First, partisan arguments have appeared in the literature that strongly favor either clinical or statistical pre- dictions. Our analysis and that of Grove et al. (2000) argue for more tem- perance on both sides. Although the statistical method is almost always the equal of the clinical method and is often better, the improvement is not overwhelming. Much more research is needed—in particular, programmatic lines of research on statistical prediction—that translates into practical appli- cations for practicing psychologists (e.g., Quinsey et al., 1998). Likewise, supporters of clinical decision making must show how their approach can be improved.

A second reason for not ignoring such a modest effect involves how it is to be used. Consider heart attack prevention. Would not any improvement here be important? The Steering Committee of the Physicians Health Study

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 367

TCP285875.qxd 3/29/2006 12:54 PM Page 367

Research Group (1988) thought so. They discontinued a randomized double- blind experiment on the use of aspirin to reduce heart attacks after finding a preliminary effect of r = .034. This small effect was nonetheless large enough that continuing to give control subjects a placebo would have been unethical. After converting this study’s d+ values (–.12) to r+ (= –.06), the effect of aspirin in reducing heart attack is half the size of the effect found in the current meta-analysis.

One area in which the statistical method is most clearly superior to the clinical approach is the prediction of violence, r = –.09. Out of 1,000 pre- dictions of violence, the statistical method should correctly identify 90 more violent clients than will the clinical method (Rosenthal, 1991). The victims of violence would not consider this effect small. Some predictions are more important; therefore, we recommend that statistical prediction techniques be developed, considered, and used for the most important types of decisions made by counseling psychologists and other mental health professionals (e.g., danger, suicide, and/or parole).

Our study went beyond the work of Grove et al. (2000) by examining how specific aspects of the prediction studies influenced the nature of the difference between clinical and statistical prediction. These study design variables were chosen because they had been hypothesized by other review- ers to explain why clinical and statistical predictions differ in accuracy. We found that in some instances, the difference between clinical and statistical prediction was influenced by study design characteristics. Knowing the type of prediction is important: Predictions of violence or academic perfor- mance were much more accurate with statistical techniques, whereas treat- ment length was predicted equally well by both methods.

We found that being familiar with the prediction setting did not help clinicians do better than statistical methods; in fact, clinicians fared worse. The type of statistical formula was also important. All statistical types, except logically constructed rules, did better than clinicians. Even with the exception, the result was a draw; the clinician did not do better. Some have argued that certain studies have “stacked the deck” against the clinician by providing less information than the formula (e.g., Holt, 1970). We found that when clinicians were given the same or more information than the statistical formula, the formula did better. Information was insufficient to assess those conditions in which clinicians had less information. In con- trast, the overall effect size was not influenced by clinicians’ access to the statistical formula. Furthermore, the overall effect size was not influenced by the outlet in which the study was published. Finally, trends were observed that future studies should address. Clinicians considered experts in a predic- tion task did better than nonexperts and did as well as statistical methods.

368 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 368

Also, having base rate information available resulted in clinicians’ approaching the prediction accuracy of statistical methods.

Implications for Practice

On the basis of our analyses, we tentatively suggest when and under what conditions counseling psychologists and other mental health practitioners might best use clinical or statistical methods to make accurate predictions about their clients (Westen & Weinberger, 2004). Certain limitations apply, however. Akin to the psychotherapy literature on empirically validated treat- ments (e.g., Waehler, Kalodner, Wampold, & Lichtenberg, 2000; Wampold, 1997), recommendations about what works for clinical predictions can log- ically be made based only on those applications that have so far been tested. Likewise, for many hypotheses that we tested, the sample sizes were so small that more research is needed before firm conclusions can be made about those judgments.

Given the convergence between our meta-analysis and the work of Grove et al. (2000), statistical rules ought to be employed when feasible. This is especially true if judgment accuracy is important and errors are costly. This recommendation, however, is made with qualification. First, as was recog- nized by an American Psychological Association task force on the use of psychological assessment (Meyer et al., 1998), prediction rules for many judg- ment tasks are scarce. That is, researchers have yet to study a large array of possible applications of statistical prediction techniques.

Second, not all statistical formulas are effective. Examples are Goldberg’s rules for predicting neurotic versus psychotic diagnoses from the MMPI. These logically constructed rules were developed from large samples and were extensively cross-validated. A review of 406 samples of patients and non- patients found that Goldberg’s index did not generalize well across samples (Zalewski & Gottesman, 1991). This finding corresponds to our results show- ing that logically constructed rules, despite being cross-validated, were not more accurate than clinician judgments. Other statistical prediction methods, in contrast, have been found to be successful (e.g., regression formulas for pre- dicting recidivism; Hilton et al., 2004; Quinsey et al., 1998).

Third, counseling psychologists should educate and familiarize themselves with available statistical prediction methods, such as regression formulas, test cutoff scores, and hit rates (for further information, see Anastasi & Urbina, 1996; Crocker & Algina, 1986; Greene, 2000). This is especially true for critical decisions in which false-negative judgments can be costly. Even a small increase in accuracy is important if one is predicting suicide, domestic violence, or postparole adjustment. For example, statistical formulas

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 369

TCP285875.qxd 3/29/2006 12:54 PM Page 369

exist to predict violent behavior (Hilton et al., 2004; Quinsey et al., 1998), as do cutoff scores and hit rates for improving classification accuracy using several psychological tests (e.g., MMPI/MMPI-2 cutoffs for malingering; Graham, Watts, & Timbrook, 1991; posttraumatic stress disorder; Bury & Bagby, 2002; substance abuse; Rouse et al., 1999; Stein et al., 1999). Ignoring available statistical prediction schemes may do a disservice to clients and could even be unethical when false-negative outcomes carry severe consequences (e.g., Dawes, 2002). Counseling psychologists already rely on nomothetic assessment techniques when making predictions about clients (e.g., Strong Interest Inventory, MMPI-2/MMPI-A, or Millon Multiaxial Clinical Inventory-III). Use of tests and inventories as statistical strategies requires knowledge of models and test and measurement principles to effectively bridge nomothetic findings to the individual situation (for fur- ther discussion, see Faust, 1997). Furthermore, counseling psychologists already rely on test cutoff scores, hit rates, and decision trees to aid accu- rate classification (e.g., the Substance Abuse Subtle Screening Inventory; Miller, 1985). Our findings suggest that counseling psychologists should highly weigh scores from valid and reliable psychological tests in their clinical decision making.

Fourth, when counseling psychologists have familiarized themselves with available statistical formulas and prediction techniques, they should use those formulas and techniques to improve their prediction accuracy. While this is especially true when judgment accuracy is critical, such as predicting future violence and offense, it is also true for predictions of prog- nosis, psychological adjustment, and academic performance. Several suggestions can be drawn from the literature reviewed here. For instance, in studies examining prognostic predictions (e.g., improvement in psychother- apy; Barron, 1953; Bolton, Butler, & Wright, 1968; Kaplan, 1962; Wirt, 1956; Wittman & Steinberg, 1944), the statistical prediction involved cutoff scores from psychological tests in almost all instances (e.g., MMPI profile within normal limits and high scores on Barron’s ego strength scale). Thus on the basis of these findings, counseling psychologists should place sub- stantial weight on valid and reliable instruments when making prognostic predictions about their clients. Furthermore, counseling psychologists should rely on past academic accomplishment and scores from aptitude tests when predicting academic success, especially as it relates to completion of under- graduate training (Alexakos, 1966; Conrad & Satter, 1954; Melton, 1952; Sarbin, 1942; Watley, 1966). We call on research to develop statistical for- mulas especially relevant to counseling psychology practice to perform these and other tasks. For instance, empirical tests of the relevance of GPA and Graduate Record Examination scores in predicting success in counseling

370 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 370

psychology programs are needed, either alone or in combination with other hypothesized predictors of successful training.

Fifth, counseling psychologists should use available base rate informa- tion when making decisions to aid their accuracy. Prevalence data routinely reported in epidemiological studies (e.g., National Comorbidity Survey; Kessler et al., 2003) and reported in Diagnostic and Statistical Manual of Mental Disorders (4th ed., text rev.; DSM-IV-TR; American Psychiatric Association, 2000) are useful, as are normative data for specific settings or client type. Prevalence data may aid diagnostic accuracy such that more prevalent diagnoses (higher base rate) are more likely to be accurate than less prevalent diagnoses, given certain symptoms. Spengler et al. (1995) hypothesized that teaching counseling psychologists how to use base rates in their decision making would increase judgment accuracy, and they found support for this in a pre- and posttest control group comparison of a 6-week educational course on clinical decision making (see Spengler & Strohmer, 2001). Further research is needed on the use of base rates for increasing the accuracy of clinical judgments.

Sixth, counseling psychologists should be skeptical about the relative accu- racy of their clinical judgments, even when they are working with familiar cases in familiar contexts. Studies show that clinical predictions were worse than statistical predictions in these conditions. These counterintuitive results may be because of clinicians’ overconfidence or habituation to the setting (e.g., Arkes, 1981, 1991; Faust, 1986). When working with familiar cases, clinicians may become overly confident in their decision making and use a positive hypothesis-testing strategy. Such strategies serve to confirm hypothe- ses by recalling supporting information and discounting or forgetting con- tradictory information. Incorporating statistical methods in decision making involving familiar cases could reduce this bias.

Finally, in several instances, clinical judgment was the equal of statistical methods. Our findings show that clinicians can rely on either their own infer- ential processes (the clinical method) or on statistical methods when detecting brain impairment from psychological tests, determining personality charac- teristics, predicting length of hospitalization and treatment, and diagnosing. In these predictions, clinicians were as accurate as the statistical approaches. Two reasons might account for these findings. First, experience with these tasks may have provided clinicians with information regarding validity of relations between client data and outcome (Dawes, 1994; Ericsson & Lehman, 1996; Faust, 1991; Lichtenberg, 1997; Spengler, 1998). In fact, these pre- diction tasks, in comparison with the tasks in which mechanical methods were more accurate, tend to offer a greater opportunity for immediate feed- back to the practicing clinician. Second, these prediction tasks may have more

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 371

TCP285875.qxd 3/29/2006 12:54 PM Page 371

objective tools available to aid judgment accuracy, such as psychological tests and systematic guidelines (e.g., use of DSM-IV-TR and test cutoff score for classification), than other prediction tasks in which statistical methods surpassed the clinical method.

Implications for Training

Counseling psychology has given much attention to the importance of the scientist-practitioner model of training. Epistemological issues in research and practice have been discussed, and the use of empirically validated treat- ment strategies for specific clients and problems has been encouraged. Our analysis supports the argument that training should also encourage the use of statistical methods to decrease judgment biases and errors. A portion of pre- and postdoctoral training could be used to show trainees how frequently they have the opportunity to use statistical methods of prediction and how these methods could improve their predictions. Test cutoff scores, hit rates, and decision trees are all statistical strategies and can improve clinical predic- tions. Adequate training in statistics and probability theory has been found to increase judgment accuracy (Nisbett & Ross, 1980), as has education about inferential errors and common heuristics (e.g., Arkes, 1981, 1991; Spengler & Strohmer, 2001). Methods for demystifying statistical prediction should be incorporated into counseling psychology curricula. Furthermore, trainees should be familiarized with the construction of simple regression models to aid judgment accuracy. Some research on the benefits of training counselors in effective decision-making strategies already exists (Berven, 1985; Berven & Scofield, 1980; Falvey & Hebert, 1992; Kurpius, Benhamin, & Morran, 1985; Spengler & Strohmer, 2001). More research is needed on how best to imple- ment statistical decision making and comparing its effectiveness in different contexts before educators and trainers can most effectively introduce statis- tical models of decision making.

In their proposed scientist-practitioner model for assessment, Spengler et al. (1995) suggested that counseling psychologists collect local clinical data for decisions that might be considered routine; these data would be used to evaluate and improve the effectiveness of their practice. Psychologists could then use these data to mechanize routine decisions. Counseling psy- chologists could also adjust existing statistical formulas to fit their client type and setting, basing their adjustment on local data. Some of the many decisions that might be assisted by statistical formulas include amount of treatment, school-to-work transition, career choice, and predictions of dan- gerousness to self or others. Blocher (1987) provided a simple and accessible model for counseling psychologists to learn how to empirically evaluate

372 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 372

their practice. Likewise, Stricker (e.g., Stricker & Trierweiler, 1995) has written extensively about training psychologists as local clinical scientists who collect local practice data to improve their decision making.

Counseling psychologists should be trained in statistical methods for evaluating their effectiveness with individual clients. Examples of promis- ing statistical prediction techniques for clinical practice come from a newer area of psychotherapy research called patient-focused research (Howard, Moras, Bril, Martinovich, & Lutz, 1996). By using statistical techniques, such as probit analysis, survival analysis, and hierarchical linear modeling, an individual client’s progress is compared with expected recovery curves to improve clinical decision making. Lambert, Hansen, and Finch (2001) developed a methodology based on a data set of 10,000 cases to identify clients likely to fail in treatment (i.e., signal cases). They used the Outcome Questionnaire-45 (Lambert et al., 1996) to monitor client weekly progress. Providing clinicians with the simple feedback to change the course of their treatment, compared with a no-feedback group, led to statistically and clini- cally significant improvements in recovery rates for the signal cases. Another area of statistical prediction research relevant to counseling training appli- cations is the use of statistical techniques to measure clinically significant and reliable change (Ogles, Lambert, & Masters, 1996).

Implications for Research

This meta-analysis represents only the second meta-analysis conducted in this area of the literature (cf. Grove et al., 2000). The present findings are not without limitations. The arguments in favor of the small, but reliable, edge of statistical prediction techniques are strong, but we are struck by the limits of these studies. Few programmatic lines of research have accumulated bodies of evidence for specific applications (e.g., clinical versus statistical methods to aid suicide assessment risk). More systematic studies, such as those per- formed by Goldberg (1965) and Quinsey et al. (1998), are clearly needed. Likewise, more models should be developed for clinical practice. Otherwise, these findings will remain only an academic issue of little practical interest to mental health professionals.

A recent proposal by Katsikopoulos, Machery, Pachur, and Wallin (2004) suggested development of user-friendly models, arguing that statistical models must be context based, simple, and friendly for the clinician. Katsikopolous et al. proposed constructing statistical methods they termed “friendly heuris- tics.” These friendly heuristics are simple guidelines that rely on a limited amount of client data and do not need complex integration. Often, these friendly heuristics mirror the cognitive processes underlying clinical judgment.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 373

TCP285875.qxd 3/29/2006 12:54 PM Page 373

Examples include relying on few but important client data to render a prediction (e.g., if X communicates a desire to die and has access to a gun, X is likely to commit suicide). Similarly, test cutoff scores and hit rates reported for many psychological tests are simple and easy to use and there- fore friendly. To further aid judgment accuracy and save time, more friendly heuristics for use in clinical settings must be developed and tested.

A common criticism of the body of studies on clinical versus statistical pre- diction accuracy is lack of ecological validity (e.g., Holt, 1970; Rock et al., 1987; Westen & Weinberger, 2004). This criticism has some merit. In various studies, the clinicians made global dichotomous judgments about events that they seldom encounter and about which they may have limited knowledge or training. More research is clearly needed on predictions common to counsel- ing psychologists that incorporate statistical and clinical prediction methods.

In conclusion, we do not argue against clinical prediction as a decision- making strategy (cf. Hammond, 1996). Counseling psychologists make far too many decisions in which the absolute right answer is not the issue and in which determining statistical decision rules may never be practical (e.g., moment-to-moment decisions; Spengler et al., 1985). For these decisions, the clinical strategy is necessary. However, after reviewing 56 years of research, we conclude that clinical prediction should not be the only method. After being shown by two meta-analyses and several independent analyses to be at least equal and often superior to clinical decision making in counseling psychology and mental health contexts, statistical methods must be one of the strategies of the careful clinician. Quoting Meehl (1954), “We have no right to assume that entering the clinic has resulted in some miraculous mutations and made us singularly free from the ordinary human errors which characterized our psychological ancestors” (p. 28).

NOTES

1. A complete description of the search process can be found in Spengler et al. (2005). To limit the retrieval of studies to a manageable, yet representative, sample, studies that appeared between 1970 and 1996 were included in the search. Electronic databases included PsychInfo, ERIC, Dissertation Abstracts, BRS, MEDLINE, and Social Science Index. Unavailable dis- sertations and journal articles were purchased, and authors were contacted to obtain material that was difficult to retrieve. After we identified likely clinical judgment studies, forward and backward cross-referencing was conducted until no new studies were obtained. By using this strategy, more than 35,000 articles were identified; 4,617 were coded and 1,135 met our inclu- sion criteria for the project. We chose this open-ended strategy to maximize the number of studies that would be reviewed. Because our search did not include studies published after 1996, we performed a file drawer analysis. In this analysis, we can project how many studies reporting significantly different effect sizes will be needed to change our overall results. This analysis indicated that to reduce our effect size to zero (within the 95% confidence interval

374 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 374

boundaries), indicating no difference between clinical and statistical prediction, 99 additional studies are needed using cross-validated formulas and producing nonoutlying effect sizes.

2. Hit rates reported as percentage correct were directly transformed into d using the DSTAT program (Johnson, 1993). When hit rates were reported as a correlation of the predic- tion with the criterion (r) for clinical and statistical prediction, the r values were first converted to Z scores. Differences in Z scores were then transformed into a chi-square value. The chi- square value of Z score differences was transformed into d (Rosenthal, 1991).

3. These results are not reported but are available on request by contacting the first author. 4. Johnson (1993) and Hedges and Olkin (1985) recommend outlier analyses in meta-

analyses when moderating variables fail to explain observed heterogeneity of studies. In this procedure, outliers are sequentially removed until the hypothesis of homogeneity cannot be rejected (i.e., the probability of Hedges’s Q exceeds .05). In the current meta-analysis, the study design characteristics that we hypothesized would influence the overall effect size (e.g., pre- diction task) did not reduce the heterogeneity among the studies. This made the interpretation of the effect size difficult because the many studies differed from each other in terms of mag- nitude and direction of the effects. Therefore to make our findings more interpretable, we per- formed an outlier analysis.

REFERENCES

*References marked with an asterisk indicate studies included in the meta-analysis.

Adams, K. M. (1974). Automated clinical interpretation of the neuropsychological battery: An ability-based approach. Unpublished doctoral dissertation, Wayne State University, Detroit, Michigan.

*Alexakos, C. E. (1966). Predictive efficiency of two multivariate statistical techniques in comparison with clinical predictions. Journal of Educational Psychology, 57, 297-306.

American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text rev.). Washington, DC: Author.

Arkes, H. R. (1981). Impediments to accurate clinical judgment and possible ways to minimize their impact. Journal of Consulting and Clinical Psychology, 49, 323-330.

Arkes, H. R. (1991). Costs and benefits of judgment errors: Implications for debiasing. Psychological Bulletin, 110, 486-498.

Anastasi, A., & Urbina, S. (1996). Psychological testing (7th ed.). New York: Prentice Hall. *Astrup, C. A. (1975). Predicted and observed outcome in followed-up functional psychosis.

Biological Psychiatry, 10, 323-328. *Barron, F. (1953). Some test correlates of response to psychotherapy. Journal of Consulting

Psychology, 17, 235-241. Berven, N. L. (1985). Reliability and validity of standardized case management simulations.

Journal of Counseling Psychology, 32, 397-409. Berven, N. L., & Scofield, M. E. (1980). Evaluation of clinical problem-solving skills through

standardized case-management simulations. Journal of Counseling Psychology, 27, 199-208. Blocher, D. H. (1987). Process models for professional counseling. In The professional

counselor. New York: MacMillan. Blumetti, A. E. (1972). A test of clinical versus actuarial prediction: A consideration of accuracy

and cognitive functioning. Unpublished doctoral dissertation, University of Florida, Gainesville. Bolton, B. F., Butler, A. J., & Wright, G. N. (1968). Clinical versus statistical prediction of client

feasibility. Wisconsin Studies in Vocational Rehabilitation [Monograph VII], University of Wisconsin Regional Rehabilitation Research Institute, Madison, WI.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 375

TCP285875.qxd 3/29/2006 12:54 PM Page 375

Bury, A. S., & Bagby, R. M. (2002). The detection of feigned uncoached and coached posttraumatic stress disorder with the MMPI-2 in a sample of workplace accident victims. Psychological Assessment, 14, 472-484.

*Carlin, A. S., & Hewitt, P. L. (1990). The discrimination of patient-generated and randomly generated MMPIs. Journal of Personality Assessment, 54, 24-29.

Cohen, J. (1988). Statistical power analysis for the behavior sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Conrad, H. S., & Satter, G. A. (1954). The use of test scores and quality-classification ratings in predicting success in electrician’s mates school. Project N-106: Research and develop- ment of the Navy’s aptitude testing program. Princeton, NJ: Research and Statistical Laboratory College Entrance Examination Board.

*Cooke, J. K. (1967a). Clinicians’ decisions as a basis for deriving actuarial formulae. Journal of Clinical Psychology, 23, 232-233.

*Cooke, J. K. (1967b). MMPI in actuarial diagnosis of psychological disturbance among college males. Journal of Counseling Psychology, 14, 474-477.

Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Orlando, FL: Harcourt Brace Jovanovich.

*Danet, B. N. (1965). Prediction of mental illness in college students on the basis of “nonpsy- chiatric” MMPI profiles. Journal of Counseling Psychology, 29, 577-580.

Dawes, R. M. (1994). House of cards: Psychology and psychotherapy built on myth. New York: Free Press.

Dawes, R. M. (2002). The ethics of using or not using statistical prediction rules in psycho- logical practice and related consulting activities. Philosophy of Science, 69, S178-S184.

Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95-106.

Dawes, R. M., Faust, D., & Meehl, P.E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.

*Devries, A. G., & Shneidman, E. S. (1967). Multiple MMPI profiles of suicidal persons. Psychological Reports, 21, 401-405.

*Dickerson, J. H. (1958). The Biographical Inventory compared with clinical prediction of post counseling behavior of V.A. hospital counselors. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.

*Dunham, H. W., & Meltzer, B. N. (1946). Predicting length of hospitalization of mental patients. American Journal of Sociology, 52, 123-131.

Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician, 37, 36-48.

Ericsson, K. A., & Lehmann, A. C. (1996). Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology, 47, 273-305.

*Evenson, R. C., Altman, H., Sletten, I. W., & Cho, D. W. (1975). Accuracy of actuarial and clinical predictions for length of stay and unauthorized absence. Diseases of the Nervous System, 36, 250-252.

Falvey, J. E., & Hebert, D. J. (1992). Psychometric study of clinical treatment planning simulation (CTPS) for assessing clinical judgment. Journal of Mental Health Counseling, 14, 490-507.

Faust, D. (1986). Research on human judgment and its application to clinical practice. Professional Psychology, Research, and Practice, 17, 420-430.

Faust, D. (1991). What if we had really listened? Present reflections on altered pasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol. I. Matters of public interest (pp. 185-217). Minneapolis: University of Minnesota Press.

376 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 376

Faust, D. (1997). Of science, meta-science, and clinical practice: The generalization of a generalization to a particular. Journal of Personality Assessment, 68, 331-354.

Faust, D. (2003). Holistic thinking is not the whole story: Alternative or adjunct approaches for increasing the accuracy of legal evaluations. Assessment, 10, 428-411.

Fero, D. D. (1975). A lens model analysis of the effects of amount of information and mechan- ical decision making aid on clinical judgment and confidence. Unpublished doctoral dis- sertation, Bowling Green State University, Bowling Green, OH.

Garb, H. N. (1989). Clinical judgment, clinical training, and professional experience. Psychological Bulletin, 105, 387-396.

Garb, H. N. (1994). Toward a second generation of statistical prediction rules in psychodiag- nosis and personality assessment. Computers in Human Behavior, 10, 377-394.

Garb, H. N. (1998). Studying the clinician: Judgment research and psychological assessment. Washington, DC: American Psychological Association.

*Gardner, W., Lidz, C. W., Mulvay, E. P., & Shaw, E. C. (1996). Clinical versus actuarial pre- dictions of violence in patients with mental illnesses. Journal of Consulting and Clinical Psychology, 64, 602-609.

*Goldberg, L. R. (1965). Diagnosticians vs. diagnostic signs: The diagnosis of psychosis vs. neurosis from the MMPI. Psychological Monographs: General and Applied, 79(9), 1-27.

*Goldberg, L. R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving clinical inferences. Psychological Bulletin, 73, 422-432.

*Goldstein, S. G., Deysach, R. E., & Kleinknecht, R. A. (1973). Effect of experience and amount of information on identification of cerebral impairment. Journal of Consulting and Clinical Psychology, 41, 30-34.

Gottfredson, D. M., & Snyder, H. N. (2005). The mathematics of risk classification: Changing data into valid instruments for juvenile courts (NCJ 209158). Washington, DC: National Center for Juvenile Justice, Office of Juvenile Justice and Delinquency Prevention.

Graham, J. R., Watts, D., & Timbrook, R. E. (1991). Detecting fake-good and fake-bad MMPI-2 profiles. Journal of Personality Assessment, 57, 264-277.

*Grebstein, L. C. (1963). Relative accuracy of actuarial prediction, experienced clinicians, and graduate students in a clinical judgment task. Journal of Consulting Psychology, 27, 127-132.

Greene, R. L. (2000). The MMPI-2: An interpretive manual (2nd ed.). Boston: Allyn & Bacon. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological

Bulletin, 82, 1-20. Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impres-

sionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293-323.

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical vs. mechanical prediction: A meta-analysis. Psychological Assessment, 12, 19-30.

*Gustafson, D. H., Greist, J. H., Stauss, F. F., Erdman, H., & Laughren, T. (1977). A proba- bilistic system for identifying suicide attempters. Computers and Biomedical Research, 10, 83-89.

*Halbower, C. C. (1955). A comparison of actuarial versus clinical prediction to classes discriminated by the Minnesota Multiphasic Personality Inventory. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.

*Hall, G. C. N. (1988). Criminal behavior as a function of clinical and actuarial variables in a sexual offender population. Journal of Consulting and Clinical Psychology, 56, 773-775.

Hammond, K. R. (1996). Human judgment and social policy: Irreducible uncertainty, inevitable error, unavoidable justice. New York: Oxford University Press.

Hare, R. D. (1991). The revised psychopathy checklist. Toronto, Canada: Multi-Health Systems.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 377

TCP285875.qxd 3/29/2006 12:54 PM Page 377

Harvey-Cook, J. E., & Taffler, R. J. (2000). Biodata in professional entry-level selection: Statistical scoring of common format applications. Journal of Occupational and Organizational Psychology, 73, 103-118.

*Heaton, R. K., Grant, I., Anthony, W. Z., & Lehman, R. A. (1981). A comparison of clinical and automated interpretation of the Halstead-Reitan battery. Journal of Clinical Neuropsychology, 3, 121-141.

Hedges, L. V. (1987). How hard is hard science, how soft is soft science: The empirical cumu- lativeness of research. American Psychologist, 42, 443-455.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.

Hilton, N. Z., Harris, G. T., Rice, M. E., Lang, C., Cormier, C. A., & Lines, K. J. (2004). A brief actuarial assessment for the prediction of wife assault recidivism: The Ontario Domestic Assault Risk Assessment. Psychological Assessment, 16, 267-275.

*Holland, T. R., Holt, N., Levi, M., & Beckett, G. E. (1983). Comparison and combination of clinical and statistical predictions of recidivism among adult offenders. Journal of Applied Psychology, 68, 203-211.

Holt, R. R. (1958). Clinical and statistical prediction: A reformulation and some new data. Journal of Abnormal and Social Psychology, 56, 1-12.

Holt, R. R. (1970). Yet another look at clinical and statistical prediction: Or, is clinical psychology worthwhile? American Psychologist, 25, 337-349.

*Hovey, H. B., & Stauffacher, J. C. (1953). Intuitive versus objective prediction from a test. Journal of Clinical Psychology, 9, 341-351.

Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z. M., & Lutz, W. (1996). Efficacy, effec- tiveness, and patient progress. American Psychologist, 51, 1059-1064.

Hunter, J. E., & Smith, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.

Johnson, B. T. (1993). DSTAT 1.10: Software for the meta-analytic review of research literature. Hillsdale, NJ: Lawrence Erlbaum.

Johnson, B. T., & Eagly, A. H. (2000). Quantitative synthesis of social psychological research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social psychology. London: Cambridge University Press.

*Johnston, R., & McNeal, B. F. (1967). Statistical versus clinical prediction: Length of neuro- psychiatric hospital stay. Journal of Abnormal Psychology, 72, 335-340.

Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 801, 237-251.

*Kaplan, R. L. (1962). A comparison of actuarial and clinical predictions of improvement in psychotherapy. Unpublished doctoral dissertation, University of California, Los Angeles.

Katsikopoulos, K., Machery, E., Pachur, T., & Wallin, A. (2004). The search for models of clin- ical judgment: Fast, frugal, and friendly in Paul Meehl’s spirit. Unpublished manuscript. Retrieved October 15, 2004, from http://jeannicod.ccsd.cnrs.fr/documents/disk0/00/00/05/33/ ijn_00000533_00/ijn_00000533_00.pdf

Kelly, E. L., & Fiske, D. W. (1950). The prediction of success in the VA training program in clinical psychology. American Psychologist, 5, 395-406.

Kessler, R. C., Berglund, P., Demler, O., Jin, R., Koretz, D., Merikangas, K. R., et al. (2003). The epidemiology of major depressive disorder: Results from the National Comorbidity Survey Replication (NCS-R). Journal of the American Medical Association, 289, 3095-3106.

*Klehr, R. (1949). Clinical intuition and test scores as a basis for diagnosis. Journal of Consulting Psychology, 13, 34-38.

*Kleinmuntz, B. (1967). Sign and seer: Another example. Journal of Abnormal Psychology, 72, 163-165.

378 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 378

Kleinmuntz, B. (1990). Why we still use our heads instead of formulas: Toward an integrative approach. Psychological Bulletin, 107, 296-310.

*Klinger, E., & Roth, I. (1965). Diagnosis of schizophrenia by Rorschach patterns. Journal of Projective Techniques and Personality Assessment, 29, 323-335.

Kurpius, D. J., Benjamin, D., & Morran, D. K. (1985). Effect of teaching a cognitive strategy on counselor trainee internal dialogue and clinical hypothesis formulation. Journal of Counseling Psychology, 32, 262-271.

Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159-172.

Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G., et al. (1996). Administration and scoring manual for the Outcome Questionnaire (OQ 45.2). Wilmington, DE: American Professional Credentialing Services.

*Leli, D. A., & Filskov, S. B. (1981). Clinical-actuarial detection and description of brain impairment with the W-B Form 1. Journal of Clinical Psychology, 37, 623-629.

*Leli, D. A., & Filskov, S. B. (1984). Clinical detection of intellectual deterioration associated with brain damage. Journal of Clinical Psychology, 40, 1435-1441.

Lemerond, J. N. (1977). Suicide prediction for psychiatric patients: A comparison of the MMPI and clinical judgments. Unpublished doctoral dissertation, Marquette University, Madison, WI.

*Lewis, E. C., & MacKinney, A. C. (1961). Counselor vs. statistical prediction of job satisfac- tion in engineering. Journal of Counseling Psychology, 8, 224-230.

Lefkowitz, M. B. (1973). Statistical and clinical approaches to the identification of couples at risk in marriage. Unpublished doctoral dissertation, University of Florida, Gainesville.

Lichtenberg, J. W. (1997). Expertise in counseling psychology: A concept in search of support. Educational Psychology Review, 9, 221-238.

*Lindsey, G. R. (1965). Seer versus sign. Journal of Experimental Research in Personality, 1, 17-26. Lyle, O., & Quast, W. (1976). The Bender Gestalt: Use of clinical judgment versus recall scores

in prediction of Huntington’s disease. Journal of Consulting and Clinical Psychology, 44, 229-232.

McHugh, R. B., & Apostolakos, P. C. (1959). Methodology for the comparison of clinical with actuarial predictions. Psychological Bulletin, 56, 301-309.

Meehl, P. E. (1954). Clinical vs. statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.

Meehl, P. E. (1959). A comparison of clinicians with five statistical methods of identifying psychotic MMPI profiles. Journal of Counseling Psychology, 6, 102-109.

Meehl, P. E. (1986). Causes and effects of my disturbing little book. Journal of Personality Assessment, 50, 370-375.

*Melton, R. S. (1952). A comparison of clinical and actuarial methods of prediction with an assessment of the relative accuracy of different clinicians. Unpublished doctoral disserta- tion, University of Minnesota, Minneapolis.

Meyer, K. (1973). The effect of training in the accuracy and appropriateness of clinical judgment. Unpublished doctoral dissertation, Adelphi University, Garden City, NY.

Meyer, J. J., Finn, S. E., Eyde, L. D., Kay, G. G., Kubiszyn, T. W., Moreland, K. L., et al. (1998). Benefits and costs of psychological assessment in healthcare delivery: Report of the Board of Professional Affairs Psychological Assessment Work Group (Part I). Washington, DC: American Psychological Association.

*Miller, D. E., Kunce, J. T., & Getsinger, S. H. (1972). Prediction of job success for clients with hearing loss. Rehabilitation Counseling Bulletin, 16, 21-29.

Miller, G. A. (1985). The Substance Abuse Subtle Screening Inventory (SASSI): Manual. Bloomington, IN: Spencer Evening World.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 379

TCP285875.qxd 3/29/2006 12:54 PM Page 379

Nisbett, R. E., & Ross, L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, NJ: Prentice Hall.

Ogles, B., Lambert, M. J., & Masters, K. S. (1996). Assessing outcome in clinical practice. Needham Heights, MA: Allyn & Bacon.

*Oskamp, S. (1962). The relationship of clinical experience and training methods to several criteria of clinical prediction. Psychological Monographs: General and Applied, 76, 1-27.

Oxman, T. E., Rosenberg, S. D., Schnurr, P. P., & Tucker, G. J. (1988). Diagnostic classi- fication through content analysis of patients’ speech. American Journal of Psychiatry, 145, 464-468.

Pepinsky, H. B., & Pepinsky, N. (1954). Counseling theory and practice. New York: Ronald Press. *Perez, F. I. (1976). Behavioral analysis of clinical judgment. Perceptual and Motor Skills,

43, 711-718. *Popovics, A. J. (1983). Predictive validities of clinical and actuarial scores of the Gesell

Incomplete Man Test. Perceptual and Motor Skills, 56, 864-866. Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (1998). Violent offenders:

Appraising and managing risk. Washington, DC: American Psychological Association. Rock, D. L., Bransford, J. D., & Maisto, S. A. (1987). The study of clinical judgment: An

ecological approach. Clinical Psychology Review, 7, 645-661. Rosenthal, R. (1991). Meta-analytic procedures for social research (Rev. ed). Newbury Park,

CA: Sage. Rosenthal, R., & Rubin, D. (1982). A simple general purpose display of magnitude of exper-

imental effects. Journal of Educational Psychology, 74, 166-169. Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge

in psychological science. American Psychologist, 44, 1276-1284. Rouse, S. V., Butcher, J. N., & Miller, K. B. (1999). Assessment of substance abuse in psycho-

therapy clients: The effectiveness of the MMPI-2 substance abuse scales. Psychological Assessment, 11, 101-107.

*Russell, E. W. (1995). The accuracy of automated and clinical detection of brain damage and lateralization on neuropsychology. Neuropsychology Review, 5, 1-68.

Sarbin, T. L. (1942). A contribution to the study of actuarial and individual methods of pre- diction. American Journal of Sociology, 48, 593-602.

Sawyer, J. (1966). Measurement and prediction, clinical and statistical. Psychological Bulletin, 66, 178-200.

Schmidt, F. L., & Hunter, J. E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.

Shaffer, J. W., Perlin, S., Schmidt, C. W., & Stephens, J. H. (1974). The prediction of suicide in schizophrenia. Journal of Nervous and Mental Disease, 150, 349-355.

Shagoury, P., & Satz, P. (1969). The effect of statistical information on clinical prediction. Proceedings of the 77th Annual Convention of the American Psychological Association, 4, 517-518.

Sines, J. O. (1970). Actuarial versus clinical prediction in psychopathology. British Journal of Psychiatry, 116, 129-144.

Spengler, P. M. (1998). Multicultural assessment and a scientist-practitioner model of psy- chological assessment. The Counseling Psychologist, 26, 930-938.

Spengler, P. M., & Strohmer, D. C. (2001, August). Empirical analyses of a scientist-practitioner model of assessment. Paper presented at the annual meeting of the American Psychological Association, San Francisco.

Spengler, P. M., Strohmer, D. M., Dixon, D. N., & Shivy, V. A. (1995). A scientist-practitioner model of psychological assessment: Implications for training, practice, and research. The Counseling Psychologist, 23, 506-534.

380 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 380

Spengler, P. M., White, M. J., Ægisdóttir, S., Maugherman, A. S., Anderson, L. A., Cook, R. S., et al. (2005). The meta-analysis of clinical judgment project: Effects of experience on judg- ment accuracy. Manuscript submitted for publication.

Steering Committee of the Physicians Health Study Research Group. (1988). Preliminary report: Findings from the aspirin component of the ongoing physicians’ health study. New England Journal of Medicine, 318, 262-264.

Stein, L. A. R., Graham, J. R., Ben-Porath, Y. S., & McNulty, J. L. (1999). Using the MMPI-2 to detect substance abuse in an outpatient mental health setting. Psychological Assessment, 11, 94-100.

Stricker, G. (1967). Actuarial, naïve clinical, and sophisticated clinical prediction of pathology from figure drawings. Journal of Consulting Psychology, 31, 492-494.

Stricker, G., & Trieweiler, S. J. (1995). The local clinical scientist: A bridge between science and practice. American Psychologist, 50, 995-1002.

Sullivan, E., Cirincione, C., Nelson, K., & Wallis, J. (2001). Classifying inmates for strategic programming. Washington, DC: National Criminal Justice Service, U.S. Department of Justice.

*Szuko, J. J., & Kleinmuntz, B. (1981). Statistical versus clinical lie detection. American Psychologist, 36, 488-496.

*Taulbee, E. S., & Sisson, B. D. (1957). Configurational analysis of MMPI profiles of psy- chiatric groups. Journal of Consulting Psychology, 21, 413-417.

*Thompson, R. E. (1952). A validation of the Glueck Social Prediction Scale for proneness to delinquency. Journal of Criminal Law, Criminology, and Police Science, 43, 451-470.

Waehler, C. A., Kalodner, C. R., Wampold, B. E., & Lichtenberg, J. W. (2000). Empirically supported treatments (ESTs) in perspective: Implications for counseling psychology train- ing. The Counseling Psychologist, 28, 657-671.

*Walters, G. D., White, T. W., & Greene, R. L. (1987). The use of MMPI to identify malinger- ing and exaggeration of psychiatric symptomatology in male prison inmates. Journal of Consulting and Clinical Psychology, 1, 111-117.

Wampold, B. E. (1997). Methodological problems in identifying efficacious psychotherapies. Psychotherapy Research, 7, 21-43.

Watley, D. J. (1966). Counselor variability in making accurate predictions. Journal of Counseling Psychology, 13, 53-62.

*Watley, D. J., & Vance, F. L. (1964). Clinical versus actuarial prediction of college achieve- ment and leadership activity. Unpublished doctoral dissertation, University of Minnesota, Minneapolis.

Webb, S. C., Hultgen, D. D., & Craddick, R. A. (1977). Predicting occupational choice by clinical and statistical methods. Journal of Counseling Psychology, 24, 98-110.

*Wedding, D. (1983). Clinical and statistical prediction in neuropsychology. Clinical Neuro- psychology, 5, 49-55.

*Weinberg, G. H. (1957). Clinical versus statistical prediction with a method of evaluating a clinical tool. Unpublished doctoral dissertation, Columbia University, New York.

Werner, P. D., Rose, T. L., Yesavage, J. A., & Seeman, K. (1984). Psychiatrists’ judgment of dangerousness in patients on an acute care unit. American Journal of Psychiatry, 141, 263-266.

Westen, D., & Weinberger, J. (2004). When clinical description becomes statistical prediction. American Psychologist, 59, 595-613.

Wiggins, J. S. (1981). Clinical and statistical prediction: Where are we and where do we go from here? Clinical Psychology Review, 1, 3-18.

Wiggins, N., & Kohen, E. S. (1971). Man versus model of man revisited: The forecasting of graduate school success. Journal of Personality and Social Psychology, 19, 100-106.

Ægisdóttir et al. / CLINICAL VERSUS STATISTICAL PREDICTION 381

TCP285875.qxd 3/29/2006 12:54 PM Page 381

*Wirt, R. D. (1956). Actuarial prediction. Journal of Consulting Psychology, 20, 123-124. *Wittman, M. P., & Steinberg, L. (1944). Follow-up of an objective evaluation of prognosis in

dementia praecox and manic-depressive psychosis. The Elgin Papers, 5, 216-227. Zalewski, C. E., & Gottesman, I. I. (1991). (Hu)man versus mean revisited: MMPI group data

and psychiatric diagnosis. Journal of Abnormal Psychology, 100, 562-568.

382 THE COUNSELING PSYCHOLOGIST / May 2006

TCP285875.qxd 3/29/2006 12:54 PM Page 382

<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJDFFile false /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams true /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveEPSInfo true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true ] /NeverEmbed [ true ] /AntiAliasColorImages false /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox false /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (U.S. Web Coated \050SWOP\051 v2) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown /SyntheticBoldness 1.000000 /Description << /FRA <FEFF004f007000740069006f006e00730020007000650072006d0065007400740061006e007400200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000500044004600200064006f007400e900730020006400270075006e00650020007200e90073006f006c007500740069006f006e002000e9006c0065007600e9006500200070006f0075007200200075006e00650020007100750061006c0069007400e90020006400270069006d007000720065007300730069006f006e00200061006d00e9006c0069006f007200e90065002e00200049006c002000650073007400200070006f0073007300690062006c0065002000640027006f00750076007200690072002000630065007300200064006f00630075006d0065006e007400730020005000440046002000640061006e00730020004100630072006f0062006100740020006500740020005200650061006400650072002c002000760065007200730069006f006e002000200035002e00300020006f007500200075006c007400e9007200690065007500720065002e> /JPN <FEFF3053306e8a2d5b9a306f30019ad889e350cf5ea6753b50cf3092542b308000200050004400460020658766f830924f5c62103059308b3068304d306b4f7f75283057307e30593002537052376642306e753b8cea3092670059279650306b4fdd306430533068304c3067304d307e305930023053306e8a2d5b9a30674f5c62103057305f00200050004400460020658766f8306f0020004100630072006f0062006100740020304a30883073002000520065006100640065007200200035002e003000204ee5964d30678868793a3067304d307e30593002> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e0020005000440046002d0044006f006b0075006d0065006e00740065006e0020006d00690074002000650069006e006500720020006800f60068006500720065006e002000420069006c0064006100750066006c00f600730075006e0067002c00200075006d002000650069006e0065002000760065007200620065007300730065007200740065002000420069006c0064007100750061006c0069007400e400740020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f0062006100740020006f0064006500720020006d00690074002000640065006d002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /PTB <FEFF005500740069006c0069007a006500200065007300740061007300200063006f006e00660069006700750072006100e700f5006500730020007000610072006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006d00200075006d00610020007200650073006f006c007500e700e3006f00200064006500200069006d006100670065006d0020007300750070006500720069006f0072002000700061007200610020006f006200740065007200200075006d00610020007100750061006c0069006400610064006500200064006500200069006d0070007200650073007300e3006f0020006d0065006c0068006f0072002e0020004f007300200064006f00630075006d0065006e0074006f0073002000500044004600200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002c002000520065006100640065007200200035002e0030002000650020007300750070006500720069006f0072002e> /DAN <FEFF004200720075006700200064006900730073006500200069006e0064007300740069006c006c0069006e006700650072002000740069006c0020006100740020006f0070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f8006a006500720065002000620069006c006c00650064006f0070006c00f80073006e0069006e006700200066006f00720020006100740020006600e50020006200650064007200650020007500640073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /NLD <FEFF004700650062007200750069006b002000640065007a006500200069006e007300740065006c006c0069006e00670065006e0020006f006d0020005000440046002d0064006f00630075006d0065006e00740065006e0020007400650020006d0061006b0065006e0020006d00650074002000650065006e00200068006f0067006500720065002000610066006200650065006c00640069006e00670073007200650073006f006c007500740069006500200076006f006f0072002000650065006e0020006200650074006500720065002000610066006400720075006b006b00770061006c00690074006500690074002e0020004400650020005000440046002d0064006f00630075006d0065006e00740065006e0020006b0075006e006e0065006e00200077006f007200640065006e002000670065006f00700065006e00640020006d006500740020004100630072006f00620061007400200065006e002000520065006100640065007200200035002e003000200065006e00200068006f006700650072002e> /ESP <FEFF0055007300650020006500730074006100730020006f007000630069006f006e006500730020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006e0020006d00610079006f00720020007200650073006f006c00750063006900f3006e00200064006500200069006d006100670065006e00200070006100720061002000610075006d0065006e0074006100720020006c0061002000630061006c006900640061006400200061006c00200069006d007000720069006d00690072002e0020004c006f007300200064006f00630075006d0065006e0074006f00730020005000440046002000730065002000700075006500640065006e00200061006200720069007200200063006f006e0020004100630072006f00620061007400200079002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /SUO <FEFF004e00e4006900640065006e002000610073006500740075007300740065006e0020006100760075006c006c006100200076006f0069006400610061006e0020006c0075006f006400610020005000440046002d0061007300690061006b00690072006a006f006a0061002c0020006a006f006900640065006e002000740075006c006f0073007400750073006c00610061007400750020006f006e0020006b006f0072006b006500610020006a00610020006b007500760061006e0020007400610072006b006b007500750073002000730075007500720069002e0020005000440046002d0061007300690061006b00690072006a0061007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f006200610074002d0020006a00610020004100630072006f006200610074002000520065006100640065007200200035002e00300020002d006f0068006a0065006c006d0061006c006c0061002000740061006900200075007500640065006d006d0061006c006c0061002000760065007200730069006f006c006c0061002e> /ITA <FEFF00550073006100720065002000710075006500730074006500200069006d0070006f007300740061007a0069006f006e00690020007000650072002000630072006500610072006500200064006f00630075006d0065006e00740069002000500044004600200063006f006e00200075006e00610020007200690073006f006c0075007a0069006f006e00650020006d0061006700670069006f00720065002000700065007200200075006e00610020007100750061006c0069007400e00020006400690020007300740061006d007000610020006d00690067006c0069006f00720065002e0020004900200064006f00630075006d0065006e00740069002000500044004600200070006f00730073006f006e006f0020006500730073006500720065002000610070006500720074006900200063006f006e0020004100630072006f00620061007400200065002000520065006100640065007200200035002e003000200065002000760065007200730069006f006e006900200073007500630063006500730073006900760065002e> /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f00700070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f80079006500720065002000620069006c00640065006f00700070006c00f80073006e0069006e006700200066006f00720020006200650064007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f0067002000730065006e006500720065002e> /SVE <FEFF0041006e007600e4006e00640020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006e00e40072002000640075002000760069006c006c00200073006b0061007000610020005000440046002d0064006f006b0075006d0065006e00740020006d006500640020006800f6006700720065002000620069006c0064007500700070006c00f60073006e0069006e00670020006f006300680020006400e40072006d006500640020006600e50020006200e400740074007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e0020006b0061006e002000f600700070006e006100730020006d006500640020004100630072006f0062006100740020006f00630068002000520065006100640065007200200035002e003000200065006c006c00650072002000730065006e006100720065002e> /ENU <FEFF005500730065002000740068006500730065002000730065007400740069006e0067007300200066006f00720020006300720065006100740069006e00670020005000440046002000660069006c0065007300200066006f00720020007300750062006d0069007300730069006f006e00200074006f002000540068006500200053006800650072006900640061006e002000500072006500730073002e002000540068006500730065002000730065007400740069006e0067007300200063006f006e006600690067007500720065006400200066006f00720020004100630072006f006200610074002000760036002e0030002000300038002f00300036002f00300033002e> >> >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice