A L A N B RY M A N University of Leicester
A B S T R A C T This article seeks to move beyond typologies of the ways in which quantitative and qualitative research are integrated to an examination of the ways that they are combined in practice. The article is based on a content analysis of 232 social science articles in which the two were combined. An examination of the research methods and research designs employed suggests that on the quantitative side structured interview and questionnaire research within a cross-sectional design tends to predominate, while on the qualitative side the semi-structured interview within a cross-sectional design tends to predominate. An examination of the rationales that are given for employing a mixed-methods research approach and the ways it is used in practice indicates that the two do not always correspond. The implications of this finding for how we think about mixed-methods research are outlined.
K E Y W O R D S : qualitative research, quantitative research, mixed-methods research, multi-strategy research, typologies
There can be little doubt that research that involves the integration of quanti- tative and qualitative research has become increasingly common in recent years. While some writers express unease about the ‘whatever works’ position that underpins it (e.g. Buchanan, 1992; Pawson and Tilly, 1997), so far as research practice is concerned, combining quantitative and qualitative research has become unexceptional and unremarkable in recent years. Indeed, for some writers it has come to be seen as a distinctive research approach in its own right that warrants comparison with each of quantitative and qualitative research. In this sense, we end up with three distinct approaches to research: quantitative; qualitative; and what is variously called multi-methods (Brannen, 1992), multi-strategy (Bryman, 2004), mixed methods (Creswell,
A RT I C L E 97
Integrating quantitative and qualitative research: how is it done?
Qualitative Research Copyright © 2006 SAGE Publications (London, Thousand Oaks, CA and New Delhi) vol. 6(1) 97–113.
2003; Tashakkori and Teddlie, 2003), or mixed methodology (Tashakkori and Teddlie, 1998) research. In the field of evaluation research, and indeed in several other applied fields, the case for a multi-strategy research approach seems to have acquired especially strong support (Tashakkori and Teddlie, 2003).
Typologies of mixed-methods research
The discussion of the integration of quantitative and qualitative research has increasingly been taken over by a formalized approach which is especially apparent in the discussion and proliferation of typologies of integration. This has been a particular emphasis among North American contributors to the field. Creswell et al. (2003) argue that giving types of mixed-methods research names has certain advantages. It conveys a sense of the rigour of the research and provides guidance to others about what researchers intend to do or have done (for example, funding bodies and journal editors). To that extent, the typologies of mixed-methods or multi-strategy research can be helpful to researchers and writers in clarifying the nature of their intentions or of their accomplishments.
However, the variety and range of typologies has reached the point where these exercises have become almost too refined, bearing in mind that the range of concrete examples of multi-strategy research is not great. Indeed, most of the typologies have been constructed in largely theoretical terms and have not been apparently influenced in a systematic way by examples of multi-strategy research. To a large extent, they are exercises in logically possible types of inte- gration, rather than being built up out of examples.
However, the dimensions out of which the typologies are constructed are instructive, in that they draw attention to the different aspects of multi- strategy research:
1. Are the quantitative and qualitative data collected simultaneously or sequentially? (Morgan, 1998; Morse, 1991).
2. Which has priority – the quantitative or the qualitative data? (Morgan, 1998; Morse, 1991).
3. What is the function of the integration – for example, triangulation, explanation, or exploration? (Creswell, 2003; Creswell et al., 2003; Greene et al., 1989).
4. At what stage(s) in the research process does multi-strategy research occur? (Tashakkori and Teddlie, 1998). It may be at stages of research question formulation, data collection, data analysis, or data interpret- ation.
5. Is there more than one data strand? (Tashakkori and Teddlie, 2003). With a multi-strand study, there is more than one research method and hence source of data. With a mono-strand study, there is one research method
Qualitative Research 6(1)98
and hence one source of data. However, whether a mono-strand study can genuinely be regarded as a form of mixing methods is debatable.
A further issue with the use of these typologies is that they imply some forward commitment to a type of design, much like a decision to employ an experimental research design entails a commitment to uncovering data of a particular type. However, as some authors observe (e.g. Erzberger and Kelle, 2003), the outcomes of multi-strategy research are not always predictable. While a decision about design issues may be made in advance and for good reasons, when the data are generated, surprising findings or unrealized poten- tial in the data may suggest unanticipated consequences of combining them.
What do we know about mixing quantitative and qualitative research?
The exercise of specifying typologies co-exists with unease among some authors about what we actually know about the ways in which quantitative and qualitative research are combined in practice. For example, it has been suggested that there are relatively few guidelines about ‘how, when and why different research methods might be combined’ (Bryman, 1988: 155). Maxwell has suggested that ‘the theoretical debate about combining methods has prevented us from seeing the different ways in which researchers are actually combining methods’ (Maxwell, 1990: 507, cited in Maxwell and Loomis, 2003: 251). He and Loomis have argued further that:
Uncovering the actual integration of qualitative and quantitative approaches in any particular study is a considerably more complex undertaking than simply classifying the study into a particular category on the basis of a few broad dimen- sions or characteristics. (Maxwell and Loomis, 2003: 256)
Remarks such as these suggest that the formalization of approaches to multi- strategy research through typologies has moved too far ahead of a systematic appreciation of how quantitative and qualitative research are combined in practice. The writers who adopt a formalized strategy use many examples to illustrate their ‘types’ but we have relatively little understanding of the prevalence of different combinations, though there are some exceptions to this statement (e.g. Greene et al., 1989; Niglas, 2004).
An investigation of articles combining quantitative and qualitative research
With these kinds of consideration in mind, an investigation was undertaken of the ways that quantitative and qualitative research are combined in published journal articles. The findings reported in this article derive from only one phase of this research project, albeit a major component of it – namely, a content
Bryman: Integrating quantitative and qualitative research 99
analysis of articles based on multi-strategy research. Journal articles do not encapsulate all possible contexts in which projects reporting multi-strategy research might be found. Conference papers and books are other possible sites. However, journal articles are a major form of reporting findings and have the advantage that, in most cases, the peer review process provides a quality control mechanism. By contrast, conference papers and books are sometimes not peer reviewed.
The approach to gleaning a sample was to search the Social Sciences Citation Index (SSCI) for articles in which relevant key words or phrases such as ‘quantitative’ and ‘qualitative’, or ‘multi(-)method’, or ‘mixed method’, or ‘triangulation’ appeared in the title, key words, or abstract. This means that the sample comprises articles which to some degree foreground the fact that the study is based on both quantitative and qualitative research. Searches using other kinds of key words, such as ‘survey’ and ‘ethnography/ic’, produce a far larger sample of articles than could be dealt with within the purview of this investigation. In conducting the search, the emphasis was on uncovering articles in five fields: sociology; social psychology; human, social and cultural geography; management and organizational behaviour; and media and cultural studies. The analysis was restricted to the 10-year period of 1994–2003. The fact that the findings are based on a large corpus of articles suggests that the sample is unlikely to be overly atypical, although claims of representativeness would be impossible to sustain. Judgments about whether articles fell within the purview of the investigation, in terms of whether they could be regarded as deriving from the five fields, were made on the basis of the journal title or information supplied in abstracts. In this way, a total of 232 articles was generated and content analyzed.
What was and was not an example of the combination of quantitative and qualitative research was occasionally problematic. The most notable of these occasions had to do with cases in which the researcher claimed to have used a qualitative approach or to be using qualitative data, but in fact the ‘qualitative data’ were based on a quantitative analysis of unstructured data – for example, of responses to open questions. Articles in which this occurred and where such data were the only source of the qualitative component were not included in the sample, because it is very debatable whether they can be regarded as indica- tive of a qualitative approach. This kind of quantification of qualitative data is more properly regarded as indicative of a quantitative research approach. Indeed, in some articles that were included in the sample, this kind of process was depicted by authors as indicative of a quantitative research approach rather than a qualitative one. There is clearly some confusion concerning whether the quantification of qualitative, unstructured data is indicative of a quantitative or a qualitative research approach. For the purposes of sample selection, it was taken to be the former, regardless of authors’ claims. However, this was not a very common occurrence; although a log was not kept of these cases, they number no more than five or six articles.
Qualitative Research 6(1)100
The sample is likely to be biased in the sense that by no means all authors of articles reporting multi-strategy research foreground the fact that the findings reported derive from a combination of quantitative and qualitative research, or do not do so in terms of the key words that drove the online search strategy. An alternative search strategy is to select a sample of journals and to search for articles exhibiting multi-strategy research. This tactic was employed by Niglas (2004) in her investigation of multi-strategy research in education. Her sample of 145 articles derived from 1156 articles in 15 journals. This is a very good way of generating a sample but, in the context of a study that is meant to cover five fields of study, it is difficult to implement and also results in a lot of redundancy because a large number of articles have to be read in order to establish whether they are based on both quantitative and qualitative research (only 12.5 percent of articles read for Niglas’s study were relevant to the main focus of her investigation). Moreover, foregrounding that a study is based on multi-strategy research is interesting because it implies that the fact that the different sources of data were employed is important and significant to the author(s) concerned. Since a major focus of the research was the kinds of purposes to which multi-strategy research is put, the online search strategy that was used for the study reported here was very relevant, because we might anticipate that researchers who choose to emphasize this aspect of their studies will have given greater consideration to the issues involved in combin- ing quantitative and qualitative research. In this sense, the articles from which the findings derive constitute a purposive sample. A further issue that suggests some advantages to the sampling approach taken for this article is that it allows articles in a wide variety of journals to be uncovered. Thus, while it is certainly possible to trawl through sociology journals for instances of multi-strategy research articles in sociology, such a process risks neglecting many relevant sociology articles appearing in specialist journals.
Several writers have pointed out that quantitative and qualitative research can be combined at different stages of the research process: formulation of research questions; sampling; data collection; and data analysis. Articles for this study were chosen in terms of data collection and data analysis and then content analyzed in relation to these aspects of the research process. Issues of sampling did materialize in the study, as the findings below will indicate. Data collection and analysis were emphasized because these are arguably defining features of quantitative and qualitative research. Moreover, multi-strategy research articles nearly always entail the collection and analysis of both quan- titative and qualitative data (Niglas, 2004).
BAC KG RO U N D F I N D I N G S First, a small number of background features of the articles analyzed thus far will be mentioned. When the primary discipline of each article is examined, we find that the major contributing discipline is sociology with 36 percent of all articles. This is followed by social psychology (27%); management and
Bryman: Integrating quantitative and qualitative research 101
organizational behaviour (23%); geography (8%); and media and cultural studies (7%). These findings strongly suggest that multi-strategy research is more commonly practised in some disciplines than others.
A further interesting background characteristic is the nation of the insti- tutional affiliation of the author or first author of each article. North America is the major contributor with 49 percent of all articles; the UK comes second with 27 percent; followed by Europe and Australia (8% and 7%); Middle East (4%); with Asia, Africa and Latin America contributing 3 percent between them. These figures are obviously significantly affected by the fact that only English language publications were sought and read.
R E S E A RC H M E T H O D S A N D R E S E A RC H D E S I G N S U S E D The first issue to be addressed in this article is: what research methods and research designs were employed in the articles? Each article was coded in terms of the research methods that were employed. Some of the research methods are perhaps better thought of as methods of data analysis, but they are frequently portrayed as research methods because of their distinctive approaches to sampling or capturing data (for example, content analysis, discourse analysis and conversation analysis). Table 1 presents the main methods used. The analysis presented derives from a multiple response analysis using SPSS.
A striking feature is that a small number of methods account for the vast majority of all methods employed. Survey methods and qualitative interviews account for the vast majority of methods employed in the articles. If we aggregate self-administered questionnaire, structured interview and questionnaire/structured interview (a category used when it was unclear
Qualitative Research 6(1)102
TA B L E 1 . Research methods employed
Number of articles using
Self-administered questionnaire 121 Structured interview 52 Structured observation 3 Content analysis 18 Quantification of qualitative interview questions 15 Questionnaire/structured interview 18 Semi-structured interview 159 Participant observation/ethnography 14 Unstructured interview 6 Qualitative analysis of documents 28 Answers to open questions in questionnaire 48 Focus groups 33 Language-based analysis 5 Other method 55
how survey instruments were administered), 82.4 percent of all articles coded used a survey instrument. If we aggregate semi-structured interview and unstructured interview, we find that data for 71.1 percent of articles derived from either of these two ways of conducting qualitative interviews. Further, 57.3 percent of all articles are based on a combination of a survey instrument and qualitative interviewing. In other words, one combination of research methods predominates in this data set – that is, one in which data are collected by either structured interview or questionnaire on the quantita- tive side along with either a semi-structured or unstructured interview on the qualitative side.
A further feature is that with 6.5 percent of articles, the quantitative data derive from an individual qualitative or focus group interview and that, in 20.7 percent of articles, the qualitative data derive from open questions in a struc- tured interview or self-administered questionnaire. In the former case, the quantitative data derive from a research instrument associated with qualitative data collection while, in the second case, the qualitative data derive from a research instrument associated with quantitative data collection. In other words, for around 27 percent of articles, the collection of quantitative and qualitative data was not based on the administration of separate research instruments.
This finding is interesting because some methodologists might argue that a combination of quantitative and qualitative data based on the administration of one research instrument does not represent a true integration of quantita- tive and qualitative research because one will tend to be subordinate to the other. Thus, when multi-strategy research derives from the administration of a semi-structured interview, some of whose questions are quantified, an argument might be levelled that this does not represent a genuine form of quantitative research since the data have not been gathered in line with its underlying principles. Similarly, it might be argued that asking a small number of open questions in the course of a structured interview does not really provide an instance of multi-strategy research because the qualitative data have been collected in the course of administering a research instrument that has been devised in terms of survey principles. Moreover, such a situation requires a modification of approach to answering questions on the part of respondents in the course of responding to a research instrument. However, articles adopting an approach in which quantitative and qualitative data derived from the same research instrument were included.
When we turn to research designs, the aim of the analysis was to code articles in terms of the design employed for the quantitative data and the design employed for the qualitative data. In a small number of cases (4), because of the complexity of the data, a third research design was coded. Table 2 presents the data on this aspect of the investigation using a classifi- cation that follows Bryman’s (2004) categorization of research designs. In this classification, a study is treated as a case study if it involves just a single case.
Bryman: Integrating quantitative and qualitative research 103
If it was a multiple case study, involving two or more cases, it was treated as a comparative design. Again, the analysis derived from a multiple response analysis using SPSS.
As one might expect from the findings in Table 2, the bulk of the studies employed a cross-sectional design for the collection of both the quantitative and the qualitative data. Experimental and quasi-experimental designs barely figure in the findings. Employing a cross-sectional design for the collection of both quantitative and qualitative data is by far the most common design combi- nation (62.9% of all articles). When we put the data relating to research methods and research designs together, we find that 41.8 percent of all articles included both a survey instrument and personal qualitative interviewing within a cross-sectional design for the collection of both sets of data. Some- times, although rarely, this format will have been accompanied by other sources of data.
J U S T I F I CAT I O N S F O R C O M B I N I N G Q UA N T I TAT I V E A N D Q UA L I TAT I V E R E S E A RC H A major focus of the content analysis was on the rationales proffered for combining quantitative and qualitative research. This aspect of the investi- gation was approached in several ways. First, the rationale given by authors for combining the two approaches to data collection and/or analysis was coded. For this exercise, the reasons that were given before the findings were presented were typically examined. Then, the ways in which quantitative and qualitative research were actually combined were coded. In doing so, the coding reflected authors’ reflections on what they felt had been gleaned from combining quan- titative and qualitative research, and any ways in which the two were combined which were not reflected in authors’ accounts. The purpose of discriminating between these two ways of thinking about the justification for multi-strategy research was that authors’ accounts of why they intended to combine quanti- tative and qualitative research might differ from how they actually combined
Qualitative Research 6(1)104
TA B L E 2 . Research designs employed
Number of articles using
Cross-sectional design 1 169 Cross-sectional design 2 148 Case study 1 24 Case study 2 16 Longitudinal 1 28 Longitudinal 2 19 Experimental 1 (includes quasi-experimental) 9 Experimental 2 (includes quasi-experimental) 5 Comparative 1 (includes multiple case study) 30 Comparative 2 (includes multiple case study) 19
them in practice. If a difference was sometimes found between the two accounts (that is, between the rationale and practice), this would be interesting because the scientific paper is often perceived among sociologists of science as an ex post facto reconstruction that rationalizes and injects coherence into the different elements of the research process (e.g. Gilbert and Mulkay, 1984).
In coding the justifications for combining quantitative and qualitative research, two different schemes were employed. First, the influential scheme devised in the context of evaluation research by Greene et al. (1989) was used. This scheme isolates five justifications for combining quantitative and quali- tative research:
1. Triangulation: convergence, corroboration, correspondence or results from different methods. In coding triangulation, the emphasis was placed on seeking corroboration between quantitative and qualitative data.
2. Complementarity: ‘seeks elaboration, enhancement, illustration, clarifi- cation of the results from one method with the results from another’ (Greene et al., 1989: 259).
3. Development: ‘seeks to use the results from one method to help develop or inform the other method, where development is broadly construed to include sampling and implementation, as well as measurement decisions’ (Greene et al., 1989: 259).
4. Initiation: ‘seeks the discovery of paradox and contradiction, new perspec- tives of [sic] frameworks, the recasting of questions or results from one method with questions or results from the other method’ (Greene et al., 1989: 259).
5. Expansion: ‘seeks to extend the breadth and range of enquiry by using different methods for different inquiry components’ (Greene et al., 1989: 259).
This scheme has been quite influential and was employed by Niglas (2004) in her examination of education research articles. In their analysis of evalu- ation research articles, Greene et al. (1989) coded each article in terms of a primary and a secondary rationale, a procedure that was also employed by Niglas (2004). An advantage of the Greene et al. scheme is its parsimony, in that it boils down the possible reasons for conducting multi-strategy research to just five reasons, although the authors’ analysis revealed that initiation was uncommon. A disadvantage is that it only allows two rationales to be coded (primary and secondary). Accordingly, a more detailed but considerably less parsimonious scheme was devised. It was based on an extensive review of the kinds of reasons that are frequently given in both methodological writings and research articles for combining quantitative and qualitative research. The scheme provided for the following rationales:
a) Triangulation or greater validity – refers to the traditional view that quan- titative and qualitative research might be combined to triangulate
Bryman: Integrating quantitative and qualitative research 105
findings in order that they may be mutually corroborated. If the term was used as a synonym for integrating quantitative and qualitative research, it was not coded as triangulation.
b) Offset – refers to the suggestion that the research methods associated with both quantitative and qualitative research have their own strengths and weaknesses so that combining them allows the researcher to offset their weaknesses to draw on the strengths of both.
c) Completeness – refers to the notion that the researcher can bring together a more comprehensive account of the area of enquiry in which he or she is interested if both quantitative and qualitative research are employed.
d) Process – quantitative research provides an account of structures in social life but qualitative research provides sense of process.
e) Different research questions – this is the argument that quantitative and qualitative research can each answer different research questions but this item was coded only if authors explicitly stated that they were doing this.
f) Explanation – one is used to help explain findings generated by the other. g) Unexpected results – refers to the suggestion that quantitative and quali-
tative research can be fruitfully combined when one generates surprising results that can be understood by employing the other.
h) Instrument development – refers to contexts in which qualitative research is employed to develop questionnaire and scale items – for example, so that better wording or more comprehensive closed answers can be generated.
i) Sampling – refers to situations in which one approach is used to facilitate the sampling of respondents or cases.
j) Credibility – refers to suggestions that employing both approaches enhances the integrity of findings.
k) Context – refers to cases in which the combination is rationalized in terms of qualitative research providing contextual understanding coupled with either generalizable, externally valid findings or broad relationships among variables uncovered through a survey.
l) Illustration – refers to the use of qualitative data to illustrate quantitative findings, often referred to as putting ‘meat on the bones’ of ‘dry’ quanti- tative findings.
m) Utility or improving the usefulness of findings – refers to a suggestion, which is more likely to be prominent among articles with an applied focus, that combining the two approaches will be more useful to practitioners and others.
n) Confirm and discover – this entails using qualitative data to generate hypotheses and using quantitative research to test them within a single project.
o) Diversity of views – this includes two slightly different rationales – namely, combining researchers’ and participants’ perspectives through quantitative and qualitative research respectively, and uncovering relationships between variables through quantitative research while also
Qualitative Research 6(1)106
revealing meanings among research participants through qualitative research.
p) Enhancement or building upon quantitative/qualitative findings – this entails a reference to making more of or augmenting either quantitative or qualitative findings by gathering data using a qualitative or quantita- tive research approach.
q) Other/unclear. r) Not stated.
This classification includes a larger number of categories than other schemes and as such is meant to capture in finer detail the range of reasons that are given for conducting multi-strategy research. There are clearly symmetries between the Greene et al. scheme and the more fine-grained approach just outlined. For example, ‘development of a research instrument’ and ‘for sampling/case study selection reasons’ correspond to ‘development’, while ‘to enhance or build upon quantitative/qualitative findings’ corresponds to ‘complementarity’.
Table 3 shows the distribution of articles in terms of just the primary rationale using the Greene et al. scheme (see column for ‘Rationale’). In just over a quarter of all articles, no rationale was provided. Complementarity and expansion were the most frequently cited primary rationales with 29 percent and 25 percent of all articles mentioning each of them as a primary rationale. Triangulation and development were less commonly mentioned, while initia- tion was extremely uncommon. The latter was also the case in Greene et al. (1989) but was even more the case in the data reported here. Turning to the actual uses of the integration of quantitative and qualitative research, Table 3 (column for ‘Practice’) provides the primary use in terms of the Greene et al. scheme. All the frequencies are greater because the category of ‘not stated’ virtually disappears. Most striking is that nearly half of all articles can be subsumed into the complementarity category. In terms of the Greene et al. scheme, this is by far the most prominent primary approach to the integration of quantitative and qualitative research.
Bryman: Integrating quantitative and qualitative research 107
TA B L E 3 . Uses of multi-strategy research – Greene et al. scheme
Category Rationale % Practice
Triangulation 7.8 12.5 Complementarity 28.9 44.8 Development 10.3 8.6 Initiation 0.4 1.3 Expansion 25.4 31.5 Not stated 27.2 1.3
Note: All percentages are based on 232 cases
When the data for the ‘practice’ column in Table 3 using the Greene et al. scheme are contrasted with comparable data from Greene et al. (1989) and Niglas (2004), we find that the pattern is closer to Niglas’s examination of educational articles than to that of Greene et al. Similarly to Niglas, comple- mentarity is the most common use of multi-strategy research, followed by expansion. In the examination of evaluation research articles by Greene et al., it was the other way around in that expansion was more common that complementarity. In the case of all three studies, then, these two uses of multi- strategy research were the most common forms. ‘Development’ as a use occurs with noticeably greater frequency in educational articles than in the present study and that of Greene et al. These findings suggest that there are slightly different uses being made of multi-strategy research when we compare general social research (present study), evaluation research and educational research, though the differences are not great.
When the articles are examined in terms of the more detailed scheme devised for the research, the rationale for nearly one-third of all articles could be coded in terms of ‘enhancement’ (see Table 4, ‘Rationale’ column). Quite large numbers of articles appeared under the categories: ‘completeness’ (13%), ‘triangulation’ (12.5%), and ‘sampling’ (13.4%). A sizeable number could not be coded in terms of any category because no rationale could be discerned.
When the articles are examined in terms of practice and compared with rationales, there are some fairly striking differences (see Table 4). For example,
Qualitative Research 6(1)108
TA B L E 4 . Uses of multi-strategy research – alternative scheme
Category Rationale Practice Number of articles (% of all 232 cases)
Triangulation 29 (12.5) 80 (34.5) Offset 7 (3) 4 (1.7) Completeness 31 (13) 67 (28.9) Process 5 (2.2) 6 (2.6) Different research questions 13 (5.6) 10 (4.3) Explanation 13 (5.6) 32 (13.8) Unexpected results 0 2 (0.9) Instrument development 18 (7.8) 21 (9.1) Sampling 31 (13.4) 43 (18.5) Credibility 2 (0.9) 5 (2.2) Context 8 (3.4) 10 (4.3) Illustration 4 (1.7) 53 (22.8) Utility 2 (0.9) 2 (0.9) Confirm and discover 9 (3.9) 15 (6.5) Diversity of views 26 (11.2) 35 (15.1) Enhancement 73 (31.5) 121 (52.2) Other/unclear 8 (3.4) 14 (6.1) Not stated 62 (26.7) 1 (0.4)
‘triangulation’ and ‘illustration’ are considerably more likely to occur as practice than as a rationale. ‘Unexpected results’ unsurprisingly did not appear as a rationale but was relevant to a small number of cases (2) when practice was examined.
In order to explore this issue, a contingency table analysis was undertaken relating rationale and practice having created a multiple response variable for each of these. The resulting table is extremely ungainly, so only highlights will be mentioned. Of the 29 articles that cited triangulation as a rationale, 19 used it in this way. In other words, one-third of all articles intending to use multi- strategy research for triangulation did not actually use multi-strategy research in this way or at least did not report doing so. Just as interestingly, we can look at this the other way around. Thus, 80 articles employed a triangulation approach, but for just 19 of these articles was triangulation a rationale. In other words, three-quarters of all articles reporting the triangulation of research findings provided other rationales for the use of multi-strategy research. What this seems to suggest is that, although triangulation may not always be a rationale for combining quantitative and qualitative research, when faced with the two (or in a small number of cases more than two) sets of data, some researchers find it hard to resist making allusions to the symmetry or otherwise between their findings.
Taking another example, ‘completeness’ was a rationale for 31 articles and most (83.9%) used it in this way. However, when practice is examined, completeness could be ascribed to 67 articles. Thus, 71.1 percent of all articles using completeness did not specify it as a rationale at the outset. ‘Enhance- ment’ provides a further example. Seventy-three articles specified this as a rationale, although over one-quarter of articles citing it as a rationale did not employ it in this way. More striking is that 121 articles employed the approach meaning that over one-half (56.2%) of articles using enhancement in practice had not specified it as a rationale. ‘Diversity of views’ was a rationale for 26 articles though quite a large percentage (30.8%) did not use it this way. Thirty- five articles employed a diversity of views approach to integrating quantitative and qualitative research, but nearly half (48.6%) did not specify this as a rationale. Among the 62 articles that appear in Table 4 as ‘not stated’, the most common uses were enhancement (45 articles), triangulation (14), illustration (15) and completeness (10).
Bryman: Integrating quantitative and qualitative research 109
What these findings suggest is that there is quite often a mismatch between the rationale for the combined use of quantitative and qualitative research and how it is used in practice. Multi-strategy research is something of a moveable feast. For several of the rationales, there is no evidence from the articles that quantitative and qualitative research are combined in the way that the rationale would lead one to suspect. This is not always the case. In particular, when ‘instrument development’ and ‘sampling’ are the rationales, they are nearly always used in this way. Only one article was found claiming ‘instru- ment development’ as a rationale and only one article claiming ‘sampling’ as a rationale, but they did not report combining quantitative and qualitative research in these ways.
There are several possible ways of looking at these findings, but two will be the focus of attention for this discussion. First, it may be that they reflect a tendency for the rationales for using multi-strategy research not to be thought through sufficiently. This would explain why the rationales and uses of multi- strategy research are not always aligned. One of the striking findings of this research has been that in the case of only 10 articles was there a clear indica- tion that quantitative and qualitative research had each been designed to answer specific and different research questions. In these 10 cases, there was a clear indication that the quantitative and the qualitative research were each geared to answering distinct research questions. Of course, a much larger number of articles specified research questions, though an impression gleaned from the examination of the articles is that these were not specified as commonly as textbooks might lead us to expect. This was not a dimension of the coding for the content analysis but, in the course of reading the articles, the relative infrequency of specified research questions was striking. In the field of organization studies, it has been found that only around one-fifth of articles discuss the relationship between research topics and the overall design of investigations (Grunow, 1995), so the relative lack of attention to research questions may be a reflection of this tendency which may have some general- izability beyond the confines of that discipline.
A second way of looking at these findings is to suggest that multi-strategy research provides such a wealth of data that researchers discover uses of the ensuing findings that they had not anticipated. It has been pointed out by some writers, for example, that triangulation may be an unanticipated conse- quence of conducting multi-strategy research (Deacon et al., 1998; Smith, 1986). This may arise when multi-strategy research is conducted for a reason other than triangulation but, in the course of interpreting the data, an incon- sistency between the quantitative and qualitative findings is revealed. Simi- larly, a researcher may employ multi-strategy research with a purpose like ‘diversity of views’ in mind, but find that the qualitative evidence helps to
Qualitative Research 6(1)110
explain some of the relationships uncovered through an analysis of survey data.
It is likely that there are elements of both these explanations (and probably others) which contribute to the moveable feast phenomenon. The first points to the possibility that the grounds on which multi-strategy research is under- taken have not been sufficiently articulated in the methodological literature, resulting in a lack of certainty about its uses. The second explanation is somewhat less negative in its connotations in implying that multi-strategy research frequently brings more to researchers’ understanding than they anticipate at the outset. Indeed, it could be argued that, if the latter is the case, perhaps it ought to be acknowledged. There would then be little or no point in wasting time and energy on enunciating principles for the contexts in which combining quantitative and qualitative research is and is not appropriate! However, it also has to be acknowledged that, unless there is some rationale for the use of multi-strategy research, there is the possibility of data redundancy, whereby some data are generated which are highly unlikely to shed light on the topic of interest. This would entail not just a waste of research resources but also a waste of participants’ time.
These results imply that there is indeed a case for encouraging researchers to be explicit about the grounds on which multi-strategy research is conducted but to recognize that, at the same time, the outcomes may not be predictable. This is, of course, a feature of much research. Qualitative research is often depicted as a research strategy whose emphasis on a relatively open-ended approach to the research process frequently produces surprises, changes of direction and new insights. However, quantitative research is by no means a mechanical application of neutral tools that results in no new insights. In quantitative data analysis, the imaginative application of techniques can result in new understandings. If the two are conducted in tandem, the potential – and perhaps the likelihood – of unanticipated outcomes is multiplied. More generally, however, the research reported here strongly suggests that there is considerable value in examining both the rationales that are given for combin- ing quantitative and qualitative research and the ways in which they are combined in practice. Such a distinction implies that methodological writings concerned with the grounds for combining the two approaches need to recog- nize that there may be a disjuncture between the two when concrete examples of research are examined.
A C K N O W L E D G E M E N T S
I wish to thank the referees of this article for their constructive comments. I also wish to thank the Economic and Social Research Council for funding the research project
Bryman: Integrating quantitative and qualitative research 111
‘Integrating quantitative and qualitative research: prospects and limits’ (Award number H333250003) which made possible the research on which this article is based.
R E F E R E N C E S
Brannen, J. (1992) ‘Combining Qualitative and Quantitative Approaches: An Overview’, in J. Brannen (ed.) Mixing Methods: Qualitative and Quantitative Research, pp. 3–37. Aldershot: Avebury.
Bryman, A. (1988) Quantity and Quality in Social Research. London: Unwin Hyman. Bryman, A. (2004) Social Research Methods (2nd edition). Oxford: Oxford University
Press. Buchanan, D.R. (1992) ‘An Uneasy Alliance: Combining Qualitative and Quantitative
Research’, Health Education Quarterly 19(1): 117–35. Creswell, J.W. (2003) Research Design: Qualitative, Quantitative, and Mixed Methods
Approaches (2nd edition). Thousand Oaks, CA: Sage. Creswell, J.W., Plano Clark, V.L., Gutman, M.L. and Handson, W.E. (2003) ‘Advanced
Mixed Methods Research Designs’, in A. Tashakkori and C. Teddlie (eds) Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: Sage.
Deacon, D., Bryman, A. and Fenton, N. (1998) ‘Collision or Collusion? A Discussion of the Unplanned Triangulation of Quantitative and Qualitative Research Methods’, International Journal of Social Research Methodology 1: 47–63.
Erzberger, C. and Kelle, U. (2003) ‘Making Inferences in Mixed Methods: The Rules of Integration’, in A. Tashakkori and C. Teddlie (eds) Handbook of Mixed Methods in Social and Behavioral Research, pp. 457–90. Thousand Oaks, CA: Sage.
Gilbert, G.N. and Mulkay, M. (1984) Opening Pandora’s Box: A Sociological Analysis of Scientists’ Discourse. Cambridge: Cambridge University Press.
Greene, J.C., Caracelli, V.J. and Graham, W.F. (1989) ‘Toward a Conceptual Framework for Mixed-method Evaluation Designs’, Educational Evaluation and Policy Analysis 11(3): 255–74.
Grunow, D. (1995) ‘The Research Design in Organization Studies: Problems and Prospects’, Organization Science 6(1): 93–103.
Maxwell, J.A. (1990) ‘Response to “Campbell’s Retrospective and a Constructionist’s Perspective”’, Harvard Educational Review 60: 504–8.
Maxwell, J.A. and Loomis, D.M. (2003) ‘Mixed Methods Design: An Alternative Approach’, in A. Tashakkori and C. Teddlie (eds) Handbook of Mixed Methods in Social and Behavioral Research, pp. 209–40. Thousand Oaks, CA: Sage.
Morgan, D.L. (1998) ‘Practical Strategies for Combining Qualitative and Quantitative Methods: Applications for Health Research’, Qualitative Health Research 8: 362–76.
Morse, J.M. (1991) ‘Approaches to Qualitative-Quantitative Methodological Triangula- tion’, Nursing Research 40(2): 120–3.
Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational Research. Tallinn, Estonia: Tallinn Pedagogical University Dissertation on Social Sciences.
Pawson, R. and Tilly, N. (1997) Realistic Evaluation. London: Sage. Smith, M.L. (1986) ‘The Whole is Greater: Combining Qualitative and Quantitative
Approaches in Evaluation Studies’, in D.D. Williams (ed.) Naturalistic Evaluation, pp. 37–54. San Francisco, CA: Jossey-Bass.
Tashakkori, A. and Teddlie, C. (1998) Mixed Methodology: Combining Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage.
Qualitative Research 6(1)112
Tashakkori, A. and Teddlie, C. (2003) Handbook of Mixed Methods in Social and Behavioral Research. Thousand Oaks, CA: Sage.
A L A N B RY M A N is Professor of Organisational and Social Research, Management Centre, University of Leicester. Address: Management Centre, Ken Edwards Building, University of Leicester, University Road, Leicester LE1 7RH, UK. [email: [email protected]]
Bryman: Integrating quantitative and qualitative research 113