Discussion Posts - 200 words each question - APA Format - Requires Turnitin Part 2
NeedHomeworkHelpHRE7271/Leading Learning in Human Resource Development Discussion Posts.docx
The instructor is more interested in your own arguments and your ability to defend and support them logically and analytically using the course, or other, readings. Each initial posting should be at least 200 words, with at least 2 references and written in APA format. Grading criteria on page 3
Week 2 Discussion:
Based on the readings, answer the following questions. Original post due Thursday, Midnight and two responses by Sunday, Midnight.
· How could the sample assessment and evaluation questions found in table 2 in the article 'Integrating Adult Learning Principles' be incorporated into an organizations training offerings?
· Based on the adult learning principles, how can those inform your development and delivery of training in an organizational context?
· How can you ensure that learning not only takes place but is transferred back to the employee's job?
· What does distant learning or self-paced learning using technology need in order for employee's to learning and apply their new knowledge?
· How do you ensure that employee's understand and are aware of the relevance of the training (learning) they're involved in? What measures can be put in place to ensure that the purpose of learning is known?
Week 3 Discussion:
Based on this week's reading, answer the following questions. Your post is due Thursday, Midnight and two responses Sunday, Midnight.
· How can the Universal Principles guide us in developing training that leads to the transfer of learning?
· How does the 5 Step Training Model help to develop a training session that supports organizational objectives?
· How does Metacognition factor in to effective training and the transfer of learning?
· What strategies can you take from the Biology focused metacognitive article and implement in organizational training to improve employee metacognitive skills?
Week 4 Discussion:
Based on the summary slides and readings, answer the following questions. Original post is due Thursday, Midnight and two responses by Sunday, Midnight.
· Which of the four types of learning would you most likely use most often and why?
· State your argument for or against using lectures in a learning situation?
· What is one way that you can ensure self-directed learning is being effective?
· How will technology play an important part in effective training?
Answer the following questions based on the readings from this week. Your first post is due Thursday, midnight and 2 response Sunday, midnight.
· Why are reaction measures (Ruona, Leimbach, Holton & Bates, 2002) no sufficient for determining learning impact?
· Why is transfer or learning important to organizations?
· How would you determine if the transfer climate of an organization (Hatala, 2007) will facilitate the transfer of learning?
· What do you consider to be the most important factors for learning transfer (Bates, Holton & Hatala, 2012)?
Week 6 Discussion:
Using the chapter Learning through Immersive Virtual Environments- An Organizational Context, answer the following questions. Original posts are due, Thursday, midnight and two responses, Sunday, midnight.
1. What impact can IVE have on organizations?
2. What are some subject areas that may not be conducive to IVE? Why?
3. How would IVE facilitate the transfer of learning back to the job?
Week 7 Discussion:
Watch both video's under Media and answer the following questions.
Poor Coaching / Performance Conversation
· For the poor coaching video, what issues did you identify that were more demotivating than motivating?
· For the great coaching conversation video, what was revealed through the conversation?
HRE7271/Week 2/How Adults Learn from Self‐Paced, Technology‐Based Corporate Training- New focus for learners, new focus for designers.pdf
Distance Education, Vol. 27, No. 2, August 2006, pp. 155–170
ISSN 0158-7919 (print); 1475-0198 (online)/06/020155–16 © 2006 Open and Distance Learning Association of Australia, Inc. DOI 10.1080/01587910600789506
How Adults Learn from Self-Paced, Technology-Based Corporate Training: New focus for learners, new focus for designers
Jackie Dobrovolny* University of Colorado at Denver and Health Sciences Center (UCDHSC), USA Taylor and Francis LtdCDIE_A_178898.sgm10.1080/01587910600789506Distance Education0158-7919 (print)/1475-0198 (online)Original Article2006Open and Distance Learning Association of Australia, Inc.272000000August [email protected]
How do adults learn from self-paced, technology-based corporate training, which they select based on its relevance to their current employment responsibilities? Specifically, how do adults use the following learning strategies: prior experience, reflection, metacognition, conversations, generative learning strategies, and authentic experiences? Based on a recent dissertation research investiga- tion, the author found that learning starts with, and is sustained by, metacognition which was defined as self-assessment and self-correction. While learners using metacognition is by no means a new phenomenon, learners using metacognition significantly more often than other learning strategies has important implications for the design of new generations of online distance instruc- tion. Similarly, that learners frequently use conversations to learn from self-paced, technology- based training strongly suggests that dialogs and discussions are important in the design of these new ways of learning.
Introduction
Research on the effectiveness of self-paced, technology-based instruction dates back more than three decades and indicates this type of instruction improves performance and that students master the learning objectives in significantly less time than students in group-paced instruction (Dalton, Hannafin, & Hooper, 1989; Fletcher, 1996). Unfortunately, adults above college age were rarely the focus of these studies (Bates, Seyler, & Holton, 1995; Merriam & Caffarella, 1999), making the implica- tions for corporate settings uncertain. This article (a) describes a dissertation research investigation of how adults, in a corporate setting, learn from self-paced,
* Corresponding author. University of Colorado at Denver and Health Sciences Center, 12281 E. Villanova Drive, Aurora, CO 80014, USA. Email: [email protected]
156 J. Dobrovolny
technology-based training, and (b) elaborates on the findings to enhance our under- standing of the online distance education context.
Literature Review
A constructivist theory of learning was the primary conceptual framework for this study and “learning” was defined as “the process whereby learners personalize and/ or customize new information; it is the process whereby learners make new informa- tion relevant and/or meaningful to themselves” (L. K. Anderson & Thomas, 1992; Candy, 1991; Fosnot, 1996; Mezirow, 1997; Roth, 1997). Constructivism and defining learning as a process of personalization provides an important theory for designing the new wave of online, distance instruction. It challenges designers to ask themselves, “What can I do to help my learners personalize this content?”
Adult learning theory was the secondary conceptual framework for this study. Both constructivism and adult learning theory emphasize the social construction of mean- ing and the importance of reflection, prior experiences, and authentic experiences in the learning process (Merriam & Caffarella, 1999; Sutherland, 1997). Both theories also emphasize the importance of self-monitoring (Garrison, 1997) and the use of generative learning strategies (Grabowski, 1996). Thus, this literature review summa- rizes research in the following areas: sociocultural learning, reflection, metacognition, prior experience, authentic experiences, and generative learning strategies.
Sociocultural learning: As adults, we learn not only by ourselves, as individuals, but also with others. We learn through discussions with colleagues, friends, and family (Cobb & Bowers, 1999; Salomon & Perkins, 1996). We construct knowledge by conversing with others, analyzing problems together, identifying solutions together, and meeting goals together. Sociocultural constructivists argue that learning is a process of acculturation into an established community of practice (Duffy & Cunning- ham, 1996) and other research indicates corporate employees use collaboration, narration, and improvisation to transform information into corporate knowledge (Brown & Duguid, 1991, 2000). For designers of the new types of online, distance instruction, one challenge is how to incorporate multi-user environments in such a way that they support and enhance sociocultural learning.
Reflection: Reflection is careful, deliberate thinking that helps us make sense of experiences and supports our knowledge construction process (Jonassen & Reeves, 1996). It is the interpretative process of abstracting meaning in an effort to under- stand reality (Boud, Keogh, & Walker, 1985; Jonassen & Reeves, 1996). It is thinking about the implications and consequences of applying the instruction (Von Wright, 1992). In the next wave of online, distance instruction, where learners are geograph- ically dispersed and demographically diverse, what is the role of reflection and to what extent is it the responsibility of our learners?
Metacognition: Metacognition is the process of self-monitoring, that is, self-assess- ment and self-correction (Grabinger, 1996; Schraw, 1998). It is the process of regu- lating and modifying our cognitive activity (Von Wright, 1992), planning and selecting strategies, monitoring the progress of learning, correcting errors, and changing
How Adults Learn from Self-Paced, Technology-Based Corporate Training 157
strategies when necessary (Ridley, Schutz, Glanz, & Weinstein, 1992; Schraw, 1998). Metacognition includes knowledge of our strengths and weaknesses as a learner, knowl- edge about learning strategies, and when and where to use those strategies (Schraw, 1998). How do adults learn metacognition and what technologies and instructional strategies can designers of online, distance instruction use to facilitate metacognition?
Prior experiences: Prior experiences are the foundation on which adult learners construct new information (Bereiter & Scardamalia, 1989; Merriam & Caffarella, 1999; Mezirow, 1995; Pillay, 1998). Drawing on the “rich emotional associations” of the prior experiences of adult learners is a powerful strategy (Sutherland, 1997) perhaps because none of us can think about anything without thinking about prior experiences and information about those experiences (Dewey, 1938). Prior experi- ence also provides the baseline against which learners compare and contrast new information. Given the diversity of our learners and thus the diversity of their prior experiences, what are the implications for designing examples and analogies for online, distance instruction?
Authentic experiences: Authentic experiences are opportunities for learners to practice new skills and knowledge. It is not enough, however, for learners to merely have oppor- tunities to practice because learning does not occur in a vaccum; it is always situated in a context. Thus, learning is ‘in part, a product of the activity, context, and culture in which it is developed and used’ (Brown, Collins and Duguid, 1989, p. 32). Numer- ous constructivist models identify authentic experiences as important in the learning process including cognitive apprenticeship (Clark, 1996; Collins, Brown, & Holum, 1991), problem-based learning (Savery & Duffy, 1996), anchored instruction (CTGV & Vanderbilt, 1990, 1993), and situated cognition (Brown, Collins, & Duguid, 1989). One of the challenges for designers of online, distance instruction is how to use these models to incorporate authentic experiences in online, distance instruction.
Generative learning strategies: Both constructivism and generative learning theory emphasize the active role learners (Duffy & Cunningham, 1996; Fosnot, 1996; Grabowski, 1996; Winn & Snyder, 1996; Wittrock, 1992) and the importance of prior experiences (Grabowski, 1996; Jonassen & Reeves, 1996; Wittrock, 1992). Both posit that making connections across experiences is an important knowledge construction activity (Fosnot, 1996; Grabowski, 1996; Wittrock, 1992) and metacog- nition is an important part of the learning process (Duffy & Cunningham, 1996; Grabinger, 1996; Grabowski, 1996). Generative learning strategies are activities that involve the actual creation of meaning and/or relationships (Grabowski, 1996). They can be part of the instruction, for example, questions or examples, or they can be strategies learners initiate, for example, note-taking or creating analogies. What is the role of generative learning strategies in the new types of online, distance instruction and how can the new technologies help learners use generative learning strategies?
Research Questions
The major research question was, “In what ways do adults, in a corporate setting, construct their own knowledge during and after using a self-paced, technology-based
158 J. Dobrovolny
course that is relevant to their current employment responsibilities?” More specifi- cally, “How do adult learners use the following knowledge construction techniques?”
(1) conversations (2) reflection (3) metacognition (4) prior experience (5) authentic experiences (6) generative learning strategies.
Methodology and Participants
Fletcher (1996) suggested we need to investigate not only if instructional technology works, but how it works. In reviewing the self-paced, technology-based, learner control research, Reeves (1993) recommended, “the qualitative, interpretivist para- digm should precede the quantitative if we are to identify meaningful hypotheses to investigate empirically” (p. 44). Using these recommendations, this research was an exploratory, qualitative study designed to investigate how adults, in a corporate context, learn from self-paced, technology-based instruction.
Because this research focused on how adults learn, the methodology was phenom- enological. The purpose of phenomenology is to describe experiences (Anderson, 1991; Bergum, 1991; Cohen & Omery, 1994; Morse, 1994) and interview questions “center around meaning (What is the meaning of an experience?) and analogy (What is it like to experience…?)” (Ray, 1994, p. 128). Phenomenological research methods include audio-taped conversations and written descriptions of personal experiences (Morse, 1994). Researchers studying metacognition and reflection typically use “think aloud” protocols, self-reports (Ridley et al., 1992), and journals (Dennison, 1999; Kottkamp, 1990; Roth, 1997). As the new generation of online, distance instruction is implemented, we should investigate what it is like to experience this new way of learning and for these studies our research methodology will be phenomenology.
For this study, interviews were audio-taped and included a think aloud protocol and the development of a Post-It™ note diagram. Participants also kept a critical incident journal. Critical incident journals focus on specific important experiences (Kottkamp, 1990) and for this study were used to document each time participants thought about, talked about, or used the information they learned from the course they selected, after they completed it. The journal and the Post-It™ note diagram were strategies to document, both verbally and visually, respectively, how the partic- ipants learned.
Controlling who is selected to be interviewed, that is, “primary selection,” is an important characteristic of qualitative research (Morse, 1991). Participants for this study were employed by corporations who offered self-paced, technology-based training to their employees and the purpose of at least some of those courses was to improve employee performance. Participants in this study selected a specific perfor- mance improvement course, offered by their corporation, because it was relevant to
How Adults Learn from Self-Paced, Technology-Based Corporate Training 159
their career or current organizational responsibilities. In other words, participants had to meet specific criteria to participate in this study.
Sample sizes are typically small in qualitative, phenomenological studies. “Primary selection is clearly efficient, and because of this efficiency, the sample size is as small as possible” (Morse, 1991, p. 136). Generally, phenomenological research requires at least six participants (Morse, 1994), depending on the quality of the data. The more useable data a researcher obtains from each participant, the fewer the number of participants needed (Morse, 2000). There were seven partici- pants in this study and they produced 2,818 coded passages. Two study participants were managers and one had been a manager but was not currently in a managerial role. Three participants had technical jobs, working with complex software. Two participants worked in marketing and customer support. One participant managed the production of training materials and conducted usability analyses, and another participant managed the implementation of a new product. The education of the participants ranged from high school diploma to master of business administration (MBA) and their ages ranged from 32 to 59. All participants were Caucasian, American citizens. There were two male and five female participants.
Data Generation Process
The six steps in the data generation process were:
(1) The author conducted a phone or email discussion with each participant to explain the research methodology, clarify expectations, answer questions, and collect demographic information.
(2) Each participant took the self-paced, technology-based course he or she selected. Participants selected their course based on its relevance to their career.
(3) The author met with each participant in a conference room at the participant’s office. The focus of this discussion was the participant’s knowledge construction process during and immediately after completing the course. Participants described their knowledge construction process using both verbal and visual descriptions. During the verbal part of the discussion, the author asked a series of questions focusing on (a) how the participant personalized the new informa- tion and made it meaningful, and (b) the strategies the participant used to determine his or her understanding of the instruction.
During the visual half of the discussion, participants created a diagram, using flip chart paper and yellow and blue Post-It™ notes, to show the relationship between topics in the course and how they personalized each of those topics. Yellow Post-It™ notes were labeled “instruction”; blue Post-It™ notes were labeled “personalization.” Mean time between course completion and this discus- sion was 54 hours. Mean time of this second discussion was about 50 minutes.
(4) Participants recorded their knowledge construction process in their journals. Specifically, they documented each time they thought about, talked about, or used the information from the course.
160 J. Dobrovolny
(5) The author met again with each participant in a conference room at the partici- pant’s office. The participant’s journal was the focus of this discussion, which investigated the participant’s knowledge construction process since the last discussion. Like the previous discussion, there was both a verbal and a visual part of the discussion. During the verbal part, participants elaborated on each journal entry. During the visual part, participants updated their Post-It™ note diagram to reflect each journal entry. Mean number of days between the first discussion at the participant’s office and the second discussion was about 21 days. Mean time of this discussion was about 31 minutes.
(6) Four of the seven participants continued recording their knowledge construction process for 2 weeks after the last face-to-face discussion. Rather than document- ing those experiences in their journals, participants emailed those descriptions to the author.
An important characteristic of this data generation process is it assumed learning occurred and the focus was on how that learning occurred. Perhaps that is the same assumption we should make about the new wave of online, distance instruction. Learners will learn; they will construct knowledge but to what extent and in what ways will they use the six learning strategies described above?
Data Analysis Procedures
Three volunteers participated in a pilot study. The goals of the pilot were to (a) assess the strengths and weaknesses of the primary selection strategy; (b) assess the flow and duration of the two face-to-face discussions; (c) evaluate the effectiveness of the technical interview tools; (d) determine the number of face-to-face discussions needed for each participant; and (e) evaluate the NVivo™ software.
The pilot study indicated the primary selection sampling strategy worked well but the criteria by which participants selected their course needed to be more rigorous. The face-to-face discussions went well but participants needed access to the instruc- tion to create their Post-It™ note diagrams.
The technical interview tools included the Post-It™ notes, the flip chart paper, and the audio-tape recorder. The Post-It™ notes worked well as nodes on the diagram, but directions on how to use them needed improvement and participants had difficulty differentiating between the two different types of Post-It™ notes, that is, those for “instruction” and those for “personalization strategies.” For the full study, the author used both labels and colors to distinguish between the two types of Post-It™ notes. The pilot study indicated two face-to-face interviews were sufficient for collecting the necessary data and NVivo™ was an efficient and effective tool for analyzing the data.
Before starting the full study, the author created a code in NVivo™ for each knowledge construction technique: prior experience, metacognition, reflection, conversations, generative learning strategies, and authentic experiences. As the author coded the data, she created sub-codes to categorize how the participants used each technique. These sub-codes were operational definitions of the major code. For
How Adults Learn from Self-Paced, Technology-Based Corporate Training 161
example, metacognition is the process of self-assessment and self-correction and one of the 12 metacognition sub-codes was “answer questions in the text.”
Approximately one-fourth of the way through the coding process, the author esti- mated the interrater reliability of the data coding. Three raters received a set of 24 passages, from the transcripts and the participants’ journals, and a description of the codes and sub-codes the author used. The raters also received a set of directions, which contained example passages and an explanation of how the author coded those examples. These three raters had between 20 and 28 years experience in instructional design. All three raters coded the 24 passages twice. The Interrater Reliability Quotient (IRQ) on the first coding ranged from 66.6% agreement, between the raters and the author, to 87.5% agreement. For the second coding, the IRQ ranged from 84% to 95%.
During the coding process, the author identified two themes, “Relevance” and “Other Features of the Course,” that did not fit in any of the major codes. For each of these two new themes, the author created a code and a set of “how” sub-codes. Also during the coding process, the author identified themes across the major codes. For example, the concept of “review” was a sub-code in metacognition, authentic experi- ences, and relevance. The author created “sets” to analyze themes that cut across the major codes. The “sets” were “review,” “terminology,” “job aids,” and “big picture.”
Results
The author coded 2,818 passages using 133 sub-codes (see Table 1 for the number of passages coded in each major category). The study results are summarized in Figure 1, the Dobrovolny model, which indicates the knowledge construction process began during the time participants took the courses they each selected and it continued after participants finished those courses. Once participants finished their selected courses (see diamond in the model labeled “Finished with course?”), the course became another “prior experience,” and their knowledge construction process continued. Figure 1. Dobrovolny model of how adults learn from self-paced, technology-based corporate training
The implication for the new wave of online, distance instruction (hereafter abbreviated to merely “Implication”): Learning is the process of personalizing new information and that process continues after learners “complete” their instruction. How can we help learners personalize new information before, during, and after the instructional experience?
The model in Figure 1 also indicates how corporate employees used metacogni- tion, reflection, prior experiences, authentic experiences, and conversations to construct knowledge. Previous research indicated these were important strategies. This research not only verified that, but described how adults used those knowledge construction strategies. This research also indicated that metacognition was the most important strategy. That is, knowledge construction not only started with metacognition but it was sustained by metacognition.
Metacognition and prior experience: Participants assessed their understanding during and after completing their respective courses and they used their own internal self- assessments and self-check questions, simulations, and practice exercises in the courses
162 J. Dobrovolny
they took, to assess their knowledge. They also compared their prior experiences with examples in their courses, looking for similarities and differences, as a way to check their understanding. If their self-assessment was positive, that is, “I understand,” they continued to read and reflect on the usefulness, relevancy, and big picture of the course content (see the box to the right of the Metacognition diamond in Figure 1).
If the participants’ self-assessment was negative, that is, “I’m confused,” they reviewed parts of the course and tried to resolve their confusion by: (a) comparing the instruction to their prior experiences, (b) reflecting on the usefulness of the instruction, and (c) looking for familiar terminology (see large diamond in the middle of Figure 1). If participants resolved their confusion, that is, “I understand,” they continued reading and reflecting on the content. If they were still confused, they asked themselves how important it was to continue to struggle (see diamond labeled “Work on it some more?” in Figure 1).
Implication: What kinds of strategies will help learners who are asking themselves, “Should I work on this any more?”
Reviewing the course content was a frequent knowledge construction technique for the participants, originating from their need to self-assess and self-correct. Specifically,
Table 1. Number of sub-codes and passages per major code or set
Major code or “set” No. of sub-codes No. and % of passages
Metacognition 12 602 22.9% Reflection 10 476 18.2% Prior experience 9 273 10.4% Authentic experiences 6 265 10.1% Other features of the course (new) 9 117 4.47% Review or refresh (“set”) 9 112 4.28% Relevance of the course (new) 5 108 4.13% Terminology concerns (“set”) 7 107 4.09% Conversations 6 104 3.97% Questions in the coursea 10 87 3.32% Job aids (“set”) 4 73 2.79% Big picture (“set”) 3 59 2.25% Examples and analogiesa 7 42 1.6% Simulations or animations in the coursea 4 30 1.15% Interactive techniques in the coursea 4 30 1.15% Paper copy of the coursea 4 29 1.1% Note-takinga 10 27 1.03% Graphics in the coursea 4 24 0.92% When to take a break (“set”) 3 23 0.88% Sections and tables of contents in instructiona 3 19 0.73% Highlight paper copya 4 11 0.42% Totals 133 2,818 100%
aThis is a type of generative learning strategy.
How Adults Learn from Self-Paced, Technology-Based Corporate Training 163
questions or self-checks, simulations, paper copies of the course, taking notes, authen- tic experiences, and prior experiences all related to reviewing the content.
Implication: Learners want and need to review the instruction, including tests and self-checks. How can we make that an efficient and effective process for them?
Start
Metacognition (Self- Assessment)
I Understand
I’m Confused
Link to Prior Experiences
Reflection: Meet My Needs?
Look for Familiar Terminology
I U
n d e rs
ta n d
1. Keep reading 2. Reflect on how to use course content 3. Reflect on relevance and big picture
No, I’m Still Confused
Work on it Some More?
No
Yes
Skim and Keep Reading
Need a Job Aid?
Yes
No
Create Job Aid
Finished with Course?
No
Yes
Back to Metacognition
“Future”
Practice: Authentic Experiences
How Relevant is This?
Conversations: Seek Help
Review and Re-Read
Authentic Experiences
Metacognition (Back to Start)
Conversations
Reflection
Prior Experiences
Figure 1. Dobrovolny model of how adults learn from self-paced, technology-based corporate training
164 J. Dobrovolny
Reflection: Participants were keenly aware of how their course applied to their past, present, and future. They frequently thought about the past and their prior experiences, the present and how the new information applied to their current job responsibilities, and the future, represented by the “Future” cloud in the figure, and how the instruction might apply to new responsibilities. Participants looked for similarities and differences between their past experiences and the examples, exer- cises, and simulations in their courses.
Implication: The diversity of our learners makes the design of useful examples and analogies extremely challenging. Can our new technologies help us resolve this challenge?
Authentic experiences: Participants used what they learned in their courses to improve their efficiency and/or help colleagues or subordinates. They also explained the course content to colleagues. Notice that these authentic experiences were outside the “formal” instruction.
Implication: Should designers incorporate more authentic experiences in the new generation of online, distance instruction or assume learners will provide their own authentic experiences?
Conversations: All participants discussed their course content. Specifically, partic- ipants used conversations in five ways to construct knowledge. They (a) asked or answered questions about their course; (b) discussed problems that could be solved by applying the course content; (c) discussed general concepts and issues covered in or related to the courses they took; (d) taught others something they learned in the course; and (e) discussed confusing or misleading terminology related to the course. Again, all of these conversations occurred outside the “formal” training.
Implication: Will (and should) the new multi-user environments enable designers to incorporate conversations into the “formal” training?
Generative learning strategies: Participants used the following generative learning strat- egies: (a) self-check questions, as part of their metacognition and for motivation; (b) examples, to learn procedures, as part of their metacognition, and to compare examples with their prior experiences; (c) simulations and interactivity, to practice procedures; (d) paper copies of the course, as job aids, to discuss the content with mentors, and to study the course offline; and (e) their notes, to clarify areas of confusion, and record important procedures or concepts. Typically, participants wanted more questions, examples, interactive simulations, and diagrams in their courses. That is, the most popular generative learning strategies were those that helped learners self-assess and self-correct.
Implication: To what extent can the new technologies help us integrate generative learning strategies into the new generation of online, distance instruction?
Relevance and job aids: Since most participants thought their courses were more relevant to future positions, not their current position, they frequently discussed relevance and job aids. If they thought a section or passage was relevant, or likely to be relevant in the future, they typically spent more time on it, either reviewing it and/ or taking notes. Participants created job aids, to help them recall the course content,
How Adults Learn from Self-Paced, Technology-Based Corporate Training 165
terminology, and acronyms, by printing the course and/or by taking notes (see “Need a job aid?” diamond and “Create job aid” box in Figure 1).
Implication: To what degree are learners in online, distance instruction responsible for creating their own job aids and identifying the relevance of the training for their own needs? How can designers assist learners with relevance and job aids?
Terminology: Learning the definitions for new terms and acronyms was an impor- tant goal for most participants. They discussed terminology in terms of their (a) metacognition, (b) linking the course to their prior experiences, (c) conversations with colleagues, (d) note-taking, (e) daily challenges at work, and (f) reasons for making a paper copy of their course.
Implication: What is the unique terminology in each of our courses and how are we going to help learners personalize it?
Big picture: Participants wanted the “big picture” of both the content and the design of the courses they took. To understand the big picture of the content, partic- ipants used diagrams, conversations, and historical information. To understand the big picture of the course design, participants used or wanted tables of contents, section headings, and “trail maps.”
Implication: What are effective strategies for providing two different types of “big picture” information?
Discussion and Implications
The theme of this special issue is, “Online Distance Education: New Ways of Learn- ing? New Models of Teaching?” Using this research to answer those questions, consider the differences between: (a) self-paced and group-paced instruction and (b) adult learning and the learning of K-12 populations. Distance education can be either group-paced or self-paced. The results of this research apply only to self-paced instruction. Furthermore, the “New models of teaching” question is a question for instructional designers of self-paced instruction, rather than teachers or instructors, who design and deliver group-paced instruction.
Distance education can also be designed for nearly any age group, from elemen- tary school children to adults in a corporate setting. This research was conducted with adults, who selected technology-based training they thought was relevant to their current or future professional responsibilities. Accordingly, this research is only applicable to distance instruction designed to improve the performance of employees.
Given these caveats, this research provides the following insights into new ways of learning and new ways of “teaching.” First, learning, for adults taking self-paced, technology-based training, starts with, and is sustained by, metacognition (see Figure 1). Thus, instructional designers need to create frequent opportunities for adults to self-assess and self-correct, that is, include questions or self-checks, prac- tice exercises, and/or simulations (“interactivity”) in all instruction.
Interactivity is frequently recommended (Jones, 1999; Milheim, 1995; Park & Hannafin, 1993; Sims, 1999, 2000; Wagner, 1994; Yacci, 2000) yet often ignored.
166 J. Dobrovolny
Perhaps two reasons it is ignored are because (a) we have not clarified the relation- ship between interactivity and metacognition and (b) developing effective interactiv- ity is difficult and thus time-consuming.
Part of the difficulty in writing effective interactive exercises is writing useful feed- back. Metacognition is not only self-assessment, but also self-correction. To be effective, each question, practice exercise, or simulation must provide “correct” or “incorrect” feedback and information about the correct answer for those who answered the question incorrectly (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Dempsey, Driscoll, & Swindell, 1993; Schwier & Misanchuk, 1993). Additionally, adults often know or can visualize more than one way to solve the same problem and they are impatient and suspicious when instruction does not acknowledge these alternative solutions.
Another difficult aspect of developing effective feedback is including comparisons between correct answers, and/or incorrect answers, and the learners’ prior experi- ences. This type of comparison is also useful with examples and analogies but to work effectively, designers must collect information about their learners through some type of audience analysis (Fenrich, 1997; Morrison, Ross, & Kemp, 2001; Steinberg, 1991). This too is time-consuming and thus, often ignored.
One interesting strategy to help adults understand the relevance of a course is to provide historical information about the content along with timelines linked to other historical events relevant to learners. While focusing on the relevance of a course is a frequently recommended instructional strategy (Alessi & Trollip, 2001; Grabinger, 1996; Keller, 1999; Knowles, Holton, & Swanson, 1998; Mayer, 1993; Wlod- kowski, 1999), describing the history or evolution of a topic is generally not viewed as effective. Nevertheless, one of the participants in this research linked the history of a specific telecommunications technology to his prior experiences in order to create a “big picture” of the information he was learning. Perhaps the fact he did this on his own is what made the activity memorable but perhaps instructional designers should consider linking historical information about course content with other historical events that are relevant to their learners.
Participants in this study frequently returned to the courses, after they completed them, to refresh their memories or answer questions that arose as part of their work responsibilities, that is, their learning continued after they completed the training. Instructional designers should, therefore, consider how to meet this post-course requirement. Enabling learners to easily print individual sections or an entire course, allows adults to create personalized job-aids. Other strategies to help adults continue learning after they finish their training include providing (a) the names and contact information of mentors or experts, (b) frequently asked questions (FAQs) documents, (c) access to the course after officially “completing” it, and (d) comprehensive search capabilities within the training.
Finally, designers of self-paced instruction for adult learners must remember their learners ultimately control the instruction. Adult learners customize instruction to meet their needs, based on their prior experiences, their current responsibilities, and their expectations of future responsibilities. They may skip sections of the course they
How Adults Learn from Self-Paced, Technology-Based Corporate Training 167
find irrelevant. They may skim the instructional information and focus on the self- assessment exercises. They may read only parts of the instruction applicable to questions they answered incorrectly. Instructional designers should, therefore, design online distance instruction that is easy for learners to manipulate and personalize.
Future research, focused on the new types of online, distance instruction, should validate the Dobrovolny model or generate a new model. That is, to what extent will this model change due to new technologies and new instructional strategies? To what extent will this model change due to a more diverse learner population?
Summary
When discussing distance education and new modes of teaching and learning, it is important to be clear about the specifics. Is it group-paced instruction, facilitated by an instructor, or is it self-paced? Who are the learners and who decided they needed this instruction? What is the learning context? Is it corporate training or K-12 education?
This research focused on self-paced, technology-based training for employees of large corporations. Results indicated learning starts with, and is sustained by, metacognition (self-assessment and self-correction) for learners in this instructional context. Practice exercises and self-checks are more than interactive experiences or opportunities for designers to assess learners. They are first, and foremost, self- assessment opportunities for learners. Thus, designers of distance education for adults taking self-paced, technology-based corporate training must evaluate all their courses in terms of how well they facilitate and support metacognition.
Notes on Contributor
Jackie Dobrovolny teaches part time and conducts research, at the University of Colorado at Denver and Health Sciences Center (UCDHSC), USA.
References
Alessi, S. M., & Trollip, S. R. (2001). Multimedia for learning: Methods and development (3rd ed.). Boston: Allyn and Bacon.
Anderson, J. M. (1991). The phenomenological perspective. In J. M. Morse (Ed.), Qualitative nursing research (pp. 25–38). Newbury Park, CA: Sage.
Anderson, L. K., & Thomas, R. G. (1992). Innovation in parent education: Self-reflection and dialogue as avenues of learning. Paper presented at the American Educational Research Association, San Francisco, CA.
Bangert-Drowns, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213–238.
Bates, R. A., Seyler, D. L., & Holton, E. F. I. (1995, March 2–5). Principles of CBI design and the adult learner: The need for further research. Paper presented at the Academy of Human Resource Development, St Louis, MO.
Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction. In L. B. Resnick (Ed.), Knowing, learning, and instruction: Essays in honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates.
168 J. Dobrovolny
Bergum, V. (1991). Being a phenomenological researcher. In J. M. Morse (Ed.), Qualitative nursing research (pp. 55–71). Newbury Park, CA: Sage.
Boud, D., Keogh, R., & Walker, D. (1985). Promoting reflection in learning: A model. In D. Boud, R. Keogh, & D. Walker (Eds.), Reflection: Turning experience into learning. New York: Kogan Page, Long/Nichols.
Brown, J. S., Collins, A., & Duguid, P. (1989, January–February). Situated cognition and the culture of learning. Educational Researcher, 32–42.
Brown, J. S., & Duguid, P. (1991). Organizational learning and communities-of-practice: Toward a unified view of working, learning, and innovation. Organizational Science, 2(1), 40–57.
Brown, J. S., & Duguid, P. (2000). The social life of information. Boston, MA: Harvard Business School Press.
Candy, P. C. (1991). Self-direction for lifelong learning. San Francisco: Jossey-Bass. Clark, R. (1996). Designing cognitive apprenticeship training programs to accelerate expertise. Paper
presented at the Interactive ’96 conference, Atlanta, GA. Cobb, P., & Bowers, J. (1999). Cognitive and situated learning perspectives in theory and practice.
Educational Researcher, 28(2), 4–15. Cohen, M. Z., & Omery, A. (1994). Schools of phenomenology: Implications for research. In J.
M. Morse (Ed.), Critical issues in qualitative research methods (pp. 136–156). Thousand Oaks, CA: Sage.
Collins, A., Brown, J. S., & Holum, A. (1991, Winter). Cognitive apprenticeship: Making thinking visible. American Educator, 19–46.
CTGV, & Vanderbilt, C. a. T. G. a. (1990). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19(6), 2–10.
CTGV, & Vanderbilt, C. a. T. G. a. (1993). Anchored instruction and situated cognition revisited. Educational Technology, 33(3), 52–70.
Dalton, D. W., Hannafin, M. J., & Hooper, S. (1989). Effects of individual and cooperative computer-assisted instruction on student performance and attitudes. Educational Technology Research and Development, 37(2), 15–24.
Dempsey, J. V., Driscoll, M. P., & Swindell, L. K. (1993). Text-based feedback. In J. V. Dempsey & G. C. Sales (Eds.), Interactive instruction and feedback (pp. 21–54). Englewood Cliffs, NJ: Educational Technology Publications.
Dennison, R. S. (1999). Interest, fear, fate, luck, motivation, epistemology, and monitoring: Measuring and promoting metacognitive monitoring. Paper presented at the AERA Conference, Montreal, Canada.
Dewey, J. (1938). Experience and education. New York: Simon & Schuster. Duffy, T. M., & Cunningham, D. J. (1996). Constructivism: Implications for the design and delivery
of instruction. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.
Fenrich, P. (1997). Practical guidelines for creating instructional multimedia applications. Fort Worth, TX: Harcourt Brace.
Fletcher, J. D. (1996). Does this stuff work? Some findings from applications of technology to education and training. Paper presented at the Teacher Education and the Use of Technology-Based Learning Systems Conference, Warrenton, VA.
Fosnot, C. T. (1996). Constructivism: A psychological theory of learning. In C. T. Fosnot (Ed.), Constructivism: Theory, perspectives, and practice. New York: Teachers College Press.
Garrison, D. R. (1997). Self-directed learning: Toward a comprehensive model. Adult Education Quarterly, 48(1), 18–33.
Grabinger, R. S. (1996). Rich environments for active learning. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.
Grabowski, B. L. (1996). Generative learning: Past, present, and future. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.
How Adults Learn from Self-Paced, Technology-Based Corporate Training 169
Jonassen, D. H., & Reeves, T. C. (1996). Learning with technology: Using computers as cognitive tools. In D. H. Jonassen (Ed.), Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.
Jones, M. (1999). What can we learn from computer games: Strategies for learner involvement. Paper presented at the AECT Conference, Houston, TX.
Keller, J. M. (1999). Motivational systems. In H. D. Stolovitch & E. J. Keeps (Eds.), Handbook of human performance technology: Improving individual and organizational performance worldwide (2nd ed., pp. 373–394). San Francisco: Jossey-Bass.
Knowles, M., Holton, E. F. I., & Swanson, R. A. (1998). The adult learner: The definitive classic in adult education and human resource development (5th ed.). Woburn, MA: Butterworth-Heinemann.
Kottkamp, R. B. (1990). Means for facilitating reflection. Education and Urban Society, 22(2), 182–203.
Mayer, R. E. (1993). Problem-solving principles. In M. Fleming & W. H. Levie (Eds.), Instruc- tional message design (2nd ed.). Englewood Cliffs, NJ: Educational Technology Publications.
Merriam, S. B., & Caffarella, R. S. (1999). Learning in adulthood: A comprehensive guide. San Francisco: Jossey-Bass.
Mezirow, J. (1995). Transformation theory of adult learning. In M. R. Welton (Ed.), In defense of the lifeworld. New York: State University of New York Press.
Mezirow, J. (1997). Cognitive processes: Contemporary paradigms of learning. In P. Sutherland (Ed.), Adult learning: A reader. Sterling, VA: Kogan Page.
Milheim, W. (1995). Interactivity and computer-based instruction. Journal of Educational Technology Systems, 24(3), 225–233.
Morrison, G. R., Ross, S. M., & Kemp, J. E. (2001). Designing effective instruction. New York: John Wiley.
Morse, J. M. (1991). Strategies for sampling. In J. M. Morse (Ed.), Qualitative nursing research (pp. 127–145). Newbury Park, CA: Sage.
Morse, J. M. (1994). Designing funded qualitative research. In N. K. Denzin & Y. Lincoln (Eds.), Handbook of qualitative methods. Newbury Park, CA: Sage.
Morse, J. M. (2000). Determining sample size. Qualitative Health Research, 10(1), 3–5. Park, I., & Hannafin, M. J. (1993). Empirically-based guidelines for the design of interactive
multimedia. Educational Technology Research and Development, 41(3), 63–85. Pillay, H. (1998). Adult learning in a workplace context. In P. Sutherland (Ed.), Adult learning: A
reader. Sterling, VA: Kogan Page. Ray, M. A. (1994). The richness of phenomenology: Philosophic, theoretic, and methodologic
concerns. In J. M. Morse (Ed.), Critical issues in qualitative research methods (pp. 117–133). Thousand Oaks, CA: Sage.
Reeves, T. C. (1993). Pseudoscience in computer-based instruction: The case of learner control research. Journal of Computer-Based Instruction, 20(2), 39–46.
Ridley, D. S., Schutz, P. A., Glanz, R. S., & Weinstein, C. E. (1992). Self-regulated learning: The interactive influence of metacognitive awareness and goal setting. Journal of Experimental Education, 60(4), 293–306.
Roth, G. L. (1997). Learning to learn in the workplace. Paper presented at the Academy of Human Resource Development Conference, Atlanta, GA.
Salomon, G., & Perkins, D. (1996). Learning in wonderland: What do computers really offer education? In S. T. Kerr (Ed.), Technology and the future of schooling (Vol. 95, Yearbook of the National Society for the Study of Education, Part II). Chicago: University of Chicago Press.
Savery, J. R., & Duffy, T. M. (1996). Problem-based learning: An instructional model and its constructivist framework. In B. G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design. Englewood Cliffs, NJ: Educational Technology Publications.
Schraw, G. (1998). On the development of adult metacognition. In M. C. Smith & T. Pourchot (Eds.), Adult learning and development. Mahwah, NJ: Lawrence Erlbaum Associates.
170 J. Dobrovolny
Schwier, R. A., & Misanchuk, E. R. (1993). Interactive multimedia instruction. Englewood Cliffs, NJ: Educational Technology Publications.
Sims, R. (1999). Interactivity: A forgotten art? Computers in Human Behavior, 13(2), 157–180. Sims, R. (2000). An interactive conundrum: Constructs of interactivity and learning theory.
Australian Journal of Educational Technology, 16(1), 45–57. Steinberg, E. R. (1991). Computer-assisted instruction: A synthesis of theory, practice and technology.
Hillsdale, NJ: Lawrence Erlbaum Associates. Sutherland, P. (1997). Experiential learning and constructivism: Potential for a mutually beneficial
synthesis. In P. Sutherland (Ed.), Adult learning: A reader. Sterling, VA: Kogan Page. Von Wright, J. (1992). Reflections on reflection. Learning and Instruction, 2, 59–68. Wagner, E. D. (1994). In support of a functional definition of interaction. American Journal of
Distance Education, 8(2), 6–29. Winn, W., & Snyder, D. (1996). Cognitive perspectives in psychology. In D. H. Jonassen (Ed.),
Handbook of research for educational communications and technology. New York: Simon & Schuster Macmillan.
Wittrock, M. C. (1992). Generative learning processes of the brain. Educational Psychologist, 27(4), 531–541.
Wlodkowski, R. J. (1999). Enhancing adult motivation to learn. San Francisco: Jossey-Bass. Yacci. (2000). Interactivity demystified: A structural definition for distance education and intelli-
gent CBT. Educational Technology, 40(4), 5–16.
HRE7271/Week 2/Integrating Adult Learning Principles.pdf
Integrating Adult Learning Principles Into Training for Public Health Practice
Rebecca L. Bryan, MPH Matthew W. Kreuter, PhD, MPH
Ross C. Brownson, PhD
For example, only 44% of the public health workforce has formal public health education (Turnock, 2001).
In partnership with the various practitioner groups and agencies that are seeking to improve the health of the public, many programs are now being implemented with “grass-roots” community partners (sometimes called “community-based participatory research”; Israel, Schulz, Parker, & Becker, 1998). For these community-based pre- vention efforts, it is critical to involve in the process both members of the community itself as well as representa- tives of organizations that work with or serve members of the community. Most of these community partners will lack formal training and/or experience in a variety of public health disciplines, including health promotion, epidemiology, and evaluation.
Therefore, both practitioners and community partners can benefit from training programs designed to enhance adult learning. Despite considerable research and well- accepted theories and models describing ways to enhance adult learning, many adult educators have little training in adult education (Henschke, 1998; Imel, 1994; Merriam & Caffarella, 1999). To facilitate understanding and con- sistent application of adult learning theory by those who plan and conduct such training programs, this article identifies and explains five key principles of adult learn- ing and describes how each can be used to enhance the effectiveness of training and other planned learning expe- riences. Specifically, we explore ways that each adult learning principle can be applied during three essential steps in the training process: (a) assessing trainee needs,
Providing training and planned learning experiences to practitioners and community partners is fundamental to effective public health. The extent to which principles of adult learning currently guide such training is unknown and likely varies widely. The purpose of this article is to introduce five principles of adult learning and discuss how each can be applied in assessing trainee needs, planning and delivering training, and evaluating training processes and outcomes. Training guided by these principles should facilitate adult learning, collaborative efforts, and mutual respect between agencies, practitioners, and community partners.
Keywords: training; adult learning; public health prac- tice; evaluation; planning
D elivery of effective programs and policies in public health requires a well-prepared workforce. The practice of public health is a large and diverse
enterprise encompassing the activities of 59 state and terri- torial health departments, more than 3,000 local health departments, and myriad federal agencies and community- based organizations with both discrete and overlapping responsibilities. It is estimated that the governmental public health workforce (“practitioners”) alone numbers more than 430,000, with another 15,000 in voluntary agencies (Institute of Medicine [IOM], 2003). The public health workforce also cuts across multiple professions with highly varied preparation in the biological and social sciences and other technical fields (IOM, 2003).
Health Promotion Practice October 2009 Vol. 10, No. 4, 557-563 DOI: 10.1177/1524839907308117 ©2009 Society for Public Health Education
Authors’ Note: This project was supported by the Cancer Prevention and Control Research Network, through Cooperative Agreement No. 5 US48 DP000060 from the Centers for Disease Control and Prevention and the National Cancer Institute. Please address all cor- respondence to Dr. Matthew W. Kreuter, Health Communication Research Laboratory, School of Public Health, Saint Louis University, 3545 Lafayette Avenue, St. Louis, MO 63104; e-mail: [email protected].
557
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
(b) planning and delivering training, and (c) evaluating the process and impact of training activities.
>>WHAT IS AN ADULT LEARNER?
Adult can be defined in biological, legal, social, or psychological terms (Mogelonsky, 1996), and no single
definition is consistently used to identify adult learners. For the purposes of this article, we use Malcolm Knowles’s definition: “We become adult psychologically when we arrive at a self-concept of being responsible for our own lives, of being self-directing” (Knowles, Holton, & Swanson, 1998, p. 64).
>>FIVE PRINCIPLES OF ADULT LEARNING
There are numerous theories and models that seek to describe or explain how adults learn (Cyr, 1999; Imel, 1998; Merriam, 2001; Merriam & Caffarella, 1999). Although dif- ferent models have different areas of emphasis (e.g., char- acteristics of the learner, the process of learning, changes in consciousness of the learner), there is general consensus about the central principles (Lawler, 2003). In reviewing existing theories and models, we identified recurring themes and synthesized them into five key principles. These principles appear in some form in many theories and models of adult learning (see Table 1) and are major com- ponents of two leading models, andragogy (Knowles, 1980, 1984; Knowles et al., 1998) and self-directed learning (Knowles, 1975; Tough, 1967, 1971). The principles are:
558 HEALTH PROMOTION PRACTICE / October 2009
The Authors
Rebecca L. Bryan, MPH, is a senior research assistant in the Health Communication Laboratory at the School of Public Health at Saint Louis University, St. Louis, Missouri.
Matthew W. Kreuter, PhD, MPH, is professor of community health and director of the Health Communication Laboratory in the School of Public Health at Saint Louis University, St. Louis, Missouri.
Ross C. Brownson, PhD, is professor of community health and director of the Prevention Research Center in the School of Public Health at Saint Louis University, St. Louis, Missouri.
TABLE 1 Five Key Principles of Adult Learning, by Selected Theories and Models
Adult Learning Principle
Adults Are Adults’ Learning Adults Need Adults Motivated Previous Approaches to Be Need to to Learn Experience Should Match Actively
Know Why by the Need Must Be Adults’ Involved in They Are to Solve Respected, Background, the Learning
Theory or Model Key Citationa Learning Problems Built Upon Diversity Process
Andragogy Knowles, X X X X X Holton, and Swanson (1998)
Thiagi’s laws of Zemke (2002) X X X X X learning
Teachers as Lawler (2003) X X X X learnersb
Self-directed Knowles (1975) X X X learning
Adult basic Imel (1998) X X education principlesb
Constructivist Daley (2001) X learning
a. Selected from many citations. b. Model not formally named.
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
1. Adults need to know why they are learning. 2. Adults are motivated to learn by the need to solve
problems. 3. Adults’ previous experience must be respected and
built upon. 4. Adults need learning approaches that match their
background and diversity. 5. Adults need to be actively involved in the learning
process.
Although these principles are straightforward and intu- itive, we propose that integrating them into training requires thoughtful planning, not simply relying on one’s instincts. To help guide such planning, we will briefly describe each principle and how it is applied in assessing trainee needs, preparing and delivering training, and eval- uating the processes and outcomes of training. Sample
questions for assessing trainee needs and evaluating adher- ence to each principle are provided in Table 2.
Principle 1: Adults need to know why they are learning. Adults will spend more time and energy learning when they see a reason for learning (Knowles et al., 1998; Tough, 1967). Although many adults—especially those participating in learning activities voluntarily—will enter a learning situation with a clear sense of why it is important for them or their organization, others will not (Lieb, 1991). For these latter adults, it is important to identify potentially compelling reasons for them to fully engage in the learning process. Such reasons can often be gleaned from an assessment of the learner pop- ulation. For example, it is often useful to learn about current job responsibilities of the learners as well as their individual and organizational goals and priorities.
Bryan et al. / ADULT LEARNING 559
TABLE 2 Sample Assessment and Evaluation Questions for Applying Adult Learning Principles
in Training for Public Health Practice
Principle
1. Adults need to know why they are learning.
2. Adults are motivated to learn by the need to solve problems.
3. Adults' previous experience must be respected and built upon.
4. Adults need learning approaches that match their background and diversity.
5. Adults need to be actively involved in the learning process.
Sample Assessment Questions
What is your organization's mission? How do your job responsibilities
contribute to that mission? What areas of your organization need
improvement? What specific problem(s) or
challenge(s) do you expect this training will help you address?
In current or past positions, how have you used local data to help understand or solve a local problem?
What challenges did you face working to address this problem?
How did you overcome those challenges?
Do you learn better by listening and watching or doing?
Do you learn better working along or in a group?
What do you need to learn to achieve the goals of the training?
If you were responsible for planning the training, what would you include and how would you deliver it?
Sample Evaluation Questions
Were learning objectives clearly stated? How well did trainers understand your reasons
for wanting to learn this information?
How realistic were the problems presented? Were the problems presented ones you have
encountered in your organization? How will the training help you address problems
or challenges you currently face? How well did trainers understand your past
experience and current job demands? Was the content of the training too simple, too
complicated, or about right for your level of experience?
How easy or difficult was it to integrate the content of the training into your existing knowledge and experience?
Were different training methods used to address different learning objectives?
In what ways did the methods used help or hinder your personal learning?
What input and control did you have over what you learned and how you learned it?
How did that level of control help or hinder your learning?
Did the training include too much, about the right amount, or too little participation from trainees?
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
Such an assessment should also identify constraints and obstacles to achieving these goals and priorities. For example, in training on evidence-based public health (Brownson, Baker, Leet, & Gillespie, 2003), the first session of the course includes a discussion of ways in which decisions are made in public health practice and barriers to effective decision making. This discus- sion relies largely on the “real-world” experience of the class participants. To help identify trainees’ reasons for learning, assessment questions might include: What is your organization’s mission? How do your job respon- sibilities contribute to that mission? What are your per- sonal and professional goals? What knowledge do you hope to gain from this training? What areas of your organization need improvement? Answers to questions like these can help infer reasons for learning that might be compelling for a specific adult learner or group of learners.
Because planned learning activities usually have spe- cific objectives, it is important for trainers to make explicit how these objectives will address learners’ reasons for attending a training (Lieb, 1991). For example, the reasons for learning about evidence-based health promotion strate- gies might be very different for a local health officer who must make strategic decisions about investing limited resources than for a health education practitioner seeking to enhance the delivery of an existing program or service. Demonstrating how a given learning activity will address a variety of different learner goals will help engage different trainees. Because some learners will be goal oriented (Houle, 1961), it follows that incorporating clear goals will also enhance motivation to learn and attention to training.
Evaluation efforts can help determine how well a training or other planned learning experience identi- fied and addressed learners’ reasons for wanting to gain new knowledge or skills. Evaluators might ask learners: Were learning objectives clearly stated? How were the stated objectives similar to or different from your own personal learning goals for the training? How well did the trainers seem to understand your reasons for want- ing to learn this information?
Principle 2: Adults are motivated to learn by the need to solve problems. Adults are practical and therefore eager to apply new knowledge to solve problems or challenges they face (Lawler, 2003; Merriam, 2001). For this reason, adults become more ready to learn when they have a specific problem to solve (Knowles, 1984). “Problems” in this sense can be wide ranging, including technical, logistical, tactical, strategic, or administrative challenges. They may also include uncertainty or transition, such as a changing or expanding role or expectations in one’s job. To assure
that planned learning experiences or training sessions ade- quately address the problems adult learners are hoping to solve, pretraining assessments should gather such infor- mation. The basic question is this: What problem(s) or challenge(s) do you expect this training will help you address?
Because one’s professional practice influences how they make meaning of new knowledge (Daley, 2001), the problems identified by different learners will vary consid- erably, especially in context and detail. It may be imprac- tical to address each learner’s unique problems, so those responsible for planning a training session often must identify broader problem themes that capture the range of challenges reported by learners. For example, in a given training, some learners might express difficulty in estab- lishing community partnerships, others with sharing responsibility and accountability in partnerships, and still others with reconciling different priorities among partner organizations. For planners, these concerns might be con- solidated into a single theme relating to effective collabo- ration, which could then be directly addressed within the training. Problem-based learning, a method in which learning occurs by working through problems (Lohman, 2002), is particularly well suited to do this. By building learning experiences around real problems facing a given group, a more learner-centered environment is adopted and learners are more likely to be engaged, participate in discussion, and share personal insights and experience (Knowles et al., 1998; Lawler, 2003), all factors that facili- tate learning.
An evaluation of how well a training adhered to this principle could assess whether the problems introduced and addressed were realistic and relevant to trainees’ job responsibilities or organizational priorities, whether the solutions proposed were realistic and feasible, and the extent to which incorporating problems into learning activities enhanced effectiveness of the training. To gather this information, evaluators might ask adult learners: In your experience, how realistic were the problems pre- sented? Were the problems presented ones you have encountered in your organization? How, if at all, will the information you learned be helpful in addressing prob- lems or challenges you are currently facing? How, if at all, did the problems presented in the training help facilitate your learning?
Principle 3: Adults’ previous experience must be respected and built upon. Adults bring valuable personal and pro- fessional experience into a learning setting (Knowles, 1984; Knowles et al., 1998; Merriam & Caffarella, 1999). Because new knowledge is learned within the context of existing knowledge, relating new material to what learners
560 HEALTH PROMOTION PRACTICE / October 2009
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
already know can aid in the learning process (Merriam & Caffarella, 1999). For example, Daley (2001) showed that knowledge became meaningful when adults linked new information with their professional practice, suggesting they may learn more when instruction is contextualized (Daley, 2001). Identifying, acknowledging, and validating learners’ past experience is thus essential for adult learn- ing (Knowles, 1984).
Accordingly, an important first step in planning effective training for adult learners is to take inventory of trainees’ experiences, particularly as they relate to the learning topics to be addressed or learning objec- tives to be achieved. Such information can be obtained by asking participants what they know from past expe- rience or local data about the problem of interest. Questions might include: In your current or past posi- tions, how have you used local data to help understand or solve a local problem? What challenges did you face working to address this problem? How did you over- come those challenges? Identifying and understanding the experience and knowledge adult learners bring to training will not only increase efficiency by avoiding redundancy in content but also make possible a wider range of interactive learning activities.
Trainers can and should capitalize on the past experi- ence of learners to facilitate learning (Zemke, 2002). For example, trainers can employ experiential learning tech- niques such as case studies or discussions that draw on learners’ previous experiences. Local data—both qualita- tive and quantitative—can be especially useful in devel- oping such case studies and usually can be gathered prior to a training session. For example, local quantitative data might be used to describe the magnitude of a problem while qualitative data could seek to capture trainees’ tacit knowledge about the problem (e.g., percep- tions of the effectiveness of different solutions). In the National Cancer Institute’s Using What Works training program, trainee experiences are elicited and integrated into training content (Boyle & Homer, 2006). Trainees are first asked: “Have you had any successful partnerships with organizations with competing messages or nontradi- tional program partners?” (Boyle & Homer, 2006). The trainer then uses trainee responses as a talking point for illustrating how having different types of partners can benefit a program.
Using this approach has at least three benefits: It (a) involves learners directly by having them talk about their experiences and reflect on local data, (b) acknowl- edges and validates learners’ expertise and experience by making it part of the planned learning, and (c) provides learners with the opportunity to extend planned content (e.g., “Is there anything I left out that you have found to be important in your experience?”). In addition, trainees can
be encouraged to discuss how the new knowledge fits into what they already know. What are they learning that is new? How well, if at all, does the content of the training fit their personal experience? Does it affirm or change their beliefs? How, if at all, will it change what they do or how they do it?
An evaluation of training based on this principle will determine the extent to which learners feel their experi- ences were valued and perceive instructors as providing opportunities to share their experiences and apply new knowledge and skills to current or past situations. Sample questions might include: How well did the train- ers seem to understand your past experience and current job demands? Would you say the content of the training was too simple, too complicated, or about right given your level of experience? How easy or difficult was it to integrate the content of the training into your own exist- ing knowledge and previous experience? In what ways, if any, did the use of local data enhance your learning? How much, if at all, did you learn from other participants’ sharing their experiences?
Principle 4: Adults need learning approaches that match their background and diversity. Because adults have accu- mulated a greater set and wider range of life experiences, they are more diverse as a group in their interests and pre- ferred styles of learning (Knowles et al., 1998). A multi- tude of learning styles exists, and different ones may predominate over others depending on the learning situa- tion (Delahoussaye, 2002). Furthermore, trainees’ educa- tional background and orientation to learning may range from a strong aversion to certain subjects (e.g., using math and/or interpreting data) to an eagerness to go above and beyond the presented material. This individual variability may be more pronounced given the diversity of the public health workforce. To create effective learning experiences, adult education and training programs must develop and use multiple methods, represent different perspectives, contextualize content in a variety of ways, and most important, know when each variant is appropriate for a given group of learners, or even a particular subgroup or individual within a group.
Assessment, even just asking how a person likes to learn (about the training topic) or which of several dif- ferent learning styles best describes them, helps iden- tify the diverse needs and learning styles of trainees. Specific questions to ask might include: Do you learn better by listening and watching or doing? Do you learn better working alone or in a group? With this informa- tion, training planners are better able to select or develop appropriate methods to maximize learning.
Because the diversity of any adult learning group is likely to be great, it may be necessary to prepare different
Bryan et al. / ADULT LEARNING 561
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
training approaches that vary across training content and learning styles. A modular approach, in which different components of a training are distinguished by their focus on specific content or learning style used, allows trainees to focus on subtopics they are most interested in or choose the learning style they prefer. Although creating multiple versions of training lessons requires significantly more preparation time, it can pay off in increased effectiveness if learning activities are better suited to trainee needs. For example, training in evidence-based public health (Brownson et al., 2003) has been adapted to the educa- tional backgrounds, language, and local needs for a vari- ety of public health practitioners nationally and globally. Evaluation of training based on this principle will reveal how well individual trainee differences were accounted for in the training approach. Specific evaluation questions might include: Were different training methods used to address different learning objectives? In what ways did the methods used help or hinder your personal learning?
Principle 5: Adults need to be involved in the learning process. Adults often consider themselves self-directing and thus prefer to control their learning (Knowles, 1984). For many adults, organized learning represents a rigid environment where they have little choice of what and how they learn. Such an environment can inhibit adults’ motivation to learn (Lawler, 2003). Allowing adults to become stakeholders in the training process by sharing control over content and methods will lead to greater moti- vation (Garrison, 1997) and presumably increase the like- lihood of training success. Ideally, trainees will have an opportunity to provide input into the training curriculum (Knowles et al., 1998). In some cases, a high degree of indi- vidualized self-direction may not be feasible within a diverse population of learners or in a single training ses- sion. Trainers can involve the trainees in setting the cur- riculum by incorporating their responses to the following types of questions: What do you need to learn to achieve the goals of the training? If you were responsible for plan- ning the training, what would you include and how would you deliver it? For example, in a series of 1-day courses to improve physical activity interventions (Brownson et al., 2007), each training session was planned with one or more representatives of eight states to ensure that the needs of each state were met. Among the benefits of these preses- sions, trainers were able to analyze and present local data, as suggested within Principle 3.
The level of desired self-direction will vary among trainees. Some will prefer no self-direction and others will prefer to be completely self-directing (Pratt, 1988). Providing a participatory learning environment allows
learners to adopt their preferred level of autonomy. Learning modules, as described earlier, in the form of breakout sessions or smaller working groups can allow for self-directed learning within smaller learning groups. Using teaching techniques such as discussions or case studies that engage learners on a personal level also allows some learner control over the pace and direction of the training.
Evaluation of training based on this principle should show how much control trainees perceived they had and to what extent that level of self-direction enhanced train- ing effects. Possible evaluation questions include: What input and control did you have over what you learned and how you learned it? How did that level of control help or hinder your learning? Did you feel the trainer asked for an appropriate amount of participation from the trainees?
>>CONCLUSION
There are now numerous adult learning programs being implemented for a range of public health practi- tioners and community partners (Bartholomew, Parcel, Kok, & Gottlieb, 2006; Brownson et al., 2003; Franks et al., 2005). Understanding and adhering to the principles we have outlined should enhance not only learning but also reciprocal respect and trust between trainers and trainees. Assessment and evaluation using items like those pro- vided (see Table 2) can help determine how well the five principles were integrated into a training and will yield useful data for refining and improving future iterations of a training.
REFERENCES
Bartholomew, L. K., Parcel, G. S., Kok, G., & Gottlieb, N. H. (2006). Health promotion planning: An intervention mapping approach (2nd ed.). San Francisco: Jossey-Bass.
Boyle, L., & Homer, M. (2006). Using what works: Adapting evidence- based programs to fit your needs. Retrieved January 5, 2007, from http://cancercontrol.cancer.gov/use_what_works/start.htm
Brownson, R. C., Baker, E. A., Leet, T. L., & Gillespie, K. N. (2003). Evidence-based public health. New York: Oxford University Press.
Brownson, R. C., Ballew, P., Brown, K. L., Elliott, M. B., Haire-Joshu, D., Heath, G. W., et al. (2007). The effect of disseminating evidence-based interventions that promote physical activity to health departments. American Journal of Public Health, 97, 1900-1907.
Cyr, A. V. (1999). Overview of theories and principles relating to characteristics of adult learners: 1970s-1999. Columbus, OH: ERIC Clearinghouse on Adult, Career, and Vocational Education. (ERIC Document Reproduction Service No. ED435817)
Daley, B. J. (2001). Learning and professional practice: A study of four professions. Adult Education Quarterly, 52, 39-54.
Delahoussaye, M. (2002). The perfect learner: An expert debate on learning styles. Training, 39(5), 28-36.
562 HEALTH PROMOTION PRACTICE / October 2009
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
Franks, A. L., Brownson, R. C., Bryant, C., Brown, K. M., Hooker, S. P., Pluto, D. M., et al. (2005). Prevention research centers: Contributions to updating the public health workforce through training. Preventing Chronic Disease, 2(2), A26.
Garrison, D. R. (1997). Self-directed learning: Toward a compre- hensive model. Adult Education Quarterly, 48, 18-33.
Henschke, J. A. (1998). Modeling the preparation of adult educa- tors. Adult Learning, 9(3), 11-13.
Houle, C. O. (1961). The inquiring mind. Madison: University of Wisconsin Press.
Imel, S. (1994). Guidelines for working with adult learners. Columbus, OH: ERIC Clearinghouse on Adult, Career, and Vocational Education. (ERIC Document Reproduction Service No. ED377313)
Imel, S. (1998). Using adult learning principles in adult basic and literacy education. Columbus, OH: ERIC Clearinghouse on Adult, Career, and Vocational Education. (ERIC Document Reproduction Service No. ED425336)
Institute of Medicine. (2003). Who will keep the public healthy? Educating public health professionals for the 21st century. Washington, DC: National Academies Press.
Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Annual Review of Public Health, 19, 173-202.
Knowles, M. S. (1975). Self-directed learning. New York: Associ- ated Press.
Knowles, M. S. (1980). The modern practice of adult education: From pedagogy to andragogy (2nd ed.). New York: Cambridge Books.
Knowles, M. S. (1984). The adult learner: A neglected species (3rd ed.). Houston, TX: Gulf.
Knowles, M. S., Holton, E. F., & Swanson, R. A. (1998). The adult learner: The definitive classic in adult education and human resource development (5th ed.). Houston, TX: Gulf.
Lawler, P. (2003). Teachers as adult learners: A new perspective. New Directions for Adult and Continuing Education, 98, 15-22.
Lieb, S. (1991). Principles of adult learning. Retrieved May 19, 2006, from http://honolulu.hawaii.edu/intranet/committees/ FacDevCom/guidebk/teachtip/adults-2.htm
Lohman, M. C. (2002). Cultivating problem-solving skills through problems-based approaches to professional development. Human Resource Development Quarterly, 13, 243-261, 263-269.
Merriam, S. B. (2001). Andragogy and self-directed learning: Pillars of adult learning theory. New Directions for Adult and Continuing Education, 89, 3-14.
Merriam, S. B., & Caffarella, R. S. (1999). Learning in adulthood: A comprehensive guide (2nd ed.). San Francisco: Jossey-Bass.
Mogelonsky, M. (1996). The rocky road to adulthood. American Demographics, 18(5), 26-35, 56.
Pratt, D. D. (1988). Andragogy as a relational construct. Adult Education Quarterly, 38, 160-172.
Tough, A. (1967). Learning without a teacher (Educational Research Series, No. 3). Toronto, Canada: Ontario Institute for Studies in Education.
Tough, A. (1971). The adult’s learning projects. Toronto, Canada: Ontario Institute for Studies in Education.
Turnock, B. J. (2001). Public health: What it is and how it works. Gaithersburg, MD: Aspen.
Zemke, R. (2002). Who needs learning theory anyway? Training, 39(9), 86-91.
Bryan et al. / ADULT LEARNING 563
at Sheridan on December 1, 2016hpp.sagepub.comDownloaded from
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Warning /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /DetectCurves 0.1000 /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedOpenType false /ParseICCProfilesInComments true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams false /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveDICMYKValues true /PreserveEPSInfo true /PreserveFlatness false /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /ACaslon-Bold /ACaslon-BoldItalic /ACaslon-Italic /ACaslon-Ornaments /ACaslon-Regular /ACaslon-Semibold /ACaslon-SemiboldItalic /AdobeCorpID-Acrobat /AdobeCorpID-Adobe /AdobeCorpID-Bullet /AdobeCorpID-MinionBd /AdobeCorpID-MinionBdIt /AdobeCorpID-MinionRg /AdobeCorpID-MinionRgIt /AdobeCorpID-MinionSb /AdobeCorpID-MinionSbIt /AdobeCorpID-MyriadBd /AdobeCorpID-MyriadBdIt /AdobeCorpID-MyriadBdScn /AdobeCorpID-MyriadBdScnIt /AdobeCorpID-MyriadBl /AdobeCorpID-MyriadBlIt /AdobeCorpID-MyriadLt /AdobeCorpID-MyriadLtIt /AdobeCorpID-MyriadPkg /AdobeCorpID-MyriadRg /AdobeCorpID-MyriadRgIt /AdobeCorpID-MyriadRgScn /AdobeCorpID-MyriadRgScnIt /AdobeCorpID-MyriadSb /AdobeCorpID-MyriadSbIt /AdobeCorpID-MyriadSbScn /AdobeCorpID-MyriadSbScnIt /AdobeCorpID-PScript /AGaramond-BoldScaps /AGaramond-Italic /AGaramond-Regular /AGaramond-RomanScaps /AGaramond-Semibold /AGaramond-SemiboldItalic /AGar-Special /AkzidenzGroteskBE-Bold /AkzidenzGroteskBE-BoldEx /AkzidenzGroteskBE-BoldExIt /AkzidenzGroteskBE-BoldIt /AkzidenzGroteskBE-Ex /AkzidenzGroteskBE-It /AkzidenzGroteskBE-Light /AkzidenzGroteskBE-LightEx /AkzidenzGroteskBE-LightOsF /AkzidenzGroteskBE-Md /AkzidenzGroteskBE-MdEx /AkzidenzGroteskBE-MdIt /AkzidenzGroteskBE-Regular /AkzidenzGroteskBE-Super /AlbertusMT /AlbertusMT-Italic /AlbertusMT-Light /Aldine401BT-BoldA /Aldine401BT-BoldItalicA /Aldine401BT-ItalicA /Aldine401BT-RomanA /Aldine401BTSPL-RomanA /Aldine721BT-Bold /Aldine721BT-BoldItalic /Aldine721BT-Italic /Aldine721BT-Light /Aldine721BT-LightItalic /Aldine721BT-Roman /Aldus-Italic /Aldus-ItalicOsF /Aldus-Roman /Aldus-RomanSC /AlternateGothicNo2BT-Regular /AmazoneBT-Regular /AmericanTypewriter-Bold /AmericanTypewriter-BoldA /AmericanTypewriter-BoldCond /AmericanTypewriter-BoldCondA /AmericanTypewriter-Cond /AmericanTypewriter-CondA /AmericanTypewriter-Light /AmericanTypewriter-LightA /AmericanTypewriter-LightCond /AmericanTypewriter-LightCondA /AmericanTypewriter-Medium /AmericanTypewriter-MediumA /Anna /AntiqueOlive-Bold /AntiqueOlive-Compact /AntiqueOlive-Italic /AntiqueOlive-Roman /Arcadia /Arcadia-A /Arkona-Medium /Arkona-Regular /ArrusBT-Black /ArrusBT-BlackItalic /ArrusBT-Bold /ArrusBT-BoldItalic /ArrusBT-Italic /ArrusBT-Roman /AssemblyLightSSK /AuroraBT-BoldCondensed /AuroraBT-RomanCondensed /AuroraOpti-Condensed /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-Demi /AvantGarde-DemiOblique /Avenir-Black /Avenir-BlackOblique /Avenir-Book /Avenir-BookOblique /Avenir-Heavy /Avenir-HeavyOblique /Avenir-Light /Avenir-LightOblique /Avenir-Medium /Avenir-MediumOblique /Avenir-Oblique /Avenir-Roman /BaileySansITC-Bold /BaileySansITC-BoldItalic /BaileySansITC-Book /BaileySansITC-BookItalic /BakerSignetBT-Roman /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /Baskerville-Bold /BaskervilleBook-Italic /BaskervilleBook-MedItalic /BaskervilleBook-Medium /BaskervilleBook-Regular /BaskervilleBT-Bold /BaskervilleBT-BoldItalic /BaskervilleBT-Italic /BaskervilleBT-Roman /BaskervilleMT /BaskervilleMT-Bold /BaskervilleMT-BoldItalic /BaskervilleMT-Italic /BaskervilleMT-SemiBold /BaskervilleMT-SemiBoldItalic /BaskervilleNo2BT-Bold /BaskervilleNo2BT-BoldItalic /BaskervilleNo2BT-Italic /BaskervilleNo2BT-Roman /Baskerville-Normal-Italic /BauerBodoni-Black /BauerBodoni-BlackCond /BauerBodoni-BlackItalic /BauerBodoni-Bold /BauerBodoni-BoldCond /BauerBodoni-BoldItalic /BauerBodoni-BoldItalicOsF /BauerBodoni-BoldOsF /BauerBodoni-Italic /BauerBodoni-ItalicOsF /BauerBodoni-Roman /BauerBodoni-RomanSC /Bauhaus-Bold /Bauhaus-Demi /Bauhaus-Heavy /BauhausITCbyBT-Bold /BauhausITCbyBT-Heavy /BauhausITCbyBT-Light /BauhausITCbyBT-Medium /Bauhaus-Light /Bauhaus-Medium /BellCentennial-Address /BellGothic-Black /BellGothic-Bold /Bell-GothicBoldItalicBT /BellGothicBT-Bold /BellGothicBT-Roman /BellGothic-Light /Bembo /Bembo-Bold /Bembo-BoldExpert /Bembo-BoldItalic /Bembo-BoldItalicExpert /Bembo-Expert /Bembo-ExtraBoldItalic /Bembo-Italic /Bembo-ItalicExpert /Bembo-Semibold /Bembo-SemiboldItalic /Benguiat-Bold /Benguiat-BoldItalic /Benguiat-Book /Benguiat-BookItalic /BenguiatGothicITCbyBT-Bold /BenguiatGothicITCbyBT-BoldItal /BenguiatGothicITCbyBT-Book /BenguiatGothicITCbyBT-BookItal /BenguiatITCbyBT-Bold /BenguiatITCbyBT-BoldItalic /BenguiatITCbyBT-Book /BenguiatITCbyBT-BookItalic /Benguiat-Medium /Benguiat-MediumItalic /Berkeley-Black /Berkeley-BlackItalic /Berkeley-Bold /Berkeley-BoldItalic /Berkeley-Book /Berkeley-BookItalic /Berkeley-Italic /Berkeley-Medium /Berling-Bold /Berling-BoldItalic /Berling-Italic /Berling-Roman /BernhardBoldCondensedBT-Regular /BernhardFashionBT-Regular /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /BernhardTangoBT-Regular /BlockBE-Condensed /BlockBE-ExtraCn /BlockBE-ExtraCnIt /BlockBE-Heavy /BlockBE-Italic /BlockBE-Regular /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /Bodoni-Poster /Bodoni-PosterCompressed /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /BremenBT-Black /BremenBT-Bold /BroadwayBT-Regular /CaflischScript-Bold /CaflischScript-Regular /Caliban /CarminaBT-Bold /CarminaBT-BoldItalic /CarminaBT-Light /CarminaBT-LightItalic /CarminaBT-Medium /CarminaBT-MediumItalic /Carta /Caslon224ITCbyBT-Bold /Caslon224ITCbyBT-BoldItalic /Caslon224ITCbyBT-Book /Caslon224ITCbyBT-BookItalic /Caslon540BT-Italic /Caslon540BT-Roman /CaslonBT-Bold /CaslonBT-BoldItalic /CaslonOpenFace /CaslonTwoTwentyFour-Black /CaslonTwoTwentyFour-BlackIt /CaslonTwoTwentyFour-Bold /CaslonTwoTwentyFour-BoldIt /CaslonTwoTwentyFour-Book /CaslonTwoTwentyFour-BookIt /CaslonTwoTwentyFour-Medium /CaslonTwoTwentyFour-MediumIt /CastleT-Bold /CastleT-Book /Caxton-Bold /Caxton-BoldItalic /Caxton-Book /Caxton-BookItalic /CaxtonBT-Bold /CaxtonBT-BoldItalic /CaxtonBT-Book /CaxtonBT-BookItalic /Caxton-Light /Caxton-LightItalic /CelestiaAntiqua-Ornaments /Centennial-BlackItalicOsF /Centennial-BlackOsF /Centennial-BoldItalicOsF /Centennial-BoldOsF /Centennial-ItalicOsF /Centennial-LightItalicOsF /Centennial-LightSC /Centennial-RomanSC /Century-Bold /Century-BoldItalic /Century-Book /Century-BookItalic /CenturyExpandedBT-Bold /CenturyExpandedBT-BoldItalic /CenturyExpandedBT-Italic /CenturyExpandedBT-Roman /Century-HandtooledBold /Century-HandtooledBoldItalic /Century-Light /Century-LightItalic /CenturyOldStyle-Bold /CenturyOldStyle-Italic /CenturyOldStyle-Regular /CenturySchoolbookBT-Bold /CenturySchoolbookBT-BoldCond /CenturySchoolbookBT-BoldItalic /CenturySchoolbookBT-Italic /CenturySchoolbookBT-Roman /Century-Ultra /Century-UltraItalic /CharterBT-Black /CharterBT-BlackItalic /CharterBT-Bold /CharterBT-BoldItalic /CharterBT-Italic /CharterBT-Roman /CheltenhamBT-Bold /CheltenhamBT-BoldCondItalic /CheltenhamBT-BoldExtraCondensed /CheltenhamBT-BoldHeadline /CheltenhamBT-BoldItalic /CheltenhamBT-BoldItalicHeadline /CheltenhamBT-Italic /CheltenhamBT-Roman /Cheltenham-HandtooledBdIt /Cheltenham-HandtooledBold /CheltenhamITCbyBT-Bold /CheltenhamITCbyBT-BoldItalic /CheltenhamITCbyBT-Book /CheltenhamITCbyBT-BookItalic /Christiana-Bold /Christiana-BoldItalic /Christiana-Italic /Christiana-Medium /Christiana-MediumItalic /Christiana-Regular /Christiana-RegularExpert /Christiana-RegularSC /Clarendon /Clarendon-Bold /Clarendon-Light /ClassicalGaramondBT-Bold /ClassicalGaramondBT-BoldItalic /ClassicalGaramondBT-Italic /ClassicalGaramondBT-Roman /CMR10 /CMR8 /CMSY10 /CMSY8 /CMTI10 /CommonBullets /ConduitITC-Bold /ConduitITC-BoldItalic /ConduitITC-Light /ConduitITC-LightItalic /ConduitITC-Medium /ConduitITC-MediumItalic /CooperBlack /CooperBlack-Italic /CooperBT-Bold /CooperBT-BoldItalic /CooperBT-Light /CooperBT-LightItalic /CopperplateGothicBT-Bold /CopperplateGothicBT-BoldCond /CopperplateGothicBT-Heavy /CopperplateGothicBT-Roman /CopperplateGothicBT-RomanCond /Copperplate-ThirtyThreeBC /Copperplate-ThirtyTwoBC /Coronet-Regular /Courier /Courier-Bold /Courier-BoldOblique /Courier-Oblique /Critter /CS-Special-font /DellaRobbiaBT-Bold /DellaRobbiaBT-Roman /Della-RobbiaItalicBT /Della-RobbiaSCaps /Del-NormalSmallCaps /Delphin-IA /Delphin-IIA /Delta-Bold /Delta-BoldItalic /Delta-Book /Delta-BookItalic /Delta-Light /Delta-LightItalic /Delta-Medium /Delta-MediumItalic /Delta-Outline /DextorD /DextorOutD /DidotLH-OrnamentsOne /DidotLH-OrnamentsTwo /DINEngschrift /DINEngschrift-Alternate /DINMittelschrift /DINMittelschrift-Alternate /DINNeuzeitGrotesk-BoldCond /DINNeuzeitGrotesk-Light /Dom-CasItalic /DomCasual /DomCasual-Bold /Dom-CasualBT /Ehrhard-Italic /Ehrhard-Regular /EhrhardSemi-Italic /EhrhardtMT /EhrhardtMT-Italic /EhrhardtMT-SemiBold /EhrhardtMT-SemiBoldItalic /EhrharSemi /ELANGO-IB-A03 /ELANGO-IB-A75 /ELANGO-IB-A99 /ElectraLH-Bold /ElectraLH-BoldCursive /ElectraLH-Cursive /ElectraLH-Regular /ElGreco /EnglischeSchT-Bold /EnglischeSchT-Regu /ErasContour /ErasITCbyBT-Bold /ErasITCbyBT-Book /ErasITCbyBT-Demi /ErasITCbyBT-Light /ErasITCbyBT-Medium /ErasITCbyBT-Ultra /Euclid /Euclid-Bold /Euclid-BoldItalic /EuclidExtra /EuclidExtra-Bold /EuclidFraktur /EuclidFraktur-Bold /Euclid-Italic /EuclidMathOne /EuclidMathOne-Bold /EuclidMathTwo /EuclidMathTwo-Bold /EuclidSymbol /EuclidSymbol-Bold /EuclidSymbol-BoldItalic /EuclidSymbol-Italic /EUEX10 /EUFB10 /EUFB5 /EUFB7 /EUFM10 /EUFM5 /EUFM7 /EURB10 /EURB5 /EURB7 /EURM10 /EURM5 /EURM7 /EuropeanPi-Four /EuropeanPi-One /EuropeanPi-Three /EuropeanPi-Two /EuroSans-Bold /EuroSans-BoldItalic /EuroSans-Italic /EuroSans-Regular /EuroSerif-Bold /EuroSerif-BoldItalic /EuroSerif-Italic /EuroSerif-Regular /Eurostile /Eurostile-Bold /Eurostile-BoldCondensed /Eurostile-BoldExtendedTwo /Eurostile-BoldOblique /Eurostile-Condensed /Eurostile-Demi /Eurostile-DemiOblique /Eurostile-ExtendedTwo /EurostileLTStd-Demi /EurostileLTStd-DemiOblique /Eurostile-Oblique /EUSB10 /EUSB5 /EUSB7 /EUSM10 /EUSM5 /EUSM7 /ExPonto-Regular /FairfieldLH-Bold /FairfieldLH-BoldItalic /FairfieldLH-BoldSC /FairfieldLH-CaptionBold /FairfieldLH-CaptionHeavy /FairfieldLH-CaptionLight /FairfieldLH-CaptionMedium /FairfieldLH-Heavy /FairfieldLH-HeavyItalic /FairfieldLH-HeavySC /FairfieldLH-Light /FairfieldLH-LightItalic /FairfieldLH-LightSC /FairfieldLH-Medium /FairfieldLH-MediumItalic /FairfieldLH-MediumSC /FairfieldLH-SwBoldItalicOsF /FairfieldLH-SwHeavyItalicOsF /FairfieldLH-SwLightItalicOsF /FairfieldLH-SwMediumItalicOsF /Fences /Fenice-Bold /Fenice-BoldOblique /FeniceITCbyBT-Bold /FeniceITCbyBT-BoldItalic /FeniceITCbyBT-Regular /FeniceITCbyBT-RegularItalic /Fenice-Light /Fenice-LightOblique /Fenice-Regular /Fenice-RegularOblique /Fenice-Ultra /Fenice-UltraOblique /FlashD-Ligh /Flood /Folio-Bold /Folio-BoldCondensed /Folio-ExtraBold /Folio-Light /Folio-Medium /FontanaNDAaOsF /FontanaNDAaOsF-Italic /FontanaNDCcOsF-Semibold /FontanaNDCcOsF-SemiboldIta /FontanaNDEeOsF /FontanaNDEeOsF-Bold /FontanaNDEeOsF-BoldItalic /FontanaNDEeOsF-Light /FontanaNDEeOsF-Semibold /FormalScript421BT-Regular /Formata-Bold /Formata-MediumCondensed /ForteMT /FournierMT-Ornaments /FrakturBT-Regular /FrankfurterHigD /FranklinGothic-Book /FranklinGothic-BookItal /FranklinGothic-BookOblique /FranklinGothic-Condensed /FranklinGothic-Demi /FranklinGothic-DemiItal /FranklinGothic-DemiOblique /FranklinGothic-Heavy /FranklinGothic-HeavyItal /FranklinGothic-HeavyOblique /FranklinGothicITCbyBT-BookItal /FranklinGothicITCbyBT-Demi /FranklinGothicITCbyBT-DemiItal /FranklinGothicITCbyBT-Heavy /FranklinGothicITCbyBT-HeavyItal /FranklinGothic-Medium /FranklinGothic-MediumItal /FranklinGothic-Roman /Freeform721BT-Bold /Freeform721BT-BoldItalic /Freeform721BT-Italic /Freeform721BT-Roman /FreestyleScrD /FreestyleScript /Freestylescript /FrizQuadrataITCbyBT-Bold /FrizQuadrataITCbyBT-Roman /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura /FuturaBlackBT-Regular /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldCondensed /FuturaBT-BoldCondensedItalic /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-ExtraBlack /FuturaBT-ExtraBlackCondensed /FuturaBT-ExtraBlackCondItalic /FuturaBT-ExtraBlackItalic /FuturaBT-Heavy /FuturaBT-HeavyItalic /FuturaBT-Light /FuturaBT-LightCondensed /FuturaBT-LightItalic /FuturaBT-Medium /FuturaBT-MediumCondensed /FuturaBT-MediumItalic /Futura-CondensedLight /Futura-CondensedLightOblique /Futura-ExtraBold /Futura-ExtraBoldOblique /Futura-Heavy /Futura-HeavyOblique /Futura-Light /Futura-LightOblique /Futura-Oblique /Futura-Thin /Galliard-Black /Galliard-BlackItalic /Galliard-Bold /Galliard-BoldItalic /Galliard-Italic /GalliardITCbyBT-Bold /GalliardITCbyBT-BoldItalic /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Galliard-Roman /Galliard-Ultra /Galliard-UltraItalic /Garamond-Antiqua /GaramondBE-Bold /GaramondBE-BoldExpert /GaramondBE-BoldOsF /GaramondBE-CnExpert /GaramondBE-Condensed /GaramondBE-CondensedSC /GaramondBE-Italic /GaramondBE-ItalicExpert /GaramondBE-ItalicOsF /GaramondBE-Medium /GaramondBE-MediumCn /GaramondBE-MediumCnExpert /GaramondBE-MediumCnOsF /GaramondBE-MediumExpert /GaramondBE-MediumItalic /GaramondBE-MediumItalicExpert /GaramondBE-MediumItalicOsF /GaramondBE-MediumSC /GaramondBE-Regular /GaramondBE-RegularExpert /GaramondBE-RegularSC /GaramondBE-SwashItalic /Garamond-Bold /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BoldItalic /Garamond-Book /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-BookItalic /Garamond-Halbfett /Garamond-HandtooledBold /Garamond-HandtooledBoldItalic /GaramondITCbyBT-Bold /GaramondITCbyBT-BoldCondensed /GaramondITCbyBT-BoldCondItalic /GaramondITCbyBT-BoldItalic /GaramondITCbyBT-BoldNarrow /GaramondITCbyBT-BoldNarrowItal /GaramondITCbyBT-Book /GaramondITCbyBT-BookCondensed /GaramondITCbyBT-BookCondItalic /GaramondITCbyBT-BookItalic /GaramondITCbyBT-BookNarrow /GaramondITCbyBT-BookNarrowItal /GaramondITCbyBT-Light /GaramondITCbyBT-LightCondensed /GaramondITCbyBT-LightCondItalic /GaramondITCbyBT-LightItalic /GaramondITCbyBT-LightNarrow /GaramondITCbyBT-LightNarrowItal /GaramondITCbyBT-Ultra /GaramondITCbyBT-UltraCondensed /GaramondITCbyBT-UltraCondItalic /GaramondITCbyBT-UltraItalic /Garamond-Kursiv /Garamond-KursivHalbfett /Garamond-Light /Garamond-LightCondensed /Garamond-LightCondensedItalic /Garamond-LightItalic /GaramondNo4CyrTCY-Ligh /GaramondNo4CyrTCY-LighItal /GaramondThree /GaramondThree-Bold /GaramondThree-BoldItalic /GaramondThree-BoldItalicOsF /GaramondThree-BoldSC /GaramondThree-Italic /GaramondThree-ItalicOsF /GaramondThree-SC /GaramondThreeSMSIISpl-Italic /GaramondThreeSMSitalicSpl-Italic /GaramondThreeSMSspl /GaramondThreespl /GaramondThreeSpl-Bold /GaramondThreeSpl-Italic /Garamond-Ultra /Garamond-UltraCondensed /Garamond-UltraCondensedItalic /Garamond-UltraItalic /GarthGraphic /GarthGraphic-Black /GarthGraphic-Bold /GarthGraphic-BoldCondensed /GarthGraphic-BoldItalic /GarthGraphic-Condensed /GarthGraphic-ExtraBold /GarthGraphic-Italic /Geometric231BT-HeavyC /GeometricSlab712BT-BoldA /GeometricSlab712BT-ExtraBoldA /GeometricSlab712BT-LightA /GeometricSlab712BT-LightItalicA /GeometricSlab712BT-MediumA /GeometricSlab712BT-MediumItalA /Giddyup /Giddyup-Thangs /GillSans /GillSans-Bold /GillSans-BoldCondensed /GillSans-BoldExtraCondensed /GillSans-BoldItalic /GillSans-Condensed /GillSans-ExtraBold /GillSans-ExtraBoldDisplay /GillSans-Italic /GillSans-Light /GillSans-LightItalic /GillSans-LightShadowed /GillSans-Shadowed /GillSans-UltraBold /GillSans-UltraBoldCondensed /Gill-Special /Giovanni-Bold /Giovanni-BoldItalic /Giovanni-Book /Giovanni-BookItalic /Glypha /Glypha-Bold /Glypha-BoldOblique /Glypha-Oblique /Gothic-Thirteen /Goudy /Goudy-Bold /Goudy-BoldItalic /GoudyCatalogueBT-Regular /Goudy-ExtraBold /GoudyHandtooledBT-Regular /GoudyHeavyfaceBT-Regular /GoudyHeavyfaceBT-RegularCond /Goudy-Italic /GoudyOldStyleBT-Bold /GoudyOldStyleBT-BoldItalic /GoudyOldStyleBT-ExtraBold /GoudyOldStyleBT-Italic /GoudyOldStyleBT-Roman /GoudySans-Black /GoudySans-BlackItalic /GoudySans-Bold /GoudySans-BoldItalic /GoudySans-Book /GoudySans-BookItalic /GoudySansITCbyBT-Black /GoudySansITCbyBT-BlackItalic /GoudySansITCbyBT-Bold /GoudySansITCbyBT-BoldItalic /GoudySansITCbyBT-Light /GoudySansITCbyBT-LightItalic /GoudySansITCbyBT-Medium /GoudySansITCbyBT-MediumItalic /GoudySans-Medium /GoudySans-MediumItalic /Granjon /Granjon-Bold /Granjon-BoldOsF /Granjon-Italic /Granjon-ItalicOsF /Granjon-SC /GreymantleMVB-Ornaments /Helvetica /Helvetica-Black /Helvetica-BlackOblique /Helvetica-Black-SemiBold /Helvetica-Bold /Helvetica-BoldOblique /Helvetica-Compressed /Helvetica-Condensed /Helvetica-Condensed-Black /Helvetica-Condensed-BlackObl /Helvetica-Condensed-Bold /Helvetica-Condensed-BoldObl /Helvetica-Condensed-Light /Helvetica-Condensed-Light-Light /Helvetica-Condensed-LightObl /Helvetica-Condensed-Oblique /Helvetica-Condensed-Thin /Helvetica-ExtraCompressed /Helvetica-Fraction /Helvetica-FractionBold /HelveticaInserat-Roman /HelveticaInserat-Roman-SemiBold /Helvetica-Light /Helvetica-LightOblique /Helvetica-Narrow /Helvetica-Narrow-Bold /Helvetica-Narrow-BoldOblique /Helvetica-Narrow-Oblique /HelveticaNeue-Black /HelveticaNeue-BlackCond /HelveticaNeue-BlackCondObl /HelveticaNeue-BlackExt /HelveticaNeue-BlackExtObl /HelveticaNeue-BlackItalic /HelveticaNeue-Bold /HelveticaNeue-BoldCond /HelveticaNeue-BoldCondObl /HelveticaNeue-BoldExt /HelveticaNeue-BoldExtObl /HelveticaNeue-BoldItalic /HelveticaNeue-Condensed /HelveticaNeue-CondensedObl /HelveticaNeue-ExtBlackCond /HelveticaNeue-ExtBlackCondObl /HelveticaNeue-Extended /HelveticaNeue-ExtendedObl /HelveticaNeue-Heavy /HelveticaNeue-HeavyCond /HelveticaNeue-HeavyCondObl /HelveticaNeue-HeavyExt /HelveticaNeue-HeavyExtObl /HelveticaNeue-HeavyItalic /HelveticaNeue-Italic /HelveticaNeue-Light /HelveticaNeue-LightCond /HelveticaNeue-LightCondObl /HelveticaNeue-LightExt /HelveticaNeue-LightExtObl /HelveticaNeue-LightItalic /HelveticaNeueLTStd-Md /HelveticaNeueLTStd-MdIt /HelveticaNeue-Medium /HelveticaNeue-MediumCond /HelveticaNeue-MediumCondObl /HelveticaNeue-MediumExt /HelveticaNeue-MediumExtObl /HelveticaNeue-MediumItalic /HelveticaNeue-Roman /HelveticaNeue-Thin /HelveticaNeue-ThinCond /HelveticaNeue-ThinCondObl /HelveticaNeue-ThinItalic /HelveticaNeue-UltraLigCond /HelveticaNeue-UltraLigCondObl /HelveticaNeue-UltraLigExt /HelveticaNeue-UltraLigExtObl /HelveticaNeue-UltraLight /HelveticaNeue-UltraLightItal /Helvetica-Oblique /Helvetica-UltraCompressed /HelvExtCompressed /HelvLight /HelvUltCompressed /Humanist521BT-Bold /Humanist521BT-BoldCondensed /Humanist521BT-BoldItalic /Humanist521BT-ExtraBold /Humanist521BT-Italic /Humanist521BT-Light /Humanist521BT-LightItalic /Humanist521BT-Roman /Humanist521BT-RomanCondensed /Humanist521BT-UltraBold /Humanist521BT-XtraBoldCondensed /Humanist531BT-BlackA /Humanist531BT-BoldA /Humanist531BT-RomanA /Humanist531BT-UltraBlackA /Humanist777BT-BlackB /Humanist777BT-BlackCondensedB /Humanist777BT-BlackItalicB /Humanist777BT-BoldB /Humanist777BT-BoldCondensedB /Humanist777BT-BoldItalicB /Humanist777BT-ExtraBlackB /Humanist777BT-ExtraBlackCondB /Humanist777BT-ItalicB /Humanist777BT-LightB /Humanist777BT-LightCondensedB /Humanist777BT-LightItalicB /Humanist777BT-RomanB /Humanist777BT-RomanCondensedB /Humanist970BT-BoldC /Humanist970BT-RomanC /HumanistSlabserif712BT-Black /HumanistSlabserif712BT-Bold /HumanistSlabserif712BT-Italic /HumanistSlabserif712BT-Roman /ICMEX10 /ICMMI8 /ICMSY8 /ICMTT8 /Iglesia-Light /ILASY8 /ILCMSS8 /ILCMSSB8 /ILCMSSI8 /Imago-Book /Imago-BookItalic /Imago-ExtraBold /Imago-ExtraBoldItalic /Imago-Light /Imago-LightItalic /Imago-Medium /Imago-MediumItalic /Industria-Inline /Industria-InlineA /Industria-Solid /Industria-SolidA /Insignia /Insignia-A /IPAExtras /IPAHighLow /IPAKiel /IPAKielSeven /IPAsans /ITCGaramondMM /ITCGaramondMM-It /JAKEOpti-Regular /JansonText-Bold /JansonText-BoldItalic /JansonText-Italic /JansonText-Roman /JansonText-RomanSC /JoannaMT /JoannaMT-Bold /JoannaMT-BoldItalic /JoannaMT-Italic /Juniper /KabelITCbyBT-Book /KabelITCbyBT-Demi /KabelITCbyBT-Medium /KabelITCbyBT-Ultra /Kaufmann /Kaufmann-Bold /KeplMM-Or2 /KisBT-Italic /KisBT-Roman /KlangMT /Kuenstler480BT-Black /Kuenstler480BT-Bold /Kuenstler480BT-BoldItalic /Kuenstler480BT-Italic /Kuenstler480BT-Roman /KunstlerschreibschD-Bold /KunstlerschreibschD-Medi /Lapidary333BT-Black /Lapidary333BT-Bold /Lapidary333BT-BoldItalic /Lapidary333BT-Italic /Lapidary333BT-Roman /LASY10 /LASY5 /LASY6 /LASY7 /LASY8 /LASY9 /LASYB10 /LatinMT-Condensed /LCIRCLE10 /LCIRCLEW10 /LCMSS8 /LCMSSB8 /LCMSSI8 /LDecorationPi-One /LDecorationPi-Two /Leawood-Black /Leawood-BlackItalic /Leawood-Bold /Leawood-BoldItalic /Leawood-Book /Leawood-BookItalic /Leawood-Medium /Leawood-MediumItalic /LegacySans-Bold /LegacySans-BoldItalic /LegacySans-Book /LegacySans-BookItalic /LegacySans-Medium /LegacySans-MediumItalic /LegacySans-Ultra /LegacySerif-Bold /LegacySerif-BoldItalic /LegacySerif-Book /LegacySerif-BookItalic /LegacySerif-Medium /LegacySerif-MediumItalic /LegacySerif-Ultra /LetterGothic /LetterGothic-Bold /LetterGothic-BoldSlanted /LetterGothic-Slanted /Life-Bold /Life-Italic /Life-Roman /LINE10 /LINEW10 /Linotext /Lithos-Black /LithosBold /Lithos-Bold /Lithos-Regular /LOGO10 /LOGO8 /LOGO9 /LOGOBF10 /LOGOSL10 /LOMD-Normal /LubalinGraph-Book /LubalinGraph-BookOblique /LubalinGraph-Demi /LubalinGraph-DemiOblique /LucidaHandwritingItalic /LucidaMath-Symbol /LucidaSansTypewriter /LucidaSansTypewriter-Bd /LucidaSansTypewriter-BdObl /LucidaSansTypewriter-Obl /LucidaTypewriter /LucidaTypewriter-Bold /LucidaTypewriter-BoldObl /LucidaTypewriter-Obl /LydianBT-Bold /LydianBT-BoldItalic /LydianBT-Italic /LydianBT-Roman /LydianCursiveBT-Regular /Machine /Machine-Bold /Marigold /MathematicalPi-Five /MathematicalPi-Four /MathematicalPi-One /MathematicalPi-Six /MathematicalPi-Three /MathematicalPi-Two /MatrixScriptBold /MatrixScriptBoldLin /MatrixScriptBook /MatrixScriptBookLin /MatrixScriptRegular /MatrixScriptRegularLin /Melior /Melior-Bold /Melior-BoldItalic /Melior-Italic /MercuriusCT-Black /MercuriusCT-BlackItalic /MercuriusCT-Light /MercuriusCT-LightItalic /MercuriusCT-Medium /MercuriusCT-MediumItalic /MercuriusMT-BoldScript /Meridien-Bold /Meridien-BoldItalic /Meridien-Italic /Meridien-Medium /Meridien-MediumItalic /Meridien-Roman /Minion-Black /Minion-Bold /Minion-BoldCondensed /Minion-BoldCondensedItalic /Minion-BoldItalic /Minion-Condensed /Minion-CondensedItalic /Minion-DisplayItalic /Minion-DisplayRegular /MinionExp-Italic /MinionExp-Semibold /MinionExp-SemiboldItalic /Minion-Italic /Minion-Ornaments /Minion-Regular /Minion-Semibold /Minion-SemiboldItalic /MonaLisa-Recut /MrsEavesAllPetiteCaps /MrsEavesAllSmallCaps /MrsEavesBold /MrsEavesFractions /MrsEavesItalic /MrsEavesPetiteCaps /MrsEavesRoman /MrsEavesRomanLining /MrsEavesSmallCaps /MSAM10 /MSAM10A /MSAM5 /MSAM6 /MSAM7 /MSAM8 /MSAM9 /MSBM10 /MSBM10A /MSBM5 /MSBM6 /MSBM7 /MSBM8 /MSBM9 /MTEX /MTEXB /MTEXH /MTGU /MTGUB /MTMI /MTMIB /MTMIH /MTMS /MTMSB /MTMUB /MTMUH /MTSY /MTSYB /MTSYH /MTSYN /MusicalSymbols-Normal /Myriad-Bold /Myriad-BoldItalic /Myriad-CnBold /Myriad-CnBoldItalic /Myriad-CnItalic /Myriad-CnSemibold /Myriad-CnSemiboldItalic /Myriad-Condensed /Myriad-Italic /MyriadMM /MyriadMM-It /Myriad-Roman /Myriad-Sketch /Myriad-Tilt /NeuzeitS-Book /NeuzeitS-BookHeavy /NewBaskerville-Bold /NewBaskerville-BoldItalic /NewBaskerville-Italic /NewBaskervilleITCbyBT-Bold /NewBaskervilleITCbyBT-BoldItal /NewBaskervilleITCbyBT-Italic /NewBaskervilleITCbyBT-Roman /NewBaskerville-Roman /NewCaledonia /NewCaledonia-Black /NewCaledonia-BlackItalic /NewCaledonia-Bold /NewCaledonia-BoldItalic /NewCaledonia-BoldItalicOsF /NewCaledonia-BoldSC /NewCaledonia-Italic /NewCaledonia-ItalicOsF /NewCaledonia-SC /NewCaledonia-SemiBold /NewCaledonia-SemiBoldItalic /NewCenturySchlbk-Bold /NewCenturySchlbk-BoldItalic /NewCenturySchlbk-Italic /NewCenturySchlbk-Roman /NewsGothic /NewsGothic-Bold /NewsGothic-BoldOblique /NewsGothicBT-Bold /NewsGothicBT-BoldCondensed /NewsGothicBT-BoldCondItalic /NewsGothicBT-BoldExtraCondensed /NewsGothicBT-BoldItalic /NewsGothicBT-Demi /NewsGothicBT-DemiItalic /NewsGothicBT-ExtraCondensed /NewsGothicBT-Italic /NewsGothicBT-ItalicCondensed /NewsGothicBT-Light /NewsGothicBT-LightItalic /NewsGothicBT-Roman /NewsGothicBT-RomanCondensed /NewsGothic-Oblique /New-Symbol /NovareseITCbyBT-Bold /NovareseITCbyBT-BoldItalic /NovareseITCbyBT-Book /NovareseITCbyBT-BookItalic /Nueva-BoldExtended /Nueva-Roman /NuptialScript /OceanSansMM /OceanSansMM-It /OfficinaSans-Bold /OfficinaSans-BoldItalic /OfficinaSans-Book /OfficinaSans-BookItalic /OfficinaSerif-Bold /OfficinaSerif-BoldItalic /OfficinaSerif-Book /OfficinaSerif-BookItalic /OnyxMT /Optima /Optima-Bold /Optima-BoldItalic /Optima-BoldOblique /Optima-ExtraBlack /Optima-ExtraBlackItalic /Optima-Italic /Optima-Oblique /OSPIRE-Plain /OttaIA /Otta-wa /Ottawa-BoldA /OttawaPSMT /Oxford /Palatino-Bold /Palatino-BoldItalic /Palatino-Italic /Palatino-Roman /Parisian /Perpetua /Perpetua-Bold /Perpetua-BoldItalic /Perpetua-Italic /PhotinaMT /PhotinaMT-Bold /PhotinaMT-BoldItalic /PhotinaMT-Italic /PhotinaMT-SemiBold /PhotinaMT-SemiBoldItalic /PhotinaMT-UltraBold /PhotinaMT-UltraBoldItalic /Plantin /Plantin-Bold /Plantin-BoldItalic /Plantin-Italic /Plantin-Light /Plantin-LightItalic /Plantin-Semibold /Plantin-SemiboldItalic /Poetica-ChanceryI /Poetica-SuppLowercaseEndI /PopplLaudatio-Italic /PopplLaudatio-Medium /PopplLaudatio-MediumItalic /PopplLaudatio-Regular /ProseAntique-Bold /ProseAntique-Normal /QuaySansEF-Black /QuaySansEF-BlackItalic /QuaySansEF-Book /QuaySansEF-BookItalic /QuaySansEF-Medium /QuaySansEF-MediumItalic /Quorum-Black /Quorum-Bold /Quorum-Book /Quorum-Light /Quorum-Medium /Raleigh /Raleigh-Bold /Raleigh-DemiBold /Raleigh-Medium /Revival565BT-Bold /Revival565BT-BoldItalic /Revival565BT-Italic /Revival565BT-Roman /Ribbon131BT-Bold /Ribbon131BT-Regular /RMTMI /Rockwell /Rockwell-Bold /Rockwell-BoldItalic /Rockwell-Italic /Rockwell-Light /Rockwell-LightItalic /RotisSansSerif /RotisSansSerif-Bold /RotisSansSerif-ExtraBold /RotisSansSerif-Italic /RotisSansSerif-Light /RotisSansSerif-LightItalic /RotisSemiSans /RotisSemiSans-Bold /RotisSemiSans-ExtraBold /RotisSemiSans-Italic /RotisSemiSans-Light /RotisSemiSans-LightItalic /RotisSemiSerif /RotisSemiSerif-Bold /RotisSerif /RotisSerif-Bold /RotisSerif-Italic /RunicMT-Condensed /Sabon-Bold /Sabon-BoldItalic /Sabon-Italic /Sabon-Roman /SackersGothicLight /SackersGothicLightAlt /SackersItalianScript /SackersItalianScriptAlt /Sam /Sanvito-Light /SanvitoMM /Sanvito-Roman /Semitica /Semitica-Italic /SIVAMATH /Siva-Special /SMS-SPELA /Souvenir-Demi /Souvenir-DemiItalic /SouvenirITCbyBT-Demi /SouvenirITCbyBT-DemiItalic /SouvenirITCbyBT-Light /SouvenirITCbyBT-LightItalic /Souvenir-Light /Souvenir-LightItalic /SpecialAA /Special-Gali /Sp-Sym /StempelGaramond-Bold /StempelGaramond-BoldItalic /StempelGaramond-Italic /StempelGaramond-Roman /StoneSans /StoneSans-Bold /StoneSans-BoldItalic /StoneSans-Italic /StoneSans-PhoneticAlternate /StoneSans-PhoneticIPA /StoneSans-Semibold /StoneSans-SemiboldItalic /StoneSerif /StoneSerif-Italic /StoneSerif-PhoneticAlternate /StoneSerif-PhoneticIPA /StoneSerif-Semibold /StoneSerif-SemiboldItalic /Swiss721BT-Black /Swiss721BT-BlackCondensed /Swiss721BT-BlackCondensedItalic /Swiss721BT-BlackExtended /Swiss721BT-BlackItalic /Swiss721BT-BlackOutline /Swiss721BT-BlackRounded /Swiss721BT-Bold /Swiss721BT-BoldCondensed /Swiss721BT-BoldCondensedItalic /Swiss721BT-BoldCondensedOutline /Swiss721BT-BoldExtended /Swiss721BT-BoldItalic /Swiss721BT-BoldOutline /Swiss721BT-BoldRounded /Swiss721BT-Heavy /Swiss721BT-HeavyItalic /Swiss721BT-Italic /Swiss721BT-ItalicCondensed /Swiss721BT-Light /Swiss721BT-LightCondensed /Swiss721BT-LightCondensedItalic /Swiss721BT-LightExtended /Swiss721BT-LightItalic /Swiss721BT-Medium /Swiss721BT-MediumItalic /Swiss721BT-Roman /Swiss721BT-RomanCondensed /Swiss721BT-RomanExtended /Swiss721BT-Thin /Swiss721BT-ThinItalic /Swiss921BT-RegularA /Symbol /Syntax-Black /Syntax-Bold /Syntax-Italic /Syntax-Roman /Syntax-UltraBlack /Tekton /Times-Bold /Times-BoldA /Times-BoldItalic /Times-BoldOblique /Times-Italic /Times-NewRoman /Times-NewRomanBold /TimesNewRomanPS-BoldItalicMT /TimesNewRomanPS-ItalicMT /TimesNewRomanPSMT /Times-Oblique /Times-PhoneticAlternate /Times-PhoneticIPA /Times-Roman /Times-RomanSmallCaps /Times-Sc /Times-SCB /Times-special /TimesTenGreekP-Upright /TradeGothic /TradeGothic-Bold /TradeGothic-BoldCondTwenty /TradeGothic-BoldCondTwentyObl /TradeGothic-BoldOblique /TradeGothic-BoldTwo /TradeGothic-BoldTwoOblique /TradeGothic-CondEighteen /TradeGothic-CondEighteenObl /TradeGothicLH-BoldExtended /TradeGothicLH-Extended /TradeGothic-Light /TradeGothic-LightOblique /TradeGothic-Oblique /Trajan-Bold /TrajanPro-Bold /TrajanPro-Regular /Trajan-Regular /Transitional521BT-BoldA /Transitional521BT-CursiveA /Transitional521BT-RomanA /Transitional551BT-MediumB /Transitional551BT-MediumItalicB /Univers /Universal-GreekwithMathPi /Universal-NewswithCommPi /Univers-BlackExt /Univers-BlackExtObl /Univers-Bold /Univers-BoldExt /Univers-BoldExtObl /Univers-BoldOblique /Univers-Condensed /Univers-CondensedBold /Univers-CondensedBoldOblique /Univers-CondensedOblique /Univers-Extended /Univers-ExtendedObl /Univers-ExtraBlackExt /Univers-ExtraBlackExtObl /Univers-Light /Univers-LightOblique /UniversLTStd-Black /UniversLTStd-BlackObl /Univers-Oblique /Utopia-Black /Utopia-BlackOsF /Utopia-Bold /Utopia-BoldItalic /Utopia-Italic /Utopia-Ornaments /Utopia-Regular /Utopia-Semibold /Utopia-SemiboldItalic /VAGRounded-Black /VAGRounded-Bold /VAGRounded-Light /VAGRounded-Thin /Viva-BoldExtraExtended /Viva-Regular /Weidemann-Black /Weidemann-BlackItalic /Weidemann-Bold /Weidemann-BoldItalic /Weidemann-Book /Weidemann-BookItalic /Weidemann-Medium /Weidemann-MediumItalic /WindsorBT-Elongated /WindsorBT-Light /WindsorBT-LightCondensed /WindsorBT-Roman /Wingdings-Regular /WNCYB10 /WNCYI10 /WNCYR10 /WNCYSC10 /WNCYSS10 /WoodtypeOrnaments-One /WoodtypeOrnaments-Two /ZapfCalligraphic801BT-Bold /ZapfCalligraphic801BT-BoldItal /ZapfCalligraphic801BT-Italic /ZapfCalligraphic801BT-Roman /ZapfChanceryITCbyBT-Bold /ZapfChanceryITCbyBT-Demi /ZapfChanceryITCbyBT-Medium /ZapfChanceryITCbyBT-MediumItal /ZapfChancery-MediumItalic /ZapfDingbats /ZapfDingbatsITCbyBT-Regular /ZapfElliptical711BT-Bold /ZapfElliptical711BT-BoldItalic /ZapfElliptical711BT-Italic /ZapfElliptical711BT-Roman /ZapfHumanist601BT-Bold /ZapfHumanist601BT-BoldItalic /ZapfHumanist601BT-Demi /ZapfHumanist601BT-DemiItalic /ZapfHumanist601BT-Italic /ZapfHumanist601BT-Roman /ZapfHumanist601BT-Ultra /ZapfHumanist601BT-UltraItalic /ZurichBT-Black /ZurichBT-BlackExtended /ZurichBT-BlackItalic /ZurichBT-Bold /ZurichBT-BoldCondensed /ZurichBT-BoldCondensedItalic /ZurichBT-BoldExtended /ZurichBT-BoldExtraCondensed /ZurichBT-BoldItalic /ZurichBT-ExtraBlack /ZurichBT-ExtraCondensed /ZurichBT-Italic /ZurichBT-ItalicCondensed /ZurichBT-Light /ZurichBT-LightCondensed /ZurichBT-LightCondensedItalic /ZurichBT-LightExtraCondensed /ZurichBT-LightItalic /ZurichBT-Roman /ZurichBT-RomanCondensed /ZurichBT-RomanExtended /ZurichBT-UltraBlackExtended ] /NeverEmbed [ true ] /AntiAliasColorImages false /CropColorImages false /ColorImageMinResolution 150 /ColorImageMinResolutionPolicy /OK /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageMinDownsampleDepth 1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /CropGrayImages false /GrayImageMinResolution 150 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /CropMonoImages false /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox false /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (U.S. Web Coated \050SWOP\051 v2) /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown /CreateJDFFile false /Description << /DAN <FEFF004200720075006700200064006900730073006500200069006e0064007300740069006c006c0069006e006700650072002000740069006c0020006100740020006f0070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f8006a006500720065002000620069006c006c00650064006f0070006c00f80073006e0069006e006700200066006f00720020006100740020006600e50020006200650064007200650020007500640073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e0020005000440046002d0044006f006b0075006d0065006e00740065006e0020006d00690074002000650069006e006500720020006800f60068006500720065006e002000420069006c0064006100750066006c00f600730075006e0067002c00200075006d002000650069006e0065002000760065007200620065007300730065007200740065002000420069006c0064007100750061006c0069007400e400740020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f0062006100740020006f0064006500720020006d00690074002000640065006d002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /ENU (Use these settings for creating PDF files for submission to The Sheridan Press. These settings configured for Acrobat v6.0 08/06/03.) /ESP <FEFF0055007300650020006500730074006100730020006f007000630069006f006e006500730020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006e0020006d00610079006f00720020007200650073006f006c00750063006900f3006e00200064006500200069006d006100670065006e00200070006100720061002000610075006d0065006e0074006100720020006c0061002000630061006c006900640061006400200061006c00200069006d007000720069006d00690072002e0020004c006f007300200064006f00630075006d0065006e0074006f00730020005000440046002000730065002000700075006500640065006e00200061006200720069007200200063006f006e0020004100630072006f00620061007400200079002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /FRA <FEFF004f007000740069006f006e00730020007000650072006d0065007400740061006e007400200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000500044004600200064006f007400e900730020006400270075006e00650020007200e90073006f006c007500740069006f006e002000e9006c0065007600e9006500200070006f0075007200200075006e00650020007100750061006c0069007400e90020006400270069006d007000720065007300730069006f006e00200061006d00e9006c0069006f007200e90065002e00200049006c002000650073007400200070006f0073007300690062006c0065002000640027006f00750076007200690072002000630065007300200064006f00630075006d0065006e007400730020005000440046002000640061006e00730020004100630072006f0062006100740020006500740020005200650061006400650072002c002000760065007200730069006f006e002000200035002e00300020006f007500200075006c007400e9007200690065007500720065002e> /ITA <FEFF00550073006100720065002000710075006500730074006500200069006d0070006f007300740061007a0069006f006e00690020007000650072002000630072006500610072006500200064006f00630075006d0065006e00740069002000500044004600200063006f006e00200075006e00610020007200690073006f006c0075007a0069006f006e00650020006d0061006700670069006f00720065002000700065007200200075006e00610020007100750061006c0069007400e00020006400690020007300740061006d007000610020006d00690067006c0069006f00720065002e0020004900200064006f00630075006d0065006e00740069002000500044004600200070006f00730073006f006e006f0020006500730073006500720065002000610070006500720074006900200063006f006e0020004100630072006f00620061007400200065002000520065006100640065007200200035002e003000200065002000760065007200730069006f006e006900200073007500630063006500730073006900760065002e> /JPN <FEFF3053306e8a2d5b9a306f30019ad889e350cf5ea6753b50cf3092542b308000200050004400460020658766f830924f5c62103059308b3068304d306b4f7f75283057307e30593002537052376642306e753b8cea3092670059279650306b4fdd306430533068304c3067304d307e305930023053306e8a2d5b9a30674f5c62103057305f00200050004400460020658766f8306f0020004100630072006f0062006100740020304a30883073002000520065006100640065007200200035002e003000204ee5964d30678868793a3067304d307e30593002> /NLD <FEFF004700650062007200750069006b002000640065007a006500200069006e007300740065006c006c0069006e00670065006e0020006f006d0020005000440046002d0064006f00630075006d0065006e00740065006e0020007400650020006d0061006b0065006e0020006d00650074002000650065006e00200068006f0067006500720065002000610066006200650065006c00640069006e00670073007200650073006f006c007500740069006500200076006f006f0072002000650065006e0020006200650074006500720065002000610066006400720075006b006b00770061006c00690074006500690074002e0020004400650020005000440046002d0064006f00630075006d0065006e00740065006e0020006b0075006e006e0065006e00200077006f007200640065006e002000670065006f00700065006e00640020006d006500740020004100630072006f00620061007400200065006e002000520065006100640065007200200035002e003000200065006e00200068006f006700650072002e> /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f00700070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f80079006500720065002000620069006c00640065006f00700070006c00f80073006e0069006e006700200066006f00720020006200650064007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f0067002000730065006e006500720065002e> /PTB <FEFF005500740069006c0069007a006500200065007300740061007300200063006f006e00660069006700750072006100e700f5006500730020007000610072006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006d00200075006d00610020007200650073006f006c007500e700e3006f00200064006500200069006d006100670065006d0020007300750070006500720069006f0072002000700061007200610020006f006200740065007200200075006d00610020007100750061006c0069006400610064006500200064006500200069006d0070007200650073007300e3006f0020006d0065006c0068006f0072002e0020004f007300200064006f00630075006d0065006e0074006f0073002000500044004600200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002c002000520065006100640065007200200035002e0030002000650020007300750070006500720069006f0072002e> /SUO <FEFF004e00e4006900640065006e002000610073006500740075007300740065006e0020006100760075006c006c006100200076006f0069006400610061006e0020006c0075006f006400610020005000440046002d0061007300690061006b00690072006a006f006a0061002c0020006a006f006900640065006e002000740075006c006f0073007400750073006c00610061007400750020006f006e0020006b006f0072006b006500610020006a00610020006b007500760061006e0020007400610072006b006b007500750073002000730075007500720069002e0020005000440046002d0061007300690061006b00690072006a0061007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f006200610074002d0020006a00610020004100630072006f006200610074002000520065006100640065007200200035002e00300020002d006f0068006a0065006c006d0061006c006c0061002000740061006900200075007500640065006d006d0061006c006c0061002000760065007200730069006f006c006c0061002e> /SVE <FEFF0041006e007600e4006e00640020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006e00e40072002000640075002000760069006c006c00200073006b0061007000610020005000440046002d0064006f006b0075006d0065006e00740020006d006500640020006800f6006700720065002000620069006c0064007500700070006c00f60073006e0069006e00670020006f006300680020006400e40072006d006500640020006600e50020006200e400740074007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e0020006b0061006e002000f600700070006e006100730020006d006500640020004100630072006f0062006100740020006f00630068002000520065006100640065007200200035002e003000200065006c006c00650072002000730065006e006100720065002e> >> /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ << /AsReaderSpreads false /CropImagesToFrames true /ErrorControl /WarnAndContinue /FlattenerIgnoreSpreadOverrides false /IncludeGuidesGrids false /IncludeNonPrinting false /IncludeSlug false /Namespace [ (Adobe) (InDesign) (4.0) ] /OmitPlacedBitmaps false /OmitPlacedEPS false /OmitPlacedPDF false /SimulateOverprint /Legacy >> << /AllowImageBreaks true /AllowTableBreaks true /ExpandPage false /HonorBaseURL true /HonorRolloverEffect false /IgnoreHTMLPageBreaks false /IncludeHeaderFooter false /MarginOffset [ 0 0 0 0 ] /MetadataAuthor () /MetadataKeywords () /MetadataSubject () /MetadataTitle () /MetricPageSize [ 0 0 ] /MetricUnit /inch /MobileCompatible 0 /Namespace [ (Adobe) (GoLive) (8.0) ] /OpenZoomToHTMLFontSize false /PageOrientation /Portrait /RemoveBackground false /ShrinkContent true /TreatColorsAs /MainMonitorColors /UseEmbeddedProfiles false /UseHTMLTitleAsMetadata true >> << /AddBleedMarks false /AddColorBars false /AddCropMarks false /AddPageInfo false /AddRegMarks false /BleedOffset [ 0 0 0 0 ] /ConvertColors /NoConversion /DestinationProfileName (sRGB IEC61966-2.1) /DestinationProfileSelector /NA /Downsample16BitImages true /FlattenerPreset << /PresetSelector /MediumResolution >> /FormElements true /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles true /MarksOffset 6 /MarksWeight 0.250000 /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PageMarksFile /RomanDefault /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /LeaveUntagged /UseDocumentBleed false >> ] /SyntheticBoldness 1.000000 >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice
HRE7271/Week 2/Using Adult Learning Theory for New-Hire Training.pdf
M PA EA Journal of Adult Education
V olume X X X V I, Number 1, Spring 2007
44
Using Adult Learning Theory for New-Hire Training
Chris A Woodard
Abstract
To test if adult learning theory can inform a training program for newly-hired employees in industry, a training program was set up using Knowles’ concepts of andragogy. Evaluation results from before and after the new training program indicate that the perceptions of those in the new training program changed in a positive direction. This indicates that the concept of andragogy does translate to the workplace.
Introduction
There is too often a difference between theory and practice. This is especially true in corporate America where practice must have a positive impact on the bottom line, and there is little room to experiment with theory that may or may not work in certain contexts. One such context is corporate new-hire training for those newly hired (new hire), and one such theory is adult learning. At the foundation of adult learning theory, Malcolm Knowles’ assumptions of the adult learner hold true for the self- directed person solving everyday life problems. However, it was not known if Knowles’ assumptions of the adult learner would translate into effective new-hire training at the corporate level. Based on the belief that andragogy would transfer to the context of new-hire training in the business world, the trainer at one company used Knowles’ assumptions of the adult learner, along with various other adult learning theories, as a guide to develop a new-hire training process with the following results.
Problem
The problem at this company was that the evaluations indicated the ______
Chris A. W oodard is a Senior Trainer, American Electric Power, Tulsa, Oklahoma.
45
training was inadequate and did not properly prepare new-hires to perform effectively on the job. To alleviate this problem the new-hire trainer was asked to create a new comprehensive training process that matched the complexity of the job and that provided the skills required for a new hires to do their job effectively. To determine the effectiveness of the new training process, evaluations from before the new training were compared to those after the new training model was implemented.
Context
The context in which this project took place was the new-hire customer service department in a large electric utility. Overall, the company employees around 20,000 employees and about 600 are telephone customer service representatives spread out over 6 states in the United States. In the customer service centers where the telephone representatives work, the training department trains approximately 200 new-hire employees each year.
Andragogy
Knowles’ (1980) andragogical model was originally based on four assumptions of adult learners and how they develop:
• Concept of learner--Their self-concept moves from one of being a dependant person to one who is self-directed.
• Role of learners’ experience--As individuals grow, they accumulate a reservoir of experience that becomes an increasingly rich resource for learning.
• Readiness to learn--Learners see education as a process for developing increased competence to achieve their full potential in life.
• Orientation to learning--As real life problems occur some learning situations require immediate attention. (pp. 43-44)
Knowles (1998) later added two additional andragogical assumptions. • Motivation--Adults tend to be more motivated to learn things that help
them solve problems in their lives or results in internal rewards. • Adults need to know why they need to learn something before
undertaking learning it. Knowing why an adult needs to learn something is the key to giving them a sense of volition about their learning. (p. 149)
46
Using Concepts of Andragogy
Concept of Learner: Some adults have a high need to be self-directed, but for the sake of consistency, the new-hire material has to ensure everything is covered. Time was allowed each day for the trainee to focus on whatever they felt they needed most. Based on individual need, the trainee had the autonomy to self-direct this additional learning experience.
Role of Learner’s Experience: Since not all new-hires have job specific experience and in order to ensure everyone has experiences to draw on, time was set aside for the new-hires to observe the job, and even to perform the job while sitting side-by-side with experienced employees. This “mentor” program was essential because it gave the trainees valuable experiences to use throughout the training program. The trainer often had trainees reflect upon these experiences.
Readiness to Learn: It is hard to know whether learners in the context of new-hire training are learning to achieve their full potential in life or learning the bare minimum just to get by. However, it is likely many trainees see new-hire training as part of the process for developing competence to achieve their full potential. This assumption was the underlying philosophy that permeated the entire training process initiative.
Orientation to Learning: In new-hire training, the trainee’s “problem” is that they know nothing about the new job. The goal of the trainer is to work with the trainees to help them immediately solve the problem of knowing nothing to becoming fully functional, productive employees.
Motivation: On the first day of training the desired behaviors and characteristics of a successful employee are communicated. Throughout the training program, these behaviors and characteristics are modeled through words and actions from everyone involved in the training. Additionally, the trainees continuously receive feedback in the form of positive reinforcement or correction. All of these things are designed to motivate the trainee. Whether or not this motivation is intrinsic or extrinsic is based on the individuals’ perception, values, and work ethic. The individual will decide whether or not the motivation is intrinsic, extrinsic, or most likely, a combination of both (Schein, 1980).
Adults Need to Know Why: Learning objectives were communicated before each section of the new-hire training manual. The objectives not only explained what was to be learned but also why it is important to learn that subject. Thoroughly explaining the objectives and desired outcomes address “why they are learning something.”
47
Table 1: Pre- and Post-Evaluation Results
Answer Pre Post Change Strongly Agree 14% 27% +48% Agree 57% 63% +10% Neutral 18% 10% -44%
Disagree 9% 0% -100% Strongly Disagree 2% 0% -100%
Results
As Table 1 indicates, there was substantial improvement on the evaluation question “My initial training provided the skills required to do my job effectively.” None of the respondents either strongly disagree or disagree with the statement after the new training model was implemented compared to 11% under the old training model. Ninety percent of the respondents either strongly agree or agree with the statement for the new training compared to 71% for the old.
Conclusion
Based on the evaluation results, the new-hires’ perception of training changed in a positive direction. In this case, theory and practice did go hand-in-hand. The logical conclusion, therefore, is that Knowles’ concept of andragogy and his notion of the adult learner does translate to the workplace to increase the effectiveness of new-hire training.
References
K nowles, M . S. (1980). The m odern practice of adult education: From andragogy
to pedagogy. New York: Follett.
K nowles, M . S., Holton, E.F. III, & Swanson, R.A. (1998). The adult learner.
H ouston: Gulf.
Schein, E. H. (1980). O rganizational psychology. Englewood Cliffs, NJ: Prentice-
H all.
HRE7271/Week 3/Promoting Student Metacognition.pdf
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 1/11
+
CBE-Life Sciences Education www.lifescied.org
doi: 10.1187/cbe.12-03-0033 CBE Life Sci Educ vol. 11 no. 2 113-120
Features
Approaches to Biology Teaching and Learning
Promoting Student Metacognition
Kimberly D. Tanner*
Affiliations
Learning how to learn cannot be left to students. It must be taught. (Gall et al., 1990)
Imagine yourself as the instructor of an introductory undergraduate biology course. Two students from your course independently visit your office the week after the first exam. Both students are biology majors. Both regularly attend class and submit their assignments on time. Both appear to be eager, dedicated, and genuine students who want to learn biology. During each of their office hours visits, you ask them to share how they prepared for the first exam. Their stories are strikingly different (inspired by Ertmer and Newby, 1996).
During office hours, Josephina expresses that she was happy the exam was on a Monday, because she had a lot of time to prepare the previous weekend. She shares that she started studying after work on Saturday evening and did not go out with friends that night. When queried, she also shares that she reread all of the assigned textbook material and made flashcards of the bold words in the text. She feels that she should have done well on the test, because she studied all Saturday night and all day on Sunday. She feels that she did everything she could do to prepare. That said, she is worried about what her grade will be, and she wants you to know that she studied really hard, so she should get a good grade on the exam.
Later in the week, Maya visits your office. When asked how she prepared for the first exam, she explains that she has regularly reviewed the PowerPoint slides each evening after class since the beginning of the term 4 weeks ago. She also read the assigned textbook pages weekly, but expresses that she spent most of her time comparing the ideas in the PowerPoint slides with the information in the textbook to see how they were similar and different. She found several places in which things seemed not to agree, which confused her. She kept a running list of these confusions each week. When you ask what she did with these confusions, she shares that she brought them to her weekly study group with peers from her course lab section. There, she says, she got most of her questions answered and lots of her confusions cleared up. She has come to office hours to ask you about a couple of things that she did not figure out before the exam that she thinks she probably missed. She is not too worried about her score on the exam, because most of the material related to problems and concepts that she felt had been thinking about a lot.
So, what is different about Josephina and Maya? No doubt many things, including their educational histories, their personalities, and more. However, one key difference in their approach to their studies is evident from their stories. They appear to be strikingly different in knowing how to learn, being able to monitor their own understanding, being reflective about what they understand and do not understand, and being able to strategize about how to resolve their confusions. They are different in their ability to use metacognitive approaches in their learning.
INTRODUCING METACOGNITION
The importance of metacognition in the process of learning is an old idea
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 2/11
The importance of metacognition in the process of learning is an old idea that can be traced from Socrates' questioning methods to Dewey's twentieth- century stance that we learn more from reflecting on our experiences than from the actual experiences themselves (Dewey, 1933). What is more recent is the coining of the term “metacognition” and the emergence of a metacognition research field in the last four decades. Credited to developmental psychologist John Flavell in a publication from the 1970s, metacognition is used in different disciplines in different ways, and a common, succinct definition appears to be elusive in the literature. Below is an excerpt from Flavell's original writing, as well as several additional definitions and conceptualizations from different sources:
Metacognition refers to one's knowledge concerning one's own cognitive processes or anything related to them, e.g., the learning- relevant properties of information or data. For example, I am engaging in metacognition if I notice that I am having more trouble learning A than B; if it strikes me that I should double check C before accepting it as fact. (Flavell, 1976)
Metacognition: awareness or analysis of one's own learning or thinking processes. (Merriam-Webster, 2012)
Metacognition also includes self-regulation—the ability to orchestrate one's learning: to plan, monitor success, and correct errors when appropriate—all necessary for effective intentional learning… Metacognition also refers to the ability to reflect on one's own performance. (National Research Council, 2000)
Students learn to monitor and direct their own progress, asking questions such as “What am I doing now?,” “Is it getting me anywhere?,” “What else could I be doing instead?” This general metacognitive level helps students avoid persevering in unproductive approaches… (Perkins and Salomon, 1989)
These multiple perspectives on what metacognition might entail—which expand on Flavell's original definition to include an emphasis on planning, monitoring, and evaluating one's own learning processes—are likely related to the relative youth of the metacognition research field and the associated growing pains of an emerging discipline (Flavell, 1979; Schraw, 1998). Delineation of distinct aspects of metacognition, development of tools for measuring these aspects, and strategies for teaching them to students are all active areas of inquiry among researchers across several social science disciplines (Zohar, 2009; Schraw et al., 2006). In addition, there are complex overlaps between metacognition research and other research arenas focused on self-regulated learning (an individual's ability to take control of his or her learning; Schraw et al., 2006) and self-efficacy (an individual's conceptualization of his or her own competency; Bandura, 1977). Because the goal of this feature is to translate ideas from other disciplines that may have immediate, practical relevance for biology education, I will leave these intriguing overlaps and areas of active inquiry for the exploration of interested readers.
So, let us reconsider Josephina and Maya. Their stories are likely familiar to anyone who has taught college biology even for a short period of time. And the reactions from faculty to these two kinds of students might be briefly summarized as exasperation with Josephina and elation with Maya. Faculty are often perplexed by students like Josephina, who do not seem to have mastered learning how to learn, and some faculty will assert that it is their job to “teach biology, not study strategies.” Yet metacognition, which represents more than just study skills, has been linked to improving thinking skills and promoting conceptual change in younger students (Nickerson et al., 1985; White and Gunstone, 1989; Georghiades, 2000). Additionally, there is evidence that improved metacognition is associated with promoting young students' overall academic success (Adey and Shayer, 1993; Kuhn and Pearsall, 1998). Evidence indicates that individuals with poor metacognitive skills perform less well academically than peers (Kruger, 1999; Dunning et al., 2003). But there remains much to be learned about the influence of metacognition on learning, especially among college-age students and within particular disciplinary contexts (e.g., biology vs. physics vs. music theory). So, how can we as biology educators use what is currently known about metacognition to our and our students' advantage to support biology
teaching and learning? What could integrating student metacognition into a
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 3/11
View this table: In this window In a new window
teaching and learning? What could integrating student metacognition into a college biology course look like? And how might active learning look different with more emphasis on metacognition?
USING METACOGNITION TO HELP STUDENTS LEARN TO THINK LIKE BIOLOGISTS
To make an individual metacognitively aware is to ensure that the individual has learned how to learn. (Garner, 1988)
With the recent publication of the 2011 American Association for the Advancement of Science (AAAS) report, Vision and Change for Undergraduate Biology Education, and the 2012 President's Council of Advisors on Science and Technology (PCAST) report, Engage to Excel, considerable attention is being paid to transforming the learning experiences of undergraduate students in the sciences (AAAS, 2011; PCAST, 2012). An example of our collective aspirations as a biology education community for what we want students to be able to do at the conclusion of their undergraduate biology education is stated as follows in Vision and Change:
Biology in the 21st century requires that undergraduates learn how to integrate concepts across levels of organization and complexity and to synthesize and analyze information that connects conceptual domains.
This aspiration can be approximated by the assertion that we want undergraduate learning experiences to help students learn to think like biologists. Promoting student metacognition—teaching students to think about how they are thinking about biology and how they approach learning about biology—would seem to be a useful strategy in striving to reach these kinds of goals for students (NRC, 2000; D'Avanzo, 2003; Crowe et al., 2008). Below, I describe potential approaches to increasing attention to metacognition in undergraduate biology classrooms, including: 1) explicitly teaching students metacognitive strategies, and 2) more generally building a classroom culture grounded in metacognitive strategies by modifying what we are already doing.
EXPLICITLY TEACHING STUDENTS METACOGNITIVE STRATEGIES IN BIOLOGY COURSES
There is a need to teach for metacognitive knowledge explicitly…we are continually surprised at the number of students who come to college having very little metacognitive knowledge; knowledge about different strategies, different cognitive tasks, and particularly, accurate knowledge about themselves. (Pintrich, 2002)
Teaching students to use metacognition to understand how they are thinking about biology provides an important step on the path to thinking like a biologist (AAAS, 2011). In the context of undergraduate biology teaching, this need not take much time, and it is an effort that is in the service of learners and learning, as well as teachers and teaching. Table 1 provides examples of self-questions that metacognitive undergraduate biology learners might ask in the process of planning, monitoring, and evaluating their learning in the context of a single class session, a homework assignment, an exam, or an entire course. While this collection of questions by no means represents the entire landscape of what metacognition could involve, it does provide starting points for faculty who wish to talk with students explicitly about metacognitive strategies. These questions can be shared directly with students and/or embedded into particular assignments. Several examples of how these student self-questions can be explicitly used in teaching a biology course are considered below.
Table 1.
Sample self-questions to promote student metacognition
about learning
Preassessments—Encouraging Students to Examine Their Current Thinking
The importance of instructors knowing what students are thinking about a
a
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 4/11
The importance of instructors knowing what students are thinking about a topic prior to trying to teach them something new has been written about extensively. However, preassessment can also be helpful for the learner and is a wonderful opportunity for promoting metacognition among students. “What do I already know about this topic that could guide my learning?” is an example of a self-question that it at the core of most preassessments used by instructors. It takes no more than a few simple statements by an instructor to transform an existing preassessment prompt—be it a homework assignment, an index card, or a clicker question—into a metacognitive activity for students, directing them not only to complete the task as part of the course, but also to be metacognitive in doing so and to use the information given on the preassessment to help them begin thoughtful planning of how they might approach learning this new idea.
The Muddiest Point—Giving Students Practice in Identifying Confusions
One long-standing, active-learning strategy that has been used across many disciplines in classrooms of any size is the Muddiest Point (Angelo and Cross, 1993). Usually done as an in-class, quick-write on an index card, students are asked to write for a brief period of time—1, 3, or 5 min, usually at the end of a class session—to address the self-question “What was most confusing to me about the material being explored in class today?” Similar to preassessments, the Muddiest Point is incredibly useful to instructors in gauging what was challenging for or unclear to students. However, the oft- missed opportunity is for this activity to explicitly charge students to identify what they are confused about and then to embrace, work on, and wrestle with that confusion as they participate in the learning activities of the course. For many students, it is an unusual experience for an instructor to invite them to share confusions aloud in a science classroom, in which the conversation is often limited to students who are offering the scientifically most accurate answer. Students who are confused risk scorn by raising a question or revealing confusion, unless instructors explicitly invite the sharing of confusions and create a safe learning environment in which to do so. Regular use of the Muddiest Point in classrooms, which requires only a few minutes, sets a tone that confusion is a part of learning and that articulating confusions is not done solely to inform the instructor, but also to inform students themselves; students can use identified confusions to drive their independent learning or to generate dialogue in review sessions.
Retrospective Postassessments—Pushing Students to Recognize Conceptual Change
Cognitive psychologists and science education researchers conceptualize learning as a student-centered activity in which students change their ideas about a topic (Posner et al., 1982). This view implies that students will not really learn new information if they do not go through a metacognitive realization that requires them to examine how they thought about the topic before and how they are thinking differently about that topic now; this is similar to Dewey's assertion that reflection on an experience is the key step in learning (Dewey, 1933). A simple tool for explicitly charging students to think about how their ideas are (or are not) changing is a retrospective postassessment. As its moniker implies, this tool is a postassessment and occurs after learning may have taken place. It is retrospective, in that students are asked to recall how they were thinking about the topic prior to course learning activities and compare that with how they are now thinking about the same topic afterward. As an example, students might be asked to complete the phrase: “Before this course, I thought evolution was… Now I think that evolution is…” Alternatively, they may be asked to write about three ways in which their thinking about a given topic has changed over a given period of time. Either of these explicit approaches to teaching metacognition is a mechanism of training students to self-question, “How is my thinking changing (or not changing) over time?”
Reflective Journals—Providing a Forum in Which Students Monitor Their Own Thinking
In the case of Josephina, one of the metacognitive strategies that she simply does not seem to possess is to be analytical about what did or did not work well for her in studying for the last exam, and to then use that information in preparing for future exams. Instructors can assign something as simple as a
low-stakes, low-points writing assignment after a first exam, asking
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 5/11
View this table: In this window In a new window
low-stakes, low-points writing assignment after a first exam, asking students to reflect and write a brief letter to their future selves covering: “What about my exam preparation worked well that I should remember to do next time? What did not work so well that I should not do next time or that I should change?” If an instructor assigns such writing, either in conjunction with an exam or as part of a specific reflective writing assignment, he or she is explicitly giving students a strategy for developing metacognitive approaches, as well as practice using that approach in the context of their disciplinary course. To extend this, instructors can also assign a reread of this writing before the next exam and a second writing assignment on how well students followed their own advice to themselves. In addition, students can be asked to share their strategies with fellow students and to identify at least two new exam preparation strategies used by their peers. If such writing about their metacognitive, thinking, and learning strategies is done regularly, students can create a reflective/biologist journal and can be rewarded with some form of credit, as for other course activities.
BUILDING A BIOLOGY CLASSROOM CULTURE GROUNDED IN METACOGNITION
Making the discussion of metacognitive knowledge part of the everyday discourse of the classroom helps foster a language for students to talk about their own cognition and learning. (Pintrich, 2002)
While using specific individual assignments to teach students metacognitive strategies is one explicit approach, there are more subtle ways that metacognition can be integrated into the fabric of any course and become part of the everyday language of both teacher and students. This is particularly useful in helping students to become aware of when it is appropriate to apply their own metacognitive strategies—for example, identifying confusions—that they may have learned through previous assignments. The point at which students have both learned metacognitive strategies and have become aware of when to apply these strategies is hypothetically the point at which they have matured into lifelong learners within their disciplines. Below are several starting points for thinking about how the language and habit of metacognition could become part of everyday classroom culture. In addition, Table 2 provides some sample prompts that can be used to add a metacognitive aspect to learning activities that may already be in use in your teaching, such as pair discussions after clicker questions, a variety of types of homework assignments, and the ever-present exams and quizzes. Simply adding one additional question or using some of the language in the table in making the assignment can demonstrate to students the value you as an instructor place on their efforts to develop metacognitive habits of mind as a biology student. Below are four general ways that instructors might build a classroom culture that promotes metacognition and conveys that culture to students.
Table 2.
Sample prompts for integrating metacognition into course
activities
Give Students License to Identify Confusions within the Classroom Culture
While most faculty welcome questions from students in or out of class, it is generally not in the culture of college science courses for students to share their confusions; rather, there is a focus on right answers and on being scientifically correct (Tobias, 1990; Steele and Aronson, 1995; Seymour and Hewitt, 1997). Simply giving students permission to be confused is one way to provide the impetus for students to be metacognitive and to ask themselves what they do not understand. Sometimes all that is required is for an instructor to explicitly share with students that an upcoming topic has proved confusing to students in the past and that confusion is to be expected. Even slight alterations in the verbal directions for course activities could serve to give students the license to share and display what is confusing to them, as opposed to hiding it. For example, during in-class pair discussion of a clicker question, the direction to not only compare chosen answers with a colleague but also to pose one question that relates to
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 6/11
answers with a colleague but also to pose one question that relates to something you found confusing about the question could immediately increase the willingness and comfort level of students to discuss confusion, which demands them to be metacognitive during the activity.
Integrating Reflection into Credited Course Work
Integrating reflection into any course can be achieved by a relatively simple tweaking of existing course assignments. In addition to having students respond to homework questions or solve problems, instructors need only add one or more questions that push students to consider their own thinking (see Table 1). These questions can be as simple as “What was most challenging to you about this assignment?” to “What questions arose during your work on this assignment that you had not considered before?” The instructor's decision to make these kinds of questions part of an assignment —and part of the grading scheme for the assignment—can prompt students to bring a more metacognitive stance to their everyday coursework. Similarly, for assignments that involve diagramming or concept mapping, instructors can encourage (or require) students to indicate in their work what questions arose and which concepts they found most confusing. In this more subtle approach, what changes is not the assignment itself, but the nature of the assignment.
Metacognitive Modeling by the Instructor for Students
As a professional, practicing biologist, it can be almost impossible to remember a time when you did not think biologically, to remember the nature of your own biological confusions as a student, and to be able to offer up self-reflective examples of your own transitions in thinking for your students. As researchers, we think metacognitively all the time, reflecting on our current understanding of our research system, what the burning questions are, and how our thinking has changed over the years with new data. Showing students explicitly how you, as a biologist, think procedurally in solving a problem—how you start, how you decide what to do first and then next, how you check your work, how you know when you are done—is one example of metacognitive modeling. A teaching colleague of mine shared that he was perplexed as to why students were unable to make accurate predictions about the proportions of different phenotypes in the offspring from a specific cross, as required in response to a homework question. But when he asked all of the students to do a problem in class one day, he noticed that only a minority of them were drawing a Punnett square. When he asked several students why they did not have pencil and paper out, they said they thought they should just be able to do it mentally. My colleague then went to the stage and proceeded to metacognitively present how he thinks through a problem similar to their homework question. His first step—always, even as a practiced biologist—is to get out a pencil and a piece of paper and to translate the problem into a Punnett square! Showing students how we think about a biology concept, or how biologists more generally have thought about a concept over the history of biology, illustrates how the entire field of biology has changed its collective understanding. For example, what biologists think about how plants grow and build mass has undergone multiple revisions over time. In addition, our collective understanding of how genetic information is transferred from parent to offspring across all species is ripe for analysis of how “thinking like a biologist” looked different in Mendel's time versus the modern era.
ON INSTRUCTOR METACOGNITION AND BIOLOGY TEACHING
We began this exploration of metacognition by considering two contrasting students—Josephina and Maya. Now, imagine that you have the opportunity to talk to two of your biology faculty colleagues about their approaches to teaching. Both are research-active, full professors in biology. Both regularly teach introductory courses for biology majors. Both appear to be genuinely eager to help their students succeed in their biology courses. In your conversation with each of them, you begin by asking them about how their teaching is going this semester. In addition, you ask each of them how they prepare for class each week. Their stories are strikingly different.
Kara expresses dissatisfaction with the students in her upper-division biology course. She thinks that the students are getting worse every year, even though she works harder and harder to bring them more cutting-edge
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 7/11
View this table: In this window In a new window
even though she works harder and harder to bring them more cutting-edge research in the field. She shares that she has committed to updating all of her PowerPoint lectures this semester, even though she already has tenure, and has often stayed up very late the night before to make sure that her slides are really clear. When queried about how she gets insights into how students are thinking, she shares that she has added an additional exam between the mid-term and the final to motivate students to keep up with the reading. She is also very frustrated that no students come to her office hours. She feels like she is doing everything she can to help students understand the material, but they do not seem to be willing to work as hard in a course as she did when she was an undergraduate. She is worried about her student evaluation scores, which have declined over the years, and she thinks it is not fair to be evaluated by students who do not seem to care about their learning.
In contrast, another faculty colleague, Aerial, seeks your input on a new series of clicker questions she has developed as the basis of a classroom activity she is trying out with her students the next day. From prior experience, she knows that few students are able to connect the ideas of photosynthesis with those of climate change, and she wants to start her new unit on transformation of matter and energy with an assessment question that will really get students thinking about their prior ideas. She has changed this unit of her course each time she has taught it over the last several years, based on all the information she has collected from students about their ideas on the topic. She is aware that the more she knows about how students are thinking, the more ideas she has about new things to try in her teaching. She also shares that many of the homework writing assignments students have already submitted before the midnight deadline show that they have identified exactly the confusions she wants to alert them to tomorrow! When you ask her if she is concerned about how students will react to her new clicker-based classroom activity, she is not too worried. She regularly shares with students her own rationale for why she has developed a particular learning activity for them and gets their feedback on it through an index card or homework assignments so that she will have insights for the next time she teaches the same activity.
So, what is different about Kara and Aerial? No doubt many things, but one key difference is their ability as faculty members to be metacognitive about their teaching. Similar to the contrast between Josephina and Maya's abilities to be metacognitive about their learning, there is a difference in the extent to which each of these faculty members is thinking about how they think about their teaching. While instructors no doubt bring a deeply metacognitive approach to their field of scientific research, cultivating a metacognitive lens toward one's teaching does not appear to automatically or easily transfer. However, developing a metacognitive stance toward one's own teaching— thinking about how you think about teaching—can be a wonderfully natural entry point into iteratively changing one's own teaching practice. Self- analysis about one's own ideas about teaching could include: What assumptions do I hold about students? To what extent do I have evidence for those assumptions? Why do I make the instructional decisions that I make? What do I know about teaching? What would I like to learn? What am I confused about? These analyses can also become more specific to particular granularities, ranging from an individual class session to the scope of an entire course. Table 3 provides some starting points in the form of sample self-questions for faculty that may aid them in becoming more metacognitive about their teaching.
Table 3.
Sample self-questions to promote faculty metacognition
about teaching
Postscript 1: Using Metacognition to Make the Most of Active Learning —Learning from History
As stated above, attention to improving undergraduate biology education is high at present, and active-learning strategies are a central approach among suggested changes (AAAS, 2011). However, what different instructors mean
by active learning and what active learning actually looks like in a different
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 8/11
by active learning and what active learning actually looks like in a different classrooms has not been well documented or investigated (Ebert-May et al., 2011; Tanner, 2011). Metacognition is not generally central, or even included, in discussions and articles about active learning. In fact, the term “active learning” is prominent and often used in the Vision and Change for Undergraduate Biology Education report, whereas “metacognition” does not make an appearance (AAAS, 2011). One possible difference in the effectiveness of active-learning pedagogies in the hands of different instructors may lie in the extent to which these instructors consider student metacognition when they implement active-learning strategies.
During the 1980s, K–12 science education experienced a period of intense focus on hands-on learning, which might be considered parallel to the recent rise in emphasis on active learning in undergraduate biology education. However, there was a general dissatisfaction, with reports that K–12 students were doing a lot of activities but not necessarily very much thinking. The hands-on era in K–12 science education was followed a shift in both the language and emphasis in policy documents to minds-on and inquiry-based learning in the 1990s (National Research Council, 1996). One aspect of this shift in emphasis in K–12 science education reform was an increased emphasis on student metacognition, students thinking about what they were thinking while they were doing, as opposed to just doing hands-on, active things without the thinking. As such, attention to student metacognition may be especially salient at this moment in the history of the undergraduate biology education revolution. To avoid repeating the trajectory of K–12 science education reform, explicit attention to integrating metacognition into undergraduate biology classrooms could help keep a focus on the learning part of active learning.
Postscript 2: On Thinking about Your Thinking about This Article…
Why, in the first place, did you choose to read this feature? Was it the title? The term “metacognition”? What did you already know or think about metacognition before reading this feature? How, if at all, have your ideas changed? What in this article was most intriguing to you? What are you thinking about in terms of how you might use those ideas? What in the article was most confusing? How do you plan to follow up on that to clarify your ideas and learn more? Will you? Why or why not? As you read, what, if anything, came to mind that you already do with your students that may promote their use of metacognitive strategies? Are you thinking about how explicit you are with your students about the thinking strategies and processes that you yourself use as a practicing biologist? What is the most important thought you had in reading this article? Did it even have anything to do with metacognition?
Footnotes
↵ *Address correspondence to: Kimberly Tanner ([email protected]).
“ASCB®” and “The American Society for Cell Biology®” are registered trademarks of The American Society of Cell Biology.
REFERENCES
Adey P, Shayer M. An exploration of long-term far-transfer effects following an extended intervention program in the high school science curriculum. Cogn Instr 1993;11:1-29. CrossRef Google Scholar American Association for the Advancement of Science. Vision and Change: A Call to Action, Final Report . Washington, DC: AAAS; 2011. http://visionandchange.org/finalreport.
Angelo T, Cross K. Classroom Assessment Techniques: A Handbook for College Teachers . 2nd ed. San Francisco, CA: Jossey-Bass; 1993.
Google Scholar
Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 1977;84:191-215. CrossRef Medline Google Scholar Coutinho SA. The relationship between goals, metacognition, and academic success. Educate 2007;7:39-47. Google Scholar Crowe A, Dirks C, Wenderoth MP. Biology in Bloom: implementing Bloom's Taxonomy to enhance student learning in biology. CBE Life Sci Educ 2008;7:368-381. Abstract/FREE Full Text
D'Avanzo C. Application of research on learning to college teaching:
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 9/11
D'Avanzo C. Application of research on learning to college teaching: ecological examples. BioSciences 2003;53:1121-1128. CrossRef
Google Scholar
Dewey J. How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process . Boston: Heath; 1933. Google Scholar Dunning D, Johnson K, Ehrlinger J, Kruger J. Why people fail to recognize their own incompetence. Curr Directions Psychol Sci 2003;12:83-87.
CrossRef Google Scholar
Ebert-May D, Derting TL, Hodder J, Momsen JL, Long TM, Jardeleza SE. What we say is not what we do: effective evaluation of faculty professional
development programs. BioScience 2011;61:550-558. CrossRef Google Scholar
Ertmer PA, Newby TJ. The expert learner: strategic, self-regulated, and reflective. Instr Sci 1996;24:1-24. CrossRef Google Scholar Flavell JH. Metacognition and cognitive monitoring: a new area of psychological inquiry. Am Psychol 1979;34:906-911. CrossRef
Google Scholar
Gall MD, Gall JP, Jacobsen DR, Bullock TL. Tools for Learning: A Guide to Teaching Study Skills . Alexandria, VA: Association for Supervision and Curriculum Development; 1990. Google Scholar
Garner R. Metacognition and Reading Comprehension . Norwood, NJ: Ablex; 1988. Google Scholar
Georghiades G. Beyond conceptual change learning in science education: focus on transfer, durability, and metacognition. Educ Res 2000;42:119- 139. CrossRef Google Scholar
Kruger J, Dunning D. Unskilled and unaware of it: how differences in recognizing one's own incompetence lead to inflated self-assessments. J Personality Soc Psychol 1999;77:1121-1134. CrossRef Medline
Google Scholar
Kuhn D, Pearsall S. Relations between metastrategic knowledge and strategic performance. Cogn Dev 1998;13:227-247. CrossRef
Google Scholar
Merriam-Webster. 2012. www.merriam- webster.com/dictionary/metacognition (accessed 14 March 2012).
National Research Council (NRC). National Science Education Standards . Washington, DC: National Academies Press; 1996.
NRC. How People Learn: Brain, Mind, Experience, and School . Washington, DC: National Academies Press; 2000. Nickerson RS, Perkins DN, Smith EE. The Teaching of Thinking . Hillsdale, NJ: Lawrence Erlbaum; 1985. Google Scholar
Perkins DN, Salomon G. Are cognitive skills context-bound? Educ Res 1989;18:16-25. Google Scholar Pintrich P. The role of metacognitive knowledge in learning, teaching, and assessing. Theory Pract 2002;41:219-226. CrossRef Google Scholar Posner GJ, Strike KA, Hewson PW, Gertzog WA. Accommodation of a scientific conception: towards a theory of conceptual change. Sci Educ 1982;66:211-227. CrossRef Google Scholar President's Council of Advisors on Science and Technology. Report to the President—Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering, and Mathematics. 2012. www.whitehouse.gov/administration/eop/ostp/pcast (accessed 13 March 2012). Schraw G. Promoting general metacognition awareness. Instr Sci 1998;26:113-125. Google Scholar Schraw G, Crippen K, Hartley K. Promoting self-regulation in science education: metacognition as part of a broader perspective on learning. Res Sci Educ 2006;36:111-139. CrossRef Google Scholar Seymour E, Hewitt NM. Talking About Leaving: Why Undergraduates Leave the Sciences . Boulder, CO: Westview; 1997. Google Scholar Steele CM, Aronson J. Stereotype threat and the intellectual test performance of African Americans. J Pers Soc Psychol 1995;69:797-811.
CrossRef Medline Google Scholar
Tanner KD. Reconsidering “what works”. CBE Life Sci Educ 2011;10:329- 333. FREE Full Text
Tobias S. They're not dumb. They're different. A new tier of talent for science. Change 1990;22:11-30. Google Scholar White RT, Gunstone RF. Metalearning and conceptual change. Int J Sci Educ 1989;11:577-586. CrossRef Google Scholar Zohar. Paving a clear path in a thick forest: a conceptual analysis of a metacognitive component. Metacognition Learning 2009;4:177-195.
Expand
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 10/11
metacognitive component. Metacognition Learning 2009;4:177-195. CrossRef
Articles citing this article
Characterizing Student Perceptions of and Buy-In toward Common Formative Assessment Techniques
cellbioed 2016 15:ar73 Abstract Full Text Full Text (PDF)
Effective Educational Videos: Principles and Guidelines for Maximizing Student Learning from Video Content
cellbioed 2016 15:es6 Abstract Full Text Full Text (PDF)
Scientist Spotlight Homework Assignments Shift Students Stereotypes of Scientists and Enhance Science Identity in a Diverse Introductory Science Class
cellbioed 2016 15:ar47 Abstract Full Text Full Text (PDF)
Colleagues as Change Agents: How Department Networks and Opinion Leaders Influence Teaching at a Single Research University
cellbioed 2016 15:ar15 Abstract Full Text Full Text (PDF)
Writing on the board as students' preferred teaching modality in a physiology course
Adv. Physiol. Educ. 2016 40:229-233 Abstract Full Text Full Text (PDF)
Development of a framework for graph choice and construction Adv. Physiol. Educ. 2016 40:123-128
Full Text Full Text (PDF)
Increasing the Use of Student-Centered Pedagogies from Moderate to High Improves Student Learning and Attitudes about Biology
cellbioed 2016 15:ar3 Abstract Full Text Full Text (PDF)
A Portal into Biology Education: An Annotated List of Commonly Encountered Terms
cellbioed 2015 14:fe2 Full Text Full Text (PDF)
Test-Enhanced Learning: The Potential for Testing to Promote Greater Learning in Undergraduate Science Courses
cellbioed 2015 14:es4 Abstract Full Text Full Text (PDF)
Scientific Teaching: Defining a Taxonomy of Observable Practices cellbioed 2015 14:ar9
Abstract Full Text Full Text (PDF)
Teaching Assistant Professional Development in Biology: Designed for and Driven by Multidimensional Data
cellbioed 2014 13:212-223 Abstract Full Text Full Text (PDF)
Student Perceived and Determined Knowledge of Biology Concepts in an Upper-Level Biology Course
cellbioed 2014 13:322-330 Abstract Full Text Full Text (PDF)
Writing Assignments with a Metacognitive Component Enhance Learning in a Large Introductory Biology Course
cellbioed 2014 13:311-321
Expand
12/1/2016 Promoting Student Metacognition
http://www.lifescied.org/content/11/2/113.full 11/11
cellbioed 2014 13:311-321 Abstract Full Text Full Text (PDF)
Considering the Role of Affect in Learning: Monitoring Students' Self-Efficacy, Sense of Belonging, and Science Identity
cellbioed 2014 13:6-15 Full Text Full Text (PDF)
How Can We Improve Problem Solving in Undergraduate Biology? Applying Lessons from 30 Years of Physics Education Research
cellbioed 2013 12:153-161 Abstract Full Text Full Text (PDF)
CREATE Cornerstone: Introduction to Scientific Thinking, a New Course for STEM-Interested Freshmen, Demystifies Scientific Thinking through Analysis of Scientific Literature
cellbioed 2013 12:59-72 Abstract Full Text Full Text (PDF)
Using Learning Principle in Faculty Development Workshops cellbioed 2012 11:335-336
Full Text Full Text (PDF)
Expand
HRE7271/Week 3/The 5 Step Training Model.pdf
71
What if we offered you a simple, easy-to-use training tool that significantly increased the probability of learning success with any group, of any size, on any topic? Would you want it? In this chapter, that’s exactly what we present to you. No strings attached, no caveats, no maybes or sometimes. In a way, this chap- ter is the heart and soul of Telling Ain’t Training: Updated, Expanded, and Enhanced. We’re not diminishing the importance of the other chapters, but things do come together here. So, prepare yourself.
First, here’s a brief review. In chapter 1, we set up some challenges to establish the central theme of this book: Telling ain’t training. We also wanted you to immedi- ately experience the style of this volume: fun, challenging, participative, and con- versational.
Chapter 2 provided you with some basic vocabulary—training, instruction, edu- cation, and learning—and presented a focus, a mantra: “learner centered, per- formance based.” It also stressed that the medium isn’t the message and that the
Chapter highlights:
u Six universal principles from learning research
u Model for structuring training
u Worksheets to guide and support application
u Means to retrofit existing training to the model.
Chapter 6
A Five-Step Model for Creating Terrific Training Sessions
u Section 2: What You Must Know to Be a Better Trainer
72
content of this book applies to all forms of instruction, regardless of the delivery vehicle. In chapter 3, we visited the senses, the brain, and memory to acquire an understanding of the learning characteristics and limitations of our learners.
Chapter 4 focused on why we often have difficulty communicating our knowledge to our learners, even though we know a lot. It emphasized how differ ently experts and novices process information and described the fundamental dis tinctions be- tween declarative and procedural knowledge with all of the inherent implications. Finally, chapter 5 provided a structured overview of adult learning and exempli- fied four key adult learning principles.
The stage now is set for building effective learning sessions. You have had enough information and argumentation to convince you that we require a structuring mech anism that differs from the one we observe in most work settings. Where do we turn for this? Once again, research on learning helps direct us.
Six Universal Principles From Research on Learning
How would you classify yourself as a learner?
£ more auditory
£ more focused on details
£ more visual
£ more focused on the whole
£ more social
£ more right brain
£ more independent
£ more left brain.
When we observe individuals at work and play, we notice differences among them. Each person appears to possess a unique set of capabilities and traits that sets him or her apart. We naturally assume that they have their own style of learning. Also, our observations suggest that, ideally, we should tailor our learning sessions to each learner. Obviously, this is an awesome challenge and most likely not feasible, espe cially with so many learners and so few resources.
What then can we do? Must we compromise? Must we accept less?
We have good news and bad news to share with you, depending on your point of view. We have boxed each of them below so that you can self-select the one you prefer.
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
73
Now that we have given you both the good and bad news, we arrive at a single con clusion: If we can derive some overall, “universal” principles from research on learn ing, we can mold them into a model for teaching most learners most subjects with a relatively high success rate. What are those universals? They make such good sense that they are almost embarrassing to share.
Here are six words that sum up a lot of findings from research on learning: why, what, structure, response, feedback, and reward. Let’s examine each of them.
Why As reasonable as it may seem, if the learner knows “why” he or she is supposed to learn something and the reason makes sense to—is valued by—the learner, the prob ability of learning increases. This sounds similar to the readiness principle from the previous chapter. Readiness suggests that the adult learner learns more easily if his or her mind is open and ready to take in new information. The key is to show what’s in it for the learner.3
Research in which different learner groups received instruction with and without a meaningful “why” produced different learning results. In the research studies, “why” is frequently represented by the terms “expectancy value” or “task value,” referring to what the learners perceive investing in the learning effort offers them. Groups with strong rationales that convincingly explained how the learners would benefit from the instruction paid closer attention and retained what they
The Bad News
Sorry. We humans are not as unique as we like to think we are. Research in learning indicates that there are significant differ- ences in the way individual learners are affected by different types of instructional approaches. However, the detectable dif- ferences in the research findings do not translate into a major overall impact on learning. We are alike in more ways than we are different.1
The Good News
The good news is akin to the bad. As much as we would like to believe that each of us is incredibly unique, unless we have some form of perceptual or cogni- tive disability, we are all very much alike in how we perceive, process, store, and retrieve information. Well-designed and well-delivered instruction seems to have a broadly similar impact. This allows us to design sound instruction based on a universal set of principles and to achieve a high degree of effectiveness with a wide variety of learners.2
u Section 2: What You Must Know to Be a Better Trainer
74
had learned more accurately. This appeared to be true regardless of the type of learner. The clearer and more mean ingful the “why” offered, the better and more long lasting the learning.4
What There’s an old saying, “If you don’t know where you’re going, you’ll probably end up someplace else.” This is true also of learning. Have you ever been in a class in which the instructor/teacher/professor wandered aimlessly through the course material? You sat there trying to figure out where this person was heading, and you felt lost. Research on learning demonstrates the value of clarifying to the learners what it is they will be able to do by the end of the lesson, module, or course. Such early infor mation acts as a set of guideposts or a map. The clearer and more mean- ingful it is for the learners, the higher the probability they will learn it.
However, this should not be confused with provision of specific instructional ob- jectives at the front end of a course when the objectives may be meaningless to the learners. Studies done on “specific instructional objectives,” their use, and their placement in instruction had confusing and contradictory results.5
Structure Examine the array of symbols below for 15 seconds.
Ready? Go!
Stop! Cover the array with a piece of paper.
Now reproduce the array in the same order in the box.
Done all you can? Compare the two arrays and give yourself one point for
each symbol you placed in the correct sequence. The maximum number of
points is 25.
Jot your score down here.
Now, repeat the exercise using the array below. Once again,
you will have 15 seconds to “learn” it. Ready? Go!
$?$*#*$?£*£##?$?*££?*#£#$
$$$$$ ????? ***** £££££ #####
75
Cover the array and reproduce what you remember in the space below.
Then score yourself again. As before, you get one point for each symbol
placed in the correct sequence.
Compare arrays. Enter your score in the box.
Let’s examine the results. Did you do better in the first or the second trial? When we try this out with adult learners, we rarely discover scores above four or five in the case where the symbols are all jumbled up.
However, when these same symbols are placed in an easy-to-understand, structured order, most people score a perfect 25. Amazing! Same symbols (or content) and different structures produce dramatically different results.
Humans seek order. Where there is none, they will create it artificially. Think about gazing at clouds. Don’t you see shapes in what are really random patterns? And what about the man in the moon? The research tells us that the clearer the structure of the content is for the learners, the more easily they will grasp and retain it.6
Here’s one more example of this all-important structure issue. Imagine that we offered you $10,000 to name all the states of the United States without using any references or getting any assistance. One error, and you receive nothing. What would you do to be 100 percent sure you get all the states right? Check off the most prob able strategy for you. If none fit, add your own.
£ Just randomly name them as they pop into my mind.
£ Use the alphabet as a guide and name them alphabetically.
£ Break up the United States into regions and name states by region.
£ Start on the east (or west) coast and name all the states in order down
the seaboard and along the southern and northern borders. Then fill in
spiraling toward the center.
£ Use a rhyme, song, or other memory device to organize my recollection
of all the states.
£ My method:_____________________________.
We have asked this of hundreds of adult learners. Every one of them selects some structured and systematically organized method. Not one person chooses to name them randomly. The need to bring order or structure to what we deliberately try to learn and recall is universal among all types of learners, although the nature of the structure can vary.
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
u Section 2: What You Must Know to Be a Better Trainer
76
Response The more learners actively respond to learning the content, the better they learn and retain it. Response can take the form of answering a question, filling in a blank, labeling something, solving a problem, making a decision, or even discuss- ing and arguing. It can take any form that elicits an active response to learning the content. Before we share a little more about the research on active responding, here’s a quick challenge. Check off your choice below.
£ Learners learn better if the response they emit is out loud or written
down—an overt response.
£ Learners learn better if the response they emit is in their heads—a silent
or covert response.
£ Learners learn better if they respond. There is no significant difference
between overt or covert responding.
Ready for what may appear to be the surprising answer? The correct one is the sec- ond choice. Almost everyone selects the first choice, but what the research shows is that active responding is the critical ingredient. What is also important is that the re- sponse be a meaningful one. We have seen so-called interactive e-learning in which learners move objects; click on items; and enter numbers, letters, and even words that have no meaning with respect to what they are supposed to be mastering. This is empty responding. It has some limited value in that it may maintain the learners’ attention for a while, but it does little to clarify meaning or assist retention.
Examples of this, both live and mediated, are learning games in which the gaming aspect becomes so dominant that the learning content fades out. The response is about the game, not the content, and it ceases to be relevant or meaningful.
Here is a note on the research concerning active responding and covert versus overt responses. Most of the studies were conducted in the 1960s and 1970s. We decided to delve back into these to reassure ourselves about what we were affirming. Sure enough, the preponderance of research findings supports covert responding, mostly because the mental engagement compared favorably with some empty types of overt respond- ing (for example, raising hands, clicking on something, or repeating text). Two con- clusions emerge here. First, response must be meaningful. Second, there must be an element of reflection before deciding on a response.7
Concerning meaningfulness and its importance, think of yourself performing rou tine tasks in which you are responding but are no longer mentally engaged. Have any of the following ever happened to you? Check off those that you have experienced.
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
77
£ You’ve driven for several minutes and then suddenly realized you were on
“automatic” and can’t remember what you’ve done or how you ended up on a
frequently traveled route you didn’t intend to take.
£ You performed your morning routine (shower, shave/make-up, hair, and so
forth), and then had to check whether or not you had put on deodorant.
£ You studied for an exam and read several paragraphs or pages only to realize you
can’t recall a thing about what you’ve just read.
£ You’ve been introduced to new people at a social event, smiled, shaken hands,
and then noticed that you couldn’t remember the names of the people you had
just met.
£ You finished your meal then couldn’t remember what you just ate.
Both of us checked off all the items, so don’t worry if you felt that you were begin- ning to lose your mind. You were simply on automatic, a normal mechanism that allows you to perform unconsciously. The problem is that during these periods you are not mentally engaged in your responses. No new learning occurs. Even with a gun to your head, you cannot recall what you did although you responded appro- priately. Active, conscious response during initial learning—overt or covert—is es- sential for comprehension and recall, but the learner must be completely mentally engaged.
Feedback Feedback is one of the most powerful mechanisms for learning.8 The problem is that a lot of myths are associated with feedback. Feedback is information that learn ers receive about how on or off target they are (for example, in identifying a compo nent of a system, describing a process, solving a problem, or throwing a curve ball). The learner responds in some way to a critical part of the learning or to all of the learning elements that lead to objective attainment. Feedback comes to the learners from an instructor or from the environment that informs them how on or off target their responses have been. This helps the learners to adjust or continue the responding. From an instructional perspective, feedback should be either corrective (to let the learner alter responses) or confirming (to let the learner know that he or she has attained the partial or complete objective).
Here’s what research tells us about feedback:
u Feedback that the learner perceives as directed toward the task helps improve performance.
u Feedback that the learner perceives as a criticism of himself or herself tends to hinder or reduce performance.
u Section 2: What You Must Know to Be a Better Trainer
78
u Immediate feedback helps improve performance on simple tasks. u Delayed feedback seems to be more effective on what learners perceive as
complex tasks (feedback given too soon can confuse learners by overload- ing short-term memory).
u Frequent and specific feedback helps improve performance. However, if the feedback is too detailed or specific (for example, “In your golf swing, alter the angle of your elbow by 11 degrees, turn your left foot out by four degrees and advance it two inches, adjust the angle of the front surface of your club 2 degrees…”), it confuses the learner and may have an adverse effect on performance.
Reward If you put on a new article of clothing and receive compliments about your appear- ance, what is the likelihood you will wear it again in the future? It’s relatively high if you conform to what the research tells us.9 In learning, if we achieve an objec tive— master a piece of learning—and are rewarded for our success, the probability of retaining that learning increases. Recognized success encourages most people to learn and retain. When behaviorism was in its heyday, from the mid 1950s to the mid 1970s, the value and impact of reward was almost a sacred law.
Cognitive research tends to temper some of the extreme enthusiasm for the power of reward, but almost all learning researchers still acknowledge the value of re- inforcement. There is a major distinction between intrinsic rewards—those that emerge from the sense of accomplishment when you succeed at learning some- thing—and extrinsic rewards that are associated with something tangible that you are given for learning (for example, a gold star, food, money, or removal of some- thing unpleasant). The more one can include and build in intrinsic rewards, the joy that springs from the learn ing itself, the better it is for the learner. With certain learners, however, extrinsic rewards in the form of tokens, points, privileges, and removal of unpleasant chores, such as washing dishes, can help associate learning with pleasant experiences.
Taken together, those six universals drawn from research on learning lay the foundation for a powerful instructional model. When supported by what we have learned about how people process information and adult
learning principles, we discover the following essential ingredients for creating effec tive and efficient learning:
u letting the learners know why the learning is beneficial to them u helping the learners clearly understand in a meaningful way what it is they
will be learning
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
79
u creating structured activities and information that facilitate acquisition of targeted skills and knowledge
u building into the learning some opportunities for frequent and meaning- ful responses
u providing appropriate, corrective, and confirming feedback with respect to learner responses
u including appropriate intrinsic and extrinsic rewards, which each learner values, to enhance the pleasure of the learning process and its successful outcomes.
A Universal Model for Structuring Any Learning Session
Based on the preceding essential ingredients, we now introduce you to a user- friendly, easy-to-apply model for developing any learning session. It allows for all sorts of vari ations. In this part of the chapter, we present examples of its use with different con tent, contexts, and target audiences. Applying this model can provide you with an instantly successful learning session. As you become more familiar and comfortable with its use, you can incorporate other elements into it from this vol- ume and from your own experiences and observations. The model has been tested and used in hundreds of organizations with demonstrable success.
As you will discover, it is easy to use and makes good sense for creating learner- centered, performance-based instruction.
Figure 6-1 presents, in overview, our five-step model for structuring training. Here are some details about the elements of the model.
Rationale Provide a rationale. Explain why learners should learn whatever you are present- ing to them. Early on in any learning session, the learners require an explana- tion of why they should attend the session, whether live, e-learning, video-based, or in print. If the learner knows why she or he should learn something and val- ues it, the research suggests that learners have a higher probability of learning it. This is directly tied to the readiness principle—the opening of the mind and spirit—described in chapter 5. In the rationale, the instructor or the instruction informs the learners about what is in it for them and for others (for example, peers, customers, and the company’s shareholders). The rationale either can provide an explanation or can lead the learners to discover on their own why they should learn this.
Let’s work with an example using somewhat familiar content, performance objec- tives. Imagine that the audience consists of internal subject matter experts (SMEs)
u Section 2: What You Must Know to Be a Better Trainer
80
who have been tasked with developing and delivering training sessions to customer organizations and third-party vendors who will be selling your products and services. What might we include in a rationale for acquiring skills in developing performance objectives?
Rationale: • As trainers, your success is measured by the success of your learners.
• The more concrete and verifiable what you want your learners to be
able to do and say, the more easily you can identify their successes or
short comings.
• Learning objectives are the targets toward which all of your instruction and all
of the learners’ learning are aimed. They provide concrete goals for everyone
to attain.
• The more easily you can create these objectives, the more readily and
smoothly all the other parts of your instruction will fall into place. It will make
your lesson planning much easier.
• If your learners know where they are going, the probability that they will get
there becomes higher.
• In the rationale, you provide a form of overview of where you are heading in
the ses sion. You also build a desire to learn by underscoring how useful, inter-
esting, and exciting this session will be to the learners.
Figure 6-1. Five-Step Model for Structuring Training
Rationale
Objectives
Activities
Evaluation
(corrective) Feedback
(confirming) Feedback OK?
Yes No
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
81
Performance Objective State the performance objective to the learners. Tell them clearly what they will be able to do by the end of the session. If the learners know what they are sup- posed to learn, research suggests that there’s a better chance that they will learn it. The instruc tor or, if self-paced, the instructional material states the objectives meaningfully in terms of the learner and not in terms of the trainer or training system.
Which of these statements is more appropriate as an objective?
£ You will be able to convert a service call to a sales call.
£ I will show you how to convert a service call to a sales call.
The first statement is more appropriate because it is expressed in terms of the learner. The second states what the trainer will do and as such is not a suitable learner-centered, performance-based objective.
The instructor or instructional material also states the objective in concretely verifi able (measurable or observable) terms. Select the item from these two op- tions that you believe is a better performance objective:
£ You will state the four steps for transforming a service call to a sales call.
£ You will know the steps for converting a service call to a sales call.
The first objective is better because it uses a more verifiable verb, “state,” and names a spe cific number of steps. The more concretely verifiable the objective is (without it becoming obsessive or trivial), the better the performance objective.
Continuing our example of the content SMEs learning to become trainers, here is how the performance objective might be phrased:
Performance objective: Participants will be able to create for their training sessions performance objectives that are stated in terms of the learner and
that contain verifiable verbs and specific performance standards.
Activities Create learning activities that lead to attaining the performance objectives. If learners do things that lead directly to meeting the objectives, there’s a better chance that they will attain those objectives. This means that the trainer (or train- ing designer) creates or selects only those activities that lead the learner directly to meeting each objective.
Here is one of the key benefits of this model: It is lean and focused. The rationale pro vides benefits for the learner. The performance objectives state the contract
u Section 2: What You Must Know to Be a Better Trainer
82
between the training and the learners—what they will be able to do and how well. Now, the activities cut out the extraneous and frequently disruptive noise. They focus sharply on objective attainment, nothing more or less. The activities are designed to encour age—even require—learner participation plus more. The activities also should stimu late the learners to contribute their own experiences, imagination, and judgment. After all, these people are adults!
Important to successful learning activities is that they be inherently interesting, even fun. This means that the trainer or training designer should build in ele- ments of challenge, curiosity, and fantasy. For challenge, the activities present dif- ficulties that, with effort, can be overcome to achieve hard-won success. Curiosity means not telling the learners everything at once. The activities have the learners wondering what will happen next. They’re curious but not confused.
Finally, fantasy acts like spice. It piques interest and is imaginative. It provokes creative participation. This makes the activities fun and interesting and helps pro- mote a broader type of trans fer to the job (encourages visualizing a wider range of application than if all the activ ities are narrowly focused on the immediate job). Chapter 8 contains 25 examples of interesting learning activities, many of which contain all of these elements.
In our example on performance objectives, possible activities might include these ones:
Activities: • For rationale, start with examples of vague statements and clear perfor
mance objectives. Have participants select those they prefer and articu-
late why.
• State benefits of performance objectives for training sessions with
job-related examples.
• State the performance objective of this session and analyze and discuss
with participants their expectations and the value of this objective.
• Conduct an exercise that has participants identify examples of good per
formance objectives compared with non-examples, and have them give
reasons for their selection. Summarize by highlighting critical characteris-
tics of excellent performance objectives.
• Conduct an exercise in which participants first edit poorly constructed
performance objectives, then create objectives from given content, and
finally share and correct them in teams. Provide some fantasy content
(for example, butter a bagel, pilot a flying saucer). Provide a checklist for
veri fying objectives.
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
83
• Based on their selfselected content, participants generate performance
objec tives, edit their own, and then edit others’ objectives.
• Conclude with a wrapup discussion on benefits and techniques for gen
erating performance objectives. Participants review all objectives of this
session and critique, edit, or approve them.
Evaluation Evaluate learner performance. Check to see whether learners have learned. If the learn ers are assessed on what they are supposed to learn, they have a better chance of learning it. It is important, however, to evaluate in terms of the performance objec tive and not the person. The trainer or the training system verifies the degree to which each learner has met each objective for the desired level of performance. In self-paced, computerized training, this can be automated and the results re- corded for remediation, prescription, or later review. Learning management sys- tems (LMSs) have become very advanced in helping you do all of this.
However, we caution that the results will only be as good as what you programmed the LMS to perform. In live settings, the trainer does what is fea sible. This can in- clude asking questions; requesting real or simulated demonstra tions; having learn- ers do exercises and then self-evaluate, peer correct, or evaluate in teams; and providing problems and cases and verifying both process and outcome.
The most common tools for checking attainment of performance objectives are per formance and written or oral tests, observation checklists, and performance results. (In chapter 9, we go into much more detail on tests and testing.)
Returning to our example, we might handle evaluation in this way:
Evaluation: • For the exercise on identifying examples and nonexamples of good
objectives, use an answer key. Include discussion to justify responses.
• Have participants derive and state critical characteristics of excellent per
formance objectives.
• Check all edited and generated objectives using the performance objec
tives checklist.
• Check all participantgenerated objectives for their own content using the
performance objectives checklist.
Feedback Provide feedback in terms of the performance objectives. Let learners know if they’ve got it right. Correct them when they go astray.
u Section 2: What You Must Know to Be a Better Trainer
84
If learners receive information on how well they are learning, they tend to learn bet ter. For this reason, learners must receive feedback throughout the train ing session. As mentioned earlier with respect to research on feedback, always give feedback in terms of the performance objective and not the person. Generally, the best time to provide feedback is directly after the evaluation. For difficult or complex tasks, however, feedback can be effective if it comes just prior to the next attempt or practice. This acts as a refresher in terms of learning and a just-in-time prompt. If evaluation causes anxiety, which is frequently the case with adult learn- ers, don’t keep them guessing. Provide sufficient, immediate feedback to reduce stress and encourage learning.10
Most important, feedback comes in two forms: corrective, which explains to the learners how they can attain the objective, and confirming, which informs the learn ers that they have attained the objective. Corrective feedback always must be stated positively and encouragingly.
Feedback is not always something one can specifically plan for. Nevertheless, the feedback component is essential and omnipresent in training. In our example, we might offer this feedback
Feedback: • As learners acquire skills and knowledge about performance objectives,
provide corrective and confirming feedback on a continual basis.
• Following each exercise, provide specific information on how to improve
performance or confirm the correctness of the response in relation to the
performance objective.
Figure 6-2 presents the five-step model for structuring training annotated with a summary of the main points made in this section of the text. This model, as simple as it appears, incorporates significant findings from research on learning that help learners acquire new learning efficiently and effectively.11 In the next section, we trans form the model into operational worksheets and try them out with content.
The Training Session Planning Session
Examine figure 6-3 and note how we have transformed the five-step model into a planning sheet. The planning sheet enables you to take a first cut at creating your training session. Also note two of its key characteristics. First, it is not con- tent cen tered. Rather, it forces you to think about the learners. It begins with the require ment for a rationale that provides meaningful benefits to the learners. It also requires learner-centered, performance-based objectives that are meaningful to and valued by the learner. It specifies the activities that will lead the learners to objective attain ment.
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
85
Activities must maintain at least a 50-50 balance between learner and instruc tor or instructional content in self-paced mode. It then asks how learner attainment of the objectives will be evaluated. (In chapter 9, we’ll spend a considerable amount of time on appropriate evaluation methods and tools.) The final step, corrective and confirming feedback, should be a natural outgrowth of the evaluation and sponta neously adapted to how each learner performs. It may be useful, however, to antici pate where difficulties will occur and how these can best be addressed if the learner needs to be brought back on track.
Second, note the brevity and simplicity of the planning sheet. You are asked to think about each session and then write your plan in bulleted format. Remember, as we discussed in chapter 2, our natural tendency as content specialists or SMEs is to fill up a training session plan with content. Here, instead, we are asking you to focus first on the customer. When you have planned your training-learning strat- egy, you can go for the necessary, relevant content that the learners can absorb and retain.
Now let’s proceed to an example that enables us to tr y out this training session plan. Ready for your first test flight? We’ll start with a fantasy setting to keep things simple.
Figure 6-2. Annotated Five-Step Model for Structuring Training
1. Rationale
2. Objectives
3. Activities
4. Evaluation
5b. (corrective) Feedback
5a. (confirming) Feedback OK?
Yes No
Inform them they have got it right. Check learning. Correct them when they have gone astray.
Explain why learners should learn this and how it applies to their work.
Inform learners of what they will be able to do.
Give learners things to do. Make these interesting and don’t bore them.
Check to see if they have learned.
u Section 2: What You Must Know to Be a Better Trainer
86
F ig
u re
6 -3
. T ra
in in
g S
e ss
io n
P la
n n
in g
S h
e e t
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
87
Sample Planning Scenario: A Ticket to the Fair
Background: Once a year, the state holds a large-scale fair. For one week, hundreds of thousands of paying visitors flock to it. Each year, the State Fair
Commission hires temporary workers for various jobs. You are responsible
for training 45 ticket sellers. They have to be accurate and fast because lines
can get long and paying visitors impatient. Accuracy and speed are the two
key success criteria. The system is totally manual. All of the potential ticket
sellers are novices, have gone through background checks, and are bonded.
Target audience: Part-time ticket sellers with education levels ranging from grade 10 to some college. All are over 18; some are as old as 70. About
two-thirds are women. None have dexterity problems or significant hearing
or vision impairments.
Session subject: Calculating admission costs, taking money, issuing tickets, and giving change.
Time allotment: Two hours, 30 minutes.
Training context: Classroom and crudely simulated ticket booths.
Given the details of the scenario, we developed the training session plan depicted in figure 6-4. We detailed this training session plan a little more fully than we might nor mally. As a first example, however, we felt that a little extra information would help you visualize the training session more clearly. Once you have read it, assess the plan using checklist 6-1.
You may have had to do some guessing to complete checklist 6-1, but overall a “yes” should be checked off for each item. If not, determine what we could have done to obtain checkmarks in all of the “yes” boxes. Improve our session, please.
The Training Session Scripting Sheet
The training session planning sheet simply was a first attempt at organizing a learner-centered, performance-based learning experience. In many instances, that may be suf ficient. All you would have to do is add the timing, plan your re- sources, and then collect and prepare your materials for tryout. Here is a simple rule of thumb for plan ning learner-centered, performance-based training sessions:
The more content expertise the trainer possesses, the less content information you require in your plan. The more training capability and experience the trainer pos- sesses, the less instructional detail you require in your plan. This is mainly true
for live, synchronous train ing. For self-paced learning of any nature, the final plan must contain more details for both content and instructional methods.
u Section 2: What You Must Know to Be a Better Trainer
88
Figure 6-4. Sample Planning Sheet
Training Session Planning Sheet
Session title: _____________________________________________________________________
Target audience: _________________________________________________________________
Allotted time: ____________________________________________________________________
Rationale:
• The most important and trickiest part of the job is selling tickets and making correct change. • Despite background noise, if you’ve got the knack, you won’t have problems. • You are responsible for your errors up to $100. Learn the job right, and you will be error free. • Every day we have a bonus for the quickest and most accurate ticket seller. • Some people get hostile when you are slow or make errors. This session will help you avoid
the pain.
Objectives:
Overall objective: Participants will be able to sell the exact number and type of tickets, collect the exact amount of money, and give the correct change for any customer without error and at an average time of 20 seconds per transaction (maximum group of eight people per transaction).
Specific objectives: • Identify the exact numbers and types of admission tickets the customer requests. • Calculate the exact total cost in 10 seconds with no errors. • Collect the correct total amount with no errors. • Give the customer the exact change with no errors.
Activities:
• Draw from participants what concerns them most about their new job. • Show how this session helps decrease or eliminate those concerns. • Present key points of rationale and discuss each one. • Show ticket price / customer job aids and demonstrate use. • Using different voices and admission requests, have participants determine exact request
and cost. • After several examples, time the exercise. • Using play money and coins, have participants practice collecting money, issuing tickets,
and giving change. This is a peer-pair activity. • In simulated ticket booths, create a practice session putting all parts together. Loudly play
audiotape of background noise.
Evaluation:
• Practice exercises with timing toward the end for each activity. • Final evaluation: In the simulated ticket booths, each learner services 10 peer customers,
each with different characteristics and requirements. An audiotape plays loud background noise. Peers talk.
Feedback:
• Provide participants with feedback on how they are doing and how they can improve through self-assessment, peer assessment, and trainer verification.
• Provide timing and accuracy information following final evaluations. Suggest ways to improve, as necessary.
Selling tickets, collecting money, and giving change
State fair ticket sellers (15 participants per session)
Two hours, 30 minutes
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
89
When greater elaboration of the training plan is called for, the five-step model can be expanded to accommodate more scripting. In this book we provide a set of train ing session scripting sheets only for live, instructor-led training because every other type of delivery method has its own unique set of detailed requirements. Lin- ear video scripting differs from random-access, learner-controlled video scripting, and both are dissimilar to all of the endless varieties of e-learning formats.
We have found the training session scripting sheets depicted in worksheet 6-1 and figures 6-5 and 6-6 to be helpful when circumstances warrant their use. These are a few of such circumstances:
u When you have relatively inexperienced trainers: It gives them a script to follow and increases their probability of success.
u When your trainers are insecure about the session: The scripting sheet becomes a “security blanket” for them.
u If you have several trainers running the same session (some of whom like to do their own thing): It provides uniformity of approach.
u In situations with a high requirement for consistency across train ers and locations: Scripting sheets lay out a common approach that facili tates monitoring of consistency.
u Where there is frequent turnover of trainers: New trainers have a ready-to- run session already prepared for them.
Checklist 6-1. Training Session Planning Sheet Assessment
Criterion Yes No
The rationale is presented in terms of the learners. £ £
The learners participate and contribute in building the rationale. £ £
The performance objectives are stated in terms of the learners. £ £
The performance objectives are verifiable. £ £
The performance objectives are appropriate to the learners and the £ £ content.
The activities are appropriate to the performance objectives (they £ £ lead the learners to attain the objectives).
The activities require learner participation at least 50 percent of £ £ the time.
Learners can participate and contribute during the activities. £ £
Evaluation is appropriate to the performance objectives. £ £
Feedback is appropriate. £ £
The session can be conducted within the allotted time. £ £
u Section 2: What You Must Know to Be a Better Trainer
90
W o
rk sh
e e t
6 -1
. T ra
in in
g S
e ss
io n
S c ri
p ti
n g
S h
e e t
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
91
F ig
u re
6 -5
. C
o n
te n
ts o
f th
e T
ra in
in g
S e ss
io n
S c ri
p ti
n g
S h
e e t
Ta ke
n f
ro m
t h e s
e ss
io n p
la n n in
g s
h e e t
Ta ke
n f
ro m
t h e s
e ss
io n p
la n n in
g s
h e e t
Ta ke
n f
ro m
t h e s
e ss
io n p
la n n in
g s
h e e t
Ta ke
n f
ro m
t h e s
e ss
io n p
la n n in
g s
h e e t.
I f
th e re
is a
n o
ve ra
ll o b je
ct iv
e ,
st at
e it
fi rs
t. T
h e n in
cl u d
e t
h e
s p
e ci
fi c
o b
je ct
iv e
s.
T h is
r e se
m b le
s st
ag e d
ir e ct
io n s
in a
p la
y. L
is t
in o
rd e r
w h at
b o th
tr
ai n e r
an d le
ar n e rs
a ct
u al
ly d
o —
w h at
c an
b e
o b se
rv e d .
T h is
is li
ke t
h e
s cr
ip t
o f
a p
la y.
Yo
u p
ro vi
d e
t h
e t
ra in
e r
w it h
a ct
u al
w
o rd
s o r
sp e
ak in
g s
u g
g e
st io
n s.
I f
th e t
ra in
e r
re q
u ir e
s co
n te
n t
h e
lp ,
d e ta
il co
n te
n t
p o
in ts
. If
t h
e t
ra in
e r
re q u ir e s
in st
ru ct
io n
al m
e th
o d
s g u id
an ce
, d
e ta
il in
st ru
ct io
n al
m
e ss
ag e s
h e
o r
sh e
is t
o s
ta te
.
T h
is is
li ke
t h
e p
ro p
s p
e ci
fi ca
ti o
n s
fo r
a p
la y.
F o
r e
ac h
in st
ru ct
io n
al
ac ti vi
ty o
r e
ve n
t, li
st t
h e
m e
d ia
o r
re so
u rc
e r
e q
u ir e
m e
n ts
.
F o
r e
ac h
in st
ru ct
io n
al a
ct iv
it y
o r
e ve
n t,
li st
t h
e e
xa ct
t im
e a
llo tm
e n
t.
u Section 2: What You Must Know to Be a Better Trainer
92
F ig
u re
6 -6
. S
a m
p le
T ra
in in
g S
e ss
io n
S c ri
p ti
n g
S h
e e t
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
93
u Section 2: What You Must Know to Be a Better Trainer
94
Using the Five-Step Model to Retrofit Existing Training Sessions
Suppose you have inherited someone else’s existing course materials and plans. You examine them and discover that they are not much more than large data dumps. They are rich in content but essentially involve telling/one-way transmis- sion. They may even include a vast number of slides with scripted text. What can you do to increase their effectiveness without starting from scratch? (This equally applies to a great deal of print and computer-based courses.)
Here is our suggestion for retrofitting existing training to the five-step model: Take one existing, content-heavy course. Examine it to determine its overall reason for existence. Derive from all of this material what people exposed to it are supposed to be able to do with the content. For example, suppose the course is about a new line of products, and it is aimed at the sales force. By reviewing all the course ma- terials, you derive the following rationale and objectives:
Rationale: The market has been crying out for a new line of sewing needles. With the population aging and sight declining, people are finding it harder to thread needles. They also are looking for needles that are better and more versatile. Coinciding with this is an upsurge in sewing hobbyists as peo ple retire and have more leisure time. Interest in needlepoint, fashion design, dressmaking, quiltmaking, and even sailmak- ing has grown. Our new needles have attractive features and benefits for wholesalers, retailers, hobby clubs, and end users that knock the socks off the competition. They of- fer you incredible opportunities to corner the market, increase sales, and significantly improve your earnings. Not only that, they are entirely innovative products…. (We think you get the idea.)
Overall objectives: By the end of this session, you will be able to identify innovative sales opportunities for the new product line, favorably position it against all of your competitors, and present the novel products in a way that increases customer profit margins by 20 percent to 30 percent and your com missions and volume by at least 20 percent to 40 percent.
Specific objectives:
u Name and describe the unique features and benefits of the new needle product line and each of the products.
u Precision target wholesale, retail, and hobby club customers for the prod- uct line and/or specific products.
Notice how you are taking an existing, content-based course and transforming it into a more learner-centered program?
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
95
Break the existing course into its individual components. Reorganize it, if nec- essary, to create a logical learning sequence, one based on the logic of learn ing, not the logic of the content. Eliminate unnecessary components or place them in a reference manual if management won’t let go of them.
u For each retained component, create a rationale and objective. u For each component, create interactive, participative activities that involve
the learners.
In our example, have the learners play games to match features and benefits to products. Instead of telling them about the appropriate products for specific cus- tomer groups, provide customer cases and, in teams, have the learners examine the product documentation and recommend suitable match es. Build a full menu of engaging, learner-centered, performance-based sim ulations and exercises.
To evaluate, create challenging quizzes; tests; competitions (after all, these are sales representatives); and especially cases for individuals, peer pairs, and teams to solve. Provide tools and checklists for peer and self-evaluations. Develop an evalu- ation activity for each performance objective.
Make sure that throughout the revamped session there is room for a lot of dia- logue and feedback that confirm and correct as appropriate.
Voila! The five-step model can become a retrofit recipe for converting dull, telling sessions into highly motivating and effective learning events.
Final Review of the Five-Step Model
This has been your longest chapter so far. It requires pulling together some key content. Figure 6-7 is a blank chart for the five-step model coupled with some key points about each part of it. To help you retain the model, fill in the blanks beside each number. Then, in the circles place the letter of the correct key point that re- lates to each element. To check your responses, turn back to figure 6-2.
Remember This
Once again, we have a quick, closing quiz. Just cross out the incorrect option in parentheses to make each sentence come out right. Then we’ll give you our take on how we would answer.
1. We are all (alike / very different) when it comes to how we perceive, process, store,
and retrieve information for learning.
2. Expectancy value helps learners determine the (why/structure) of a learning session.
u Section 2: What You Must Know to Be a Better Trainer
96
F ig
u re
6 -7
. F iv
e -S
te p
M o
d e l fo
r S
tr u
c tu
ri n
g T
ra in
in g
a. In
fo rm
t h e m
w h at
t h e y
w ill
b e
a b
le t
o d
o .
b .
In fo
rm t
h e m
t h e y
h av
e g
o t
it r
ig h
t.
c. C
h e ck
t o s
e e if
t h e y
h av
e le
ar n
e d
.
d .
C o rr
e ct
t h e m
w h e n t
h e y
h av
e g
o n
e a
st ra
y.
e .
G iv
e t
h e m
t h in
g s
to d
o .
M ak
e t
h e
se in
te re
st in
g
an d d
o n ’t
b o re
t h e m
.
f.
E xp
la in
w h y
th e y
sh o u ld
le ar
n t
h is
a n
d h
o w
it
ap p lie
s to
t h e ir w
o rk
.
g .
C h e ck
le ar
n in
g .
5 a .
O K
? Y
e s
N o
5 b
.
4 .
3 .2 .
1 .
Chapter 6: A Five-Step Model for Creating Terrific Training Sessions t
97
3. If you don’t know where you’re going in a learning session, you’ll probably (soon
figure it out / end up someplace else).
4. Humans seek order and structure in instruction. If there is none, (they will create
it artificially / accept this state and be with it).
5. Meaningful covert and overt responding (slow down a learning session / increase
the probability of learning).
6. Essential for initial learning is (active, conscious engagement / an automatic,
unconscious mental state).
7. Feedback should be (positive or negative / corrective or confirming) following
evaluation of learning.
8. A feeling of accomplishment based on something you value is an (intrinsic/extrinsic)
reward.
9. In the five-step model, “rationale” is related to the adult learning principle of
(autonomy/readiness).
10. Sophisticated LMSs provide (meaningful, usable diagnostic, instructional and
learner data automatically / learner data that is only as good as what you program
it for).
Here is our feedback:
1. We are all alike when it comes to how we perceive, process, store, and retrieve information for learning. Unless there is some physiological or pathological issue, humans treat information using the same biological mechanisms. As a species, most of how we deal with new learning is generalized across all of us.
2. Expectancy value helps learners determine the why of a learning session. Expectancy value can be colloquially stated as WIIFM—What’s in it for me?
3. If you don’t know where you’re going in a learning session, you’ll probably end up someplace else. An old, but all-too-true saying. Without a clear sense of where the instruction is headed, learners become easily lost and soon flounder or make incorrect assumptions about the learning message. Train- ing must provide learners with clear objectives that are meaningful to them.
4. Humans seek order and structure in instruction. If none exists, they will cre- ate it artificially. We seek to make order out of chaos. New skills and knowl- edge are more easily stored in long-term memory if they are logical and organized for connection to prior knowledge. Then, through practice, the learning becomes progressively easier to retrieve.
5. Meaningful covert and overt responding increase the probability of learning. There is no research evidence to suggest that active learner responding af- fects the length of a learning session. However, there is massive evidence that such response engagement produces more effective learning and retention.
u Section 2: What You Must Know to Be a Better Trainer
98
6. Essential for initial learning is active, conscious engagement. An unconscious state of mind does not allow us to register what is being taught. When we “zone out,” we are no longer aware of what is happening. Learning goes practically nowhere. Active mental engagement, however, increases the prob- ability of learning, especially during initial stages when we are still seeking to make sense of new material.
7. Feedback should be corrective or confirming following evaluation of learn- ing. Positive and negative feedback have connotations that are incompatible with supporting learning. Feedback should be focused on task, not person, and either provide the right dose of information to place the learner back on track—corrective—or let the learner know that she or he has met the desired goal—confirming.
8. A feeling of accomplishment based on something you value is an intrinsic reward. Intrinsic rewards are internally generated based on personally mean- ingful success. Extrinsic rewards originate outside of the learner. These are valued more than accomplishment of the learning task itself and compensate for the lack of internal interest in goal accomplishment.
9. In the five-step model, “rationale” is related to the adult learning principle of readiness. The rationale gives the learner a valued reason for learning. It helps open the mind to the learning session and is therefore most related to the readiness principle.
10. Sophisticated LMSs provide learner data that is only as good as what you pro- gram it for. While LMSs can do wonderful things, particularly those that have been around for a while and have versatile capabilities, what you ask it to do gets you what it gives back to you. You must configure your LMS to produce the right types of data in the most usable, comprehensible formats. How you program it determines what you will obtain from it.
To close out on this central chapter, bear in mind that although we are attracted to the notion of how unique each of us is, when it comes to learning we are far more similar than we are different.
If adult learners know why they should learn, what they will be able to do as a result of learning, see how all the learning pieces fit together, practice, get feedback, and are rewarded for their learning … they learn.
By applying the five-step model—rationale, objectives, activities, evalua tion, and corrective and confirming feedback—you increase the probability of learning.
Now we can turn to how we can help make learning stick, the subject of the next chapter.
HRE7271/Week 4/ENHANCING CORPORATE TRAINING WITH TECHNOLOGY.pdf
ENHANCING CORPORATE TRAINING WITH TECHNOLOGY
In an environment where knowledge and technology are constantly growing and
evolving, training and development are key factors in organizational success. Smart
companies invest in employee training and development to enhance job performance,
improve productivity, and increase morale—all of which can positively impact an
organization’s bottom line.
THE NEED FOR TECHNOLOGY IN TRAINING
The professional development of employees is crucial for a business to retain clients, stay
competitive, and keep a firm grasp on company goals. In 2011, The American Society
for Training and Development found American businesses spent around $156 billion
on employee training, but it was discovered that when employees returned to their jobs,
90% of the skills they had acquired were lost. With little continued follow-up and
consistent reinforcement of significant testing, only 10% of the information was retained.
In the past decade, technological advancements are offering new corporate training
capabilities, and while many ‘stand-and-deliver’ methods can be exhausting and tedious
for employees, technology is bringing life to corporate learning. New instructional
methods are requiring specialized instructors, particularly those with technology know-
how. INTERACTIVE TECHNOLOGY HEIGHTENS EMPLOYEE ENGAGEMENT
Interaction with others and relevant material is one of the foundations of basic learning,
so why wouldn’t training involve real, hands-on interaction? Interactive technology is
changing the game of learning, and the benefits for corporate training are immense.
Lecture-style training is not always the best method of corporate education, but the
introduction of several types of technology is proving valuable to corporate learning.
• Immediate Response Systems, such as eInstruction Clickers, allow immediate feedback to the
instructor. Handheld devices permit the instructor to test the class and also provide the
employee with instant answers to proposed questions. Opinions can be gathered on the fly,
and results can later be analyzed to improve future training. The learning environment itself
becomes interactive, and employee progress can be monitored. Handheld devices can also
provide anonymity for students who want to ask questions without recognition.
• Video conferencing systems allow instructors to reach a vast, often distant, audience. The
popular video conferencing platform, Skype, had 663 million registered users by the end of
2010, and instructors both in the education and corporate world have found it to be a
wonderful alternative to traditional face-to-face meetings. Business took video conferencing
to another level with the introduction of GoToMeeting in 2004, offering the words “Work
with anyone, anywhere,” and using the same foundation, GoToTraining was founded in
2010. Video conferencing allows for one-on-one meetings as easily as group meetings, and
corporate training has found new reach with this type of technology.
• Everyone enjoys playing games, so introducing game play technology to training is an
excellent way to engage employees during training, as well as boost their knowledge
retention after training. Interactive games can be played on a variety of devices (including
laptops or television consoles, and intermingling more advanced technologies, such as
augmented reality eyewear or true-to-life simulators), can leave a lasting impression on
employees. Training games have been found to provide faster and easier instruction delivery,
higher test scores and lasting retention, and overall consistent instructional quality. As an
added bonus, interactive games offer a significantly lower cost for materials and curriculum.
A BLEND OF TRAINING METHODS
Surely human interaction can never be replaced. Without extensive technology,
employees are driven towards communication with each other and the instructor, and
some knowledge can only be gained through conversation or interaction through online
forums. Corporate training seminars should not only instruct attendees, but also invite
them to interrelate with their fellow employees and gain wisdom through the learning
process. Technology adds interaction with the material that allows the employees to gain
and maintain the required knowledge. New instructional methods are requiring more
specialized instructors, particularly those with technology know-how, and a Master of
Science in Education in Learning Design and Technology is proving to be a
commodity in the changing world of corporate and educational learning.
As technology progresses, learning and training methods will continue to evolve. Those
with specialized knowledge pertaining to the fields of Learning Design and Technology
will be called upon to lead. For those serious about a career in education, now is the
time to further their knowledge in advancing educational methods, particularly
technological methods of instruction. Retrieved from http://online.purdue.edu/ldt/learning-design-technology/resources/enhancing- corporate-training-with-technology
HRE7271/Week 4/Four Major Tyeps of Training.pdf
By Harold D. Stolovitch & Erica J. Keeps [email protected] & [email protected]
We've often heard the expression, "different strokes for different folks." In a broad sense, the four major types of training we describe here are for different types of learners. Let us caution you right away. The same learner may be an excellent candidate for all the types of training - but at different stages of her or his learning with respect to a specific skill or knowledge area. We'll examine each of the types and determine how you can use them appropriately.
The four types of training are receptive, directive, guided discovery and exploratory. Various learning researchers and professionals use other classification systems and names. We find these four, which Ruth Clark (1998) employs, to be convenient and
useful.
Receptive Training
This type of training falls into the "telling" mode. Essentially, the view of the learner is that of a vessel into which good, wholesome, nutritional information is poured. The danger in using this type of training is that there is an expectation that learners will be able to convert what they hear and see into usable skills and knowledge.
There is, however, some limited value to the receptive mode. Basically, it makes learners "aware." Well done and presented in an interesting manner, it can demonstrate value to the learners and build their motivation to accept, learn, support and desire to discover more. For highly knowledgeable learners, it may be sufficient for them to make connections with and adapt prior knowledge to new circumstances. The training content is frequently transmitted in one direction. Learners have little or no control, except to tune out, turn off or daydream.
Sadly, much of what is called training in the workplace is of a receptive nature. Here is a true case we experienced in a very progressive, world-renowned high-tech company:
A team of highly qualified software engineers (PhDs) had just completed a major overhaul of an operating system (O/S). They were tasked with going out to train the worldwide group of engineers who would support and troubleshoot the new, improved O/S. Their approach to the training was to provide each participant with a manual (approximately 1,200 pages) and tell them about the new and upgraded O/S, how it was developed, how it differed from its forerunner, what their challenges had been, how they have overcome them, what they had had to leave out, and so forth. When we asked if they were going to have participants engage in troubleshooting practice, they were taken aback. They hadn't planned on it.
Given their target audience of system troubleshooters, is that how you would have gone about the training? The receptive type of training can have a limited role in introducing something new, presenting fascinating anecdotes, and building awareness and enthusiasm, but it soon wears thin with any audience. Our bottom-line advice with respect to receptive-type learning is to use it sparingly. Always seek an alternative, except in the case of short, consciousness-raising sessions.
Directive Training
This approach to training is akin to the U.S. Army infantry slogan: "Follow me." As its name suggests, this method directs. In this approach, you analyze the knowledge and skills required to lead learners from where they are to where you want them to be. You create clear performance objectives and matching test items. Then you organize and sequence learning blocks or units that direct the learners from their starting positions to defined intermediate and final finishing lines. There's little learner control, but, unlike in the receptive type of training, learners are actively and meaningfully engaged as they progress along predetermined paths.
The directive approach is particularly well suited for learners who have little experience with the learning content, require support to build competence and confidence, and will later perform on the job in ways that are identical or very similar to what they learn in the training.
Guided Discovery
In this type of training, control is shared between the learner and the trainer or training program. Guided discovery is generally case-based. Learners immediately plunge into cases, scenarios, or problems. They may require some initial input, but mostly they take the initiative. The learners may reach out to a variety of information sources or support tools, either physically material or electronic, to deal with the situation. The learners themselves have to discover what to do and when to do it. They seek and identify appropriate information and tools to proceed. The instructor or instructional program offers assistance in the forms of cues, prompts, suggestions, and corrective feedback along the way or redirection, consulting services, and debriefings. The amount of guidance or support as well as its nature depend on the skill and knowledge level of the learners. For less knowledgeable, less independent, or less confident learners, the training may include a great deal of guidance. The more capable and knowledgeable (or skilled) the learner is, the more independently he or she can function. In that case, the role of the trainer or training is to confirm, debrief, add editorial nuances, proffer variations, and, of course, reward. When learners achieve high degrees of competence and confidence, they are ready for the next type of training.
Exploratory Learning
Here we build and organize a rich learning and informational environment for the learners and then truly get out of the way. The learners are in control. They know what is required and set out to search for whatever exists to resolve the issue and help them progress to the next level. Usually, there are only general goals set (often by the learners). Large databases of information integrated into a knowledge management system provide an ideal environment for exploratory learning. At a less sophisticated level, a workshop with all the tools, materials, and manuals does the same thing. Learning is usually individualized. The trainer can monitor and provide feedback or support and debrief as required. In exploratory learning, the trainer is essentially a resource for the learners.
Pulling the Four Types of Training Together
There is a natural progression among the four types of training with respect to learner control and sophistication. To conclude the discussion of these types of training, here are some summary points:
• All of the types are different ways of approaching training. • All of the types have a place in training, but the receptive approach is the most frequently
used method - and it should be the least often employed. Essentially, it's just telling, and telling ain't training.
• Directive learning provides the trainer and the organization with the greatest control. The side effect is decreased learner initiative and more narrow, nearer transfer-of-learning potential.
• Guided discovery is an excellent, balanced training approach for encouraging learner initiative under safe conditions. Learning results are usually stronger and more fluid (for example, transfer to a broader range of situations). Learning results are less predictable, and learning time may increase.
• Exploratory learning is powerful for sophisticated, capable learners. It allows for greater individualization and personalization of learning. However, it requires sufficient resources, decreases trainer control, and is unpredictable in terms of specific outcomes.
Our recommendation is to use the receptive type very seldom. Focus on the other three. Mix and match your approaches to fit the needs of both the learners and learning. Above all, keep the training active.
This article is an excerpt from Harold Stolovitch and Erica Keeps' award-winning, best-selling book, Telling Ain't Training.
HRE7271/Week 4/Summary Slides.pptx
Training Approaches and a Cornucopia of Learning Activities
Rationale
Making your training session learner centered. You will now learn about four different training methods that are commonly utilized. You will need to be able to adapt them to your audience in the field.
Four Major Types of Training
Receptive
Directive
Guided Discovery
Exploratory
Receptive Training
Telling mode
Expectation to convert what is heard
Makes learners aware
Ability to demonstrate value to the learner and build motivation
Little to no learner control
Involves didactic methods
View of the learner as a vessel into which good, wholesome, nutritional information is poured
Expectation that learners will convert what they hear into usable KSAs
Limited value to receptive mode- Makes learners aware
If well done, it can demonstrate value to the learners and build their motivation to accept, learn, support, and desire to discover more
May be sufficient for highly knowledgeable learners- to make connections with and adapt prior knowledge to new circumstances
Training content is generally transmitted in one direction
Involves Didactic Training
4
Didactic Training Defined
“Involving Lecture and Textbook Instruction Rather Than Demonstration and Laboratory Study” (Merriam Webster Dictionary, 2013).
Didactic Compared to Other Methods
Open Learning
Montessori Method of Learning
Experiential Learning
Active Learning
Class Discussion
Learning Cell
Collaborative Learning Group
Open Learning
Open learning is supposed to allow pupils self-determined, independent and interest-guided learning.
Montessori Example
Active Learning
Umbrella term that refers to several models of instruction that focus the responsibility of learning on learners
Examples
A learning cell is a process of learning where two students alternate asking and answering questions on commonly read materials. To prepare for the assignment, the students will read the assignment and write down questions that they have about the reading. At the next class meeting, the teacher will randomly put the students in pairs. The process begins by designating one student from each group to begin by asking one of their questions to the other. Once the two students discuss the question, the other student will ask a question and they will alternate accordingly. During this time, the teacher is going around the class from group to group giving feedback and answering questions. This system is also referred to as a student dyad.
A collaborative learning group is a successful way to learn different material for different classes. It is where you assign students in groups of 3-6 people and they are given an assignment or task to work on together. This assignment could be either to answer a question to present to the entire class or a project. Make sure that the students in the group choose a leader and a note-taker to keep them on track with the process. This is a good example of active learning because it causes the students to review the work that is being required at an earlier time to participate.
6
Effective Lecturing
Effective Lecturing | Ineffective Lecturing |
Educator-Student Interaction Two-Way Communication Educator-Student Questions Shared Responsibility for Active Learning Small Group, Problem-Solving Activities Variety of Supporting Media | 100% Educator Talk One-Way Communication Few if Any Questions Student Depends on Educator for All Information No Student Activities No Supporting Media |
Sullivan & McIntosh (1996) "Delivering effective lectures."
Primary Purpose Is to Transfer Information From the Instructor to the Student
7
When to Lecture
Appropriate | Inappropriate |
Disseminating Information Quickly to a Large Audience | Presenting Complex, Detailed, Or Abstract Information |
Presenting New information Before Using Other Activities | Dealing With Information Concerning Feelings and Attitudes |
Providing An Overview of a Topic | Training In Psychomotor Skills |
Arousing Interest in a Topic | Teaching High-Level Cognitive Skills |
Sullivan & McIntosh (1996) "Delivering effective lectures."
Components of a Lecture
Use an Opening Statement
Present Key Terms
Offer Examples
Use Visual Backups
Sullivan & McIntosh (1996) "Delivering effective lectures."
Three Major Parts of a Lecture
Introduction
Purpose is to Capture Interest and Attention of Students
Deliver Instructor’s Expectations
Body
The Core of the Information Presented
Conclusion
Draws Together Critical Information
Sullivan & McIntosh (1996) "Delivering effective lectures."
Interaction During a Lecture
Ask Questions to The Entire Group
Target A Question to a Specific Student
Use Student Names When Asking a Question
Provide Positive Reinforcement
Repeat Students’ Questions and Answers
Use The Socratic Method
Sullivan & McIntosh (1996) "Delivering effective lectures."
Lecture Presentation Skills
Open With a Good Introduction
Communicate on a Personal Level
Maintain Eye Contact
Exhibit Enthusiasm About the Topic
Project Your Voice
Avoid Using Slang or Repetitive Words
Sullivan & McIntosh (1996) "Delivering effective lectures."
Lecture Presentation Skills
Use a Variety of Audio Visual Aids
Ask A Number of Questions and Encourage Questions
Provide Positive Feedback
Sullivan & McIntosh (1996) "Delivering effective lectures."
Directive Training
Analyze KSAs to get learners from one point to another
Clear performance objectives and matching test items
Learning organized and sequenced to direct the learner
Learners are actively engaged
But still progress along predetermined paths
Suited for learners with little knowledge of experience with the learning content
This method directs “Akin to US Army infantry slogan “Follow Me”
You analyze the KSAs required to get learners from point A to point B
You create clear performance objectives and matching test items
Sequence learning blocks or units that direct the learners from their starting positions to defined intermediate and final finishing lines
Little learner control, but learners are actively and meaningfully engaged
Well suited for learners who have little experience with learning content, require support to build competence and confidence, and will later perform on the job in ways that are identical or very similar to what they learn in training
14
Guided Discovery
Control is shared
Generally case based
Learners take the initiative
Have to discover what to do and when to do it
Amount of support is dependent on the KSAs of the learners
Control is shared between the learner and the trainer or training program
Generally case based- learners immediately plunge into cases, scenarios, or reach out to a variety of information sources or support tools (physical or electronic)
Seek to discover what to do and when to do it
The instructor provides corrective feedback or redirection- offers cues, prompts, suggestions
Amount of guidance or support depends on KSA levels of the learners
15
Exploratory Learning
Trainer builds and organizes rich learning environment
Learners are in control
Only general goals are set
Trainer is a resource for the learner
Self-Directed Learning (SDL)
Choosing Module A or Module B is not SDL
Individuals take initiative to pursue a learning experience
Learners are responsible for completing learning
Learners are accountable for defining the learning experience
And following through to it’s conclusion
Does not mean the learner learns alone or in isolation
Learner drives the total learning experience beginning with recognizing a need to learn
Pulling the Four Together
Receptive is essentially just telling
Directive provides the greatest control
Side effect is decreased learner initiative
Guided discovery is a balanced training approach
Encourages learner initiative under safe conditions
Learning results are stronger and more fluid
Learning results are less predictable and timely
Exploratory Learning
Allows individualization and personalization of learning
Requires sufficient resources, reduces trainer control, and is unpredictable in terms of specific outcomes
Levels of Learner Involvement
Learner Control
Type of Training | Main Use | Amount of Learner Control | Assumed Learner Characteristics | Dangers |
Receptive | Build Awareness Inform Motivate | Practically None, Except for Q&A Learner can choose to tune in or out | Learner is self motivated Learner as prior knowledge Telling is enough for transfer of learning | Without control, learners feel like targets Learners tune out Little sticks in learner’s brain Belief that telling is training |
Directive | Provide a strong, rational path and sufficient feedback for effective initial learning Quickly build basic required KSAs Create initial competence and confidence Predict learning outcomes | Little; learning path predetermined Alternative paths can be offered | Learner is not necessarily motivated Little prior knowledge Weaknesses in metacognitive skills Lack of initiative or confidence to assume control KSAs will be applied in the workplace similar to training | May turn off more independent learners May imply one way of doing things Does not encourage exploration or creativity Limits more advanced learning |
Type of Training | Main Use | Amount of Learner Control | Assumed Learner Characteristics | Dangers |
Guided Discovery | Encourage initiative in safe environment Case-Based Build wider transfer of KSAS Build independence in learning Next step after directive training | Moderate to relatively high | Confidence to engage in discovery Some prior knowledge Good metacognitive skills Self-motivated to learn | Less Confidence-Possible stress and confusion Time intensive Outcomes less predictable |
Exploratory | Create an environment for self-initiated learning Provide maximum freedom for learners Respond to individualized needs | The learning goal, the resources, and the paths to explore are at a maximum | Highly self-motivated to learn Strong prior knowledge in content and/or self-initiated learning Well developed metacognitive skills Knows what is needed and how to find it | Learner can get lost Learner may waste time Not suited for learners lacking appropriate characteristics Learners may not learn what is necessary |
Action Learning
Dynamic process involving small groups
Involves a real world problem
Focus on learning and application of learning
Strategic instead of tactical
Six Components
A Problem
Learning Group or Team
Insightful Questioning and Reflective Listening
Taking Action on a Problem
A Commitment to Learning
An Action Learning Coach
A problem- centers around problem, project, challenge, issue/task, the resolution of which is of high importance to individual, team, and/or organization
Needs to provide opportunity for the group to generate learning opportunities, to build KSAs- can involve a single or multiple problems
Core entity is an action learning group (4-8 individuals) who examine an organizational problem that has no easily identifiable solution
Group should have a diverse background and experience
Emphasizes questions and reflection above statements and opinions. Focus is on the question instead of right answers
Tackles problems by first asking questions to clarify the nature of the problem
Groups must have power to take action themselves or be assured recommendations will be implemented
If group only makes recommendations, it looses significance- No real or practical learning until action is taken and reflected upon
Action begins with taking steps to reframe the problem and determining the goal, and only then determining strategies and taking action
Solving an organizational problem provides immediate, short-term benefits to the company. Greater long-term benefit is the learning gained by each group member as well as the group as a whole and how those learnings are applied on a systems-wide basis
Coaching is necessary for the group to focus on the important-Action learning coach helps team members reflect on what they are learning and how they are solving problems
24
Training and Technology
What is Technology
Two Definitions
Artifacts and Tools
Application of Scientific and Organized Knowledge
Aim to Solve Practical Problems
Two Definitions
Artifacts and Tools for transmitting messages
Other is concerned with the application of scientific and organized knowledge
Both have a single goal to solve practical problems
With learning the practical problem is how to inrease the efficiency and effectiveness of training
26
Efficiency And Effectiveness
Efficiency To Ensure the Measure is Fast and Cheap
Effectiveness How Well the Learning Goal is Achieved
Aim is to Put the Two Together
Efficiency: Getting the most with the least energy expenditure; ratio of useful work to energy expended
Effectiveness: Actual Desired Accomplishment, the Degree to Which Objectives Have Been Met
27
Reasonable Expectations of Technology
Can Improve Efficiency of Training
Little Impact on Effectiveness of Learning
Training Design determines Effectiveness
The Goal
Reduces time in designing, developing, delivering, accessing, updating, and recording learning results
Decreases the cost related to training
And Good; Building Powerful Training that results in valued learning and on the job performance
The dream is to obtain great learning outcomes quickly and at low cost
Reality
Time and budget factors force us to focus on how quickly we can produce, implement, and deliver training
Where we fall is on measuring how well the people we trained to perform as desired and were able to execute successfully back on the job
Reasonable Expectations
Media and Technology can substantially improve the efficiency of training and learning
However little to no impact of effectiveness of learning
The exception is when skills and knowledge to be acquired are of a technological nature and are taught using technology for which the skills and knowledge are required
Effectiveness relies on the design of training
28
Technology Benefits
Accessibility
Instantaneous Response and Feedback
Instantaneous Testing and Feedback
Consistency of Message
Rapidity of Delivery
Accessibility:
More so than any other means, learning content via computers, the web, etc. is infinitely easier to access than the past
Visiting libraries and attending a formal class is no longer required
Learning management systems (Moodle) make locating courses a touch of the fingers exercise
High speed Internet access allows instantaneous delivery
Instantaneous Response and Feedback
Learning can be delivered quickly and the system can adapt based on the design of the learning program based on learner progress
If learners require help, tools can be included, for example a glossary
Feedback on learner responses requires no discernable delay
Feedback can simply confirm or reject a response OR It can elaborate on a learner’s response
Go Over Adaptive Technology
Instantaneous Testing and Feedback
Can not only capture and record test results and responses but can have these interpreted and programmed to adapt the presentation based on how the learner responded to test items
Consistency of Message
Rapidity of Delivery
Can deliver training to over 1,000 people at the same time
Cost and Feasibility prohibits delivering in person to this many people
29
Technology Benefits
Simultaneity of Training Delivery
Ease of Update
Reusability
Flexibility of Use
Interactivity
Adaptability
Simultaneity of Training Delivery
All employees can receive training in a short period of time
No borders to cross to deliver the training
Electronic highways have few barriers to delay transmission
Ease of Update
Use example of removing a dead video link and replacing it on Moodle
Reusability
Parts of training can be reused and reworked into handy sets of on-the-job application tools
Parts of courses can be lifted and adapted or repurposed for different learning groups
Flexibility of Use
Synchronous Training
Asynchronous Training
Requires significantly more time and cost to develop
Allows for flexibility of use
Anyone at any time can sign up and take the training on his or her own schedule
Interactivity
Technology delivered training requires active responding on part of the learner
Leads to increased attention and retention
The key to interactivity is the Instructional Design
The key is in the design and what is programmed into the technological medium
Adaptability
Easily adapt a course to different learning populations, location, and needs
Instructions and Guidelines are simple to reword
ADA
30
10 Promises Technology Companies Make
Enormous Savings
Desktop Delivery
Greater Active Engagement
More Tailored and Targeted Instruction
More Up to Date Instruction
Just In Time Learning
Start with example of Moodle (cost, equipment, Mainframe, etc.)
31
Promises Continued
Any Time, Any Place Learning
Built-In Testing
Accurate Up-To-Date Training Records
Reusability of Training
Tools for Live Training
PowerPoint
Handouts
Videos
Audio Clips
Audience Participation Technologies
Use Clickers Example
33
HRE7271/Week 5/A revised LTSI 2012.pdf
A revised learning transfer system inventory: factorial replication and validation
Reid Batesa*, Elwood F. Holton IIIa and John Paul Hatalab
aHuman Resource, Leadership and Organization Development, School of Human Resource Education & Workforce Development, College of Human Sciences and Education, Louisiana State University, Baton Rouge, LA, USA; bDepartment of Service Systems, School of International Hospitality and Service Innovation, College of Applied Science & Technology, Rochester Institute of Technology, Rochester, NY, USA
(Received 9 March 2012; final version received 23 August 2012)
The learning transfer system inventory (LTSI) is an empirically derived self-report 16-factor inventory designed to assess individual perceptions of catalysts and barriers to the transfer of learning from work-related training. Although a good deal of research has been done addressing various dimensions of the LTSI’s construct validity, minor discrepancies in factor solutions in several studies together with problematic fit of some items suggest that further construct validity research is needed. Using data collected in 17 countries and utilizing 14 different language versions of the LTSI, the research objectives for this research were to (1) determine the number and nature of common factors involved to account for the pattern of correlations among the measured variables in LTSI version 3 using exploratory factor analysis (EFA) and (2) test via confirmatory factor analysis the validity of the factorial structure of the LTSI that emerged from the EFA and scale refinement efforts. Results provided strong support for the five- and 11- factor structure of the program-specific and training-general domains of a 48-item LTSI.
Keywords: learning transfer; personnel training; test validity
Transfer of learning is defined as the extent to which knowledge, skills and abilities learned in work-related training are generalized and maintained on the job. Reviews of learning transfer research over the years (e.g. Baldwin and Ford 1988; Baldwin, Ford, and Blume 2009; Ford and Weissbein 1997) consistently point to the complexity of the learning transfer process: it is a dynamic one that moves from pre- training experiences to the acquisition of knowledge and skills, to the capability to apply new learning to job-related tasks, to the application of learning to tasks and activities beyond those that were initially targeted by the training. This complex process is influenced by a variety of factors, a ‘transfer system’ (Holton 2003), which includes individual differences and perceptions (e.g. motivation, efficacy beliefs, expectations), factors within the training event (e.g. job relevant content, appropriate
*Corresponding author. Email: [email protected]
Human Resource Development International Vol. 15, No. 5, November 2012, 549–569
ISSN 1367-8868 print/ISSN 1469-8374 online ! 2012 Taylor & Francis http://dx.doi.org/10.1080/13678868.2012.726872 http://www.tandfonline.com
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
learning activities) and contextual factors (e.g. supervisor and co-worker support, group norms about change, rewards). The implication is that learning transfer can only be understood and fostered by examining a relatively complete system of influences.
Although a good deal of progress has been made in our understanding of some dimensions of the learning transfer process such as training design factors (Bell and Kozlowski 2008; DeRouin, Fritzsche, and Salas 2004; Ford and Kraiger 1995), knowledge of other factors including trainee perceptions and contextual factors has lagged behind (Baldwin et al. 2009). We contend that transfer practice and research would benefit from a more concerted effort to investigate, compare and report a core set of individual, training design and contextual factors known to be critical for successful transfer.
This view prompted the development of the learning transfer system inventory (LTSI) more than 15 years ago. The development of this tool grew out of the recognition that a wide variety of measures were being used in transfer research to assess various factors affecting learning transfer. Not only there was little overlap in instruments used from one study to another but there was also wide variation in the quality of measurement tools being used. Very few measurement tools had undergone rigorous development and testing and, as a result, the psychometric qualities of many instruments were open to question (Holton, Bates, and Ruona 2000). Still today there are differences in measurement instruments and instrument quality that continue to limit our ability to generalize findings across studies and draw valid conclusions about the factors affecting learning transfer in the workplace.
The LTSI is an empirically derived self-report inventory designed to assess individual perceptions of catalysts and barriers to the transfer of learning from work- related training. The goal in the ongoing development of the LTSI has been to provide researchers and practitioners with an instrument that contains a core set of rigorously developed and validated transfer system scales reflecting those elements critical to promoting effective learning transfer in the workplace.
Research with the LTSI
Research done with the LTSI in recent years has provided evidence of the convergent and divergent validity of the LTSI scales (Holton, Bates, and Bookter 2007) as well as evidence of the instrument’s criterion-related validity. Studies have shown LTSI scales to be correlated with self-reported learning transfer (Devos et al. 2007), intent to transfer (Holton and Bates 2011) and perceived utility of training (Ruona et al. 2002). Other studies have provided evidence of LTSI scales’ capacity to predict learning and post-training knowledge retention (Myers 2009), motivation to transfer (Seyler et al. 1998), changes in job performance as a result of learning transfer (Bates, Kauffeld, and Holton 2007; Bates et al. 2000; Fitzgerald 2002), organizational performance (Bates et al. 2007) and organizational innovation (Bates and Khasawneh 2005). Finally, research suggests the LTSI can distinguish different configurations of learning transfer systems across organizations (Holton, Chen, and Naquin 2003) and training types (Khasawneh, Bates, and Holton 2006).
The current version of the LTSI (version 3) emerged from a research program initiated in 1997 with version 1, a 63-item instrument that assessed nine constructs (see Holton et al. 1997). Version 1 was originally constructed with data from a sample of workers in a petro-chemical manufacturing plant in the southern US.
550 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
Following this research, several revisions were made to version 1. New items to measure additional scales were added and the instrument was fit to the conceptual structure of the HRD Research and Evaluation Model (Holton 1996). The result was version 2, an instrument expanded to 112 items and measured 16 constructs.
The scales in version 2 were created to assess factors affecting a trainee’s ability to transfer learning, motivation to transfer and key transfer environment factors. The 16 constructs represented two distinct construct domains: 11 constructs represented factors affecting a specific-training program (program-specific factors) and five factors were classified as training-general factors because they represented perceptions of factors extending across training programs (training-general factors). The constructs in these two domains represent four categories of elements consistent with those in the HRD Research and Evaluation Model: (1) secondary elements, (2) ability/enabling factors, (3) motivation factors and (4) work environment factors.
Exploratory factor analysis (EFA) of the version 2 instrument (Holton et al. 2000) conducted with a large and diverse sample yielded version 3 of the LTSI, a 68- item instrument that reproduced the same 16 constructs. Subsequent to this analysis, and in an effort to more clearly define and assess some of the constructs, an additional 21 items were added to create version 3 of the LTSI, an 89-item instrument.
Validation research with different language versions
In the last 10 years, version 3 has been translated into 17 languages. Validation studies with several of the translated versions have provided evidence supporting the 16-factor structure. For example, studies in Belgium with the French version (Devos et al. 2007), Germany (German version) (Bates, Kauffeld, and Holton 2007), Greece (Greek version) (Holton and Bates 2011) and Thailand (Thai version) (Yamnill 2001) using a consistent EFA methodology (common factor analysis with oblique rotation) produced factor solutions equivalent to both the 11-factor structure of the program-specific domain and the 5-factor structure of the training-general domain identified in the Holton et al. (2000) study.
Other validation studies have produced factor solutions that, although consistent with these findings, show minor factorial variations. A study using the Jordanian/ Arabic version concluded that a 12-factor structure was optimal for the program- specific domain (Khasawneh et al. 2006). Ten of the 12 factors matched those in the Holton et al. (2000) program-specific domain. The other two factors included a three-item factor that combined items from the personal capacity and negative personal outcomes scale. This scale showed marginal loadings (.33–.46) and a low reliability estimate (a¼ .48). The twelfth factor was an unstable two-item factor also with a low reliability estimate (a¼ .55). Analysis of the training-general domain produced a six-factor solution in which the performance coaching scale separated into two factors reflecting different modes of feedback (verbal advice and active assistance). Another EFA construct validation study using the Taiwan/Chinese version (Chen, Holton, and Bates 2006) showed, for the program-specific domain, an 11-factor structure in which 10 factors matched the Holton et al. (2000) structure. However, a new program-specific factor emerged which combined items associated with the transfer design and opportunity to use learning constructs. There were also several items from the opportunity to use scale that reflected problematic fit. Exploratory factor analysis of the training-general domain showed a five-factor
Human Resource Development International 551
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
structure highly consistent with the five identified in the Holton et al. (2000) study. An EFA with data collected with the Ukrainian translation (Yamkovenko, Holton, and Bates 2007) produced 11 interpretable factors that paralleled the 11 factors in the program-specific domain found by Holton et al. (2000). Analysis of this domain also pointed to problematic fit with items from the opportunity to use, personal capacity and personal outcomes positive scales. Exploratory factor analysis of the training-general domain items produced results that matched the factor structure of the training-general domain in the Holton et al. (2000) study.
In summary, exploratory factor analyses with eight translated versions of the 89- item version 3 LTSI have provided results generally consistent with the expected 16- factor structure. However, minor discrepancies in factor solutions in some studies together with problematic fit of particular items suggest that further construct validity research is needed more to fully establish the psychometric soundness of the LTSI. In addition, without exception, the factorial validity of the LTSI in published studies has been examined using only an EFA approach. In light of the deficiencies associated with EFA approaches (e.g., Bollen 1989), it is clear that a confirmatory factor analysis (CFA) approach would provide a much more powerful test of factorial validity.
This study was designed to address these issues and had two primary objectives: (1) to test the number and nature of common factors needed to account for the pattern of correlations among the measured variables in the 89-item LTSI using EFA; (2) to cross-validate via CFA the validity of the factorial structure of the LTSI that emerged from the EFA and scale refinement efforts.
Method
Instrument
Version 3 of the LTSI is an 89-item self-report survey designed to assess individual perceptions of barriers and catalysts to the transfer of work-related learning. Each item is rated on a five-point Likert-type scale ranging from one (strongly disagree) to five (strongly agree).
Sample
A total of 5990 participants drawn from a variety of organizations provided data for this study. The sampling strategy was purposive insofar as the data were collected from individuals who had recently attended or attending organization-sponsored training programs. Data were collected over a period of 10 years by the authors or by other researchers or practitioners who used the LTSI. Respondents completed the LTSI in either paper form or through an electronic survey administered at the conclusion of training. The sample included data collected in 17 countries using 14 different language versions of the LTSI. Respondents represented a variety of industries including health care, banking, insurance, information technology, municipal and state governments, manufacturing, engineering, health, higher education, telecommunications, petroleum, retail, insurance, hotel and transporta- tion. They came from a variety of work roles such as nurses, high-school teachers, mechanical and electrical engineers, technicians, manufacturing operators, secre- taries, mid- and upper-level management, customer service and customer relations, housekeeping, sales, direct service staff/mental health, supervisors and junior and
552 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
senior managers. Respondents attended a wide range of training programs that addressed a variety of knowledge and skill elements including those related to interpersonal communication, technical specialties (e.g. engineering), administration, counseling, information and computer technology, supervision, management and leadership and ethical decision making. The descriptive data available for the sample are shown in Table 1.
The data in this study included data from previously published studies and unpublished data. This approach provided a large, heterogeneous sample which minimized the impact sample specific variance might have on factorial results and increased statistical power. As EFA is a large-sample procedure and is prone to error in small samples, the most generalizable and replicable results are obtained with large diverse samples (Costello and Osborne 2005). In addition, the underlying statistical theory for CFA assumes that elements in the sample covariance matrix are equal to the population values. For this reason, large sample sizes are necessary for proper estimation in CFA (Kahn 2006). From a factorial validation perspective, a large sample provides the opportunity to subdivide the sample and yet retain two large working sub-samples, allowing one sub-sample for EFA and the second for CFA of the EFA results. This approach is widely viewed as more useful than doing a single analysis with the entire sample (Floyd and Widaman 1995).
Analysis
Exploratory factor analysis and item reduction
Data from the full sample were randomly divided into two sub-samples (A and B) of approximate equal size in preparation for subsequent analysis. Data from sub- sample A were used for the initial EFA and the data from sub-sample B were used for CFA. Random assignment was important for cross-validation purposes in order to control for differences across the sub-samples that might affect the factor structures within the separate sub-samples.
Exploratory factor analysis using common factor analysis (principle axis factoring) is generally regarded as preferable to principle components analysis (Costello and Osborne 2005). Because the LTSI factors are assumed to be correlated, oblique rotation (direct oblimin) was selected as the rotation method. The analysis was conducted using SPSS version 19. Exploratory factor analysis is a data-driven methodology in which no a priori specification of the number of factors is made. Nevertheless, it is not uncommon for researchers using EFA to have expectations about the latent structure that may emerge. That was certainly the case in this study. However, the subjectivity and uncertainty associated with the various criteria used for factor extraction in EFA often lead to unreliable factor solutions. In an attempt to address this shortcoming several criteria were used to guide decisions on the number of factors to extract. This included the eigenvalue 41 criterion, the scree plot to determine the point at which the slope of eigenvalues approached zero, and the number of variables that had strong factor loadings (".50) on a factor. At least three variables per factor were required to identify common factors that are stable (Comrey 1988).
In addition to EFA, analysis was directed at evaluating items in an effort to refine and focus scale content while adequately assessing the construct domain. Item evaluation and retention decisions were based on both strict criteria as well as the expert judgment of the authors based on their relatively long research history with
Human Resource Development International 553
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
1 .
D es cr ip ti v e d a ta .
C o u n tr y
N L T S I
la n g u a g e
In d u st ry
ty p e( s)
T ra in in g ty p e
T ra in ee
ty p e
C a n a d a
2 8 7
F re n ch /
Q u eb ec
7 7
7
C h in a *
5 9 0
M a n d a ri n
C h in es e
H o te l
C u st o m er
re la ti o n s,
co m p u te r sk il ls a n d
te ch n ic a l sk il ls
N o n -m
a n a g er ia l fr o n t-
li n e em
p lo y ee s (e .g .
fr o n t d es k ,
h o u se k ee p in g )
G er m a n y *
6 7 4
G er m a n
E le ct ri ca l en g in ee ri n g ,
m ec h a n ic a l
en g in ee ri n g ,
co n st ru ct io n , fi n a n ci a l
se rv ic es , so ci a l
se rv ic es , h ea lt h ,
p h a rm
a ce u ti ca ls ,
a u to m o ti v e
m a n u fa ct u ri n g , IT
,
7 7
G re ec e
4 3 0
G re ek
B a n k in g
T ec h n o lo g y ,
m a n a g em
en t a n d
cu st o m er
se rv ic e
M a n a g er s a n d
cu st o m er
se rv ic e
re p re se n ta ti v es
Ic el a n d
1 2 0
Ic el a n d ic
7 7
7 Ir a n
1 9 5
F a rs i
B a n k in g
A d m in is tr a ti o n ,
fi n a n ci a l
m a n a g em
en t, h u m a n
re so u rc es ,
m a n a g em
en t a n d
st ra te g ic
ch a n g e
S u p er v is o rs
a n d
m a n a g er s
Ir el a n d
1 5 0
E n g li sh
F in a n ci a l se rv ic es
L ea d er sh ip
d ev el o p m en t
Ju n io r a n d se n io r
m a n a g er s
Jo rd a n *
3 2 5
A ra b ic
M a n u fa ct u ri n g , h ig h -
te ch
a n d b a n k in g ,
in su ra n ce
In te rp er so n a l sk il ls ,
cu st o m er
re la ti o n s,
n ew
em p lo y ee
(c o n ti n u ed )
554 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
1 .
(C o n ti n u ed ).
C o u n tr y
N L T S I
la n g u a g e
In d u st ry
ty p e( s)
T ra in in g ty p e
T ra in ee
ty p e
o ri en ta ti o n , w eb p a g e
d es ig n , co m p u te r
sk il ls , sa fe ty
a n d
a cc id en t a n d
co m p en sa ti o n
M a la y si a
2 3 6
B a h a sa
M a la y si a n
7 7
7
T h e N et h er la n d
1 4 2
D u tc h
7 7
7 P o rt u g a l*
4 8 9
P o rt u g u es e
In d u st ri a l
m a n u fa ct u ri n g ,
b u si n es s a n d fi n a n ci a l
se rv ic es
a n d in su ra n ce
M a n a g em
en t,
te ch n o lo g y a n d
in te rp er so n a l sk il ls
7
S o u th
A fr ic a
1 5 1
E n g li sh
B a n k in g
F in a n ci a l se rv ic es ,
cu st o m er
se rv ic e a n d
m a n a g em
en t
M a n a g er s a n d
cu st o m er
se rv ic e
re p re se n ta ti v es
T a iw a n *
6 9 8
T a iw a n
C h in es e
E d u ca ti o n , ci v il se rv ic e,
el ec tr o n ic s, in su ra n ce ,
p et ro le u m , so ci a l
w o rk , re ta il ,
tr a n sp o rt a ti o n a n d
te le co m m u n ic a ti o n s
C o m p u te r sk il ls ,
cu rr ic u lu m
d ev el o p m en t,
cu st o m er , se rv ic e, m id -
le v el m a n a g em
en t, n ew
em p lo y ee
o ri en ta ti o n ,
cu st o m er
se rv ic e,
q u a li ty
m a n a g em
en t,
sa fe ty , sp ir it u a l,
sy st em
s o p er a ti o n s,
a cc o u n ti n g , tr a in -t h e-
tr a in er
a n d m a ch in e
m a in te n a n ce
7
T u rk ey
3 9 8
T u rk is h
7 7
7
(c o n ti n u ed )
Human Resource Development International 555
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
1 .
(C o n ti n u ed ).
C o u n tr y
N L T S I
la n g u a g e
In d u st ry
ty p e( s)
T ra in in g ty p e
T ra in ee
ty p e
U k ra in e*
3 0 0
U k ra in ia n
H ea lt h , h ig h er
ed u ca ti o n , re ta il a n d
a g ri cu lt u re
S a le s, m a rk et in g ,
co m p u te r sk il ls ,
te a ch in g sk il ls ,
ex te n si o n se rv ic es ,
cu st o m er
se rv ic e,
m ed ic a l eq u ip m en t
a n d p h a rm
a ce u ti ca l
U K
1 8 4
E n g li sh
7 7
7 U S
6 2 1
E n g li sh
M en ta l h ea lt h , h ig h
te ch n o lo g y , m en ta l
h ea lt h co u n se li n g ,
se co n d a ry
ed u ca ti o n
a d m in is tr a ti o n a n d
co u n se li n g
T ec h n o lo g y tr a in in g ,
co n fi g u ri n g
te ch n o lo g y ,
a d m in is tr a ti o n ,
tr o u b le -s h o o ti n g
h a rd w a re
a n d
so ft w a re , et h ic a l
d ec is io n m a k in g ,
p o si ti v e b eh a v io u r
in te rv en ti o n s,
fu n ct io n a l a ss es sm
en t,
cl a ss ro o m
in te rv en ti o n s a n d
cu rr ic u lu m -b a se d
m ea su re m en t
D ir ec t se rv ic e st a ff
(m en ta l h ea lt h ),
se rv ic e en g in ee rs ,
h ig h -s ch o o l
co u n se lo r a n d
a d m in is tr a to rs
T o ta l 5 9 9 0
N o te : * In d ic a te s p u b li sh ed
st u d y . E m p ty
ce ll s in d ic a te
th e d es cr ip ti v e d a ta
w er e n o t a v a il a b le
fr o m
th e re se a rc h er s.
556 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
the instrument. Criteria for item deletion included cross-loading".32 (Costello and Osborne 2005). In cases where an inter-item correlation was more than moderate (r 4 .69, Westgard 1999), the item with the lower factor loading was removed. Scale reliability estimates were also considered in making item retention decisions. Although it is desirable to maximize coefficient alpha, there are limitations with this as a strategy for item retention as it can increase scale homogeneity, increase item redundancy and reduce the breadth and dimensions of a construct assessed by a measurement scale (Coste et al. 1997). Therefore, expert judgment was used to identify those items that would enhance heterogeneity, avoid item redundancy in a scale and strengthen the practical significance of the scales. Enhancing item heterogeneity within a factor can contribute to validity across cultural settings: if all the items in a scale are similar and a given item elicits discrepant responses across cultures then item redundancy could increase measurement error (Boyle 1991).
Confirmatory factor analysis
Exploratory factor analysis is most appropriately used to explore a data set and researchers are cautioned against drawing substantive conclusions based on EFA results (Floyd and Widaman 1995). Confirmatory factor analysis, on the other hand, allows the testing of theoretically or empirically based hypotheses about the number of factors and which variables correspond to which factors. Among factor analytic procedures CFA is considered a powerful test of factorial validity and is seen as the approach most capable of confirming hypothesized factor structures (Bentler 1989; Byrne 1993). Therefore CFA, using AMOS 7, was used to confirm the hypothesized factor structure that emerged from the EFA and scale refinement efforts. In CFA, the LTSI factor structure was tested for fit with the observed covariance structure of the measured variables. Confirmatory factor analysis also allows the testing of alternative models and can thus be used to further refine instruments. Since initially specified models nearly always fail to provide acceptable fit, the models must be respecified, tested again using the same data and then the final model cross-validated with another sample (Anderson and Gerbing 1988). Therefore, two sub-samples were created for the CFA (CFA1 and CFA2) from sub-sample B data. Confirmatory factor analysis 1 was used to test the EFA results and to suggest model respecifications, if needed. Confirmatory factor analysis 2 was set aside to test the final model.
Separate CFA models were run for the program-specific domain and the training-general domains. The CFA models depicting the 11-factor structure of the program-specific domain and the five-factor structure of the training-general domain hypothesized that (1) responses to the items in these domains could be explained by 11 and five factors, respectively, (2) each item would have a non-zero loading on the LTSI factor it was designed to measure and zero loadings on all other factors, (3) the 11 factors in the training-specific domain would be correlated as would the five factors in the training-general domain, and (4) measurement error terms would be uncorrelated.
Model fit was assessed via examination of the chi-square likelihood ratio (w2) and several descriptive goodness-of-fit indices. Lower values of chi-square indicate better fit and should be non-significant. However, in large samples the use of chi-square as an index of fit is contraindicated. Therefore, it was important to use descriptive fit indices to judge acceptable model fit that were not adversely affected by sample size.
Human Resource Development International 557
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
For this study, CFA models were judged to have acceptable fit if the incremental fit index (IFI), the comparative fit index (CFI) and the Tucker-Lewis index (TLI) equaled or exceeded the cutoff value of .95 recommended by Hu and Bentler (1999). For all of these indices, values close to one indicate very good fit. The root mean square error of approximation (RMSEA) was also examined with values falling below .5 indicating a close fit of the model in relation to the degrees of freedom.
Results
Exploratory factor analysis and scale refinement
Using data from sub-sample A, the factor structure of the 89-item LTSI was assessed using EFA. The raw data were used as input. List-wise deletion was used for missing data. The 11-factor program-specific domain (63 items) and the five-factor training- domain (26 items) were factor analysed separately.
Program-specific domain
The measure of sampling adequacy (MSA), a measure of the data set’s appropriateness for factor analysis, was .94 for the program-specific data. Values ".90 are considered strong and indicate the data are suitable for factoring (Hair et al. 1998). The initial EFA for this domain resulted in a 12-factor solution that explained 60.97% of the common variance. However, examination of the solution indicated a two-item factor which did not meet the three-item per factor minimum required for stable factors. Closer examination of the two items that loaded as a separate factor showed these to be negatively worded items that had created problematic fit issues in previous validation studies of the LTSI. As a result, the two problematic items were deleted and the EFA was run again.
The MSA for the subsequent EFA was also .94. The results indicated an exceptionally clean and interpre table 11-factor structure in which all the items loaded on the anticipated factors. The solution explained 60.28% of the common variance. Table 2 shows factor loadings for all of items in the program-specific domain. Inter-scale correlations (Pearson’s) and reliability estimates are shown in Table 3. Reliability estimates ranged from .71 to .85 all in the acceptable range (Nunnally and Bernstein 1994).
Subsequent to the EFA additional analysis was directed at evaluating scale items and, where possible, refining scale content to reduce item homogeneity, better reflect the underlying construct, and where reasonable, to reduce scale length. Shorter scales are generally preferable from both a research and applied practice perspective because they increase organizational and respondent acceptance, minimize comple- tion time and diminish respondent fatigue. For most scales, the three highest loading items were retained as this is the minimum needed for stable scales (Comrey 1988). However, there were two noteworthy exceptions. First, item 46 (.67) was selected for the supervisor opposition scale even though its pattern coefficients were lower than several other items. Other items with higher pattern coefficients (item 34/.80, item 38/ .74 and item 42/.73) were not retained because they were closely similar in meaning to one of the other retained items (items 36 or 35). Item 41 (.73) was not retained because it was less consistent with the conceptualization of the supervisor opposition construct. Item 46 was selected because it was consistent with the construct definition and enhanced item heterogeneity in a desirable direction. Second, for the personal
558 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
2 .
E x p lo ra to ry
fa ct o r a n a ly si s ro ta te d fa ct o r p a tt er n a n d p a tt er n co effi
ci en t fo r it em
s in
th e p ro g ra m -s p ec ifi c d o m a in .
It em
O p p o rt u n it y
to u se
S u p er v is o r
o p p o si ti o n
S u p er v is o r
su p p o rt
P er so n a l
o u tc o m es
p o si ti v e
P er so n a l
o u tc o m es
n eg a ti v e
P er so n a l
ca p a ci ty
L ea rn er
re a d in es s
P ee r
su p p o rt
M o ti v a ti o n
to tr a n sf er
T ra n sf er
d es ig n
C o n te n t
v a li d it y
5 6
.5 3
7 .0 9
7 .0 1
7 .0 2
7 .0 1
.0 8
.0 0
.1 0
.0 1
.1 4
7 .0 8
5 5
.5 1
7 .0 8
7 .0 1
.0 1
7 .0 3
.0 1
.0 2
.0 7
7 .1 6
.1 3
7 .0 6
5 1
.5 0
7 .1 3
7 .0 1
7 .0 2
7 .0 2
7 .0 1
7 .0 1
.1 1
7 .0 8
.1 8
7 .0 3
6 0
.4 6
7 .0 1
7 .0 6
7 .0 4
7 .0 2
7 .0 1
.0 6
.0 3
.0 0
.0 8
7 .0 8
6 2
.4 6
7 .0 1
7 .0 4
7 .0 3
.0 0
7 .0 6
.0 5
.0 3
.0 2
.0 2
7 .1 0
5 0
.4 4
7 .0 5
7 .0 7
.0 3
7 .0 5
.0 3
.0 2
.0 2
7 .0 9
.0 7
7 .2 2
1 9
.3 4
.1 4
7 .0 4
7 .1 0
.0 3
7 .3 4
.1 0
.0 6
7 .0 7
7 .0 4
.0 1
3 6
7 .0 2
.8 3
7 .0 4
.0 1
7 .0 2
7 .0 1
.0 1
7 .0 1
.0 0
7 .0 2
7 .0 2
3 5
.0 1
.8 2
7 .0 2
.0 1
7 .0 2
7 .0 2
.0 0
7 .0 2
7 .0 3
7 .0 1
.0 0
3 4
7 .0 2
.8 0
7 .0 1
.0 3
.0 2
7 .0 2
.0 0
.0 1
7 .0 5
7 .0 2
7 .0 2
3 8
.0 3
.7 4
.0 1
.0 0
7 .0 5
.0 2
.0 0
7 .0 3
.0 3
.0 1
.0 3
4 2
7 .0 5
.7 3
.0 8
7 .0 3
.0 4
.0 1
.0 2
.0 1
.0 8
.0 0
7 .0 6
4 1
7 .0 3
.7 3
.0 1
7 .0 3
.0 2
.0 5
.0 1
7 .0 2
.0 2
.0 0
7 .0 3
4 6
7 .0 3
.6 7
.0 2
7 .0 1
7 .0 9
.0 2
7 .0 1
.0 1
7 .0 3
7 .0 6
7 .0 7
4 5
7 .0 1
.6 6
7 .0 2
.0 3
7 .0 3
.0 5
7 .0 1
.0 1
.0 3
.0 5
.0 5
4 4
.0 3
.4 0
.0 0
7 .0 3
.0 0
.0 7
7 .0 2
.0 3
7 .0 4
.0 6
.0 8
3 3
7 .0 4
.0 6
7 .7 8
.0 1
7 .0 6
7 .0 7
.0 1
.0 2
7 .0 2
7 .0 5
7 .0 7
3 2
7 .0 7
.0 4
7 .7 7
.0 0
7 .0 8
.0 0
.0 1
7 .0 1
7 .0 3
7 .0 3
7 .0 7
4 3
.0 2
.0 5
7 .7 1
7 .0 6
.0 3
7 .0 1
.0 2
7 .0 4
.0 3
.0 5
7 .0 1
4 0
.0 1
7 .0 6
7 .7 0
7 .0 3
.0 3
.0 8
7 .0 1
.0 8
7 .0 7
.0 2
.0 4
3 9
.0 1
.0 0
7 .6 9
7 .0 4
7 .0 5
7 .0 2
.0 2
.0 2
7 .0 4
.0 2
.0 4
3 7
.0 8
7 .1 3
7 .6 6
7 .0 2
.0 5
.0 7
7 .0 1
.0 8
.0 3
.0 6
.0 6
8 7 .0 3
.0 2
7 .0 1
7 .8 6
.0 6
.0 4
.0 3
.0 2
.0 5
.0 2
.0 1
7 7 .0 2
7 .0 4
7 .0 2
7 .8 2
.0 6
7 .0 2
.0 3
.0 2
7 .0 3
.0 0
.0 0
6 .0 1
.0 4
7 .0 3
7 .7 8
.0 0
.0 3
.0 2
7 .0 7
7 .0 3
7 .0 3
7 .0 8
2 2
.0 3
.0 1
7 .0 3
7 .6 4
7 .2 3
7 .0 4
7 .0 3
7 .0 4
7 .0 1
.0 0
7 .0 6
1 6
7 .0 1
.0 6
7 .0 6
7 .6 2
7 .1 1
.0 4
.0 0
.0 5
.0 3
.0 3
7 .0 3
1 5
.0 4
7 .0 7
7 .0 6
7 .4 6
7 .0 7
7 .0 2
.0 2
.1 2
7 .1 2
.0 7
.0 8
(c o n ti n u ed )
Human Resource Development International 559
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
2 .
(C o n ti n u ed ).
It em
O p p o rt u n it y
to u se
S u p er v is o r
o p p o si ti o n
S u p er v is o r
su p p o rt
P er so n a l
o u tc o m es
p o si ti v e
P er so n a l
o u tc o m es
n eg a ti v e
P er so n a l
ca p a ci ty
L ea rn er
re a d in es s
P ee r
su p p o rt
M o ti v a ti o n
to tr a n sf er
T ra n sf er
d es ig n
C o n te n t
v a li d it y
1 8
.0 4
7 .0 3
7 .0 7
7 .3 4
7 .0 9
7 .0 3
.0 0
.1 7
7 .2 0
.0 7
.0 9
2 3
7 .0 1
.0 8
7 .0 6
7 .0 1
7 .7 6
7 .0 4
.0 0
7 .0 1
.0 0
.0 1
.0 1
2 1
.0 3
.0 5
.0 0
.0 1
7 .7 6
.0 9
7 .0 1
7 .0 2
7 .0 2
7 .0 4
7 .0 4
1 4
7 .0 2
7 .0 1
.0 0
7 .0 4
7 .6 4
.0 4
.0 2
7 .0 2
.0 4
7 .0 3
7 .0 2
2 4
.0 1
7 .0 3
7 .0 6
7 .0 1
7 .5 3
7 .0 5
.0 3
.1 5
.0 0
.0 1
.0 1
1 7
.0 7
.0 8
.0 2
7 .3 0
7 .3 7
.0 5
.0 0
7 .0 3
7 .0 2
7 .0 1
7 .0 3
1 1
.0 8
.1 3
7 .0 2
7 .0 5
.0 7
.6 9
7 .0 1
7 .0 3
.0 7
7 .1 1
7 .0 3
1 2
.1 1
.1 2
7 .0 8
7 .0 1
7 .0 2
.6 7
.0 1
7 .0 8
.0 5
7 .0 9
7 .0 2
2 0
7 .0 2
.0 2
7 .0 2
7 .0 2
7 .0 5
.6 5
.0 3
7 .0 4
.0 1
7 .0 4
.0 0
2 6
7 .0 5
.1 0
7 .0 2
.0 1
7 .1 1
.3 9
.0 0
.0 1
7 .0 6
7 .0 1
.0 4
2 7
7 .1 4
.0 0
.0 8
7 .0 4
7 .0 6
.3 6
.0 1
.1 2
7 .0 6
.0 8
.0 6
2 5
.3 3
.1 3
.0 0
7 .0 9
7 .0 3
7 .3 4
.0 6
.1 1
7 .0 9
7 .0 1
.0 1
1 0
7 .0 2
7 .0 1
.0 0
.0 3
.0 2
7 .0 4
.7 7
.0 0
.0 4
7 .0 4
7 .0 3
9 .0 6
7 .0 5
.0 0
7 .0 8
.0 1
.0 2
.6 1
.0 3
7 .0 9
7 .0 7
7 .0 1
1 .0 2
.0 4
.0 1
7 .0 5
.0 0
.0 2
.5 7
.0 0
7 .1 1
7 .0 6
7 .0 3
1 3
7 .0 4
7 .0 1
7 .0 4
.0 4
7 .0 4
.0 4
.4 8
.0 1
.0 5
.1 6
.0 3
2 9
.0 1
.0 7
7 .0 5
7 .0 4
.0 1
7 .0 2
.0 3
.8 1
.0 3
7 .0 4
7 .0 6
2 8
.0 1
7 .0 1
.0 4
7 .0 2
.0 3
.0 0
.0 1
.7 3
7 .0 6
.0 2
7 .0 1
3 0
.0 2
.0 2
7 .0 3
.0 1
7 .1 0
.0 0
.0 0
.6 9
7 .0 1
.0 2
7 .0 4
3 1
.0 7
7 .0 4
7 .1 4
.0 1
.0 0
7 .0 2
.0 3
.5 1
.0 4
7 .0 4
7 .0 7
4 .0 5
7 .0 4
.0 2
.0 2
7 .0 1
7 .0 3
.0 5
7 .0 4
7 .7 6
.0 1
.0 1
3 .0 3
.0 0
7 .0 1
7 .0 3
7 .0 2
.0 2
7 .0 1
.0 1
7 .7 0
.0 0
7 .0 6
2 7 .0 2
.0 3
7 .0 5
7 .0 3
.0 2
.0 0
.0 9
7 .0 1
7 .6 6
7 .0 1
7 .0 1
5 7 .0 5
.0 1
7 .0 5
7 .0 1
.0 4
.0 0
7 .0 3
.0 6
7 .5 8
.0 5
7 .0 2
5 4
7 .0 2
.0 2
7 .0 4
7 .0 4
.0 6
7 .0 3
.0 0
.0 2
.0 3
.7 5
7 .0 5
5 5
.0 6
.0 2
.0 0
7 .0 4
.0 2
7 .0 7
.0 2
.0 0
7 .0 5
.6 8
.0 0
5 3
.0 3
.0 1
7 .0 5
.0 1
7 .0 1
7 .0 4
.0 5
.0 2
.0 0
.6 4
7 .0 7
5 2
.1 4
7 .0 1
7 .0 4
.0 0
.0 2
7 .0 1
.0 0
7 .0 1
7 .1 4
.5 4
7 .0 6
(c o n ti n u ed )
560 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
2 .
(C o n ti n u ed ).
It em
O p p o rt u n it y
to u se
S u p er v is o r
o p p o si ti o n
S u p er v is o r
su p p o rt
P er so n a l
o u tc o m es
p o si ti v e
P er so n a l
o u tc o m es
n eg a ti v e
P er so n a l
ca p a ci ty
L ea rn er
re a d in es s
P ee r
su p p o rt
M o ti v a ti o n
to tr a n sf er
T ra n sf er
d es ig n
C o n te n t
v a li d it y
5 9
.1 9
7 .0 1
7 .0 2
.0 1
7 .0 4
7 .0 3
.0 7
7 .0 3
7 .0 5
.4 0
7 .2 0
5 8
.2 2
7 .0 4
.0 0
7 .0 5
7 .0 7
7 .0 3
.0 8
.0 0
7 .1 2
.2 9
7 .1 9
4 8
7 .0 1
7 .0 1
.0 1
7 .0 5
7 .0 3
.0 0
.0 2
.0 6
7 .0 1
.0 3
7 .7 9
4 7
.0 2
7 .0 1
.0 0
7 .0 2
7 .0 1
.0 3
.0 5
.0 8
.0 2
.0 7
7 .6 2
4 9
.0 3
.0 2
7 .0 7
7 .0 1
7 .0 2
7 .0 3
.0 1
.0 4
7 .1 7
.0 9
7 .5 6
N o te : In it ia l ei g en v a lu es
a n d a m o u n t o f v a ri a n ce
ex p la in ed
(i n p a re n th es es ): o p p o rt u n it y to
u se : 1 3 .1 1 (2 1 .5 0 ); su p er v is o r o p p o si ti o n : 7 .8 4 (1 2 .8 6 ); su p er v is o r su p p o rt : 3 .0 9
(5 .0 7 ); p er so n a l o u tc o m es
p o si ti v e: 2 .1 9 (3 .5 8 ); p er so n a l o u tc o m es
n eg a ti v e: 1 .9 0 (3 .1 2 ); p er so n a l ca p a ci ty
1 .8 0 (2 .9 5 ); le a rn er
re a d in es s: 1 .6 9 (2 .7 8 ); p ee r su p p o rt : 1 .5 7 (2 .5 6 );
m o ti v a ti o n to
tr a n sf er : 1 .3 2 (2 .1 6 ); tr a n sf er
d es ig n : 1 .1 7 (1 .9 2 ); co n te n t v a li d it y : 1 .0 9 (1 .7 8 ). F a ct o r lo a d in g s o f th e it em
s th a t co m p ri se
ea ch
sc a le a re
sh o w n in
sh a d ed
fo rm
. N ¼
2 5 9 9 .
Human Resource Development International 561
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
T a b le
3 .
In te r- sc a le
co rr el a ti o n s a n d re li a b il it y es ti m a te s (C
ro n b a ch ’s a lp h a ).
S ca le
n 1
2 3
4 5
6 7
8 9
1 0
1 1
1 2
1 3
1 4
1 5
1 6
1 C o n te n t v a li d it y
2 9 8 6
.8 0
2 T ra n sf er
d es ig n
2 9 9 5
.4 6 *
.8 0
3 P er so n a l ca p a ci ty
2 9 9 2
7 .1 3 *
7 .2 1 *
.7 8
4 O p p o rt u n it y to
u se
2 9 7 8
.5 0 *
.5 5 *
7 .2 9 *
.7 9
5 M o ti v a ti o n to
tr a n sf er
3 0 0 4
.3 6 *
.3 8 *
7 .1 5 *
.4 1 *
.7 8
6 L ea rn er
re a d in es s
2 9 9 3
.3 3 *
.2 5 *
7 .0 7 *
.3 0 *
.3 8 *
.7 1
7 S u p er v is o r su p p o rt
2 9 9 6
.3 4
.2 7 *
.0 0
.3 3 *
.2 9 *
.2 6 *
.8 4
8 S u p er v is o r
o p p o si ti o n
3 0 0 1
7 .0 2
7 .1 6 *
.4 4 *
7 .2 0 *
7 .0 3
.0 3 *
.0 6 *
.8 3
9 P ee r su p p o rt
3 0 0 8
.3 8 *
.3 6 *
7 .1 1 *
.4 3 *
.3 7 *
.2 8 *
.4 4 *
7 .0 4 *
.8 3
1 0 P er so n a l o u tc o m es
p o si ti v e
2 9 8 9
.2 7 *
.2 1 *
.0 7 *
.2 9 *
.3 6 *
.2 8 *
.4 2 *
.2 1 *
.3 7 *
.8 3
1 1 P er so n a l o u tc o m es
n eg a ti v e
3 0 0 0
.1 8 *
.0 1
.2 6 *
.0 7 *
.1 3 *
.1 2 *
.3 1 *
.3 7 *
.2 5 *
.4 5 *
.8 1
1 2 P er fo rm
a n ce
se lf -e ffi ca cy
3 0 0 0
.2 8 *
.3 4 *
7 .1 4 *
.3 5 *
.3 0 *
.2 3 *
.2 4 *
7 .0 7 *
.2 5 *
.1 6 *
.0 6 *
.7 5
1 3 T ra n sf er
eff o rt
p er fo rm
a n ce
ex p ec t
3 0 0 4
.3 1 *
.4 2 *
7 .1 5 *
.4 0 *
.4 4 *
.2 3 *
.2 6 *
7 .1 3 *
.3 7 *
.2 3 *
.0 6 *
.4 0 *
.7 5
1 4 P er fo rm
a n ce
o u tc o m e
ex p ec t
3 0 0 5
.3 1 *
.2 8 *
7 .0 2 *
.3 6 *
.3 4 *
.2 3 *
.4 3 *
.0 6 *
.3 8 *
.5 0 *
.2 3 *
.2 8 *
.4 0 *
.7 2
1 5 P er fo rm
a n ce
co a ch in g
3 0 0 7
.3 0 *
.2 4 *
.0 6 *
.2 8 *
.2 4 *
.2 2 *
.4 6 *
.1 3 *
.3 7 *
.3 9 *
.3 0 *
.2 2 *
.2 5 *
.4 6 *
.8 5
1 6 R es is ta n ce
to ch a n g e
2 9 9 7
7 .1 0 *
7 .1 1 *
.4 1
7 .2 0 *
7 .0 0
7 .0 3 *
7 .0 6 *
.4 3 *
7 .1 4 *
.0 6 *
.1 6 *
7 .0 8 *
7 .1 0 *
7 .0 5 *
7 .0 4 *
.8 0
N o te : * p 5
.0 5 (t w o -t a il ed ); sc a le
re li a b il it y es ti m a te s (C
ro n b a ch ’s a lp h a ) a re
o n th e d ia g o n a l. S ca le s in
th e tr a in in g -g en er a l d o m a in
a re
sh a d ed . D a ta
fr o m
su b -s a m p le
A .
562 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
outcomes positive scale, item 22 (.64) was selected over item six (.78) because the wording of the former more effectively broadened the scales’ conceptual reach. In sum, the selection of both item 46 and item 22 represented efforts to increase item heterogeneity and the breadth of measurement of their respective scales while maintaining a minimum of items.
Training-general domain
The MSA for the training-general domain data was .90 indicating its suitability for factor analysis. The EFA for these items resulted in an exceptionally clean and interpretable five-factor solution in which all the items loaded on the anticipated factors. The solution accounted for 56.08% of the common variance. Again, scale refinement efforts in this domain led to the retention of three items per scale. Table 4 shows the pattern coefficients for all items that ranged from .00 to .83. Reliability
Table 4. Exploratory factor analysis rotated factor pattern and pattern coefficients for items in the training-general domain.
Item Performance coaching
Resistance to change
Performance self-efficacy
Transfer effort
performance expectation
Performance outcome
expectations
81 .83 .02 7.07 7.03 7.07 80 .81 7.03 7.07 7.04 7.04 86 .80 .00 .01 .01 7.01 89 .55 .04 .12 7.06 .03 88 .47 7.14 .12 .06 .20 79 .44 .03 .03 7.07 .20 87 .30 7.07 .26 7.01 .12 77 .01 .82 .03 .00 .05 76 .02 .77 .02 .05 .02 74 .11 .70 7.02 .04 .04 73 7.08 .57 7.01 7.05 .04 78 .16 7.37 .06 7.04 .15 75 .20 7.33 .01 7.02 .17 83 7.01 .06 .73 .06 7.02 84 .00 .00 .72 .02 .07 85 .05 .01 .68 7.07 7.02 82 7.08 7.09 .63 7.17 7.10 66 .00 .00 .01 7.71 7.01 71 .01 7.02 .02 7.69 .07 65 .04 .00 .02 7.67 7.05 69 .00 7.01 .03 7.66 .07 67 .03 .03 .00 7.01 .69 68 .05 .09 .02 7.23 .57 72 .21 .13 .05 7.01 .52 70 .03 7.03 .01 7.20 .49 64 .08 .26 7.01 7.01 7.37
Note: Initial eigenvalues and amount of variance explained (in parentheses): performance coaching: 6.71 (25.82); resistance to change: 2.74 (10.53); performance self-efficacy: 2.41 (9.28); transfer effort performance expectations: 1.56 (6.00); performance outcome expectations: 1.16 (4.45). Factor loadings of the items that comprise each scale are shown in shaded form. N ¼ 2894.
Human Resource Development International 563
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
estimates (Cronbach’s alpha) ranged from .72 to .85, all in the acceptable range (Nunnally and Bernstein 1994) (see Table 3).
The EFA findings suggested an 11-factor structure for the program-specific domain and a five-factor structure for the training-general domain with all the items loading together on the correct factors. All scales were reduced to three items and displayed acceptable reliability estimates. As seen in Table 3, the estimated correlations between the factors were low to moderate ranging from .00 to .55 with an average inter-scale correlation of .24. Overall, the EFA results strongly support the 16-factor structure of the LTSI.
Confirmatory factor analysis
One of the limitations of EFA is that it cannot assess the extent to which a hypothesized model fits the data. This must come from a CFA approach (Bollen 1989). In this study, two sub-samples were created for the purpose of CFA. The first sub-sample, CFA1, was used to test the EFA results containing only three-item scales and to indicate any model respecifications in the event of poor fit. The second sub-sample, CFA2, was created to test the final model emerging from CFA1. For both these analyses, separate CFA models were run for the factors in the program- specific domain and those in the training-general domain, yielding four separate sets of CFA results.
CFA1 program-specific results
Confirmatory factor analysis 1 results for the program-specific domain (n¼1484) suggested the 11-factor model was an acceptable fit for the data: w2 (df¼440) ¼1448.67, p 5 .001; w2/df¼3.29; CFI¼ .95; TLI¼ .94; IFI¼ .95; RMSEA¼ .04. A significant chi-square was anticipated given the large sample but the w2/df was below the five benchmark suggested by Schumacker and Lomax (1998). All of the fit indices were at the .95 cutoff suggested by Hu and Bentler (1999) with the exception of the TLI which was below the .95 level.
CFA1 training-general results
Confirmatory factor analysis 1 results for the training-general domain (n¼1484) indicated the five-factor model was a good fit for the data: w2 (df¼80)¼203.22, p 5 .001; w2/df¼2.54; CFI¼ .98; TLI¼ .97; IFI¼ .98; RMSEA¼ .03. A significant chi-square was anticipated given the large sample but the w2/df was below the five benchmark. All of the fit indices were above the more rigorous cutoff of .95 suggested by Hu and Bentler (1999).
Given the acceptable fit of the hypothesized models for both the program-specific and training-general models and with no empirical or theoretical basis for model respecification, no effort was made to further improve these models. Therefore, the CFA2 sample data were used to cross-validate these results.
CFA2 program-specific results
Confirmatory factor analysis2 results for the program-specific domain (n¼1467) suggested the 11-factor model was an acceptable fit for the data: w2 (df¼440)
564 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
¼1471.38, p 5 .001; w2/df¼3.34; CFI¼ .95; TLI¼ .94; IFI¼ .95; RMSEA¼ .04. A significant chi-square was anticipated given the large sample but the w2/df was below the five benchmark. Again, with the exception of the TLI (.94), all of the fit indices were at the cutoffs suggested by Hu and Bentler (1999).
CFA 2 training-general results
Confirmatory factor analysis2 results for the training-general domain (n¼1467) indicated the five-factor model was a good fit for the data: w2 (df¼80)¼280.64, p 5 .001; w2/df¼3.51; CFI¼ .97; TLI¼ .96; IFI¼ .97; RMSEA¼ .04. A significant chi-square was anticipated given the large sample but the w2/df was again was below the five benchmark. All of the fit indices were above the cutoffs suggested by Hu and Bentler (1999). The fit statistics for all CFA are shown in Table 5.
Although chi-square for all CFA was significant, it has been argued that the chi- square goodness-of-fit statistic is contraindicated as a measure of fit with large samples because it is directly affected by sample size (Kahn 2006; Maruyama 1998). As a result, some have suggested using w2/df estimates with those between one and five indicating good fit (Schumacker and Lomax 1998). The w2/df estimates for all models fell within this range. Examination of the other fit indexes for both the 11- factor program-specific model and the five-factor training-general model indicated a good fit of the hypothesized models to the data. Both CFA1 and 2 demonstrated that responses to the items in the program-specific and training-general domains could be explained by 11 and five factors, respectively. Items specified to measure common underlying factors in both domains showed relatively high-factor pattern coefficients on the respective factor with zero loading on all other factors. For the CFA2 data these ranged from .57 to .89 with an average factor pattern coefficient of .73 for the program-specific factors. Factor pattern coefficients for the training-general factors ranged from .57 to .85 with an average of .73.
The tested models also hypothesized items would have zero loadings on all other factors and that the measurement error terms were uncorrelated. These hypotheses were supported. Finally, the models tested were recursive, hypothesizing and confirming inter-factor correlations among the 11 factors in the program-specific domain and five factors in the training-general domain.
Table 5. Goodness-of-fit statistics for the 11-factor program-specific and 5-factor training- general models for CFA1 and CFA2 samples.
Sample and Model n w2 df w2/df TLI IFI CFI RMSEA (CI)
CFA1 11-factor
program-specific
1484 1448.67 440 3.29 .94 .95 .95 .039 (.037–.042)
5-factor training- general
1484 203.22 80 2.54 .97 .98 .98 .032 (.027–.038)
CFA2 11-factor
program-specific
1467 1471.38 440 3.34 .94 .95 .95 .037 (.038–.042)
5-factor training- general
1467 280.64 80 3.51 .96 .97 .97 .044 (.039–.049)
Note: TLI ¼ Tucker-Lewis Index; IFI ¼ incremental fit index; CFI ¼ comparative fit index; RMSEA ¼ root mean square error of approximation; CI ¼ confidence interval.
Human Resource Development International 565
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
To avoid model misinterpretation, both factor pattern and factor structure coefficients should be interpreted in CFA studies involving correlated factors. When factors are correlated as in the models tested in this study, it is always the case that the factors for which the pattern coefficients are constrained to zero will have structure coefficients not equal to zero (Graham, Guthrie, and Thompson 2003). It is therefore important to examine both of these components to avoid misinterpretation of the functioning of single variables or the content or meaning of the factors themselves. In this study, examination of the standardized pattern coefficients and the factor structure coefficients (r2) for both the program-specific and training- general factors showed that none of the variables had correlations on factors with fixed zero pattern coefficients higher than the structure coefficients for factors on which the pattern coefficients were freed (tables containing these data are available from the lead author upon request). This strongly suggests there are no issues with regard to the functioning of single variables or the content or meaning of the factors themselves.
Discussion and implications
The aim of this research was to examine the dimensionality of items used to assess individuals’ perceptions of organizational learning transfer systems. The results provide strong support for the 16 factors structure of the LTSI. In the initial sample, EFAs indicated the items in the program-specific domain of the LTSI were best summarized by 11 underlying constructs and items in the training-general domain were best summarized by five underlying constructs. Although the factors were correlated, none of the estimated correlations between the factors were excessively high (4.85). As seen in Table 3, the estimated correlations between the factors were low to moderate ranging from .00 to .55 with an average inter-scale correlation of .24. These data support the discriminant validity and the distinctiveness of the factors measured by the LTSI and these are consistent with the previous construct validation research done with the LTSI.
Two separate CFAs were used to test hypotheses about how many three-item factors exist in the program-specific and training-general domains of the LTSI, which variables correspond to which factors, and whether the hypothesized factor structure of the program-specific and training-general two domains provided a good fit to the data. Results from CFA1 provided strong evidence that the hypothesized models were an acceptable fit to the data. The data did not indicate an empirical or theoretical basis for model respecification. As a result, no effort was made to further improve these models. They were, however, tested with a second CFA sample with equivalent results. In addition, relatively low inter- factor correlations among both domains of factors suggest there is little overlap among the scales and that each assesses a unique construct. Finally, the sample data used in these analyses came from a wide range of organizations, training programs and participants, and included data from 14 different language versions of the LTSI collected in 17 different countries. The heterogeneity of these data, together with the analytic results, argues strongly for the generalizability and stability of the factorial structure of the LTSI.
This research also produced a much shorter version of the LTSI (48 vs. 89 items) while maintaining the instrument’s essential characteristics. It is hoped that the
566 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
shorter version will increase organizational and respondent acceptance, minimize completion time and diminish respondent fatigue, and provide a more practical, easier-to-use, and more accessible instrument for organizations, training practitioners and researchers.
Although the 16-factor model of learning transfer system characteristics was supported in this study, further research is needed to examine whether the factor structure fits data from different groups equally well. Multiple group CFA comparing, for example, the factor structure of the LTSI across groups defined by a demographic variable (e.g. gender, ethnicity), cultural orientation, training type, organizational or individual difference variable would provide additional and important validity evidence for LTSI scores. It may also be informative to compare factor structures across groups with varying degrees of language fluency, for example, between native English speakers and second language English speakers completing the English language version of the LTSI.
Future validation research should focus on the criterion-related validity of the LTSI. Although some of this work has been done (e.g. Bates et al. 2000, 2007; Fitzgerald 2002; Myers 2009; Seyler et al. 1998) more is needed to examine the criterion-related validity for each factor of the LTSI using the latest version of the instrument.
Finally, a number of transfer system improvement interventions linked to specific LTSI scales and transfer system barriers have recently been developed. Quasi- experimental research examining how these interventions affect an organization’s learning transfer system, changes in learning, job behaviour and performance and return-on-training investment would help the HRD field move beyond the question of whether or not training works to why training works and how the transfer outcomes and training effectiveness can be improved.
References
Anderson, J.C., and D.W. Gerbing. 1988. Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin 103: 411–23.
Baldwin, T.T., and J.K. Ford. 1988. Transfer of training: A review and directions for future research. Personnel Psychology 41: 63–105.
Baldwin, T.T., J.K. Ford, and B.D. Blume. 2009. Transfer of training 1988–2008: An updated review and agenda for future research. International Review of Industrial and Organiza- tional Psychology 24: 42–70.
Bates, R.A., E.F. Holton III, D.A. Seyler, and M.A. Carvalho. 2000. The role of interpersonal factors in the application of computer-based training in an industrial setting. Human Resource Development International 3, no. 1: 19–43.
Bates, R.A., S. Kauffeld, and E.F. Holton III. 2007. Factor structure and predictive validity of the German version of the Learning Transfer Systems Inventory. Journal of European Industrial Training 31, no. 3: 195–211.
Bates, R.A., and S. Khasawneh. 2005. Organizational learning culture, learning transfer climate and perceived organizational innovation in Jordanian organizations. International Journal of Training and Development 9, no. 2: 96–109.
Bell, B.S., and S.W.J. Kozlowski. 2008. Active learning effects of core training design elements on self-regulatory processes, learning, and adaptability. Journal of Applied Psychology 93, no. 2: 296–316.
Bentler, P.M. 1989. EQS structural equation program manual. Los Angeles: BMDP Statistical Software.
Bollen, K.A. 1989. Structural equations with latent variables. New York: Wiley. Boyle, G.J. 1991. Does item homogeneity indicate internal consistency or item redundancy in
psychometric scales? Personality and Individual Differences 12, no. 3: 291–4.
Human Resource Development International 567
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
Byrne, B.M. 1993. The Maslach Burnout Inventory: Testing for factorial validity and invariance across elementary, intermediate, and secondary teachers. Journal of Occupa- tional and Organizational Psychology 66: 197–212.
Chen, H.C., E.F. Holton III, and R.A. Bates. 2006. Situational and demographic influences on transfer system characteristics in organizations. Performance Improvement Quarterly 19, no. 3: 232–48.
Comrey, A.L. 1988. Factor analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology 56: 754–61.
Coste, J., F. Guillemin, J. Pouchot, and J. Fermanian. 1997. Methodological approaches to shortening composite measurement scales. Journal of Clinical Epidemiology 50, no. 3: 247– 52.
Costello, A.B., and J.W. Osborne. 2005. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research and Evaluation, 10: 1–9. http//pareonline/getvn.asp?v¼10&n¼7.
DeRouin, R.E., B.A. Fritzsche, and E. Salas. 2004. Optimizing e-learning: Research- based guidelines for learner-controlled training. Human Resource Management 43: 147–62.
Devos, C., X. Dumay, M. Bonami, R. Bates, and E.F. Holton. 2007. The Learning Transfer System Inventory (LTSI) translated into French: Internal structure and predictive validity. International Journal of Training and Development 11, no. 3: 181–99.
Fitzgerald, C.G. 2002. Transfer of training and transfer climate: The relationship to the use of transfer maintenance strategies in an autonomous job context. Unpublished doctoral diss., University of Connecticut, Storrs, CN.
Floyd, F.J., and K.F. Widaman. 1995. Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment 7, no. 3: 286–99.
Ford, J.K., and K. Kraiger. 1995. The application of cognitive constructs and principles to the instructional systems model of training: Implications for needs assessment, design, and transfer. In International review of industrial and organizational psychology, ed. C.L. Cooper and I.T. Robertson, vol. 10, 1–48. New York: Wiley.
Ford, J.K., and D.A. Weissbein. 1997. Transfer of training: An updated review and analysis. Performance Improvement Quarterly 10, no. 2: 22–41.
Graham, J.M., A.C. Guthrie, and B. Thompson. 2003. Consequences of not interpreting structure coefficients in published CFA research: A reminder. Structure Equation Modeling 10, no. 1: 142–53.
Hair, J.F., R.R. Anderson, R.L. Tatham, and W.C. Black. 1998. Multivariate data analysis. 5th ed. Upper Saddle River, NJ: Prentice Hall.
Holton, E.F. III. 2003. What’s really wrong: Diagnosis for learning transfer system change. In Improving learning transfer in organizations, ed. E.F. Holton and T.T. Baldwin, 7–22. San Francisco: Jossey-Bass.
Holton, E.F. III. 1996. The flawed four level evaluation model. Human Resource Development Quarterly 7: 5–21.
Holton, E.F. III, and R.A. Bates. 2011. Validation of the Learning Transfer Systems Inventory (LTSI) in Greece. Unpublished technical report, Louisiana State University.
Holton, E.F. III, R.A. Bates, and A. Bookter. 2007. Convergent and divergent validity study of the Learning Transfer Systems Inventory. Human Resource Development Quarterly 18, no. 3: 385–419.
Holton, E.F. III, R.A. Bates, K.L. Seyler, and M.A. Carvalho. 1997. Toward construct validation of a transfer climate instrument. Human Resource Development Quarterly 8, no. 2: 95–113.
Holton, E.F. III, R.A. Bates, and W.E.A. Ruona. 2000. Development and validation of a generalized learning transfer system inventory. Human Resource Development Quarterly 11, no. 4: 333–60.
Holton, E.F. III, H.-C. Chen, and S.S. Naquin. 2003. An examination of learning transfer system characteristics across organizational settings. Human Resource Development Quarterly 14, no. 4: 459–82.
Hu, L., and P.M. Bentler. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 6: 1–55.
Kahn, J.H. 2006. Factor analysis in counseling psychology research, training, and practice: Principles, advances, and applications. The Counseling Psychologist 34, no. 5: 684–718.
568 R. Bates et al.
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
Khasawneh, S., R.A. Bates, and E.F. Holton III. 2006. Construct validation of an Arabic version of the Learning Transfer Systems Inventory for use in Jordan. International Journal of Training and Development 10, no. 3: 180–94.
Maruyama, G.M. 1998. Basics of structural equation modeling. Thousand Oaks, CA: Sage. Myers, M.J. 2009. Transfer of learning from training program to the workplace in a university
healthcare organization. Unpublished doctoral diss., University of Connecticut, Storrs, CN.
Nunnally, J., and I. Bernstein. 1994. Psychometric theory. New York: McGraw-Hill. Ruona, W.E.A., M. Leimbach, E.F. Holton, and R.A. Bates. 2002. The relationship between
learner utility reactions and predicted learning transfer among trainees. International Journal of Training and Development 6, no. 4: 217–27.
Schumacker, R.E., and R.G. Lomaz. 1998. A beginner’s guide to structural equation modeling. Hillsdale, NJ: Erlbaum.
Seyler, D.L., E.F. Holton III, R.A. Bates, M.F. Burnett, and M.A. Carvalho. 1998. Factors affecting motivation to use training. International Journal of Training and Development 2, no. 1: 2–16.
Westgard, J.O. 1999. Basic method validation. Berlin: Springer. Yamkovenko, B., E.F. Holton III, and R.A. Bates. 2007. The Learning Transfer System
Inventory (LTSI) in Ukraine: The Cross-Cultural Validation of the Instrument. Journal of European Industrial Training 31, no. 5, 377–401.
Yamnill, S. 2001, Factors affecting transfer of training in Thailand. Unpublished doctoral diss., University of Minnesota, Minneapolis, MN.
Human Resource Development International 569
D ow
nl oa
de d
by [A
ca de
m y
of H
um an
R es
ou rc
e D
ev el
op m
en t]
a t 0
7: 37
2 0
Fe br
ua ry
2 01
4
HRE7271/Week 5/learner_utility_reactions_and_learning_transfer_IJTD.pdf
HRE7271/Week 5/Making Transfer Climate Visible.pdf
The authors would like to thank James C. Gumm for his review and comments of the article. Correspondence should be sent to John-Paul Hatala, Louisiana State University, School of Human Resource Education and Workforce Development, 142 Old Forestry Building, Baton Rouge, LA 70803, or by email to [email protected].
Human Resource Development Review Vol. 6, No. 1 March 2007 1-31 DOI: 10.1177/1534484306297116 © 2007 Sage Publications
Making Transfer Climate Visible: Utilizing Social Network Analysis to Facilitate the Transfer of Training JOHN-PAUL HATALA PAMELA R. FLEMING Louisiana State University
In this article the authors introduce social network analysis (SNA) as a methodology for analyzing transfer climate prior to training. The literature has indicated that transfer climate is critical to a trainee’s ability to apply the new knowledge, skills, behaviors, and attitudes they gain through training back to the workplace. SNA serves as a tool for analyzing a participant’s organizational network relationships prior to training to help the facilitator, trainee, and supervisor gain an accurate picture of the transfer climate. Based on this analysis, measures can be taken to develop strategies to deal with relational barriers prior to training that will facilitate the participant’s transfer of learning back to the work environment. The process of conducting SNA is described and illustrated using a case example. The benefits of using SNA to enhance transfer climate and the implications for further research and practice are discussed.
Keywords: transfer climate; transfer of training; social network analysis; relational barriers
More than ever, organizations are attempting to capitalize on training initiatives to move their strategic agendas forward. These initiatives require that the individuals participating in training take new knowledge back to the workplace and apply what they have learned. However, a number of barriers have been identified that impede the successful transfer of training back to the job, such as a nonsupportive organizational environment and the applicability of the training (Kim, 2004;
Machles, 2002; Noe, 1988; Rossett, 1997). As a result of these barriers, it is estimated that between 10% and 20% of the training actually transfers to the workplace (Georgenson, 1982). Human Resource Development (HRD) practi- tioners are often confronted with learning transfer issues that have not demon- strated return on investment for the organization (Rouiller & Goldstein, 1993). Researchers have attempted to address the barriers to training transfer by explor- ing the factors related to this complex process (Eagan, Yang, & Bartlett, 2004; Kontoghiorghes, 2002; Lim, 2001; Lim & Johnson, 2002). The factors identified have been commonly categorized under trainee characteristics, training designs, and work environment (Baldwin & Ford, 1988; Holton & Baldwin, 2000). Although the literature explores a number of potentially important factors con- tributing to the transfer process, the process has mainly focused on a psychologi- cal and individualistic perspective (London & Flannery, 2004).
This article proposes a methodology for identifying the relational barriers to training transfer by making visible the social structure within the transfer climate. Through the use of social network analysis (SNA), the authors posit that defining the social structure prior to the delivery of training will provide the opportunity to assess transfer climate and address any relational barriers that may affect the trans- fer of training to the workplace. SNA has been identified as a useful method for analyzing network properties within an organizational context and is presented as a means for delineating the relationships within the work environment where transfer is to take place (Hatala, 2006). Research on organization, supervisory, and peer support has been linked to successful transfer (Cromwell & Kolb, 2004; London & Flannery, 2004). However, the authors are unaware of any literature that has specifically examined the network structure of an organization as a vari- able affecting transfer climate. Researchers have advocated the design and devel- opment of transfer needs assessments to facilitate the transfer of training (Goldstein, 1986; Lim & Morris, 2006). The importance of facilitating the trans- fer process before, during, and after the training intervention is paramount to training transfer taking place. This article will add to the training transfer litera- ture by proposing SNA as a methodology for assessing the relational dynamics within the transfer climate prior to, during, or after training has been delivered.
Therefore, the purpose of this article is to illustrate how SNA can be used to make visible the social structure in which the transfer of training is to take place. More specifically, by exploring the social network aspects of transfer climate prior to a training intervention, action can be taken to address the envi- ronment in which the participant will transfer new knowledge and help main- tain transfer behavior by managing relationships that support the transfer. This action includes exploring the relationship of the training participant and the individuals they interact with on the job (i.e., cohorts, supervisor, manager). If it is determined that transfer is at risk, measures can be taken to address these issues and develop strategies to increase the likelihood of transfer occurring. The article has three objectives:
2 Human Resource Development Review / March 2007
1. To conduct a review of the literature on transfer climate and existing methods for addressing transfer of training.
2. To introduce SNA as a methodology for making transfer climate visible by uncov- ering the relationships that may inhibit or facilitate successful training transfer.
3. To discuss future directions for using SNA as a methodology for testing transfer climate hypotheses.
Methodology
An integrative literature review of transfer climate and social networks was conducted through a search of the EBSCO Host’s Academic Search Primer, Ingenta, ERIC, Proquest, and JSTOR databases. The key words used in the lit- erature search included transfer climate, learning transfer, barriers to learning transfer, transfer of training, overcoming transfer barriers, social network theory, and SNA. The following questions were explored:
1. How transfer climate affects transfer of training? 2. How various types of social support affect training transfer? 3. What methods are used to overcome transfer barriers? 4. How can SNA be utilized as a methodology to measure organizational, peer, and
supervisory support? 5. What future directions for the field of HRD can be drawn from the literature?
As a result, three bodies of literature were reviewed: management, HRD, and psychology. The HRD literature was used as the foundation for transfer of train- ing theory and addressed Questions 1, 2, and 3, whereas psychology and man- agement literature were drawn on as examples of fields that support the theories addressed in Questions 1, 2, and 3. The focus of the literature review was intended to identify the social dynamics within transfer climate and how it affects training transfer. Additionally, this literature review sought to define social barriers to training transfer and to determine whether SNA could serve as a method to identify and overcome these barriers, answering Question 4. Future directions for HRD were generated from this research addressing Question 5.
Transfer of Training
From a theoretical perspective, transfer of training occurs when prior knowledge and/or skills affect the way in which new knowledge and skills are learned and executed (Taylor, 2000, p. 4). Broad and Newstrom (1992) identified training transfer as a “transfer partnership” consisting of three key members—the trainee, the trainer, and the manager or supervisor. The compi- lation of these major players is pivotal to the success of transfer of training. With consistently changing job environments and job requirements, learning transfer has become a major concern. This section presents models and frame- works that serve as the foundation for various strategies fostering the training
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 3
transfer process and specifically examines the work environment as a major element in carrying out transfer.
Baldwin and Ford’s (1988) theoretical framework examined training transfer and stated that the transfer process consists of three components: training input factors, training outcomes, and conditions to transfer. The conditions to transfer are predicated on generalizing training to the job in addition to maintaining learned skills over time on the job. The framework also looks at organizational climate factors including peer and supervisory support as having a dramatic effect on the whether transfer of knowledge has the opportunity to occur. Holton’s (1996) conceptual model focused on three primary training outcome interventions: learning, individual performance, and organizational results. More specifically, performance outcome involves the motivation to transfer, the design of training, and transfer climate (environment), where transfer climate focuses on peer and supervisory support as major factors to the transfer process. Holton (1996) recognizes that to promote positive transfer behaviors, the need for con- tinued research on the evaluation of training transfer is crucial.
Organizational Climate and Transfer Climate
Organizational climate refers to the current perceptions of people within a work environment with regard to the observable (social, political, and physical) nature of the personal relationships that affect the accomplishment of work within a particular organization (Denison, 1996). One of the distinguishing char- acteristics of an organization’s climate, as opposed to its culture, is the transitory and malleable nature of an organizational member’s perceptions (Denison, 1996). In other words, the perceived relational dynamics that exist within an organization influence the climate, and because the study of organizational cli- mate only captures a “snapshot” of an organization at one point in time change can and will likely occur without notice (Litwin & Stringer, 1968). Continued study of a particular organization as its climate changes can lead to a better understanding of the relational patterns that exist within an organization. This understanding can then be used to explain the organizational climate variables that remain distinct or converge (Lim & Morris, 2006).
An important subset of these perceptions of organizational climate relates to the transfer of training, also known as the transfer climate of an organization (Lim & Morris, 2006). These perceptions have unique properties that influence an indi- vidual’s motivation and behavior toward the transfer of training (Baldwin & Ford, 1988). More specifically, the transfer climate encompasses an individual’s per- ceptions of supervisor support, opportunity to use new training, level of peer support, supervisor sanctions, and positive or negative personal outcomes result- ing from application of training on the job (Holton, Bates & Seyler, 1997). Researchers have used the term transfer climate to describe an individual’s per- ception of the social support structure that exists within an organization (Cheng & Ho, 2001). This social support structure manifests itself in the form of peer,
4 Human Resource Development Review / March 2007
supervisory, and organizational support. Transferring new knowledge and skills back to the workplace requires a commitment not only from the employee but also from the organization. This research indicates that transfer of trained tasks back to the workplace goes far beyond the quality of the training program and the delivery method and is in large part dependent on the transfer climate of an organization (Campbell, 1988).
Rouiller and Goldstein (1993) explored the issue of organizational transfer cli- mate by conducting an empirical study that evaluated the relationship between climate and posttraining behavioral change. They defined transfer climate as con- sisting of two types of workplace cues: situation and consequence cues. They identified four types of situation cues: goal cues, social cues, task cues, and self- control cues. The consequence cues included positive reinforcement, negative reinforcement, punishment, and no feedback. Their research findings indicated that organizational transfer climate was significantly related to the transfer behav- iors of a trainee and that learning and organizational transfer climate accounted for a significant portion of the variance in learning transfer among those in the workplace. The authors suggested interventions based on an organization’s trans- fer climate as a tool for facilitating training transfer back to the workplace. By linking Rouiller and Goldstein’s transfer climate dimensions to specific individu- als within an organizational network, practical intervention strategies can be developed to make the climate more conducive to training transfer.
Relational Barriers to Training Transfer
Often during and after training sessions, participants are excited about the new knowledge; they ask questions, clarify points, and even input value-added observations that spark stimulating dialogue. Unfortunately, this excitement is all too often limited to the training environment (Rossett, 1997), and even though an initial excitement toward the training may be present, its impact on transfer may be reduced if relational barriers exist. Relational barriers to trans- fer of learning can be traced to a nonsupportive organizational climate because of the lack of peer and supervisory support. Research has shown that without supervisory, peer, and organizational support, these barriers will dissuade training transfer behaviors (Cromwell & Kolb, 2004; Lim & Morris, 2006; Taylor, 2000). More important, relational barriers may cause participants to feel that skills acquired during training are perceived by others as having little value, thereby giving participants little to no incentive to transfer the learned skills back to the job (Kim, 2004).
Holton, Bates, and Seyler (1997) defined supervisor support as the extent to which supervisors support and reinforce the use of learning on the job. According to Machles (2002), managers may not have the knowledge to support the infor- mation employees have learned, meaning higher level managers, without the benefit of experience and training, may not themselves fully comprehend the importance of assisting in transfer of training, thus creating a barrier to transfer.
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 5
This lack of understanding most likely contributes to management’s lack of com- mitment and strengthens the supervisor support variable as a critical influence on successful transfer (Lim & Morris, 2006). Additionally, Cromwell and Kolb (2004) cited organizational climate and the lack of supervisory and peer support as barriers to training transfer, and the degree of support given once the trainee returns to the workplace greatly influences whether or not transfer takes place.
Training transfer is enhanced by involving supervisors and trainees in the needs assessment phase of training development (Hatala & Gumm, 2006; Machles, 2002). This allows managers and trainees to provide their input, thereby increasing the likelihood of their support toward the training interven- tion. This support from both management and individual organizational members will aid in creating a positive transfer climate because both sides will perceive the training as worthwhile and meeting the needs of themselves and the organization. However, to accomplish this, an understanding of the rela- tional dynamics within the work group must be understood; otherwise, the opportunity to leverage the key influencers within the group may be limited. Also, by encouraging goal setting and self-management, learners will demon- strate a higher level of transfer (Gist, Bavetta, & Stevens, 1990) in that they will feel a stronger connection to the overall objective of the training.
The next section will examine the process of conducting an SNA to uncover the hidden relational barriers within a transfer climate. SNA will assist in the identification of relational barriers and how they affect training transfer.
The Network Approach
Social network theory explains the interpersonal mechanisms and social structures that exist among interacting units (i.e., small groups, large groups, departments, units, within organizations, between organizations) (Wasserman & Faust, 1994). More specifically, it is the study of how the relationships of a person, group, or organization affect beliefs or behaviors. The theory relates to a number of different levels of analysis that can be used to determine the inter- action between individuals and their environment. The term network typically refers to a set of objects or nodes, and the mapping of the interaction and rela- tionships between the objects (Wasserman & Faust, 1994). Social network theory refers to the objects as people or groups of people. By measuring the interactivity of individuals through mapping relationships, researchers can uncover the specific dynamics that exist between and within groups. Social capital is one example why social network theory is studied. By understand- ing the mappings connecting individuals to a set of others, we stand to learn much about how individuals utilize their connections to achieve desired out- comes (Coleman, 1988). From an organizational context, actors within the net- work can increase performance or move upward based on the connections they possess. In addition, the level of social capital helps to determine how
6 Human Resource Development Review / March 2007
individuals use their position within a network to accumulate power in social settings (Burt, 1997).
Some of the formal theoretical properties in the network perspective include centrality (betweenness, closeness, degree), position (structural), strength of ties (strong/weak, weighted/discrete), cohesion (groups, cliques), and division (struc- tural holes, partition) (Scott, 2000; Wasserman & Faust, 1994). These represent the building blocks for developing and conceptualizing network theory (White, 1997).
SNA
SNA is a method that can provide an empirical measure of an organization’s work environment by focusing on the relationships between people while using attribute characteristics (Scott, 2000; Wasserman & Faust, 1994). These rela- tionships include the feelings people have for one another, the exchange of infor- mation, and issues of power. By mapping these relationships, SNA helps to uncover the informal communication patterns to compare them with existing formal structures in hope of explaining various organizational phenomena. More specifically, the position of an individual within the social structure of an orga- nization helps to explain their exposure to and control over information based on the relationships that exist (Burt, 1992). Also, because the pattern of relation- ships brings individuals into contact with attitudes and behaviors of other members of the organization, these relationships may uncover the attitudes indi- viduals may have toward certain job-related issues (Feeley & Barnett, 1996; Ibarra & Andrews, 1993; Meyer, 1994). By uncovering relational information, SNA can be very useful for studying transfer climate.
The SNA approach provides formal definitions of the structural elements that exist within networks (i.e., actors, subgroup of actors, or groups) and can help to identify support relationships that may exist within the work environ- ment. This unique approach provides insight into the dynamics of the interac- tion between actors and the formation of observable patterns of information exchange between network members that may lead to successful transfer (Hatala, 2006). The ability to measure relationships will help define the trans- fer behaviors that exist and the impact they might have on the capability of transferring training to the workplace.
Wasserman and Faust (1994) identified some basic assumptions to the net- work perspective. They include (a) actors and their actions are viewed as inter- dependent rather than independent, autonomous units; (b) relational ties (linkages) between actors are channels for transfer or “flow” of resources (either material or nonmaterial); (c) network models focus on how individuals view the structural environment of a network as providing opportunities for or constraints on individual action; and (d) network models conceptualize struc- ture (social, economic, political, and so forth) as lasting patterns of relations among actors. The main focus of SNA remains on the interactional compo- nent. Attribute data such as age, gender, and race can be collected as well and
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 7
can provide profiles of network members. For a detailed review of the litera- ture on SNA, see Hatala (2006).
SNA represents an assessment method in which practitioners can examine the existing network structure prior to, during, or after a training intervention. In general, social network theory explains the mechanisms and structures that exist within any given environment (i.e., small groups, large groups, depart- ments, units, within organizations). Social network theory can help to explain the impact relations have on training transfer by observing the social structure and the relations that exist within that structure.
Making Transfer Climate Visible
The literature has strongly supported the need for organization, supervisory, and peer support if transfer of training back to the workplace is to take place (Baldwin & Ford, 1988; Bates, Holton, Seyler, & Carvalho, 2000; Clark, Dobbins, & Ladd, 1993; Cromwell & Kolb, 2004; London & Flannery, 2004; Pidd, 2004). Removing barriers that affect successful transfer of training may require changes to the relationships existing within a department, unit, or divi- sion. However, for relational change to occur, it is important to understand the dynamics that exist between the environment and the individuals participating in the training program prior to initiating any training intervention. SNA can make transfer climate visible by uncovering the network structure the learners are situated in, which will allow management and individual organizational members to anticipate any barriers to transfer prior to commencement of train- ing. SNA also identifies the flow of information from a formal and informal power situation (Cross & Parker, 2004; Ibarra & Andrews, 1993). Network maps enable the visualization of relationships and make what are normally invisible relational dynamics inside a group visible. One of the gaps on trans- fer training identified in the literature was that many studies focused on the individualistic standpoint (London & Flannery, 2004). SNA can provide a holistic view by highlighting the relationships related to transfer climate. The SNA approach investigates the organizational properties of relations between and within groups rather than simply the properties of the groups themselves. Transfer of training will be limited if relational patterns are not in keeping with successful application of new learning on the job.
Connectivity is occurring at every level of the organization structure. It can- not be taken for granted that just because a manager requires an employee to attend training transfer will occur. Competing forces are at play from both the formal and informal organization power structure (Hatala & Gumm, 2006; Torenvlied & Velner, 1998). At the formal level, these forces take the form of individual perception that management does not support the new learning and therefore has no incentive to implement the new knowledge they have acquired
8 Human Resource Development Review / March 2007
through training. Highly central employees to the social environment can be found at the informal level who may not buy-in to the training initiative and as a result not support the objectives of the program. If these forces are not known prior to the development of a training intervention, there is little likeli- hood of transferring what is learned.
One important element of transfer climate is the level of social support from both supervisors and peers that exists within an organizational unit (Baldwin & Ford, 1988; Cheng & Ho, 2001; Cromwell & Kolb, 2004; London & Flannery, 2004). The social network perspective views individuals within a network as interdependent, rather than independent and autonomous, units, which is critical to the transfer of training back to the workplace (Wasserman & Faust, 1994). The relational ties between individuals are channels for trans- fer of information. If relational barriers are present within a work group, it is likely that application of learning will be blocked by the lack of support from supervisors and coworkers.
The next section will examine the use of SNA as a methodology for illumi- nating the transfer climate within an organization. For the purposes of guiding the reader through the SNA process, we consider the case of a researcher who wishes to examine the nature of information sharing and job support among coworkers in a small sales department prior to new product training. Because of the nature of the product they sell, formal product training is an ongoing activ- ity within the department. The 14 actors in this case include a sales manager, a supervisor, two senior sales representatives, and 10 sales representatives.
Conducting the SNA
The following steps outlined by Hatala (2006) provide a general overview of the SNA process (see Table 1). Each step of SNA is explained, followed by an illustration using the sales group case.
Determining the Type of Analysis
There are two types of analyses that can be conducted; the first is an ego network analysis and the second is a complete network analysis (Scott, 2000; Wasserman & Faust, 1994). The ego network analysis focuses on a particular individual and is structured around eliciting information about the people he or she interacts with, and about the relationships with those people. This form of analysis should be chosen for individuals who are participating in a training program in isolation to their organization, department, or coworkers. The com- plete network analysis is concerned with all the relationships among a set of respondents (i.e., entire organization, complete work group), which includes managers, supervisors, and individual organizational members.
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 9
Sales Group
The sample case (sales group) in this article requires that a complete net- work analysis be conducted as the primary goal is to assess information shar- ing and job support among the complete group. Additionally, because all members of the work group will be attending the training, complete network analysis is the most appropriate for this situation.
Defining the Relationships Within the Network
Once the type of network analysis has been established, the relations that will be measured are determined by the objective of the project. Examples of the type of relations that can be measured might include communication relations (e.g., who speaks to whom), instrumental relations (e.g., who asks whom for help), power relations (e.g., who follows whom in informal groups), and interpersonal relations (e.g., who likes who) (Hatala, 2006). The researcher may be interested in determining which relationships reveal information-sharing potential, rigidity in the network, or well-being and supportiveness in the network (Cross & Parker, 2004).
Sales Group
In the case of the sales group, the goal is to determine whether relationships are present that promote information sharing and job support as well as to iden- tify the network members who are sought after for company gossip. The inten- tion of the analysis is to assess whether there is the likelihood of transfer occurring after new product training. Cromwell and Kolb (2004) found that those trainees who reported a higher level of support in the work environment were more likely to transfer their newly acquired skills and knowledge to their job. Exploring the relationships will uncover the overall structure and transfer cli- mate within the network and help explain how individuals presently support one another in applying new learning to their work environment. Additionally,
10 Human Resource Development Review / March 2007
TABLE 1: Eight Steps to Conducting a Social Network Analysis
1. Determining the type of analysis 2. Defining the relationships within the network 3. Collecting network data 4. Measuring the relationships 5. Including actor attribute information in the analysis 6. Analyzing the network data 7. Creating descriptive indices of social structure 8. Presenting the network data
SOURCE: Adapted from Hatala, 2006.
identifying network members who provide gossip may be as important as find- ing out which members are sought after for job-related information (Cross & Parker, 2004). If gossip is negatively directed to the organization, it can have a greater impact on employees transferring what they have learned from training back to the job. Informal information flow occurs regularly throughout an orga- nization and can easily be misidentified as formal communications by employ- ees (Kahn, Cross, & Parker, 2003). Formal and informal organizational communications must be monitored to ensure that information is accurate; oth- erwise the perceptions of individuals may be altered, ultimately affecting the cli- mate. By defining the relationships prior to the actual training intervention, measures can be taken to make any corrective action required to build a sup- portive transfer climate.
Collecting Network Data
Based on the research questions or objective of the project, the data collec- tion technique must fall in line with the appropriateness of the training situation (i.e., amount of time to collect data, amount of time allotted for training
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 11
TABLE 2: Sample Questions for Social Network Analysis Surveys for Transfer Climate
Who do you talk to at least once per week? To do your job well, who must you depend on? To do their job well, who generally depends on you? Who would be useful to interact with more than you do now? Who do you typically discuss new ideas you’ve learned from training with? Who do you turn to after you return from training if you are not sure of something
you have learned? Who do you share new ideas with on how to perform your job function better? Who generally is not open to new ideas when it comes to performing their job? Who do you go to for company gossip in your work group? How often do you talk with following people regarding making changes to the
work process? Who do you typically turn to for help in implementing a new job technique or
skill? How effective is each person listed below in helping you conduct your job? Who provides feedback on how you are applying what you’ve learned in training
on the job? Who provides feedback on how well you are performing your job? Who provides advice to you on how to perform your job better? Who typically will help you improve your performance if it is not what it should be? Who do you speak with on how to improve your performance? Who do you turn to when you need clarification or advice on something you
learned from training?
program, amount of preparation time, type of training). In some instances, col- lection techniques can include all, or some, of the following: observation, inter- views, surveys, or archival documents (Scott, 2000). Table 2 represents some examples of the types of questions that can be asked of participants for social network research on transfer climate. It is recommended that two to four ques- tions are used and are closely linked to the goals and objectives of the intended project (Cross & Parker, 2004).
Sales Group
In the case of the sales group, the two questions selected for analysis were: Who do you turn to after you return from training if you are not sure of some- thing you have learned? and Who do you go to for company gossip in your work group? The selection of these two questions was determined after consultation with management and geared specifically toward the product training. The first question was used to address individuals were key points of contact and pro- vided follow-up support to training interventions. The second question was intended to determine which individuals provided the latest information on com- pany news (i.e., company direction, new product allocation, budgets). The man- agers felt that if the individuals identified as central to the network for new training reinforcement were also “go-to” people for the latest news on company gossip, the message being delivered might be diluted based on their perception of the present status of the company. Therefore, the managers felt strongly that those members who were sought after for new knowledge reinforcement and company gossip could cause the most damage if they were at all misinformed.
Measuring the Relationships
Determining how the relationships will be measured is the next step of the SNA process. If the objective of the analysis is to simply determine whether a relationship exists between network members, binary measures can be employed by using a “0” or a “1.” The lack of a relationship between two actors is indicated by a “0” and the existence of a relationship is indicated by a “1” (Wasserman & Faust, 1994). Table 3 presents an example of a survey where the respondents indicated whether a relationship existed with the listed individuals. This type of survey would be administered to a group that works together on a regular basis (complete network) and is attending the same train- ing intervention, which applies specifically to our sample sales group. The goal of this data collection is to identify who the participant is connected to, based on the question proposed.
However, if the strength of the relationship has to be identified, a valued measure would help determine the extent to which the participant interacts with another individual. For example, a Likert-type scale can be used to deter- mine how often an individual shares new knowledge with others in their work
12 Human Resource Development Review / March 2007
13
TA B LE
3: S
a m
p le
:C o
m p
le te
N e tw
o rk
S u
rv e y
(N o
n d
ir e ct
io n
a l)
D ir
ec ti
on s:
B el
o w
y o u w
il l
fi n d a
l is
t o f
n a m
es o
f m
a n y
p eo
p le
w h o w
o rk
h er
e. I
n t
h is
s ec
ti o n w
e a re
i n te
re st
ed i
n y
o u r
p er
ce p ti
o n s
o f
w h o m
y o u
w o u ld
d is
cu ss
t h e
n ew
k n o w
le d ge
w it
h o
n h
o w
t o c
o n d u ct
y o u r
jo b ,
th a t
is ,
if y
o u h
a ve
c o m
p le
te d a
t ra
in in
g p
ro g ra
m w
h o d
o y
o u t
yp ic
a ll
y sp
ea k
w it
h a
b o u t
w h a t
yo u h
a ve
l ea
rn ed
a n d h
o w
t h a t
n ew
k n o w
le d ge
w il
l h el
p y
o u c
o n d u ct
y o u r
jo b m
o re
e ff
ec ti
ve ly
. P
le a se
i n d ic
a te
y o
u r
a n sw
er b
y p la
ci n g a
c h ec
k to
t h e
le ft
o f
th e
n a m
es o
f th
e p eo
p le
. If
t h er
e is
o n ly
o n e
p er
so n y
o u w
o u ld
g o t
o ,
th en
j u st
c h ec
k th
a t
o n e
p er
so n ’s
n a m
e. I
f th
er e
a re
s ev
er a l
p eo
p le
y o u m
ig h t
g o t
o ,
th en
c h ec
k a ll
t h e
n a m
es t
h a t
a p p ly
. If
t h er
e is
n o o
n e
yo u w
o u ld
g o t
o ,
th en
d o n
o t
ch ec
k a n
y n a m
es .
A ls
o ,
w e
a re
i n te
re st
ed i
n w
h o y
o u t
h in
k m
ig h t
co m
e to
y o u a
ft er
t ra
in in
g t
o d
is cu
ss t
h ei
r n ew
k n o w
le d ge
a n d h
o w
t h ey
c a n a
p p ly
i t
to t
h ei
r jo
b. P
le a se
i n d ic
a te
t h e
n a m
es o
f th
es e
p eo
p le
b y
p la
ci n g a
c h ec
k to
t h e
ri g h t
o f
th ei
r n a m
es .
A g a in
y o u c
o u ld
c h ec
k o n e
n a m
e, m
a n y
n a m
es ,
o r
n o n
a m
es a
t a ll
.
Th is p
er so
n us
ua lly
d is cu
ss es
I us
ua lly
d is cu
ss w
ith t
hi s
w ith
m e
w ha
t th
ey h
av e
pe rs
on w
ha t
I’v e
le ar
ne d
le ar
ne d
at a
t ra
in in
g se
ss io
n I co
ns ul
t th
is p
er so
n Th
is p
er so
n us
ua lly
c on
su lts
at a
t ra
in in
g se
ss io
n w
he n
w he
n th
ey g
et b
ac k
on a
r eg
ul ar
b as
is o
n ho
w t
o w
ith m
e on
a r
eg ul
ar b
as is
I ge
t ba
ck t
o th
e jo
b to
t he
jo b
co nd
uc t
m y
jo b
on h
ow t
o co
nd uc
t th
ei r
jo b
__ __
Jo hn
__ __
__ __
Jo hn
_ _ _ _
__ __
M ik
e __
__ __
__ M
ik e
_ _ _ _
__ __
K ev
in __
__ __
__ K
ev in
_ _ _ _
__ __
R ob
er t
__ __
__ __
R ob
er t
_ _ _ _
__ __
G uy
__ __
__ __
G uy
_ _ _ _
__ __
S am
__ __
__ __
S am
_ _ _ _
__ __
H ar
ry __
__ __
__ H
ar ry
_ _ _ _
__ __
B ob
__ __
__ __
B ob
_ _ _ _
__ __
T er
ry __
__ __
__ T
er ry
_ _ _ _
__ __
L ar
ry __
__ __
__ L
ar ry
_ _ _ _
__ __
R ic
k __
__ __
__ R
ic k
_ _ _ _
__ __
M an
ny __
__ __
__ M
an ny
_ _ _ _
__ __
M ar
y __
__ __
__ M
ar y
_ _ _ _
__ __
Je rr
y __
__ __
__ Je
rr y
_ _ _ _
N O
T E:
T he
n am
es u
se d
in t
hi s
ex am
pl e
ar e
fic ti ti o us
.
14
TA B LE
4: S
a m
p le
:E g o
N e tw
o rk
S u
rv e y
(D ir
e ct
io n
a l)
D ir
ec ti
on s:
R ea
d b
el o w
t h e
th re
e st
a te
m en
ts a
n d t
h in
k o f
th e
in d iv
id u a ls
y o u k
n o w
a t
w o rk
, a n d p
o ss
ib ly
i n y
o u r
p er
so n a l
li fe
, w
h o w
o u ld
im p a ct
y o u .
A ft
er l
is ti
n g t
h ei
r n a m
e, in
d ic
a te
t h ei
r jo
b r
o le
o r
re la
ti o n sh
ip i
f th
e in
d iv
id u a l
is n
o t
re la
te d t
o y
o u t
h ro
u g h w
o rk
. T
h en
c ir
cl e
ye s
if th
a t
in d iv
id u a l
w o rk
s in
t h e
sa m
e d ep
a rt
m en
t a s
yo u .
O n a
s ca
le o
f 1 t
o 5
, 1 b
ei n g n
ev er
a n d 5
b ei
n g o
ft en
, ra
te t
h e
in d iv
id u a ls
y o u l
is te
d f
o r
ea ch
o f
th e
fo ll
o w
in g q
u es
ti o n s.
Y o u c
a n l
is t
a s
m a n y
in d iv
id u a ls
a s
yo u t
h in
k re
le va
n t.
I f
yo u r
eq u ir
e m
o re
s p a ce
, si
m p ly
t u rn
t h e
p a p er
o ve
r a n d l
is t
th em
o n t
h e
b a ck
o f
th is
p ag
e.
Q 1:
T o
ge t
m y
w or
k do
ne I
r eq
ui re
t he
a ss
is ta
nc e
of t
hi s
in di
vi du
al .
Q 2:
A ft
er a
t ra
in in
g ac
ti vi
ty ,
I sh
ar e
th e
ne w
k no
w le
dg e
I ha
ve a
cq ui
re d
w it
h th
is i
nd iv
id ua
l. Q
3: T
hi s
in di
vi du
al h
as i
nf lu
en ce
o ve
r ho
w I
g et
m y
jo b
do ne
.
N am
e of
I nd
iv id
ua l
R ol
e Sa
m e
D ep
t. Q
1 Q
2 Q
3
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
Y /N
__ _
__ _
__ _
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ __
__ Y
/N __
_ __
_ __
_
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 15
group (i.e., 1 = seldom and 5 = very often) (see Table 4). This type of survey requires that the respondents assess their relationship with the individuals listed and can help determine the extent to which information is exchanged between network members. This survey would be administered to individuals who are participating in a training intervention outside their department or organization on their own (ego network analysis).
Sales Group
In the case of the sales group, the goal is to identify the frequency of con- nectivity between network members by using a valued scale. Those actors who were sought after on a frequent basis could be viewed as critical influencers of the transfer climate and addressed accordingly.
Including Actor Attribute Information in the Analysis
In addition to relational data, attribute characteristics of participants can be collected to help determine unique similarities in groups of individuals (Brass & Labianca, 1999). For example, if a complete network analysis is conducted, relational ties that demonstrate similar attribute characteristics (i.e., high per- formers) can be identified to help predict whether transfer will be supported among participants and their supervisors. Other variables may include previ- ous training exposure, educational profile, attitudes toward training, and importance of training to job function. By including attribute information in the analysis, network transfer models can be constructed to help explain why transfer takes place in one work group but not in another.
Sales Group
Performance history could be collected for the sales group example to ensure that those individuals identified as central to the network were provid- ing information that was in keeping with the department’s objectives.
Analyzing the Network Data
Once the relational and attribute data have been collected, analyzing the data is the next stage of the process. The data and network maps created for this article were generated by UCINET 6, which offers the researcher the abil- ity to compute network measures (Borgatti, Everett, & Freeman, 2002) as well as to generate network maps through its incorporated visualization software NetDraw (Borgatti, 2002) included in the package. Depending on whether for- mal research is being conducted or whether practitioners are attempting to dis- cern the network landscape of an individual or group, the analysis is guided by the research question. For example, if the question posed to the sales group
participants was, Who do you turn to after you return from training if you are not sure of something you have learned? and participants were asked to rate their connection to the individuals within their work group on a scale of 1 (seldom) and 3 (very often), the network map created would comprise direc- tional ties that are valued. In Figure 1, we can see that the strength of the rela- tionship is indicated by the number over the line connecting the network members.
Sales Group
For example, Rick, a sales representative, shares new knowledge with Robert, a senior sales representative, “very often” (3), but Robert does not share at all with Rick (this is indicated by absence of an arrowed line from Robert to Rick).
If the goal of the analysis is to determine whether a relationship exists, it can be represented by a 0 or 1. This form of measurement may be useful if the goal of the analysis is to determine whether there is connectivity among work groups. However, this measure is nondirectional and is conducted to determine whether a relationship exists between two individuals, regardless of who initiates the connection. If determining the strength of a relationship is not important, the question posed is geared to identifying a relationship among actors. The
16 Human Resource Development Review / March 2007
FIGURE 1: Directional and Valued Ties; Question: Who do you turn to after you return from training if you are not sure of something you have learned? NOTE: The names used in this example are fictitious.
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 17
FIGURE 2: Nondirectional Ties NOTE: The names used in this example are fictitious.
example used for the sales group, Who do you turn to after you return from training if you are not sure of something you have learned? is represented by the existence or nonexistence of a line between actors (see Figure 2).
Creating Descriptive Indices of Social Structure
Once the data have been collected, the creation of descriptive indices can take place. The descriptive indices of social structure are measures of central- ity, density, and cliques. Illuminating the transfer climate will depend on the questions used for the analysis.
Centrality Measures
If the goal of the analysis is to determine which member is the most sought after for clarifying information learned in a training session, centrality mea- sures can be computed to identify those individuals who clarify or reinforce the training information to the work group on a regular basis. “Betweenness” centrality measures how often a given actor sits “between” others, with “between” referring to the shortest path (Scott, 2000). An actor that is between many actors is assumed to have a higher likelihood of being able to control
information flow in the network. If there are only a few members who control information flow within the group, the climate may be negatively oriented toward transfer. Table 5 lists the centrality betweenness measures for the sales group found in Figure 1.
Sales Group
John possesses the highest centrality measure (8.33) and is in a position to control the flow of information within the group, whether positive or negative.
18 Human Resource Development Review / March 2007
TABLE 5: Individual Centrality Measures
Employee Betweeness
Question—Who do you go to for advice on performing your job? John 8.333 Mike 7.000 Kevin 4.000 Robert 4.000 Guy 0.667 Sam 0.000 Harry 0.000 Bob 0.000 Terry 0.000 Larry 0.000 Rick 0.000 Manny 0.000 Mary 0.000 Jerry 0.000
Question—Who do you go to for company gossip in your work group? John 19.167 Robert 7.000 Mike 5.167 Guy 1.333 Bob 0.333 Kevin 0.000 Harry 0.000 Sam 0.000 Terry 0.000 Larry 0.000 Rick 0.000 Manny 0.000 Mary 0.000 Jerry 0.000
NOTE: The names used in this example are fictitious.
In addition, Mike (7.00), Kevin (7.00), and Robert (4.00) are central to the network. John, Mike, and Kevin are sales representatives, whereas Robert is a senior sales representative. This raises questions as to why the sales manager (Bob), supervisor (Larry), and the other senior sales representative (Harry) are not central to the network based on their positions. Even if these individuals sup- ported the training, their influence on transfer may be limited as they are not sought out for information on a regular basis. In addition to displaying the cen- trality measures, a visual representation of the network structure is displayed in Figure 3 (central actors John, Mike, Kevin, and Robert are displayed as larger circles in the network map).
Now that the central actors have been identified, actions may be taken to support the transfer of training. Those individuals identified as central to the network can help reinforce the goals and objectives of the training to the rest of the group. However, it is imperative that the central actor’s perception and understanding of the purpose of the training is in keeping with its desired out- come. Uncovering those individuals who act as brokers of information for group members can help ensure that everyone acknowledges the applicability
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 19
FIGURE 3: Centrality—Betweeness; Question: Who do you turn to after you return from training if you are not sure of something you have learned? NOTE: The names used in this example are fictitious.
of the new learning. Prior to the training intervention, the trainer could meet with the individuals to discuss the content that will be presented during the training. Therefore, there would be five goals when meeting with John, Mike, Kevin, and Robert: (a) to discuss why training is needed, (b) to solicit input on what content should be delivered during the session, (c) decide on the best way to deliver the training, (d) how to best apply the new learning back to the work- place, and (e) ensure that they will support the transfer of new learning back to the workplace. If the new learning is in keeping with the central actors’ existing approach to their job, it is most likely that the new knowledge would be reinforced when network members seek out their advice. However, if the information contradicts the actors approach to performing their job function or if they feel that the new learning is irrelevant to increased performance, trans- fer of training is less likely to occur. By uncovering the network structure and identifying the central actors, the trainer can take steps to ensure that these individuals will support training transfer to the workplace.
The questions that guide the analysis are situational in nature and should be treated as such when deciphering the data. The central network members may change based on the type of question that is posed to the participants. For example, Figure 3 highlights the central actors for the question, Who do you turn to after you return from training if you are not sure of something you have learned? If the question were changed to, Who do you go to for company gos- sip in your work group? the central figures may differ as a result of the new question (Figure 4). This is illustrated in Figures 3 and 4; John is highly cen- tral in both situations, as well as Robert and Mike, whereas Kevin is no longer central to the network for company gossip (Figure 4). An actor who is sought after for advice on how to transfer new learning to the job and company gossip may make the transfer climate highly volatile and may require further exploration prior to the training intervention. Even if the highly central network member is in line with the training objectives but is misinformed about some negative company information, the transfer of training may not be supported. Uncovering this information prior to the training interventions may be the difference between a successful program and limited intervention effects.
Density Measure
For transfer to occur, the transfer climate must support the employees when they return to the job. Organizational social support has often been used to rep- resent transfer climate (Cheng & Ho, 2001). If the question posed through SNA involves the identification of existing relationships that support training transfer, the researcher would want to determine the existing cohesiveness between network members. Measuring for network density will help quantify the cohesiveness that exists within the work group or between groups. Density
20 Human Resource Development Review / March 2007
is the number of actual links proportionate to the total possible links that can exist and is calculated using the following equation:
where l represents the number of lines present and n represents the number of nodes within the network. The value of the density measure can range from 0 to 1, where 1 represents complete density within the network (Scott, 2000).
Sales Group
For example, the density for the sales group (see Figure 2) for the question, Who do you turn to after you return from training if you are not sure of some- thing you have learned? is .247, meaning that the actual number of ties in the network is 24.7% out of the potential number of ties. For the sales group, a density level of 24.7% problematic in that the connectivity between individu- als regarding new information sharing is minimal and may reflect the work cli- mate. If the intervention were to include team building training, it is imperative that what occurs within the training environment follows to the workplace. Measuring the groups’ density prior to the training and after will help deter- mine whether the intervention has had any impact on opening communication channels between members. If low-density measures are identified prior to training, content can be geared to addressing specific issues and individuals identified through the analysis. In addition, interviews can be conducted before the training to establish how group members perceive the overall con- nectivity between members. If common issues are identified, action can be taken prior to the commencement of the training to deal with problems asso- ciated with low connectivity, thereby maximizing the opportunity for a higher level of transfer. Benchmarking density measures over time will help deter- mine the most appropriate density level for the particular work group. Periodically measuring the density of the group will help ensure that the trans- fer climate is at the right level for training transfer to take place.
Cliques
To determine whether pockets of cohesion exist among network members, the n-clique measure can be calculated to identify cliques within the network by setting the desired level of connectedness between actors. Uncovering existing cliques can help explain why certain groups within the network struc- ture are either “for” or “against” the adoption of new learning. For the n-clique procedure, n is the maximum path length at which members of the clique will be regarded as connected (Scott, 2000). For example, if the researcher was attempting to identify connected groups of 3, he or she would employ a 2- clique measure, which refers to an actor who is either tied directly or indirectly
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 21
l n(n–1)/2
to another actor. Those actors identified through the 2-clique measure are sit- uated within a maximum of 2 degrees of separation; therefore, Actor 1 knows Actor 2 directly but may go through Actor 2 to get to Actor 3. In some orga- nizational climates, subgroups may be the norm that can be disruptive to others in a training setting. If these cliques are identified prior to training, it is possi- ble to determine whether there are similarities in performance levels so that precautions can be taken to diffuse or encourage participation in the training situation and transfer back to the job.
Presenting the Network Data
There are two forms that network data can take, matrix data or sociograms (network map). The matrix data (see Table 6) represent raw data whereas the sociogram (see Figure 1 as an example) provides a visual representation of the connections within a network. From a practical perspective, a sociogram pro- vides organizations the opportunity to visually identify the social structure that exists within their work groups, whereas matrix data provide detailed
22 Human Resource Development Review / March 2007
FIGURE 4: Centrality—Betweeness Question: Who do you go to for company gos- sip in your work group? NOTE: The names used in this example are fictitious.
measures of interconnectivity between network members. Sharing this infor- mation with both trainees and management will assist in starting a dialogue about the relationships that exist and how they influence the organizational cli- mate. Developing an action plan that deals with the social barriers can ensure that training is supported by managers and coworkers and is transferred back to the job. The network data can be presented at the beginning of a training sit- uation so that participants become aware of the hidden pressures that exist within their working groups. If an entire work group is participating in train- ing, a portion of the training can be dedicated to overcoming the potential bar- riers to using new learning. For more detail on network measures presented in this section, please see Wasserman and Faust (1994).
Discussion
SNA can serve as a methodology for identifying the relational patterns that help form the transfer climate by illuminating the network structure of an orga- nization. The example of the sales group provided in this article entailed a complete network analysis. However, in many training situations, participants attend in isolation of other work group members. In this case, ego network analysis can be performed to determine the relational barriers that may affect their ability to transfer new knowledge back to the workplace. The researcher may choose to examine their relationships not only with coworkers but also with those individuals outside the workplace who may have some influence on training transfer (i.e., suppliers, customers, competing firms) (see Table 4).
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 23
TABLE 6: Valued Data Presented in a Matrix Format
Robert Sam Guy Harry John Kevin Mike Bob Terry Larry Rick Manny Mary Jerry
Robert 0 1 1 0 1 1 1 0 1 0 0 0 0 0 Sam 1 0 0 0 1 1 1 0 2 0 0 0 0 0 Guy 2 0 0 0 1 3 1 2 1 0 0 0 0 0 Harry 2 0 0 0 1 1 1 0 2 0 0 0 0 0 John 3 1 1 0 0 3 2 0 2 0 0 0 0 0 Kevin 1 0 0 0 1 0 1 1 1 0 0 0 0 0 Mike 2 1 2 0 2 2 0 3 2 0 0 0 0 0 Bob 0 0 0 0 1 2 2 0 1 0 0 0 0 0 Terry 0 0 0 0 1 1 0 0 0 0 0 0 0 0 Larry 3 3 0 2 3 1 3 1 3 0 0 0 0 0 Rick 3 3 0 2 1 1 3 3 3 0 0 0 0 0 Manny 2 3 0 2 3 1 2 3 3 0 0 0 0 0 Mary 3 3 0 1 1 1 1 2 3 0 0 0 0 0 Jerry 2 0 0 0 1 1 3 3 3 0 0 0 0 0
Even though the ego analysis only takes into account the relationships from the perspective of the trainee, these perceptions are useful in determining the likelihood for transfer taking place and can be utilized by the trainee and supervisor to reinforce the importance of applying the new learning on the job. If relationships with key influencers (i.e., team leader, supervisor, manager) are considered poor, steps may be taken to explore transfer strategies to deal with strengthening the trainee’s interaction with these individuals so that the new learning can be applied on their return to the job. This can easily be incor- porated into the training program and followed up with once the program has been completed. In addition, supervisors can be made aware of the importance of encouraging a climate that supports the implementation of new and creative ideas once the employee has returned to the workplace (Clark et al., 1993).
SNA can also serve as an assessment tool prior to the planning of a training intervention to identify who the true “opinion leaders” within the organizational climate are and to include them in the development and implementation of the training program. In addition, these “opinion leaders” can be utilized during and after the training to reinforce the new learning. Increasing the likelihood of train- ing transfer can be accomplished at the design phase of the HRD intervention (Hatala & Gumm, 2006). Getting buy-in from centrally positioned actors in the network on the training content, delivery mechanism, and overall objective of the training program is an important element of success that is often overlooked; SNA can help to identify these central actors. Employees participating in training are more likely to utilize the new learning if they perceive it as clearly relevant to work-related activities and are supported by their peers. Including individuals who are sought out by other network members will help ensure that the input is representative of the work group and the transfer climate is supportive.
SNA will help add to our knowledge about training transfer by providing the vehicle for mapping out the social structure of an organization that will allow for the identification of relational barriers affecting the transfer climate. Once these barriers are identified, research can start to examine the process of how transfer takes place and how transfer behaviors are maintained over time.
Implications for Further Research
SNA can be used by the researcher to make the transfer climate visible by uncovering the relational barriers existing within the social structure of a work group that influence the perceptions of network members. This can be accom- plished by identifying key employees who are in a position to influence train- ing transfer. Training transfer researchers have much to gain by learning why individuals act and respond to social pressures that exist within the organiza- tional climate. Transfer climate network models can help to identify the hid- den influencers who positively or negatively affect the organizational climate.
24 Human Resource Development Review / March 2007
Possible research questions include what level of connectedness (density) is required for a work group to transfer training back to the job? How do small cliques within the larger work environment impact transfer of training? Are informal or formal peer support groups more conducive to training transfer? Or are informal organizational power structures (centrality) more influential on training transfer than formal lines of authority? These research questions can be guided by SNA for building and adding to existing training transfer theory.
The majority of transfer literature has focused on the individualistic per- spective of training transfer (London & Flannery, 2004). Only recently have researchers begun to examine organizational support as a facilitating variable to increasing the likelihood that transfer will occur (Cromwell & Kolb, 2004; London & Flannery, 2004). More specifically, SNA is supported by the find- ings of Tracey, Tannenbaum, and Kavanagh (1995) on self-selected work groups, which suggest that interventions that target supervisors, coworkers, and individuals who interact with trainees may yield the greatest dividends. Transfer research is needed to examine the relationships existing within an organization’s social structure prior to training interventions to understand the most effective relational patterns that promote a positive transfer climate.
It is also imperative that future transfer research take a holistic approach by collecting both attribute and relational data when examining transfer behaviors. Baldwin and Ford (1988) stated that the number of correlational studies examining transfer climate was problematic in that causality could not be inferred in many of the studies on transfer research. SNA can address this issue by examining the relations within a work group and how central actors influence the application of new learning to the job. SNA can help expose both the positive and negative relationships that either support or hinder transfer efforts. Network models can help to conceptualize transfer climate as lasting patterns of relations among actors and add to training transfer theory building by measuring the impact central actors have on sup- porting transfer (Hatala, 2006).
In addition, Baldwin and Ford (1998) identified the criterion problem in that the majority of studies conducted on environmental characteristics were typically self-reports of behavioral change as a major measure of transfer. In addition, Cromwell and Kolb (2004) addressed the issue of transfer behaviors over time by examining the sustainability of training behavior over various intervals. They found that training behaviors were not greatly exhibited until the 1-year point. They provide possible reasons for the delay in transfer, which included the lack of time and energy for transfer, the workload, promotional opportunities, or salary incentives. Future studies on training transfer at vari- ous intervals may consider using SNA as a means to measure transfer and its effect on social structure over time. Although individual network members may change, the goal of achieving role-specific relational patterns between
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 25
organizational positions will remain constant. Therefore, the goal of analysis may be to determine whether the relational patterns (i.e., workers seek out supervisors for training support) are consistent over time.
For example, the sales manager (Bob) in the sales group case was not sought after for new learning reinforcement at the time of the initial SNA (pretest) based on the question, Who do you turn to after you return from training if you are not sure of something you have learned? If a second SNA (posttest) were to be con- ducted some time in the future after conscious efforts were made to improve the manager’s ability to provide support to his workers, the desired results of the analysis would identify the position as central to the network. Even if the man- ager were to be replaced by the time the next analysis was conducted, the desired outcome would be the identification of the same relational pattern where the supervisor is sought out for support. Conversely, if an individual was identified in the pretest as a central actor but left the organization by the posttest, the relational patterns could still be examined to determine the impact the individual had on the group and overall supportive nature of the transfer climate itself.
Research has attempted to address the question of network changes over time and has demonstrated that supportive ties are the most likely to persist and that frequent contact between network members is also associated with the persis- tence of relationships (Feld, 1997; Morgan, Neal, & Carder, 1996; Suitor & Keeton, 1997; Wellman, Wong, Tindall, & Nazer, 1997). In light of this evi- dence, further research on developing a cohesive social structure that supports training transfer is needed. It would be expected that once a supportive network has been established, and the transfer climate is more positive, training transfer will most likely increase. This can take the form of peer support networks that continue to meet after the completion of training to reinforce transfer to the job.
Projects using SNA should be careful not to generalize network structure based on the question presented. The sales group analysis that was based on two differ- ent questions is an example of the situational nature of SNA. An employee who is popular and central to the network but who is not sought after for advice on how to implement new learning on the job can still yield some influence. The influence of this individual may at times outweigh those employees who are sought after for job advice and should not be taken lightly. Buy-in to training programs should come from both the identified subject matter experts and those who are deemed popular from the social perspective. This is a critical component to network analysis and questions should be thought out carefully and in line with theoreti- cal assumptions of the research. Training climate has focused on work environ- ment and has identified organization, supervisory, and peer support as critical factors to successful transfer (Cromwell & Kolb, 2004; London & Flannery, 2004). It is important to keep these factors in mind when forming research ques- tions about transfer climate. Aligning trainee attributes to relational data will help uncover the social pressures that exist within a given environment and help to explain why transfer occurs.
26 Human Resource Development Review / March 2007
Implications for Human Resource Development Practice
SNA can also be used by the practitioner to make the transfer climate visi- ble. The use of SNA from a practical perspective is not a complex process and can easily be built into a training program. As was mentioned earlier in the article, there are two types of network analysis—ego and complete network analysis. If the training situation deals with a number of participants from dif- ferent work groups, units, or organizations, the goal of the SNA should be to explore the relational patterns within their work environment so that when they return to the workplace they can develop a strategy with their supervisors for dealing with any relational barriers identified. Because each individual will be conducting an ego network analysis a number of questions can be asked. The examples provided in Table 2 can be used as a guideline for selecting questions for the analysis. A discussion on the importance of training transfer can be conducted at the beginning of a training session, at which time the SNA can be administered. Each participant fills out the SNA survey and separately ranks each of their network members based on their influence within the group (see Table 6). Small groups can be formed to discuss the findings and answer questions regarding the transfer climate within their own organizations. Examples may include questions such as which of the individuals listed have you labeled a 4 or 5 and why? Which individuals have you labeled as a 3 or lower and why? How important is support from those individuals who you listed as a 4 or 5? Relational barriers to transfer can be explored using the par- ticipants’ key influencers identified through the SNA and strategies can be dis- cussed on how to seek the support needed to apply their new learning on the job. In addition, the participants can be instructed to share their SNA with their supervisor to discuss the most effective ways of implementing the new train- ing at the workplace.
The complete network analysis (sales group example) is used for a work group that is involved in a training session with all members of the group pre- sent. The SNA can be completed prior to the training session and the results can be shared with the entire group at the beginning of the training program. A dis- cussion on the potential barriers to transfer can occur, followed by strategies on how to overcome them and the best way to apply the new learning on the job. Although this adds to the length of the training program, addressing transfer issues up front will increase the likelihood of transfer actually occurring.
When conducting an SNA, the size of the organization must also be taken into consideration. A work group of just 10 people represents 90 unique rela- tionships (directional), thus creating the potential for some relational issues. The size of the organization is irrelevant to the utility of SNA, but it must be considered from a methodological perspective. The number of respondents for a complete network analysis typically should not exceed 250 respondents,
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 27
because the more the actors, the greater the difficulty in the data analysis (Cross & Parker, 2004). If there are more than 250 actors, the ego network analysis can be conducted. Another option is to create a mixed method approach by conducting a complete network analysis for a work group, unit, or department and also provide the respondents the option of identifying addi- tional individuals outside their work group. Individuals identified by the respondents outside the group can easily be identified by simply looking at the number of times the name was mentioned in the surveys.
Conclusion
SNA is tool that can assist both HRD researchers and practitioners by iden- tifying the relational barriers to the transfer of new learning to the job. By identifying relational patterns within an organizational context, we can learn a lot about how training transfer occurs and the best ways to encourage a posi- tive transfer climate. The transfer literature indicates that transfer climate is a major determinant of transfer. SNA can be used to develop strategies that may increase the likelihood of training transfer taking place.
28 Human Resource Development Review / March 2007
TABLE 7: Sample: Ego Network Survey Completed Prior to Training
Directions: Read below the three statements and think of the individuals you know at work and possibly in your personal life who would impact you. After listing their name, indicate their job role or relationship if the individual is not related to you through work. Then circle yes if that individual works in the same department as you. On a scale of 1 to 5, 1 being never and 5 being often, rate the individuals you listed for each of the following questions.
You can list as many individuals as you think relevant. If you require more space, simply turn the paper over and list them on the back of this page.
Q1: To get my work done I require the assistance of this individual. Q2: After a training activity, I share the new knowledge I’ve acquired with this
individual. Q3: This individual has influence over how I get my job done.
Name of Individual Role Same Dept. Q1 Q2 Q3
John Supervisor Y/N 5 2 5 Kim Coworker Y/N 3 3 3 Julie Coworker Y/N 2 2 2 Samantha Coworker Y/N 2 3 2 Liam Manager Y/N 2 4 4 Eric Coworker Y/N 2 3 4 Camryn Manager Y/N 1 1 1 Wendy Coworker Y/N 5 5 2
References
Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and directions for future research. Personnel Psychology, 41, 63-105.
Bates, R. A., Holton, E. F., Seyler, D. L., & Carvalho, M. A. (2000). The role of interpersonal fac- tors in the application of computer-based training in an industrial setting. Human Resource Development International, 3, 19-42.
Borgatti, S. P. (2002). NetDraw: Graph visualization software. Harvard: Analytic Technologies. Borgatti, S. P., Everett, M. G., &Freeman, L. C. (2002). Ucinet for windows: Software for social
network analysis. Needham, MA: Analytic Technologies. Brass, D. J., & Labianca, G. (Eds.). (1999). Social capital, the social ledger, and social resources
management. Dordrecht, The Netherlands: Kluwer Academic. Broad, M. L., & Newstrom, J. W. (1992). Transfer of training: Action-packed strategies to ensure
high payoff from training investments. Reading, MA: Addison-Wesley. Burt, R. S. (1992). Structural holes: The social structure of competition. Cambridge, MA: Harvard
University Press. Burt, R. S. (1997). The contingent value of social capital. Administrative Science Quarterly, 42,
339-365. Campbell, J. P. (1988). Training design for performance improvement. Productivity in Organizations.
San Francisco: Jossey-Bass. Cheng, E. W. L., & Ho, D. C. K. (2001). A review of transfer studies in the past decade. Personnel
Review, 30, 102-118. Clark, C. S., Dobbins, G. H., & Ladd, R. T. (1993). Exploratory field study of training motivation:
Influence of involvement, credibility and transfer climate. Group and Organization Management, 18(3), 292-307.
Coleman, J. S. (1988). Social capital in the creation of human capital. American Journal of Sociology, 94, S95-S120.
Cromwell, S. E., & Kolb, J. A. (2004). An examination of work-environment support factors affecting transfer of supervisory skills training to the workplace. Human Resource Development Quarterly, 15(4), 449-471.
Cross, R., & Parker, A. (2004). The hidden power of social networks: Understanding how work really gets done in organizations. Boston: Harvard Business School Press.
Denison, D. R. (1996). What is the difference between organizational culture and organizational climate? A native’s point of view on a decade of paradigm wars. Academy of Management Review, 21(3), 619-655.
Eagan, T. M., Yang, B., & Bartlett, K. R. (2004). The effects of organizational learning culture and job satisfaction on motivation to transfer learning and turnover intention. Human Resources Development Quarterly, 15(3), 279-301.
Feeley, T. H., & Barnett, G. A. (1996). Predicting employee turnover from communication net- works. Human Communication Research, 23, 370-387.
Feld, S. L. (1997). Structural embeddedness and stability of interpersonal. Social Networks, 19, 91-95.
Georgenson, D. L. (1982). The problem of transfer calls for partnerships. Training and Development Journal, 36(10), 75-78.
Gist, M. E., Bavetta A. G., & Stevens, C. K. (1990). Transfer training method. Personnel Psychology, 43, 501-523.
Goldstein, I. L. (1986). Training in organizations: Needs assessment, development, and evalua- tion. Monterey, CA: Brooks Cole.
Hatala, J. P. (2006). Social network analysis in human resource development: A new methodol- ogy. Human Resource Development Review, 5, 45-71.
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 29
Hatala, J. P., & Gumm, J. C. (2006). Managing organizational cultural influences during the implementation of competency-based training. Advances in Human Resource Development, 8(2), 229-246.
Holton, E. F. (1996). The flawed four-level model. Human Resource Development Quarterly, 7, 5-25.
Holton, E. F., & Baldwin, T. T. (2000). Making transfer happen: An action perspective on learn- ing transfer systems. Advances in Developing Human Resources, 2(4), 1-6.
Holton, E. F., Bates, R. A., Seyler, D. L., & Carvalho, M. B. (1997). Toward construct validation of a transfer climate instrument. Human Resource Development Quarterly, 8(2), 92-113.
Ibarra, H., & Andrews, S. B. (1993). Power, social influence and sense making: Effects of network cen- trality and proximity on employee perceptions. Administrative Science Quarterly, 38, 277-303.
Kahn, W. A., Cross, R., & Parker, A. (2003). Layers of diagnosis for planned relational change in organizations. Journal of Applied Behavioral Sciences, 39(3), 259-280.
Kim, H. (2004). Transfer of training as a sociopolitical process. Human Resource Development Quarterly, 15(4), 497-501.
Kontoghiorghes, C. (2002). Predicting motivation to learn and motivation to transfer learning back to the job in a service organization: A new systemic model for training effectiveness. Performance Improvement Quarterly, 15(3), 114-129.
Lim, D. H. (2001). The effect of work experience and job position on international learning trans- fer. International Journal of Vocational Education and Training, 9(2), 59-74.
Lim, D. H., & Johnson, S. D. (2002). Trainee perceptions of factors that influence learning trans- fer. International Journal of Training and Development, 6, 36-48.
Lim, D. H., & Morris, M. L. (2006). Influence of trainee characteristics, instructional satisfaction, and organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17, 85-115.
Litwin, G., & Stringer, R. (1968). Motivation and organizational climate. Cambridge, MA: Harvard University Press.
London, P. A., & Flannery, D. D. (2004). Social factors that impact women’s practice of breast self-examination: A challenge to the transfer of training literature. Journal of European Industrial Training, 28(6/7), 538-549.
Machles, D. (2002). Training transfer strategies for the safety professional. Safety Professional, 47(2), 32-34.
Meyer, G. W. (1994). Social information processing and social networks: A test of social influ- ence mechanisms. Human Relations, 47, 1013-1048.
Morgan, D. L., Neal, M. B., & Carder, P. (1996). The stability of core and peripheral networks over time. Social Networks, 19, 9-25.
Noe, R. A. (1988). Trainee’s attributes and attitudes: Neglected influences on training effective- ness. Academy of Management Review, 11, 736-749.
Pidd, K. (2004). Organizational barriers to training transfer: The role of workplace social con- trols and normative behaviour in workplace development. National Center for Education and Training on Addiction.
Rossett, A. (1997). This was a great class, but…, Training and Development Journal, 51(7), 18-24. Rouiller, J. Z., & Goldstein, I. L. (1993). The relationship between organizational transfer climate
and positive transfer in training. Human Resource Development Quarterly, 4, 377-390. Scott, J. (2000). Social network analysis: A handbook (2nd ed.). London: Sage. Suitor, J., & Keeton, S. (1997). Once a friend, always a friend? Effects of homophily on women’s
support networks across a decade. Social Networks, 19, 51-62. Taylor, M. C. (2000). Partners in the transfer of learning: A qualitative study of workplace liter-
acy programs. Proceedings of the 41st Annual Adult Education Research Conference (pp. 459- 461). Vancouver: University of British Columbia.
Torenvlied, R., & Velner, G. (1998). Informal networks and resistance to organizational change: The introduction of quality standards in a transport company. Computational & Mathematical Organization Theory, 4(2), 165-188.
30 Human Resource Development Review / March 2007
Tracey, J. B., Tannenbaum, S. I., & Kavanagh, M. J. (1995). Applying trained skills on the job: The importance of the work environment. Journal of Applied Psychology, 80(2), 239-252.
Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. New York: Cambridge University Press.
Wellman, B., Wong, R., Tindall, D., & Nazer, N. (1997). A decade of network change: Turnover, persistence and stability in personal communities. Social Networks, 19, 27-50.
White, D. R. (1997). What is network theory? Retrieved April 23, 2006, from http://eclectic.ss.uci .edu/drwhite/netsyl96.htm
John-Paul Hatala is assistant professor, Louisiana State University, Baton Rouge. His academic research has focused on the transition toward reemploy- ment, social capital in the workplace, social networking behavior, and organi- zational development.
Pamela R. Fleming is a doctoral student in human resource and leadership devel- opment at Louisiana State University. Her research interest includes leadership, organization behavior, emotional intelligence, and training transfer.
Hatala, Fleming / MAKING TRANSFER CLIMATE VISIBLE 31
<< /ASCII85EncodePages false /AllowTransparency false /AutoPositionEPSFiles true /AutoRotatePages /None /Binding /Left /CalGrayProfile (Dot Gain 20%) /CalRGBProfile (sRGB IEC61966-2.1) /CalCMYKProfile (U.S. Web Coated \050SWOP\051 v2) /sRGBProfile (sRGB IEC61966-2.1) /CannotEmbedFontPolicy /Error /CompatibilityLevel 1.3 /CompressObjects /Off /CompressPages true /ConvertImagesToIndexed true /PassThroughJPEGImages true /CreateJDFFile false /CreateJobTicket false /DefaultRenderingIntent /Default /DetectBlends true /ColorConversionStrategy /LeaveColorUnchanged /DoThumbnails false /EmbedAllFonts true /EmbedJobOptions true /DSCReportingLevel 0 /EmitDSCWarnings false /EndPage -1 /ImageMemory 1048576 /LockDistillerParams true /MaxSubsetPct 100 /Optimize false /OPM 1 /ParseDSCComments true /ParseDSCCommentsForDocInfo true /PreserveCopyPage true /PreserveEPSInfo true /PreserveHalftoneInfo false /PreserveOPIComments false /PreserveOverprintSettings true /StartPage 1 /SubsetFonts true /TransferFunctionInfo /Apply /UCRandBGInfo /Remove /UsePrologue false /ColorSettingsFile () /AlwaysEmbed [ true /AGaramond-BoldScaps /AGaramond-Italic /AGaramond-Regular /AGaramond-RomanScaps /AGaramond-Semibold /AGaramond-SemiboldItalic /AGar-Special /AkzidenzGroteskBE-Bold /AkzidenzGroteskBE-BoldEx /AkzidenzGroteskBE-BoldExIt /AkzidenzGroteskBE-BoldIt /AkzidenzGroteskBE-Ex /AkzidenzGroteskBE-It /AkzidenzGroteskBE-Light /AkzidenzGroteskBE-LightEx /AkzidenzGroteskBE-LightOsF /AkzidenzGroteskBE-Md /AkzidenzGroteskBE-MdEx /AkzidenzGroteskBE-MdIt /AkzidenzGroteskBE-Regular /AkzidenzGroteskBE-Super /AlbertusMT /AlbertusMT-Italic /AlbertusMT-Light /Aldine401BT-BoldA /Aldine401BT-BoldItalicA /Aldine401BT-ItalicA /Aldine401BT-RomanA /Aldine721BT-Bold /Aldine721BT-BoldItalic /Aldine721BT-Italic /Aldine721BT-Light /Aldine721BT-LightItalic /Aldine721BT-Roman /AlternateGothicNo2BT-Regular /AmericanaBT-Bold /AmericanaBT-ExtraBold /AmericanaBT-ExtraBoldCondensed /AmericanaBT-Italic /AmericanaBT-Roman /Anna /AntiqueOlive-Bold /AntiqueOlive-Compact /AntiqueOlive-Italic /AntiqueOlive-Roman /Arkona-Medium /Arkona-Regular /AshleyScriptMT /AssemblyLightSSK /AvantGarde-Bold /AvantGarde-BoldObl /AvantGarde-Book /AvantGarde-BookOblique /AvantGarde-CondBold /AvantGarde-CondBook /AvantGarde-CondDemi /AvantGarde-CondMedium /AvantGarde-Demi /AvantGarde-DemiOblique /AvantGarde-ExtraLight /AvantGarde-ExtraLightObl /AvantGarde-Medium /AvantGarde-MediumObl /BakerSignetBT-Roman /BaskervilleBE-Italic /BaskervilleBE-Medium /BaskervilleBE-MediumItalic /BaskervilleBE-Regular /Baskerville-Bold /BaskervilleBT-Bold /BaskervilleBT-BoldItalic /BaskervilleBT-Italic /BaskervilleBT-Roman /BaskervilleMT /BaskervilleMT-Bold /BaskervilleMT-BoldItalic /BaskervilleMT-Italic /BaskervilleMT-SemiBold /BaskervilleMT-SemiBoldItalic /BaskervilleNo2BT-Bold /BaskervilleNo2BT-BoldItalic /BaskervilleNo2BT-Italic /BaskervilleNo2BT-Roman /Baskerville-Normal-Italic /BauerBodoni-Black /BauerBodoni-BlackCond /BauerBodoni-BlackItalic /BauerBodoni-Bold /BauerBodoni-BoldCond /BauerBodoni-BoldItalic /BauerBodoni-BoldItalicOsF /BauerBodoni-BoldOsF /BauerBodoni-Italic /BauerBodoni-ItalicOsF /BauerBodoni-Roman /BauerBodoni-RomanSC /BauhausITCbyBT-Bold /BauhausITCbyBT-Heavy /BauhausITCbyBT-Light /BauhausITCbyBT-Medium /Bell-GothicBoldItalicBT /BellGothicBT-Bold /BellGothicBT-Roman /Bembo /Bembo-Bold /Bembo-BoldExpert /Bembo-BoldItalic /Bembo-BoldItalicExpert /Bembo-Expert /Bembo-ExtraBoldItalic /Bembo-Italic /Bembo-ItalicExpert /Bembo-Semibold /Bembo-SemiboldItalic /Berling-Bold /Berling-BoldItalic /Berling-Italic /Berling-Roman /BernhardBoldCondensedBT-Regular /BernhardFashionBT-Regular /BernhardModernBT-Bold /BernhardModernBT-BoldItalic /BernhardModernBT-Italic /BernhardModernBT-Roman /BickhamScriptMM /BickhamScriptMM-AltI /BickhamScriptMM-AltII /BickhamScriptMM-Beg /BickhamScriptMM-End /BickhamScriptMM-Lig /BickhamScriptMM-Or /BickhamScriptMM-SwCaps /Bodoni /Bodoni-Bold /Bodoni-BoldItalic /Bodoni-Italic /Bodoni-Poster /Bodoni-PosterCompressed /Bookman-Demi /Bookman-DemiItalic /Bookman-Light /Bookman-LightItalic /Boton-Italic /Boton-Medium /Boton-MediumItalic /Boton-Regular /Boulevard /CaflischScript-Bold /CaflischScript-Regular /Caliban /Carta /Caslon224ITCbyBT-Bold /Caslon224ITCbyBT-BoldItalic /Caslon224ITCbyBT-Book /Caslon224ITCbyBT-BookItalic /Caslon540BT-Italic /Caslon540BT-Roman /CaslonBT-Bold /CaslonBT-BoldItalic /CaslonOpenFace /CaslonTwoTwentyFour-Black /CaslonTwoTwentyFour-BlackIt /CaslonTwoTwentyFour-Bold /CaslonTwoTwentyFour-BoldIt /CaslonTwoTwentyFour-Book /CaslonTwoTwentyFour-BookIt /CaslonTwoTwentyFour-Medium /CaslonTwoTwentyFour-MediumIt /CastleT-Bold /CastleT-Book /Caxton-Bold /Caxton-BoldItalic /Caxton-Book /Caxton-BookItalic /Caxton-Light /Century-Bold /Century-BoldItalic /Century-Book /Century-BookItalic /Century-Light /Century-LightItalic /CenturyOldStyle-Bold /CenturyOldStyle-Italic /CenturyOldStyle-Regular /Century-Ultra /Century-UltraItalic /ChaparralMM /ChaparralMM-Ep /ChaparralMM-It /ChaparralMM-ItEp /ChaparralMM-ItSC /ChaparralMM-Or /ChaparralMM-SC /CharterBT-Black /CharterBT-BlackItalic /CharterBT-Bold /CharterBT-BoldItalic /CharterBT-Italic /CharterBT-Roman /CheltenhamBT-Bold /CheltenhamBT-BoldItalic /CheltenhamBT-Italic /CheltenhamBT-Roman /Christiana-Bold /Christiana-BoldItalic /Christiana-Italic /Christiana-Medium /Christiana-MediumItalic /Christiana-Regular /Christiana-RegularExpert /Christiana-RegularSC /Clarendon /Clarendon-Bold /Clarendon-Light /ClassicalGaramondBT-Bold /ClassicalGaramondBT-BoldItalic /ClassicalGaramondBT-Italic /ClassicalGaramondBT-Roman /CMB10 /CMBSY10 /CMBSY5 /CMBSY6 /CMBSY7 /CMBSY8 /CMBSY9 /CMBX10 /CMBX12 /CMBX5 /CMBX6 /CMBX7 /CMBX8 /CMBX9 /CMBXSL10 /CMBXTI10 /CMCSC10 /CMCSC8 /CMSS10 /CMSS12 /CMSS17 /CMSS8 /CMSS9 /CMSSBX10 /CMSSDC10 /CMSSI10 /CMSSI12 /CMSSI17 /CMSSI8 /CMSSI9 /CMSSQ8 /CMSSQI8 /CMSY10 /CMSY5 /CMTEX9 /ComicSansMS /ComicSansMS-Bold /ConcordeNova-Italic /ConcordeNova-ItalicExp /ConcordeNova-ItalicOsF /ConcordeNova-Medium /ConcordeNova-MediumExp /ConcordeNova-MediumSC /ConcordeNova-Regular /ConcordeNova-RegularExp /ConcordeNova-RegularSC /ConduitITC-Bold /ConduitITC-BoldItalic /ConduitITC-Light /ConduitITC-LightItalic /ConduitITC-Medium /ConduitITC-MediumItalic /CooperBlack /CooperBlack-Italic /CooperBT-Bold /CooperBT-BoldItalic /CooperBT-Light /CooperBT-LightItalic /CopperplateGothicBT-Bold /CopperplateGothicBT-BoldCond /CopperplateGothicBT-Heavy /CopperplateGothicBT-Roman /CopperplateGothicBT-RomanCond /Copperplate-ThirtyThreeBC /Copperplate-ThirtyTwoBC /Coronet-Regular /Courier /Courier-Bold /Courier-BoldOblique /Courier-Oblique /Critter /CS-Special-font /Delta-Bold /Delta-BoldItalic /Delta-Book /Delta-BookItalic /Delta-Light /Delta-LightItalic /Delta-Medium /Delta-MediumItalic /DextorD /DextorOutD /DINEngschrift /DINEngschrift-Alternate /DINMittelschrift /DINMittelschrift-Alternate /DINNeuzeitGrotesk-BoldCond /DINNeuzeitGrotesk-Light /Dom-CasItalic /DomCasual /DomCasual-Bold /Dom-CasualBT /Ehrhard-Italic /Ehrhard-Regular /EhrhardSemi-Italic /EhrhardtMT /EhrhardtMT-Italic /EhrhardtMT-SemiBold /EhrhardtMT-SemiBoldItalic /EhrharSemi /ElectraLH-Bold /ElectraLH-BoldCursive /ElectraLH-Cursive /ElectraLH-Regular /ElGreco /EnglischeSchT-Bold /EnglischeSchT-Regu /ErasContour /ErasITCbyBT-Bold /ErasITCbyBT-Book /ErasITCbyBT-Demi /ErasITCbyBT-Light /ErasITCbyBT-Medium /ErasITCbyBT-Ultra /Euclid /Euclid-Bold /Euclid-BoldItalic /EuclidExtra /EuclidExtra-Bold /EuclidFraktur /EuclidFraktur-Bold /Euclid-Italic /EuclidMathOne /EuclidMathOne-Bold /EuclidMathTwo /EuclidMathTwo-Bold /EuclidSymbol /EuclidSymbol-Bold /EuclidSymbol-BoldItalic /EuclidSymbol-Italic /EuroMono-Bold /EuroMono-BoldItalic /EuroMono-Italic /EuroMono-Regular /EuropeanPi-Four /EuropeanPi-One /EuropeanPi-Three /EuropeanPi-Two /EuroSans-Bold /EuroSans-BoldItalic /EuroSans-Italic /EuroSansITC-Black /EuroSansITC-BlackItalic /EuroSansITC-Bold /EuroSansITC-BoldItalic /EuroSansITC-Book /EuroSansITC-BookItalic /EuroSansITC-Medium /EuroSansITC-MediumItalic /EuroSans-Regular /EuroSerif-Bold /EuroSerif-BoldItalic /EuroSerif-Italic /EuroSerif-Regular /Eurostile /Eurostile-Bold /Eurostile-BoldExtendedTwo /Eurostile-ExtendedTwo /ExPonto-Regular /FairfieldLH-Bold /FairfieldLH-BoldItalic /FairfieldLH-BoldSC /FairfieldLH-CaptionBold /FairfieldLH-CaptionHeavy /FairfieldLH-CaptionLight /FairfieldLH-CaptionMedium /FairfieldLH-Heavy /FairfieldLH-HeavyItalic /FairfieldLH-HeavySC /FairfieldLH-Light /FairfieldLH-LightItalic /FairfieldLH-LightSC /FairfieldLH-Medium /FairfieldLH-MediumItalic /FairfieldLH-MediumSC /FairfieldLH-SwBoldItalicOsF /FairfieldLH-SwHeavyItalicOsF /FairfieldLH-SwLightItalicOsF /FairfieldLH-SwMediumItalicOsF /Fences /Fenice-Bold /Fenice-BoldOblique /Fenice-Light /Fenice-LightOblique /Fenice-Regular /Fenice-RegularOblique /Fenice-Ultra /Fenice-UltraOblique /FlashD-Ligh /Flood /FontanaNDEeOsF /FontanaNDEeOsF-Bold /FontanaNDEeOsF-BoldItalic /FontanaNDEeOsF-Light /FontanaNDEeOsF-Semibold /FormalScript421BT-Regular /Formata-Bold /Formata-MediumCondensed /ForteMT /FrakturBT-Regular /FranklinGothic-Book /FranklinGothic-BookItal /FranklinGothic-BookOblique /FranklinGothic-Condensed /FranklinGothic-Demi /FranklinGothic-DemiItal /FranklinGothic-DemiOblique /FranklinGothic-Heavy /FranklinGothic-HeavyItal /FranklinGothic-HeavyOblique /FranklinGothic-Medium /FranklinGothic-MediumItal /FranklinGothic-Roman /FreestyleScript /FrizQuadrataITCbyBT-Bold /FrizQuadrataITCbyBT-Roman /Frutiger-Black /Frutiger-BlackCn /Frutiger-BlackItalic /Frutiger-Bold /Frutiger-BoldCn /Frutiger-BoldItalic /Frutiger-Cn /Frutiger-ExtraBlackCn /Frutiger-Italic /Frutiger-Light /Frutiger-LightCn /Frutiger-LightItalic /Frutiger-Roman /Frutiger-UltraBlack /Futura /FuturaBlackBT-Regular /Futura-Bold /Futura-BoldOblique /Futura-Book /Futura-BookOblique /FuturaBT-Bold /FuturaBT-BoldCondensed /FuturaBT-BoldCondensedItalic /FuturaBT-BoldItalic /FuturaBT-Book /FuturaBT-BookItalic /FuturaBT-ExtraBlack /FuturaBT-ExtraBlackCondensed /FuturaBT-ExtraBlackCondItalic /FuturaBT-ExtraBlackItalic /FuturaBT-Heavy /FuturaBT-HeavyItalic /FuturaBT-Light /FuturaBT-LightCondensed /FuturaBT-LightItalic /FuturaBT-Medium /FuturaBT-MediumCondensed /FuturaBT-MediumItalic /Futura-CondensedLight /Futura-CondensedLightOblique /Futura-ExtraBold /Futura-ExtraBoldOblique /Futura-Heavy /Futura-HeavyOblique /Futura-Light /Futura-LightOblique /Futura-Oblique /Futura-Thin /Galliard-Black /Galliard-BlackItalic /Galliard-Bold /Galliard-BoldItalic /Galliard-Italic /GalliardITCbyBT-Bold /GalliardITCbyBT-BoldItalic /GalliardITCbyBT-Italic /GalliardITCbyBT-Roman /Galliard-Roman /Galliard-Ultra /Galliard-UltraItalic /Garamond-Antiqua /Garamond-BoldCondensed /Garamond-BoldCondensedItalic /Garamond-BookCondensed /Garamond-BookCondensedItalic /Garamond-Halbfett /GaramondITCbyBT-Bold /GaramondITCbyBT-BoldCondensed /GaramondITCbyBT-BoldCondItalic /GaramondITCbyBT-BoldItalic /GaramondITCbyBT-BoldNarrow /GaramondITCbyBT-BoldNarrowItal /GaramondITCbyBT-Book /GaramondITCbyBT-BookCondensed /GaramondITCbyBT-BookCondItalic /GaramondITCbyBT-BookItalic /GaramondITCbyBT-BookNarrow /GaramondITCbyBT-BookNarrowItal /GaramondITCbyBT-Light /GaramondITCbyBT-LightCondensed /GaramondITCbyBT-LightCondItalic /GaramondITCbyBT-LightItalic /GaramondITCbyBT-LightNarrow /GaramondITCbyBT-LightNarrowItal /GaramondITCbyBT-Ultra /GaramondITCbyBT-UltraCondensed /GaramondITCbyBT-UltraCondItalic /GaramondITCbyBT-UltraItalic /Garamond-Kursiv /Garamond-KursivHalbfett /Garamond-LightCondensed /Garamond-LightCondensedItalic /GaramondThree /GaramondThree-Bold /GaramondThree-BoldItalic /GaramondThree-Italic /GarthGraphic /GarthGraphic-Black /GarthGraphic-Bold /GarthGraphic-BoldCondensed /GarthGraphic-BoldItalic /GarthGraphic-Condensed /GarthGraphic-ExtraBold /GarthGraphic-Italic /Geometric231BT-HeavyC /GeometricSlab712BT-BoldA /GeometricSlab712BT-ExtraBoldA /GeometricSlab712BT-LightA /GeometricSlab712BT-LightItalicA /GeometricSlab712BT-MediumA /GeometricSlab712BT-MediumItalA /Giddyup /Giddyup-Thangs /GillSans /GillSans-Bold /GillSans-BoldCondensed /GillSans-BoldItalic /GillSans-Condensed /GillSans-ExtraBold /GillSans-Italic /GillSans-Light /GillSans-LightItalic /GillSans-UltraBold /Gill-Special /Giovanni-Bold /Giovanni-BoldItalic /Giovanni-Book /Giovanni-BookItalic /Goudy /Goudy-Bold /Goudy-BoldItalic /Goudy-BoldItalicOsF /Goudy-BoldOsF /Goudy-ExtraBold /Goudy-Heavyface /Goudy-HeavyfaceItalic /Goudy-Italic /Goudy-ItalicOsF /GoudyModernMT /GoudyModernMT-Italic /GoudyOldStyleBT-Bold /GoudyOldStyleBT-BoldItalic /GoudyOldStyleBT-ExtraBold /GoudyOldStyleBT-Italic /GoudyOldStyleBT-Roman /GoudySans-Black /GoudySans-BlackItalic /GoudySans-Bold /GoudySans-BoldItalic /GoudySans-Book /GoudySans-BookItalic /GoudySans-Medium /GoudySans-MediumItalic /Goudy-SC /GoudyTextMT /GoudyTextMT-Alternate /GoudyTextMT-Dfr /GoudyTextMT-LombardicCapitals /Helvetica /Helvetica-Black /Helvetica-BlackOblique /Helvetica-Black-SemiBold /Helvetica-Bold /Helvetica-BoldOblique /Helvetica-Compressed /Helvetica-Condensed /Helvetica-Condensed-Black /Helvetica-Condensed-BlackObl /Helvetica-Condensed-Bold /Helvetica-Condensed-BoldObl /Helvetica-Condensed-Light /Helvetica-Condensed-LightObl /Helvetica-Condensed-Oblique /Helvetica-ExtraCompressed /Helvetica-Light /Helvetica-LightOblique /Helvetica-Narrow /Helvetica-Narrow-Bold /Helvetica-Narrow-BoldOblique /Helvetica-Narrow-Oblique /HelveticaNeue-Black /HelveticaNeue-BlackCond /HelveticaNeue-BlackCondObl /HelveticaNeue-BlackExt /HelveticaNeue-BlackExtObl /HelveticaNeue-BlackItalic /HelveticaNeue-Bold /HelveticaNeue-BoldCond /HelveticaNeue-BoldCondObl /HelveticaNeue-BoldExt /HelveticaNeue-BoldExtObl /HelveticaNeue-BoldItalic /HelveticaNeue-Condensed /HelveticaNeue-CondensedObl /HelveticaNeue-ExtBlackCond /HelveticaNeue-ExtBlackCondObl /HelveticaNeue-Extended /HelveticaNeue-ExtendedObl /HelveticaNeue-Heavy /HelveticaNeue-HeavyCond /HelveticaNeue-HeavyCondObl /HelveticaNeue-HeavyItalic /HelveticaNeue-Italic /HelveticaNeue-Light /HelveticaNeue-LightCond /HelveticaNeue-LightCondObl /HelveticaNeue-LightExt /HelveticaNeue-LightExtObl /HelveticaNeue-LightItalic /HelveticaNeue-Medium /HelveticaNeue-MediumCond /HelveticaNeue-MediumCondObl /HelveticaNeue-MediumItalic /HelveticaNeue-Roman /HelveticaNeue-ThinCond /HelveticaNeue-ThinCondObl /HelveticaNeue-UltraLigCond /HelveticaNeue-UltraLigCondObl /HelveticaNeue-UltraLigExt /HelveticaNeue-UltraLigExtObl /HelveticaNeue-UltraLight /HelveticaNeue-UltraLightItal /Helvetica-Oblique /HelvLight /Humanist521BT-Bold /Humanist521BT-BoldCondensed /Humanist521BT-BoldItalic /Humanist521BT-ExtraBold /Humanist521BT-Italic /Humanist521BT-Light /Humanist521BT-LightItalic /Humanist521BT-Roman /Humanist521BT-RomanCondensed /Humanist521BT-UltraBold /Humanist521BT-XtraBoldCondensed /Humanist777BT-BlackB /Humanist777BT-BlackItalicB /Humanist777BT-BoldB /Humanist777BT-BoldItalicB /Humanist777BT-ItalicB /Humanist777BT-LightB /Humanist777BT-LightItalicB /Humanist777BT-RomanB /Imago-Book /Imago-BookItalic /Imago-ExtraBold /Imago-ExtraBoldItalic /Imago-Medium /Imago-MediumItalic /IPAExtras /IPAHighLow /IPAKiel /IPAKielSeven /IPAsans /JansonText-Bold /JansonText-BoldItalic /JansonText-Italic /JansonText-Roman /JansonText-RomanSC /JoannaMT /JoannaMT-Bold /JoannaMT-BoldItalic /JoannaMT-Italic /KeplMM-Or2 /KisBT-Italic /KisBT-Roman /KlangMT /Lapidary333BT-Black /Lapidary333BT-Bold /Lapidary333BT-BoldItalic /Lapidary333BT-Italic /Lapidary333BT-Roman /LASY10 /LASY5 /LASY6 /LASY7 /LASY8 /LASY9 /LASYB10 /LCIRCLE10 /LCIRCLEW10 /LCMSS8 /LCMSSB8 /LCMSSI8 /LDecorationPi-One /LDecorationPi-Two /LegacySans-Bold /LegacySans-BoldItalic /LegacySans-Book /LegacySans-BookItalic /LegacySans-Medium /LegacySans-MediumItalic /LegacySans-Ultra /LegacySerif-Bold /LegacySerif-BoldItalic /LegacySerif-Book /LegacySerif-BookItalic /LegacySerif-Medium /LegacySerif-MediumItalic /LegacySerif-Ultra /LetterGothic /LetterGothic-Bold /LetterGothic-BoldSlanted /LetterGothic-Slanted /LINE10 /LINEW10 /Lithos-Black /Lithos-Regular /LOGO10 /LOGO8 /LOGO9 /LOGOBF10 /LOGOSL10 /LOMD-Normal /LubalinGraph-Book /LubalinGraph-BookOblique /LubalinGraph-Demi /LubalinGraph-DemiOblique /LucidaHandwritingItalic /LucidaMath-Symbol /LydianBT-Bold /LydianBT-BoldItalic /LydianBT-Italic /LydianBT-Roman /LydianCursiveBT-Regular /Marigold /MathematicalPi-Five /MathematicalPi-Four /MathematicalPi-One /MathematicalPi-Six /MathematicalPi-Three /MathematicalPi-Two /Melior /Melior-Bold /Melior-BoldItalic /Melior-Italic /Memphis-Bold /Memphis-BoldItalic /Memphis-ExtraBold /Memphis-Light /Memphis-LightItalic /Memphis-Medium /Memphis-MediumItalic /MercuriusCT-Black /MercuriusCT-BlackItalic /MercuriusCT-Light /MercuriusCT-LightItalic /MercuriusCT-Medium /MercuriusCT-MediumItalic /MercuriusMT-BoldScript /Meridien-Medium /Meridien-MediumItalic /Meridien-Roman /MexicanBorders /Minion-Black /Minion-Bold /Minion-BoldCondensed /Minion-BoldCondensedItalic /Minion-BoldItalic /Minion-Condensed /Minion-CondensedItalic /Minion-DisplayItalic /Minion-DisplayRegular /Minion-Italic /Minion-Ornaments /Minion-Regular /Minion-Semibold /Minion-SemiboldItalic /MonaLisa-Recut /MonolineScriptMT /MrsEavesAllPetiteCaps /MrsEavesAllSmallCaps /MrsEavesBold /MrsEavesFractions /MrsEavesItalic /MrsEavesPetiteCaps /MrsEavesRoman /MrsEavesRomanLining /MrsEavesSmallCaps /MSAM10 /MSAM10A /MSAM5 /MSAM6 /MSAM7 /MSAM8 /MSAM9 /MSBM10 /MSBM10A /MSBM5 /MSBM6 /MSBM7 /MSBM8 /MSBM9 /MTEX /MTEXB /MTEXH /MT-Extra /MTGU /MTGUB /MTMI /MTMIB /MTMIH /MTMS /MTMSB /MTMUB /MTMUH /MTSY /MTSYB /MTSYH /MTSYN /Myriad-Bold /Myriad-BoldItalic /Myriad-Italic /Myriad-Roman /Myriad-Tilt /NeuzeitS-Book /NeuzeitS-BookHeavy /NewBaskerville-Bold /NewBaskerville-BoldItalic /NewBaskerville-Italic /NewBaskervilleITCbyBT-Bold /NewBaskervilleITCbyBT-BoldItal /NewBaskervilleITCbyBT-Italic /NewBaskervilleITCbyBT-Roman /NewBaskerville-Roman /NewBerolinaMT /NewCaledonia /NewCaledonia-Black /NewCaledonia-BlackItalic /NewCaledonia-Bold /NewCaledonia-BoldItalic /NewCaledonia-Italic /NewCaledonia-SemiBold /NewCaledonia-SemiBoldItalic /NewCenturySchlbk-Bold /NewCenturySchlbk-BoldItalic /NewCenturySchlbk-Italic /NewCenturySchlbk-Roman /NewsGothicBT-Bold /NewsGothicBT-BoldCondensed /NewsGothicBT-BoldCondItalic /NewsGothicBT-BoldExtraCondensed /NewsGothicBT-BoldItalic /NewsGothicBT-Demi /NewsGothicBT-DemiItalic /NewsGothicBT-ExtraCondensed /NewsGothicBT-Italic /NewsGothicBT-ItalicCondensed /NewsGothicBT-Light /NewsGothicBT-LightItalic /NewsGothicBT-Roman /NewsGothicBT-RomanCondensed /New-Symbol /Nueva-BoldExtended /Nueva-Roman /NuptialScript /OceanSansMM /OceanSansMM-It /OfficinaSans-Bold /OfficinaSans-BoldItalic /OfficinaSans-Book /OfficinaSans-BookItalic /OfficinaSerif-Bold /OfficinaSerif-BoldItalic /OfficinaSerif-Book /OfficinaSerif-BookItalic /Optima /Optima-Bold /Optima-BoldItalic /Optima-ExtraBlack /Optima-ExtraBlackItalic /Optima-Italic /OttaIA /Otta-wa /Ottawa-BoldA /OttawaPSMT /Oxford /PalaceScriptMT /PalaceScriptMT-SemiBold /Palatino-Bold /Palatino-BoldItalic /Palatino-Italic /Palatino-Roman /Perpetua /Perpetua-Bold /Perpetua-BoldItalic /Perpetua-Italic /PhotinaMT /PhotinaMT-Bold /PhotinaMT-BoldItalic /PhotinaMT-Italic /PhotinaMT-SemiBold /PhotinaMT-SemiBoldItalic /PhotinaMT-UltraBold /PhotinaMT-UltraBoldItalic /Plantin /Plantin-Bold /Plantin-BoldItalic /Plantin-Italic /Plantin-Light /Plantin-LightItalic /Plantin-Semibold /Plantin-SemiboldItalic /Poetica-ChanceryI /PopplLaudatio-Italic /PopplLaudatio-Medium /PopplLaudatio-MediumItalic /PopplLaudatio-Regular /ProseAntique-Bold /ProseAntique-Normal /QuaySansEF-Black /QuaySansEF-BlackItalic /QuaySansEF-Book /QuaySansEF-BookItalic /QuaySansEF-Medium /QuaySansEF-MediumItalic /Quorum-Black /Quorum-Bold /Quorum-Book /Quorum-Light /Quorum-Medium /Revival565BT-Bold /Revival565BT-BoldItalic /Revival565BT-Italic /Revival565BT-Roman /Ribbon131BT-Bold /Ribbon131BT-Regular /RMTMI /Rockwell /Rockwell-Bold /Rockwell-BoldCondensed /Rockwell-BoldItalic /Rockwell-Condensed /Rockwell-ExtraBold /Rockwell-Italic /Rockwell-Light /Rockwell-LightItalic /RussellSquare /RussellSquare-Oblique /RuzickaFreehandLH-Bold /RuzickaFreehandLH-BoldSC /RuzickaFreehandLH-Roman /RuzickaFreehandLH-RomanSC /Sabon-Bold /Sabon-BoldItalic /Sabon-Italic /Sabon-Roman /Sanvito-Light /SanvitoMM /Sanvito-Roman /ScotchRomanMT /ScotchRomanMT-Italic /Semitica /Semitica-Italic /SerifGothic /SerifGothic-Bold /SignaCondColumn-Light /SignaCond-Light /SignaCond-LightExpert /SIVAMATH /Siva-Special /SMS-SPELA /Souvenir-Demi /Souvenir-DemiItalic /SouvenirITCbyBT-Demi /SouvenirITCbyBT-DemiItalic /SouvenirITCbyBT-Light /SouvenirITCbyBT-LightItalic /Souvenir-Light /Souvenir-LightItalic /SpecialAA /Special-Gali /SpringLP /SpringLP-Light /Sp-Sym /SpumoniLP /StempelGaramond-Bold /StempelGaramond-BoldItalic /StempelGaramond-Italic /StempelGaramond-Roman /StoneSans /StoneSans-Bold /StoneSans-BoldItalic /StoneSans-Italic /StoneSans-PhoneticAlternate /StoneSans-PhoneticIPA /StoneSans-Semibold /StoneSans-SemiboldItalic /StoneSerif /StoneSerif-Italic /StoneSerif-PhoneticAlternate /StoneSerif-PhoneticIPA /StoneSerif-Semibold /StoneSerif-SemiboldItalic /Swiss721BT-Black /Swiss721BT-BlackCondensed /Swiss721BT-BlackCondensedItalic /Swiss721BT-BlackExtended /Swiss721BT-BlackItalic /Swiss721BT-BlackOutline /Swiss721BT-BlackRounded /Swiss721BT-Bold /Swiss721BT-BoldCondensed /Swiss721BT-BoldCondensedItalic /Swiss721BT-BoldCondensedOutline /Swiss721BT-BoldExtended /Swiss721BT-BoldItalic /Swiss721BT-BoldOutline /Swiss721BT-BoldRounded /Swiss721BT-Heavy /Swiss721BT-HeavyItalic /Swiss721BT-Italic /Swiss721BT-ItalicCondensed /Swiss721BT-Light /Swiss721BT-LightCondensed /Swiss721BT-LightCondensedItalic /Swiss721BT-LightExtended /Swiss721BT-LightItalic /Swiss721BT-Medium /Swiss721BT-MediumItalic /Swiss721BT-Roman /Swiss721BT-RomanCondensed /Swiss721BT-RomanExtended /Swiss721BT-Thin /Swiss721BT-ThinItalic /Symbol /Tekton /Times-Bold /Times-BoldA /Times-BoldItalic /Times-BoldOblique /Times-Italic /Times-NewRoman /Times-NewRomanBold /TimesNewRomanMT-BoldCond /TimesNewRomanMT-Cond /TimesNewRomanMT-CondItalic /TimesNewRomanPS-BoldItalicMT /TimesNewRomanPS-BoldMT /TimesNewRomanPS-ItalicMT /TimesNewRomanPSMT /Times-Oblique /Times-PhoneticAlternate /Times-PhoneticIPA /Times-Roman /Times-Sc /Times-SCB /Times-special /TradeGothic /TradeGothic-Bold /TradeGothic-BoldCondTwenty /TradeGothic-BoldCondTwentyObl /TradeGothic-BoldOblique /TradeGothic-BoldTwo /TradeGothic-BoldTwoOblique /TradeGothic-CondEighteen /TradeGothic-CondEighteenObl /TradeGothicLH-BoldExtended /TradeGothicLH-Extended /TradeGothic-Light /TradeGothic-LightOblique /TradeGothic-Oblique /Trajan-Bold /Trajan-Regular /Univers /Universal-NewswithCommPi /Univers-Black /Univers-BlackExt /Univers-BlackExtObl /Univers-BlackOblique /Univers-Bold /Univers-BoldExt /Univers-BoldExtObl /Univers-BoldItalic /Univers-BoldOblique /Univers-Condensed /Univers-CondensedBold /Univers-CondensedBoldOblique /Univers-CondensedOblique /Univers-Extended /Univers-ExtendedObl /Univers-ExtraBlack /Univers-ExtraBlackExt /Univers-ExtraBlackExtObl /Univers-ExtraBlackObl /Univers-Italic /Univers-Light /Univers-LightOblique /Univers-LightUltraCondensed /Univers-Oblique /Univers-ThinUltraCondensed /Univers-UltraCondensed /Utopia-Regular /VAGRounded-Black /VAGRounded-Bold /VAGRounded-Light /VAGRounded-Thin /Viva-BoldExtraExtended /Viva-Regular /Weidemann-Black /Weidemann-BlackItalic /Weidemann-Bold /Weidemann-BoldItalic /Weidemann-Book /Weidemann-BookItalic /Weidemann-Medium /Weidemann-MediumItalic /WindsorBT-Elongated /WindsorBT-Light /WindsorBT-LightCondensed /WindsorBT-Roman /Wingdings-Regular /WNCYB10 /WNCYI10 /WNCYR10 /WNCYSC10 /WNCYSS10 /WoodtypeOrnaments-One /WoodtypeOrnaments-Two /ZapfCalligraphic801BT-Bold /ZapfCalligraphic801BT-BoldItal /ZapfCalligraphic801BT-Italic /ZapfCalligraphic801BT-Roman /ZapfChanceryITCbyBT-Bold /ZapfChanceryITCbyBT-Demi /ZapfChanceryITCbyBT-Medium /ZapfChanceryITCbyBT-MediumItal /ZapfChancery-MediumItalic /ZapfDingbats /ZapfDingbatsITCbyBT-Regular /ZapfElliptical711BT-Bold /ZapfElliptical711BT-BoldItalic /ZapfElliptical711BT-Italic /ZapfElliptical711BT-Roman /ZapfHumanist601BT-Bold /ZapfHumanist601BT-BoldItalic /ZapfHumanist601BT-Demi /ZapfHumanist601BT-DemiItalic /ZapfHumanist601BT-Italic /ZapfHumanist601BT-Roman /ZapfHumanist601BT-Ultra /ZapfHumanist601BT-UltraItalic /ZiptyDo /ZurichBT-Black /ZurichBT-BlackExtended /ZurichBT-BlackItalic /ZurichBT-Bold /ZurichBT-BoldCondensed /ZurichBT-BoldCondensedItalic /ZurichBT-BoldExtended /ZurichBT-BoldExtraCondensed /ZurichBT-BoldItalic /ZurichBT-ExtraBlack /ZurichBT-ExtraCondensed /ZurichBT-Italic /ZurichBT-ItalicCondensed /ZurichBT-Light /ZurichBT-LightCondensed /ZurichBT-LightCondensedItalic /ZurichBT-LightExtraCondensed /ZurichBT-LightItalic /ZurichBT-Roman /ZurichBT-RomanCondensed /ZurichBT-RomanExtended /ZurichBT-UltraBlackExtended ] /NeverEmbed [ true ] /AntiAliasColorImages false /DownsampleColorImages true /ColorImageDownsampleType /Bicubic /ColorImageResolution 300 /ColorImageDepth -1 /ColorImageDownsampleThreshold 1.50000 /EncodeColorImages true /ColorImageFilter /DCTEncode /AutoFilterColorImages true /ColorImageAutoFilterStrategy /JPEG /ColorACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /ColorImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000ColorACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000ColorImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasGrayImages false /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /GrayImageDict << /QFactor 0.15 /HSamples [1 1 1 1] /VSamples [1 1 1 1] >> /JPEG2000GrayACSImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /JPEG2000GrayImageDict << /TileWidth 256 /TileHeight 256 /Quality 30 >> /AntiAliasMonoImages false /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict << /K -1 >> /AllowPSXObjects false /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox false /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile (U.S. Web Coated \050SWOP\051 v2) /PDFXOutputCondition () /PDFXRegistryName (http://www.color.org) /PDFXTrapped /Unknown /SyntheticBoldness 1.000000 /Description << /FRA <FEFF004f007000740069006f006e00730020007000650072006d0065007400740061006e007400200064006500200063007200e900650072002000640065007300200064006f00630075006d0065006e00740073002000500044004600200064006f007400e900730020006400270075006e00650020007200e90073006f006c007500740069006f006e002000e9006c0065007600e9006500200070006f0075007200200075006e00650020007100750061006c0069007400e90020006400270069006d007000720065007300730069006f006e00200061006d00e9006c0069006f007200e90065002e00200049006c002000650073007400200070006f0073007300690062006c0065002000640027006f00750076007200690072002000630065007300200064006f00630075006d0065006e007400730020005000440046002000640061006e00730020004100630072006f0062006100740020006500740020005200650061006400650072002c002000760065007200730069006f006e002000200035002e00300020006f007500200075006c007400e9007200690065007500720065002e> /JPN <FEFF3053306e8a2d5b9a306f30019ad889e350cf5ea6753b50cf3092542b308000200050004400460020658766f830924f5c62103059308b3068304d306b4f7f75283057307e30593002537052376642306e753b8cea3092670059279650306b4fdd306430533068304c3067304d307e305930023053306e8a2d5b9a30674f5c62103057305f00200050004400460020658766f8306f0020004100630072006f0062006100740020304a30883073002000520065006100640065007200200035002e003000204ee5964d30678868793a3067304d307e30593002> /DEU <FEFF00560065007200770065006e00640065006e0020005300690065002000640069006500730065002000450069006e007300740065006c006c0075006e00670065006e0020007a0075006d002000450072007300740065006c006c0065006e00200076006f006e0020005000440046002d0044006f006b0075006d0065006e00740065006e0020006d00690074002000650069006e006500720020006800f60068006500720065006e002000420069006c0064006100750066006c00f600730075006e0067002c00200075006d002000650069006e0065002000760065007200620065007300730065007200740065002000420069006c0064007100750061006c0069007400e400740020007a0075002000650072007a00690065006c0065006e002e00200044006900650020005000440046002d0044006f006b0075006d0065006e007400650020006b00f6006e006e0065006e0020006d006900740020004100630072006f0062006100740020006f0064006500720020006d00690074002000640065006d002000520065006100640065007200200035002e003000200075006e00640020006800f600680065007200200067006500f600660066006e00650074002000770065007200640065006e002e> /PTB <FEFF005500740069006c0069007a006500200065007300740061007300200063006f006e00660069006700750072006100e700f5006500730020007000610072006100200063007200690061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006d00200075006d00610020007200650073006f006c007500e700e3006f00200064006500200069006d006100670065006d0020007300750070006500720069006f0072002000700061007200610020006f006200740065007200200075006d00610020007100750061006c0069006400610064006500200064006500200069006d0070007200650073007300e3006f0020006d0065006c0068006f0072002e0020004f007300200064006f00630075006d0065006e0074006f0073002000500044004600200070006f00640065006d0020007300650072002000610062006500720074006f007300200063006f006d0020006f0020004100630072006f006200610074002c002000520065006100640065007200200035002e0030002000650020007300750070006500720069006f0072002e> /DAN <FEFF004200720075006700200064006900730073006500200069006e0064007300740069006c006c0069006e006700650072002000740069006c0020006100740020006f0070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f8006a006500720065002000620069006c006c00650064006f0070006c00f80073006e0069006e006700200066006f00720020006100740020006600e50020006200650064007200650020007500640073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e0074006500720020006b0061006e002000e50062006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f00670020006e0079006500720065002e> /NLD <FEFF004700650062007200750069006b002000640065007a006500200069006e007300740065006c006c0069006e00670065006e0020006f006d0020005000440046002d0064006f00630075006d0065006e00740065006e0020007400650020006d0061006b0065006e0020006d00650074002000650065006e00200068006f0067006500720065002000610066006200650065006c00640069006e00670073007200650073006f006c007500740069006500200076006f006f0072002000650065006e0020006200650074006500720065002000610066006400720075006b006b00770061006c00690074006500690074002e0020004400650020005000440046002d0064006f00630075006d0065006e00740065006e0020006b0075006e006e0065006e00200077006f007200640065006e002000670065006f00700065006e00640020006d006500740020004100630072006f00620061007400200065006e002000520065006100640065007200200035002e003000200065006e00200068006f006700650072002e> /ESP <FEFF0055007300650020006500730074006100730020006f007000630069006f006e006500730020007000610072006100200063007200650061007200200064006f00630075006d0065006e0074006f0073002000500044004600200063006f006e0020006d00610079006f00720020007200650073006f006c00750063006900f3006e00200064006500200069006d006100670065006e00200070006100720061002000610075006d0065006e0074006100720020006c0061002000630061006c006900640061006400200061006c00200069006d007000720069006d00690072002e0020004c006f007300200064006f00630075006d0065006e0074006f00730020005000440046002000730065002000700075006500640065006e00200061006200720069007200200063006f006e0020004100630072006f00620061007400200079002000520065006100640065007200200035002e003000200079002000760065007200730069006f006e0065007300200070006f00730074006500720069006f007200650073002e> /SUO <FEFF004e00e4006900640065006e002000610073006500740075007300740065006e0020006100760075006c006c006100200076006f0069006400610061006e0020006c0075006f006400610020005000440046002d0061007300690061006b00690072006a006f006a0061002c0020006a006f006900640065006e002000740075006c006f0073007400750073006c00610061007400750020006f006e0020006b006f0072006b006500610020006a00610020006b007500760061006e0020007400610072006b006b007500750073002000730075007500720069002e0020005000440046002d0061007300690061006b00690072006a0061007400200076006f0069006400610061006e0020006100760061007400610020004100630072006f006200610074002d0020006a00610020004100630072006f006200610074002000520065006100640065007200200035002e00300020002d006f0068006a0065006c006d0061006c006c0061002000740061006900200075007500640065006d006d0061006c006c0061002000760065007200730069006f006c006c0061002e> /ITA <FEFF00550073006100720065002000710075006500730074006500200069006d0070006f007300740061007a0069006f006e00690020007000650072002000630072006500610072006500200064006f00630075006d0065006e00740069002000500044004600200063006f006e00200075006e00610020007200690073006f006c0075007a0069006f006e00650020006d0061006700670069006f00720065002000700065007200200075006e00610020007100750061006c0069007400e00020006400690020007300740061006d007000610020006d00690067006c0069006f00720065002e0020004900200064006f00630075006d0065006e00740069002000500044004600200070006f00730073006f006e006f0020006500730073006500720065002000610070006500720074006900200063006f006e0020004100630072006f00620061007400200065002000520065006100640065007200200035002e003000200065002000760065007200730069006f006e006900200073007500630063006500730073006900760065002e> /NOR <FEFF004200720075006b00200064006900730073006500200069006e006e007300740069006c006c0069006e00670065006e0065002000740069006c002000e50020006f00700070007200650074007400650020005000440046002d0064006f006b0075006d0065006e0074006500720020006d006500640020006800f80079006500720065002000620069006c00640065006f00700070006c00f80073006e0069006e006700200066006f00720020006200650064007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e00650020006b0061006e002000e50070006e006500730020006d006500640020004100630072006f0062006100740020006f0067002000520065006100640065007200200035002e00300020006f0067002000730065006e006500720065002e> /SVE <FEFF0041006e007600e4006e00640020006400650020006800e4007200200069006e0073007400e4006c006c006e0069006e006700610072006e00610020006e00e40072002000640075002000760069006c006c00200073006b0061007000610020005000440046002d0064006f006b0075006d0065006e00740020006d006500640020006800f6006700720065002000620069006c0064007500700070006c00f60073006e0069006e00670020006f006300680020006400e40072006d006500640020006600e50020006200e400740074007200650020007500740073006b00720069006600740073006b00760061006c0069007400650074002e0020005000440046002d0064006f006b0075006d0065006e00740065006e0020006b0061006e002000f600700070006e006100730020006d006500640020004100630072006f0062006100740020006f00630068002000520065006100640065007200200035002e003000200065006c006c00650072002000730065006e006100720065002e> /ENU <FEFF005500730065002000740068006500730065002000730065007400740069006e0067007300200066006f00720020006300720065006100740069006e00670020005000440046002000660069006c0065007300200066006f00720020007300750062006d0069007300730069006f006e00200074006f002000540068006500200053006800650072006900640061006e002000500072006500730073002e002000540068006500730065002000730065007400740069006e0067007300200063006f006e006600690067007500720065006400200066006f00720020004100630072006f006200610074002000760036002e0030002000300038002f00300036002f00300033002e> >> >> setdistillerparams << /HWResolution [2400 2400] /PageSize [612.000 792.000] >> setpagedevice
HRE7271/Week 6/Learning through Immersive Virtual Environments- An Organizational Context.pdf
1
2
Learning through Immersive Virtual Environments: An Organizational Context
Introduction
Albert Einstein once lamented that technology has exceeded our humanity. Even though this
sounds farfetched, evidence suggests technology has become a part of our daily lives. Immersive
Virtual Learning (IVL) has revolutionized how learning is conducted, from educational
institutions, to organizations, and more specifically within health care operations and to different
areas of environmental sustainability. With no consistent agreement on the term IVL we have
developed, for the purpose of this chapter a working definition. IVL is the use of advanced
computer interfaces to interact, in a simulated real-world environment while conveying
knowledge to users. Ultimately, the interaction provided by immersive virtual platforms creates
active participation and engagement which enhances learning (Ndinguri, Machtmes, Hatala, &
Coco, 2012). Consider the evolution of the smart phone from its early stages of development in
the 90’s with simple applications such as a calculator and email, to today with more sophisticated
apps that can affect learning, personal lives, even an individual’s health. These technologies have
transformed economies, cultures and how people communicate and do business (Merriam,
Caffarella, & Baumgartner, 2007). Learning is one of the many areas in which IVL plays a major
role. As technology changes, so has the shift in which learning is presented to the workforce.
Today, with the emphasis on a knowledge-based economy, how human capital is developed ties
into the strategic aspects of organizational success (Chinowsky & Carrillo, 2007; Merriam et al.,
2007; Sveiby, 2001). Specifically, the increase in IVL environments has eliminated borders and
created a global learning environment that improves resource utilization and flexibility for
information access (Symons & Stenzel, 2007). Unlike other forms of learning technology, IVL
has been credited with its ability to offer the user relatable interfaces and a simulated
3
telepresence learning environment (Nah, Eschenbrenner, & DeWester, 2011). Research
demonstrates that by using game-like devices in a simulated environment it provides resources
for increasing the knowledge transfer of the material for the learners in training (Coco, 2011). In
this case, as each participant interacts with IVL their repetitive behavior of actions through the
technology helps to signify the mastery of content (Coco, 2011). In addition, studies indicate that
participants who have trained through the IVL environment were able to transfer their learning to
a real world environmental scenario (Coco, 2011). Computer skills among participants are not
factors deterring use of the IVL environment because trainees can use game like controllers that
mimic simulated scenarios. It is documented that IVLs, through simulated realism, helps create
different levels of complexity for better conceptualization of ideas, on-line measurement during
training, control participant exposure to the environment, and type of landmarks and their
positions (Coco, 2011). Therefore, IVL environments are capable of recording and measuring
every move of the trainee and highlight areas of weakness. For example, in the health care
industry physicians immersed in an IVL environment have used the technology to understand
how to conduct debridement of a gunshot wound, act as simulator for temporal bone dissection,
orthoscopic knee surgery and provide a simulated training environment for the palpation of
subsurface breast tumors (Mantovani, Castelnuovo, Gaggioli & Riva, 2003). In addition, the
technology is used in simulation training especially for sensitive or dangerous areas such as the
military, nuclear plants, accident preparations and chemical testing (Coco, 2011; Johnson &
Levine, 2008).
However, organizations still have a long way to go before fully understanding how learning
and knowledge transfer occurs when new learning technology is used (Geng & Whinston, 2009).
One particular study examined IVL by developing a conceptual roadmap for organizations. The
4
roadmap highlights the technological processes and levels that organizations’ technology based
training must consider when using IVL. The model is not intended to provide a simplistic input-
output process but rather highlight the various levels that capture training in an organization
using established IVL environments. Therefore, the immersive virtual roadmap is intended to
provide IVL established organizations a step-based structure that guides learning using an
immersive virtual platform. Leveraging previous research and the work done from the
collaborative effort among state agencies, this chapter will provide practical ways in which an
organization can incorporate the proposed roadmap IVL levels.
Methodology
The purpose of this chapter was to develop a guiding conceptual roadmap that explores
different possible levels in which organizations’ learning takes place in an IVL-based
environment. The questions explored in this chapter are:
a) What is IVL?
b) What are the contributions of IVL to organizations?
c) What is the importance of IVL to the workforce?
d) How is IVL being applied in knowledge transfer in workforce organizational context?
e) What would a conceptual roadmap on IVL in the organization look like?
f) What is the future of IVL for the organization?
Immersive Virtual Learning and its Application
Learning as a global concept is strongly supported by world bodies such as the
Organization of Economic Cooperation and Development, European Union and United Nations
together with policy makers and governments in providing solutions for coping with the
5
challenges of globalization and the transition into knowledge-based economies (Uggla, 2008).
Therefore, it is essential that the learning activities individuals participate in throughout their
lives improve their skills, increase competence in a particular area and add knowledge driven by
personal, professional or societal factors (Aspin & Chapman, 2000; Griffin, 1999). With the need
for organizations to respond to this new environment and the transformation into a knowledge-
based economy, ongoing skill development is required of the workforce. To achieve this,
organizational learning platforms have to be flexible and engaging to meet the various learner
needs at their level of competence (Koper & Tattersall, 2004). Today, the workforce is faced
with dynamic learning environments that influence the way they learn in their work settings. As
learning platforms change, the training setting that exists within the organization needs to change
and adapt with it. Therefore, the emerging trend in education today is increasingly moving
towards learner-centered approaches (Derouin, Fritzsche, & Salas, 2004). Many workforce
centers of learning, like colleges and universities, have used this concept to describe one of their
primary goals of their institutions as they prepare their students to be life-long, self-directed
learners (Merriam et al., 2007).
What has changed?
The rapid changes in methods of learning have influenced adaptive learning to ensure
individuals maintain relevance in a changing global society. Learning has moved from a one-way
conduit of information to an exchange of ideas among and between various groups of people.
This has become even more evident as we have made the transition from an industrial-based
society to an information-based one. The rapidly changing global societies and work
environments require continuous learning, and a nontraditional student workforce are the new
majority, pursuing education for career development, job security, upward mobility, re-careering,
6
and other professional and personal reasons (Eastmond, 1998). IVL’s can be an effective and
engaging technology that can sustain skill building through interaction with a wide range of
learners who may be utilizing the self-directed learning concept. IVL has also become an area
that influences how the workforce learns individually, in addition to how they gather their
knowledge, which goes above and beyond more traditional methods (LaBrosse, 2008). IVL also
creates a shared interactive learning environment that enhances communication properties into
instructional learning environments backed by advances in technological applications (Hayes,
2006). It also fundamentally transcends borders and reaches everyone, everywhere, thus, creating
a rich environment that fosters lifelong learning and increases demand among the learners
(Koper & Tattersall, 2004)
It is also evident that multi-user virtual environments, a combination of virtual reality and
text-based charts (Ripamonti & Peraboni, 2010) assist IVLs by providing the opportunity for
individuals to connect in an intimate social context. In many regards, this is the most
sophisticated example of the ongoing transition from simple provision of content and online
tools towards platforms that facilitate interpersonal relationships (White & Cornu, 2010). The
workforce has the tendency to learn on a need-to-know basis (Wlodkowski, 2008). For this
reason, IVL fills this gap as a tool that connects and gives freedom to learn any content at any
place and time. Information and communication technology offers a “new landscape of learning”
unfettered by the enclosed spaces of traditional institutions. Educational technology can be seen
as a journey towards the ‘de-localization of learning’ where individuals are free to learn
whenever and wherever they choose (Bentley, 1998). This creates a sense of virtual inclusion
where inclusion can be said to be among the fundamental aspects of motivating adults
(Wlodkowski, 2008).
7
Services such as advanced distributed learning (making learning available at anytime and
anywhere in the world) allow worldwide access of learning materials (Fletcher, Tobias, &
Wisher, 2007). This is important as it promotes an organizations self-directed learning/training
process, which fits into a workers demanding schedule that has become more evident in the
business world today. This process focuses on the use of learning methods as a way to make
instructional materials readily accessible. Distributed learning is one of the many resources that
focus on the use of learning methods as a way to make instructional materials digital, sharable,
and reusable entities that can be used for learning and be available to learners anytime and
anywhere via the World Wide Web (Fletcher et al., 2007).
Why Organizational IVL?
With these profound changes occurring in the immersive virtual realm, the question
should be asked as to why IVL is important in organizations? The social aspect of the human
condition creates a sense of belonging and involvement that elevates the motivational level of the
adult workforce (Wlodkowski, 2008). This motivation plays a role in how the adult workforce
uses the knowledge and skills acquired in learning activities (Egan, Yang & Bartlett, 2004; Noe
& Schmitt, 1986). The deep social value for responsibility is why competence and being
effective at what one values looms so large and consistently as a force for learning among adults
(Wlodkowski, 2008). With the potential economic benefits of up-skilling the workforce, IVL
based training is also being enthusiastically promoted as a new way of combating social
exclusion (Gorard, Selwyn & Williams, 2000). The changing immersive virtual technological
aspects that are affecting different organizational and societal structures act as evidence of the
need for organizations to embrace the virtual learning methods of workforce enrichment (Bennett
& Bierema, 2010). Understanding the influence on their employees and their responses in the
8
immersive virtual world and how they adapt to a working environment is an area that requires
further exploration.
Today, organizations have brought the role of IVL technology into the core of their
operations. The role of organization IVL is evident through virtual recruiting, virtual on-boarding
and virtual training and education (Vidal, 2011). Virtual recruiting entails holding a job fair
featuring webcasts, webinars and video interviews; virtual on-boarding engages people globally
to critical information or meetings; and virtual training and education entails imparting
knowledge to the workforce using IVL platforms (Vidal, 2011). Therefore, the aspect of
knowledge transfer and the medium of transfer play a central role in the future developmental
aspects of organization learning. With the benefits offered by IVL as a medium of knowledge
transfer, it is important to understand how to take advantage of the skills imparted and
measurement abilities that are accessible through IVL environments. Research conducted by
Coco (2011) on analyzing knowledge transferability and the impact of IVL with certain
demographics exemplifies the measurement and learning capabilities of the IVL environment
(Coco, 2011; Jansen-Osmann & Wiedenbauer, 2004; Peruch, Belingrad & Thinus-Blanc, 2000).
Research Study (Coco, 2011)
Through a collaborative effort between the Louisiana Department of Transportation and
Development’s Louisiana Transportation Research Center (LTRC), the Louisiana State
University School of Human Resource Education and Workforce Development (SHREWD), and
the University of Louisiana at Lafayette’s Louisiana Immersive Technologies Enterprise (LITE)
tested the use of an IVL Environment (IVLE) on simulating real-world highway work zones
(Coco, 2011). There were 305 highway workers from government and private sources randomly
selected to participate in the research. Through a process of blind selection, half the sample
9
participated in the traditional class (control) and half participated in a blended delivery method
class which incorporated the IVL (experimental). A pretest and post-test were administered in
both the control and treatment groups. The purpose of the pretest was to reveal a pre-training
baseline understanding of flagger techniques and abstract concepts. The post-test was designed
to evaluate learning transfer. In addition to this line of measurement, data were recorded from the
IVL using built-in software which collected tracking data of the participants. The aim of
collecting the tracking data was to capture information on the number of button selections that
each individual used to complete a task (a task consisted of different activities that were part of
the regulations that participants were required to utilize), the time it took for them to have their
avatar images on the right placement area and the number of repetitive attempts before
completion of the task (Coco, 2011). Through the post-test analysis of both quantitative and
qualitative results the study concluded that participants who participated in the IVL class were
more engaged in the learning process than they had been in traditional style classes. In addition
IVL learners (experimental) displayed progressive improvement in the application of the
flagging procedures while in the Immersive Virtual Learning Environment (IVLE), as denoted in
the IVLE telemetry data.
This study highlights the capabilities accessible to user organizations of immersive
virtual environments while measuring their ability to transfer the newly acquired knowledge
back to their employees. IVLs, through simulated realism, can be used to create different levels
of complexity for better conceptualization of ideas, on-line measurement during training, control
participant exposure to the environment and the type of landmarks, including their position
(Peruch et al., 2000). Furthermore, simulation environments created through IVLs assist the
workforce in acquiring knowledge through spatial distance, which can be utilized on the job
10
(Thorndyke & Hayes-Roth, 1982). This newly acquired knowledge is useful in a workforce
setting in that it allows for practical utilization in the real world due to its precise and accurate
measurements that emanate from the sophisticated software, as shown in the LTRC example
above. IVL further facilitates effective simulation of virtual teams that not only allow
communication and information sharing across continental borders but also creates business
opportunities in terms of key operations, such as product development, strategic analysis, and
customer service (Majchrzak, Rice, Malhotra, King, & Ba, 2000). An increase in the number of
immersive virtual teams is becoming a common occurrence in organizations today (Martins,
Gilson & Maynard, 2004). Virtual teams have excelled using IVL platforms such as second life.
Companies such as Air France-KLM have embraced second life by creating virtual executive
meeting places for their shareholders as well as executives. These collaborative environments,
according to Bailenson et al. (2008), can assist in tracking nonverbal cues (i.e., movements,
gestures, expressions and sounds through simulated images hence enhancing training sessions).
As the knowledge-based business environment takes root and organizations interconnect and
create synergies globally, addressing challenges of group training and learning in major
organizations that are dispersed and time constrained can be achieved through immersive virtual
technologies (Townsend, DeMarie, & Hendrickson, 1998). The increase in dispersed training
highlights the need for a common and engaging learning transfer platform that can appeal to
different cultures and act as a learning hub for the workforce.
Dispersed organizational training makes use of IVLs that are interconnected to form
global virtual communities. In light of these virtual communities, people are increasingly
working together through face-to-face and immersive virtual situations using advanced IT
systems to fulfill organizational objectives (Creed & Zutshi, 2008). Participation in an IVL
11
classroom gives participants a membership to a community that can substantially improve
efficiency of learning (Rugelj, 2000). This membership is important in developing communities
of knowledge that can be useful in a work setting. Participation or nonparticipation of employees
in learning is seen as a function of social structure and any opportunity for a learning
environment to provide this structure will facilitate participation (Merriam et al., 2007). Despite
the many advantages associated with IVL, the biggest barrier to its adoption is the long history of
traditional learning methods. IVL environments are accused of lacking efficient coordination,
have minimal social exchanges and lack deep and active discussions (Lam, Chua, & Lee, 2007).
One way to overcome this barrier is to take advantage of the use of mentors or primarily learning
by association (Schultze & Orlikowski, 2010; Rusaw, 1995). To ensure that the organizational
users feel comfortable and embrace the new methods of learning, immersive virtual communities
provide mentorship and create new structures for them to learn. The rich diversity provided by
organizational virtual community mentors ensures that the workforce feels confident within the
learning environment. Immersive virtual mentoring can further be useful to the workers’
understanding and communication skills which creates new possibilities for the accessibility of
top mentors within organizations and leaders who are external to them (Colky, Colky, & Young,
2006). In addition, boarders do not limit the virtual communities and act as pools of information
due to their global interconnectivity.
IVL Organization Conceptual Roadmap
Given the above articulation of how, why and when IVL is an effective component of
workplace learning, an enhanced conceptual roadmap for immersive virtual workplace skill
building furthers the discussion on the development of models that could link organizational
human resource development and immersive training platforms. There is minimal understanding
12
today of how learning and knowledge transfer in organizations occur when new learning
technology is introduced to the workforce (Geng & Whinston, 2009). IVL based human resource
development can facilitate the growth or organizational competencies by encouraging
commitment to the organization (Boxall & Macky, 2009). How employees are trained and
whether transfer of knowledge occurs affects employee commitment, which then impacts
turnover, employee costs, retention, etc. (Owens, 2006). The facet of human capital development
through training brings the context of knowledge transfer which is often related to building the
organizational capability to absorb and utilize knowledge (Solomon, Sagasti, & Sachs-Jeantet,
1994). Therefore, the method used to develop the workforce can be espoused as an important
component of successful organizational knowledge acquisition.
Learning is a critical factor that helps retain and attract new talent, speeds up workforce
readiness in the job, acts as a foundation for professional development and aligns business
processes and interactions with other organizations (Manville, 2001). The paradigm of learning
platforms in organizations has developed over the years. Multiple learning platforms exist in
organizations. Some of the common platforms used by organizations to enhance learning include
classroom instruction, computer based training and on-the-job training. Each of them is applied
to a certain degree in different organizations to facilitate learning among their workforce. One of
the first computer based methods used by organizations to enhance workforce skills is e-learning.
This is an asynchronous mode of learning that has the ability to decrease distances, allow
learners flexible schedules and access to archived documents and learning materials that students
can retrieve via mail, e-mail or the school’s website for reading, downloading, and printing
(Wood, Solomon, & Allan, 2008). However, this method has shortcomings that may stifle
learning. E-learning does not extensively enhance group collaboration and its use makes
13
engagement harder for learners (Franceschi, Lee, Zanakis & Hinds, 2009). A study conducted by
Petrides (2002) reported a feeling of isolation by the learners when participating in e-learning.
On the other hand, classroom instruction and on-the-job-training require physical presence of
learners during the sessions. The cost involved in moving learners to one location in addition to
the rigidity of classes that do not intertwine in the workers’ schedule makes this method less
attractive (LaBrosse, 2008; Whisker III, 2008). Considering these shortcomings, learning in
organizations must adjust to incorporate new ways of training, preferably IVLs, which support
the virtual aspects of learning. Virtual reality provides a unique platform for learning, which
involves components that are missing from the list of current methods, such as the capability to
teach skills in the context of organizations in a simulated nature (Whisker III, 2008).
Technology for IVL has advanced to great lengths by providing 3D modeling software
and by creating an high fidelity audio visual environment where cost effective learning can occur
(Whisker III, 2008). IVL provides many of the learning characteristics both in e-learning and
classroom instruction; thus, proving to be a training driver in organizations (LaBrosse, 2008).
The realism and engagement produced by virtual environments creates an added advantage
above the two methods, e-learning and classroom instructions. IVL tries to bridge the traditional
training of the face-to-face setting and e-learning by incorporating three-dimensional graphics,
instant audio feedback, psychology and special exterior devices to produce a realistic computer
generated interactive environment (Limniou, Roberts & Papadopoulos, 2008). Within the IVL
environment, changed organizational activities can be smoothly integrated into a holistic culture
of learning opportunities (Manville, 2001).
The interaction between organizational learning and immersive virtual worlds that are
occurring in the workforce today has created the need to enhance an earlier roadmap that
14
describes the phenomenon of IVL within an organizational context (Ndinguri et al., 2012). The
agility present in organizations as they adopt virtual platforms creates a learning environment
that is ripe for competitiveness (Lin & Lin, 2001). Therefore, a roadmap that illuminates an
immersive virtual learning environment that focuses on the organization is necessary. It would be
premature to suggest that IVL in a workforce environment provides a simplistic flow as an open
input-output process that captures information, transfers it and cumulatively processes it into
new knowledge for the users. Therefore, the organization’s immersive virtual learning
conceptual roadmap (Figure 1) is intended to provide organizations a step-based structure that
guides learning using an immersive virtual platform.
Figure 1
Organization’s Immersive Virtual Learning Conceptual Roadmap
Organization’s IVL Conceptual Roadmap
The workforce and the changing global society re-emphasize the need for organization
adaptive styles and methods of knowledge acquisition. The organizational adaptation of the IVL
methods fosters workplace integration that boosts the adult workforce learning capabilities as has
been highlighted. The overall workforce learning changes, IVL benefits, IVL organizational
IVL based Workforce Dynamism
Knowledge Discharge
Cue Conceptualization
Advanced Immersive Virtual Platforms
IVL based skill Adoption
LEVEL 1 LEVEL 2
LEVEL 3
LEVEL 4
15
adaptation, IVL workforce integration and the expected learning transfer emanating from
integration of IVL serves a foundation for the levels of the framework model discussed.
Level one. The first level involves IVL workforce dynamism, which refers to changes that are
occurring in organizations that workers constantly use IVL to manage their skill development
(Caroli, Greenan, & Guellec, 2001). As the workforce uses new IVL platforms, one of the
fundamental challenges in organizations today is how to train, retrain and improve their human
capital. This creates a need to use learning environment platforms necessary to mitigate this
challenge. However, organizations must understand that these platforms are instruments of
change rather than the sources of change (Manville, 2001). This level represents the source of
change that has occurred among the workforce as they seek to improve their skills using IVL
platforms and remain competitive in the ever dynamic market.
Level two. Use of an existing IVL platform is succeeded by knowledge imparting. Human
resource development in an organization has been linked to individual and firm success (Ulrich
& Brockbank, 2005). Therefore, training of the workforce using the IVL platform becomes the
next step that aims to impart knowledge and skills. This level (skills/learning demand) deals with
IVL technology and its ability to update individual skills efficiently in order to provide a
competitive edge for the organization (Gudanescua, 2010). As human capital changes, use of
better learning platforms that are flexible, cost efficient and create engagement is evident. IVL as
a training platform is growing and taking the concept of learning in the workforce to another
level (LaBrosse, 2008).
Level three. Once learners’ access IVL platforms, they need to put it in to use. This is evident in
the third level that reflects a process in which learners in a virtual environment grasp ideas and
skills that may reinforce the knowledge they have learned. Due to realistic and simulated effects
16
in virtual worlds, learners can internalize what they learn and apply it more efficiently than more
traditional methods (de Freitas, Rebolledo-Mendez, Liarokapis, Magoulas & Poulovassilis,
2010). At this level, the organization workforce uses virtual learning environments to grasp more
complex behaviors and learning processes than traditional methods. As a result, workers may be
more likely to transfer this knowledge to the job.
Level four. The fourth level is a connecting process that propagates an interactive cyclical flow.
Once a skill is adopted, the market and other external forces result in new IVL aspects of human
resource development that keeps the continuous movement of the virtual learning process from
one level to another in the organization. The skills adopted trigger more knowledge and
innovation, which creates a new IVL workforce dynamism and the stage cycles interact again.
This flow creates a path for a framework where there are four levels around which an IVL based
organization revolves. These levels include:
A. Change in human capital development areas due to IVL platform interactions
B. Use of a immersive virtual platform to skillfully learn
C. Learner conceptualizes the learnt skills and adopts them
D. Advances and market conditions propel greater IVL changes and the stage interaction
starts all over again from the top
The road map provides a conceptual base for the learning process taking place in
organizations that have effectively adopted IVL platforms. It elaborates the steps that
homogeneously connect to describe processes that can be used to give an effective immersive
virtual training environment. As IVL expands, there is a need for organizations to adapt to new
immersive virtually based training programs. Therefore, the road map acts as a conceptual
17
foundation of workflow tools that guide the training transition involving IVL environments in an
organization.
Future Research Work and Conclusions
It is uncertain the impact in which advancements and changes in information technology
will have on the cost emanating from constant organization changes. However, using the current
technology efficiently as a learning tool in the organization is essential. To develop workforce
training materials that are effective, different units inside the organization will have to work
together, which takes a collaborative approach and can be resource intensive. Additionally, users
of IVL environments must be able to strategically manipulate and navigate through the
immersive virtual platforms that are usually accompanied by game- like devices and
accompanying equipment. This may result in wider contextual issues where the workforce
cannot relate to the virtual environment due to its appearance or the equipment used (de Freitas
et al., 2010). Therefore, organizational guidelines and careful planning of IVL platforms that suit
different organizations is necessary.
IVL as a current form of learning that is spanning borders and creating a global
connectivity that transcends beyond the traditional classroom to the organizational work place.
Most organization must adapt to the changing market conditions with a virtual, diverse and
continuous learning workforce, creative technologies and aligned business processes if they are
to remain competitive (Graf, 2009). With the increased need to learn and evolve with the skills of
today’s workforce, IVL environments provide added value to an organization’s human resource
development investment. The use of tools like the organizational IVL conceptual roadmap
discussed above show that IVL can be part of workforce development in the workplace by
providing an environment that allows simulation of working activities while providing the
18
necessary structure for guiding processes. These simulations possess unique instructional
capabilities that have the potential to enhance training effectiveness (Bell, Kanar & Kozlowski,
2008; Whisker III, 2008). The future dictates that cost effective, engaging, instant and flexible
workplace learning resources are a necessity. It is expected that the same dynamism and
innovation that has created IVL in the 21st century is required to propel the immersive virtual
organizational learning environment. Therefore, there are other areas that need to be explored to
solidify the information and constructs discussed in order to facilitate immersive virtual
workforce learning in the future:
a) In order to validate the roadmap, more work is required to understand how each level
interacts to produce an outcome that is workable for each individual organization.
b) There is also a need to further exemplify how external factors (e.g. user experience)
may influence the levels of the roadmap for different age groups.
c) Future research may also include longitudinal experimental studies to determine the
effectiveness of IVL on enhanced workplace performance versus traditional methods.
d) Additional methods on how to blend immersive virtual platforms into skill
development programs in organizations to mitigate obstacles relating to equipment
use.
e) We need to determine how organizations respond to the present day IVL environment
and what they can expect in the near future.
f) Examine how low skilled adult workers respond to training in the virtual
environment.
19
References
Aspin, D.N., & Chapman, J.D. (2000). Lifelong learning: Concepts and conceptions. International Journal of Lifelong Education, 19(1), 2–19.
Bailenson, J.N., Yee, N., Blascovich, J., Beall, A.C., Lundblad, N., & Jin, M. (2008). The use
of immersive virtual reality in the learning sciences: Digital transformations of teachers, students, and social context. The Journal of the Learning Sciences, 17, 102-141.
Bell, B. S., Kanar, A. M., & Kozlowski, S.J. (2008). Current issues and future directions in
simulation-based training in north America. International Journal of Human Resource Management, 19(8), 1416-1434.
Bennett, E. E., & Bierema, L. L. (2010). The ecology of virtual human resource development.
Advances in Developing Human Resources, 12(6), 632-647. Bentley, T. (1998). Learning beyond the classroom: Education for a changing world. London,
LDN: Routledge. Boxall, P., & Macky, K. (2009). Research and theory on high-performance work systems:
Progressing the high-involvement stream. Human Resource Management Journal, 19(1), 3-23.
Caroli, E., Greenan, N., & Guellec, D. (2001). Organizational change and skill accumulation.
Industrial & Corporate Change, 10(2), 481-506. Chinowsky, P., & Carrillo, P. (2007). Knowledge management to learning organization
connection. Journal of Management in Engineering, 23(3), 122-130. Doi: 10.1061/(ASCE)0742-597X(2007)23:3(122)
Coco, M. L. (2011). Evaluation of knowledge transfer in an IVL environment for the
transportation community (Doctoral dissertation). Retrieved from http://etd.lsu.edu/cgi- bin/ETD-browse/
Colky, D. L., Colky, M., & Young, W. H. (2006). Mentoring in the virtual organization: Keys to
building successful schools and businesses. Mentoring & Tutoring, 14 (4), 433–447. Creed, A., & Zutshi, A. (2008). The wellhouse of knowledge globalization: It and virtual
communities. Journal of Knowledge Globalization, 1(1), 29-42. de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G., & Poulovassilis, A. (2010).
Learning as immersive experiences: using the four-dimensional framework for designing and evaluating immersive learning experiences in a virtual world. British Journal of Educational Technology, 41(1), 69-85.
20
Derouin, R. E., Fritzsche, B. A., & Salas, E. (2004). Optimizing e-learning: research-based guidelines for learner-controlled training. Human Resource Management, 43(2/3), 147- 162.
Eastmond, D.V. (1998). Adult learners and internet- based distance education. New Directions
for Adult and Continuing Education, 1998(78), 33-41. Egan, T. M., Yang, B., & Bartlett, K. R. (2004). The effects of organizational learning culture
and job satisfaction on motivation to transfer learning and turnover intention. Human Resource Development Quarterly, 15(3), 279–301.
Fletcher, J. D., Tobias, S., & Wisher, R. A. (2007). Learning anytime, anywhere: advanced
distributed learning and the changing face of education. Educational Researcher, 36(2), 96-102.
Franceschi, K., Lee, R. M., Zanakis, S. H., & Hinds, D. (2009). Engaging group e-learning in
virtual worlds. Journal of Management Information Systems, 26(1), 73-100. Geng, X., Lin, L., & Whinston, A.B. (2009). Effects of organizational learning and knowledge
transfer on investment decisions under uncertainty. Journal of Management Information Systems, 26(2), 123-145.
Gorard, S., Selwyn, N., & Williams, S. (2000). Must try harder! problems facing technological
solutions to non-participation in adult learning. British Educational Research Journal, 26(4), 507-521.
Graf, T. (2009). The future of OD: developing an effective virtual organization for the OD
network. OD Practitioner, 41(3), 30-36. Griffin, C. (1999). Lifelong learning and social democracy. International Journal of Lifelong
Education. 18(5), 329-342. Gudanescua, N. (2010). Using modern technology for improving learning process at different
educational levels. Procedia - Social and Behavioral Sciences, 2(2), 5641-5645. Hayes, E.R. (2006, January). Situated learning in virtual worlds: the learning ecology of second
life. Paper presented at the Adult education research conference, Madison, Wisconsin Jansen-Osmann, P., & Wiedenbauer, G. (2004). Way finding performance in and the spatial
knowledge of a color-coded building for adults and children. Spatial Cognition and Computation, 4(4), 337–358.
Johnson, L. F., & Levine, A. H. (2008). Virtual worlds: Inherently immersive, highly social
learning spaces. Theory Into Practice, 47(2), 161-170
21
Koper, R., & Tattersall, C. (2004). New directions for lifelong learning using network technologies. British Journal of Educational Technology, 35 (6), 689–700.
LaBrosse, M. (2008). Managing virtual teams. Employment Relations Today, 35(2), 81-86. Lam, W., Chua, A., & Lee, C. (2007). Observed patterns of dysfunctional behaviour in virtual
teams. In E. Coakes & S. Clarke (Eds.), Encyclopedia of communities of practice in information and knowledge management (pp. 392-396). Hershey, PA: Idea Group Reference.
Limniou, M., Roberts, D., & Papadopoulos, N. (2008). Full immersive virtual environment
CAVETM in chemistry education. Computers & Education, 51(2), 584–593. Lin, F., & Lin, S. (2001). A conceptual model for virtual organizational learning. Journal of
Organizational Computing & Electronic Commerce, 11(3), 155-178. Majchrzak, A., Rice, R.E., Malhotra, A., King, N., & Ba, S. (2000). Technology adaptation: the
case of a computer-supported inter-organizational virtual team. MIS Quarterly, 24(4), 569-600.
Mantovani, F., Castelnuovo, G., Gaggioli, A., & Riva, G. (2003). Virtual reality training for
health-care professionals. CyberPsychology & Behavior, 6(4), 389-395. Doi: 10.1089/109493103322278772
Manville, B. (2001). Learning in the new economy. Leader to Leader (20), 36-45. Martins, L. L., Gilson, L. L., & Maynard M. T. (2004). Virtual teams: What do we know and
where do we go from here? Journal of Management, 30(6), 805-835. Merriam, S. B., Caffarella, R. S., & Baumgartner, L. M. (2007). Learning in adulthood: A
comprehensive guide (3rd Ed.). San Francisco: Jossey Bass. Nah, F., Eschenbrenner, B., & DeWester, D. (2011). Enhancing brand equity through flow and
telepresence: a comparison of 2D and 3D virtual worlds. MIS Quarterly, 35(3), 731-747. Ndinguri, E. N., Machtmes, K., Hatala, JP., & Coco, ML. (2012). Virtual learning in the
workforce: Building a roadmap for organizational impact. Proceeding of Academy of Human Resource Development, Denver, CO, February, 407-437.
Noe, R. A., & Schmitt, N. (1986). The influence of trainee attitudes on training effectiveness:
Test of a model. Personnel Psychology, 39 (3), 497–523. Owens, P. L. (2006). One more reason not to cut your training budget: the relationship between
training and organizational outcomes. Public Personnel Management, 35(2), 163-171.
22
Peruch, P., Belingard, L., & Thinus-Blanc, C. (2000). Transfer of spatial knowledge from virtual to real environments. Lecture Notes in Computer Science, 1849, 253-264.
Petrides, L.A. (2002). Web-based technologies for distributed (or distance) learning: Creating
learning-centered educational experiences in the higher education classroom. International Journal of Instructional Media, 29(1), 69 –77.
Ripamonti, L. A., & Peraboni, C. A. (2010). Managing the design-manufacturing interface in
virtual enterprises through multi user virtual environments: A perspective approach. International Journal of Computer Integrated Manufacturing, 23(8/9), 758- 776.
Rugelj, J. (2000). Collaborative virtual environments for problem based learning. Paper
Presented at the International Perspectives on Tele-Education and Virtual Learning Environments, Ashgate, Aldershot.
Rusaw, A. (1995). Learning by association: Professional associations as learning agents. Human
Resource Development Quarterly, 6(2), 215-226. Schultze, U., & Orlikowski, W. J. (2010). Virtual worlds: a performative perspective on globally distributed, immersive work. Information Systems Research, 21(4), 810–821. Solomon, J. J., Sagasti, F. R., & Sachs-Jeantet, C. (1994). The uncertain quest: Science,
technology, and development, Tokyo, JP: United Nations University Press Symons, J., & Stenzel, C. (2007). Virtually borderless: An examination of culture in virtual
teaming. Journal of General Management, 32(3), 1-17. Sveiby, K. (2001). A knowledge-based theory of the firm to guide in strategy formulation.
Journal of Intellectual Capital, 2 (4), 344-358. Thorndyke, P. W., & Hayes-Roth, B. (1982). Differences in spatial knowledge acquired from
maps and navigation. Cognitive Psychology, 14, 560–589. Townsend, A. M., DeMarie, S. M., & Hendrickson, A. R. (1998). Virtual teams: Technology
and the workplace of the future. Academy Of Management Executive, 12(3), 17- 29.
Uggla, B. K. (2008). Who is the lifelong learner? Globalization, lifelong learning and
hermeneutics. Studies in Philosophy & Education, 27(4), 211-226. Doi: 10.1007/s11217- 007-9074-y
Ulrich, D., & Brockbank, W. (2005). The HR value proposition. Boston, MA: Harvard Business
School Press.
23
Vidal, E. (2011). Virtual HR:Trends and insights on how organizations are now recruiting, on- boarding and training employees….virtually (white paper). InterCall Business Report. Retrieved from http://www.intercall.com/files/virtual-whitepaper-virtualHR.pdf.
Whisker III, V. E. (2008). Assessment of radiation awareness training in immersive virtual
environments (Doctoral dissertation). Retrieved from http://etda.libraries.psu.edu/theses /approved/WorldWideIndex/ETD
White, D., & Le Cornu, A. (2010). Eventedness and disjuncture in virtual worlds Technology-
Assisted Lifelong Learning (TALL). Department for Continuing Education, University of Oxford, 52 (2), 183–196.
Wlodkowski, R. J. (2008). Enhancing adult motivation to learn: a comprehensive guide for
teaching all adults, (3rd Ed.). Seattle, WA: Jossey- Bass. Wood, N. T., Solomon, M. R., & Allan, D. (2008). Welcome to the matrix: e-learning gets a
second life. Marketing Education Review, 18(2), 48-53.