Technical Report 1196

Program Evaluation Metrics for U.S. Army LifelongLearning Centers

Anna T. CiancioloCommand Performance Research, Inc.

March 2007 20070417199

United States Army Research Institutefor the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.

U.S. Army Research Institutefor the Behavioral and Social Sciences

A Directorate of the Department of the ArmyDeputy Chief of Staff, G1

Authorized and approved for distribution:

SCOTT E. GRAHAM MICHELLE SAMSActing Technical Director Director

Technical review by

Joseph Psotka, U. S. Army Research InstituteGregory A. Ruark U. S. Army Research Institute

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.Please address correspondence concerning distribution of reports to: U.S. ArmyResearch Institute for the Behavioral and Social Sciences, Attn: DAPE-ARI-MS,2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longerneeded. Please do not return it to the U.S. Army Research Institute for the Behavioraland Social Sciences.

NOTE: The findings in this Technical Report are not to be construed as an officialDepartment of the Army position, unless so designated by other authorized documents.

REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

March 2007 Interim January 2006 - September 2006

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBERW74V8H-04-D-0044

Program Evaluation Metrics for U.S. Army Lifelong LearningCenters 5b. PROGRAM ELEMENT NUMBER

633007

6. AUTHOR(S): 5c. PROJECT NUMBER

A792Anna T. Cianciolo (Command Performance Research, Inc.)

5d. TASK NUMBER

285

5e. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

Command Performance Research, Inc.1201 Waverly DriveChampaign, Illinois 61821

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

ARIU.S. Army Research Institute for the Behavioral and SocialSciences 11. MONITOR REPORT NUMBER2511 Jefferson Davis Highway Technical Report 1196Arlington, VA 22202-392612. DISTRIBUTION/AVAILABILITY STATEMENTApproved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTESSubject Matter POC: Kimberly A. Metcalf

14. ABSTRACT (Maximum 200 words):Lifelong Learning Centers (LLCs) comprise a suite of technologies that enable online posting of schoolhousecurricula and collaboration among distributed learners. These technologies connect the field Army to Armyschoolhouses, simultaneously improving course currency and supporting training in the field. The impact oflifelong learning on organizational excellence seems clear. However, it is unknown how LLCs promote readinessusing educational technology and how LLC effectiveness should be measured. The purpose of this research wasto develop a comprehensive, generalizable framework for conceptualizing the effectiveness of LLCs and forcapturing the drivers of success. The framework and associated metrics were used to conduct an evaluation of apilot LLC located at Fort Leavenworth. This evaluation indicated the importance of taking a causal approach. Anassessment of outcomes alone would have indicated that the initiative had achieved its goals but would haveobscured the fact that some of these goals--teaching and learning effectiveness--were achieved largelyindependently of the use of learning technologies. The basis of the framework in theory makes it generalizablenot only across current and future LLCs, but also across other blended learning initiatives, addressing a gap inthe scholarly literature regarding the effectiveness assessment of educational technology.

15. SUBJECT TERMSLifelong Learning Initiative (LLI), Lifelong Learning Center (LLC), Education, Training, Distance Learning, Blended

Learning, Program Metrics

... " SECURITY cssIFICATIONOF 19. LIMITATION 20. NUMBER 21. RESPONSIBLE PERSON__ __ _ __ _ __ _ : OF ABSTRACT OF PAGES Ellen Kinzer

16. REPORT 17. ABSTRACT 18. THIS PAGEUnclassified Unclassified Unclassified Unlimited 142 Technical Publication

Specialist703-602-8047

i

ii

Technical Report 1196

Program Evaluation Metrics for U.S. ArmyLifelong Learning Centers

Anna T. CiancioloCommand Performance Research, Inc.

Leader Development Research UnitStanley M. Halpin, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences2511 Jefferson Davis Highway, Arlington, Virginia 22202-3926

March 2007

Army Project Number Personnel Performance633007A792 and Training

Approved for public release; distribution is unlimited.

iii

ACKNOWLEDGEMENTS

The effort described in this report would not have been possible without the generouscontribution of time and input by several individuals. I am grateful to the more than fiftyinstructors, course authors, and students of the Command and General Staff School whoparticipated in focus groups and interviews to determine metrics and measures. Other LifelongLearning stakeholders, including the pilot directors and their support staff, also providedessential guidance. The staff of the Command and General Staff College Directorate ofEducational Technology provided crucial support of this effort, providing input, documentation,and data. Mr. Albert Huang of Defense Training Technologies, Inc. provided much neededresearch assistance throughout this investigation. I also wish to acknowledge the Command andGeneral Staff College Quality Assurance Office for their diligent assistance with thedevelopment and administration of the online surveys.

iv

PROGRAM EVALUATION METRICS FOR U.S. ARMY LIFELONG LEARNING CENTERS

EXECUTIVE SUMMARY

Research Requirement:

The same factors that create a greater need for Soldiers and leaders to have education andtraining that will help them to learn and adapt quickly--e.g., frequent tours of duty and rapidlychanging equipment and technology requirements--make it more difficult for Soldiers andleaders to receive the instruction they need via the traditional Army education system. Thelifelong learning concept, launched by the U.S. Army Signal Center, has been advanced by theU.S. Army Training and Doctrine Command as an Army-wide solution to this problem. LifelongLearning Centers (LLCs) comprise a suite of technologies that enable, among other things,online posting of schoolhouse curriculum materials and synchronous and asynchronouscollaboration among students, instructors, curriculum developers, and other users. Thesetechnologies connect the field Army to Army schoolhouses, simultaneously improving coursecurrency and supporting training in the field through just-in-time reachback to the institution.

The impact of lifelong learning on organizational excellence appears clear--missionreadiness will be enhanced over the short and long term, with obvious implications foroperational success. What is less well understood is how LLCs promote readiness through theuse of educational technology and how the effectiveness of LLCs should be captured. Thepurpose of the present investigation was to research and develop a comprehensive, generalizableframework for conceptualizing the effectiveness of LLCs in enhancing learning and readinessand for capturing the drivers of LLC success. The goal was to produce an assessment frameworkand associated metrics that would be applicable, with minor modification, across current andfuture LLCs.

Procedure:

Through a combination of scholarly and technical literature review and face-to-faceinterviews, phone conversations, and email exchanges with key stakeholders, an assessmentframework based on the logic modeling (or causal) approach and blended learning theory wasproduced. The framework and associated metrics developed were used to conduct a formativeevaluation of an LLC currently being piloted at the Fort Leavenworth Command and GeneralStaff College.

Findings:

The assessment framework proved to be a useful tool for capturing and representing thefunctioning of blended learning initiatives such as LLCs, from the acquisition and use ofresources to the achievement of organizational impact. Furthermore, a review of the assessmentframework by key stakeholders from each of the pilot LLCs ensured generalizability across thedifferent initiatives. The findings of the Fort Leavenworth evaluation indicated the importance oftaking a causal approach. An assessment of outcomes alone would have indicated that the

v

initiative had achieved its goals but would have obscured the fact that a subset of these goals--teaching and learning effectiveness--was achieved largely independently of the use of blendedlearning technologies.

Utilization and Dissemination of Findings:

The assessment and evaluation of LLC effectiveness is necessary to justify continuedinvestment in lifelong learning. Moreover, the optimal time to begin assessing LLCs is now,during the initial stages of their implementation. A causal assessment framework such as the oneresearched and developed in the present evaluation verifies whether the apparent success of anLLC is truly driven by the activities and outputs generated by the initiative. It also provides clearrecommendations for bridging gaps between resources, personnel activities, system outputs, andoutcomes in order to enhance impact. The basis of this framework in the theory of adult learningand blended learning methodology makes it generalizable not only across current and futureLLCs, but also across blended learning initiatives more generally, addressing a gap in thescholarly literature with regard to the effectiveness assessment of educational technology.

vi

PROGRAM EVALUATION METRICS FOR U.S. ARMY LIFELONG LEARNING CENTERS

CONTENTS

Page

IN T R O D U C T IO N ................................................................................................................ 1

WHAT TO EVALUATE? OVERVIEW OF THE APPROACH ....................................... 2Background - Evaluation Studies Measuring Effects .................................................. 2

Effectiveness Conceptualized as Learning Achievement ......................................... 3Effectiveness Conceptualized as Cost Savings ........................................................ 4

Background - Evaluation Models Addressing Causes and Effects ............................... 4A Way Forward: The Logic Modeling Approach ........................................................... 6

BUILDING THE LOGIC MODEL FOR LLC EVALUATION ........................................ 6The LLC Logic M odel .................................................................................................... 7

R eso urces .................................................................................................................. 7A ctivities .................................................................................................................... 9O u tp u t .............................................................................................. ..... . 11Outcomes (Short-, Intermediate-, and Long-term .......... . ....... .. ............... 12

METRICS AND MEASURES FOR EVALUATING LLCS ............................................. 21

APPLICATION OF THE LOGIC MODEL AND METRICS TO EVALUATING THEFORT LEAVENWORTH LIFELONG LEARNING INITIATIVE ................................... 21

C O N C L U SIO N ..................................................................................................................... 23

R E F E R E N C E S ..................................................................................................................... 25

APPENDIX A - LLC COMPREHENSIVE METRIC AND MEASURE CHART 33

APPENDIX B - EVALUATION OF THE FORT LEAVENWORTH LIFELONGLEA RN IN G IN ITIA TIV E .................................................................................................... 75

A PPEN D IX C - A CRO N Y M S ............................................................................................ 137

LIST OF FIGURES

FIGURE 1. LOGIC MODEL FOR ARMY LLC IMPACT ............................................... 8

vii

viii

INTRODUCTION

The U.S. Army's contemporary operating environment is characterized by frequent toursof duty, heavy reliance on the Reserves and the National Guard, the introduction of newequipment and technology immediately prior to and during deployment, and a volatile globalpolitical situation. All of these factors result in a greater need for Soldiers and leaders to obtainthe education and training that will help them to learn rapidly and adapt quickly. Yet these sameconditions also make it more difficult for Soldiers and leaders to receive the instruction theyneed via the traditional Army education system.

The lifelong learning concept, launched by the U.S. Army Signal Center's University ofInformation Technology, has been advanced by the U.S. Army Training and Doctrine Command(TRADOC) as an Army-wide solution to this problem (TRADOC, 2004). Lifelong learning hasbeen defined as "the ability of Soldiers to learn, grow and achieve technically and tacticallythroughout their career, wherever they serve... It's a mixture of traditional schoolhouse residenteducation with education presented in other locations at the individual's teachable moment"(TRADOC, 2004). Lifelong Learning Centers (LLCs) comprise a suite of technologies thatenable, among other things, online posting of schoolhouse curriculum materials, synchronousand asynchronous collaboration among students, instructors, curriculum developers, and otherusers, and single-source course administration and learning management. These technologiesconnect the field Army to Army schoolhouses, simultaneously improving course currency andsupporting training in the field through just-in-time reachback capability. Two LLCs, in additionto the University of Information Technology, currently are being piloted at Forts Leavenworthand Leonard Wood, with future LLCs envisioned at the other TRADOC installations.

The impact of lifelong learning on organizational excellence seems clear--missionreadiness should be enhanced over the short and intermediate term, with obvious implications foroperational success. Over the long term, lifelong learning should foster a cultural shift in theArmy toward collaboration, information sharing, and knowledge management. What is less wellunderstood is how the enablers of lifelong learning, LLCs, promote readiness and cultural shiftthrough the use of educational technology and how the effectiveness of LLCs should becaptured. Various approaches to assessing the costs and benefits of technology-assisted learninghave been advanced (e.g., Cukier, 1997; Ehrmann, 1994; Whalen & Wright, 1999), but nonehave provided the complete picture of blended learning effectiveness required to diagnoseproblems and make specific recommendations for change. For example, where assessmentapproaches were explicitly specified, they often targeted either the cost or benefit of aneducational initiative, but not both (Cukier, 1997). Some assessment approaches haveemphasized a systems approach to conceptualizing an educational initiative (as opposed torecognizing strictly its outcomes), but yet have excluded some components of the system thatdrive effectiveness (e.g., Ehrmann, 1994).

Assessment and evaluation of LLC effectiveness is necessary to justify continuedinvestment in lifelong learning. Moreover, the optimal time to begin assessing LLCs is now,during the initial stages of their implementation (Clark, 1994; Sims, Dobbs, & Hand, 2002). Thepurpose of the present investigation was to research and develop a comprehensive, generalizableframework for conceptualizing the effectiveness of LLCs in enhancing learning and readiness

and for capturing the drivers of LLC success. The goal was to produce an assessment frameworkand associated metrics that would be applicable, with minor modification, across current andfuture LLCs. The framework and metrics developed in this research were used to conduct aformative evaluation of the Fort Leavenworth pilot LLC. The results of this evaluation provideguidance for the establishment and implementation of future LLCs.

WHAT TO EVALUATE? OVERVIEW OF THE APPROACH

Educational program evaluation primarily informs two decisions: (1) whether to continuefunding for a particular initiative; and (2) where to invest resources (i.e., money, people, and/ortechnology, equipment, and supplies) in order to facilitate an ongoing initiative's effectiveness(Champagne & Wisher, 2000). To support decision-making in the former situation, theevaluation must assess program outcomes relative to program goals. In the latter case, theevaluation not only must assess goal achievement; it also must capture the causal factors drivingthe assessment results. Evaluation of a program's drivers of success--in addition to its impact--tells a story about program functioning and identifies candidate causes for shortfalls ineffectiveness and providing the leverage for change (see Cianciolo, Heiden, & Prevou, 2006;McLaughlin & Jordan, 2004).

This second type of effectiveness evaluation also is helpful for making go/no-go fundingdecisions for an initiative that shows mixed goal attainment. Equipped with a story that linksprogram outcomes to causal factors, decision makers can determine whether continued fundingwill address impact shortfalls or whether organizational resources are best invested elsewhere. Itis realistic to expect that most, if not all, educational program evaluations will produce mixedresults with regard to educational impact, especially evaluations of programs in the pilot stage ofimplementation.

Because an evaluation approach that captures both effects and causes is more (a)informative for diagnosing impact shortfalls; (b) flexible in its support of decision making; and(c) robust to the stage of implementation to which it is applied, this type of approach wasselected as the basis for the LLC evaluation framework developed in the present investigation.Background research, conducted to inform the overall design of the framework, focused on thescholarly and professional literature exploring the impact of educational technology. Assessmentframeworks that were implicit (e.g., tacit assumptions guiding quasi-experimental studies) orexplicit (e.g., formal theories) were considered.

Background - Evaluation Studies Measuring Effects

Interest in evaluating the effectiveness of educational technology is not new. Technology-assisted education has been examined extensively for its cost-effectiveness and learning impactat least since the 1970s (e.g., Clark, 1994; Cukier, 1997; Levin, 1986; Russell, 1999). In much ofthe scholarly and technical research, which focused on comparing traditional and technologicalsolutions, decisions regarding what to measure often had an implicit or empirical basis (i.e., asopposed to being theory- or model-based). In addition, these studies were characterized by asingular focus on effects, or outcomes, such as learning achievement and/or cost savings[occasionally also time savings, as in Sanders and Burnside (2001)].

2

Effectiveness Conceptualized as Learning Achievement

The majority of research exploring the effectiveness of technology-supported instruction

has used a quasi-experimental design to compare the achievement of students enrolled in acourse using a specific educational technology against students enrolled in the same course usingmore traditional means of curriculum delivery (e.g., Barry & Runyan, 1995; Gifford, 1998;Keene & Cary, 1992; Hiltz, 1990; Mclsaac, Blocher, Mahes, & Vrasidas, 1999; Sanders &Burnside, 2001). Learning effectiveness was demonstrated if students using educationaltechnology performed as well or better than students completing the course of study usingtraditional means.

The measures of learning achievement used in these studies generally corresponded to thefirst, or affective, level of Kirkpatrick's (1994) four levels of evaluation. That is, surveys orinterviews were used to capture students' reactions to/satisfaction with the course they took,including their perceptions of how much they learned or how beneficial they thought the coursewas. Several of the studies also compared student achievement using Level-2 measures--measures of actual learning. Investigators either used existing course assessment materials orthey developed special purpose measures related to course content, such as mastery tests (e.g.,Hiltz, 1990).

The large body of research demonstrating that technology can have a positive (or nonegative) effect on Level-i and Level-2 measures of academic achievement generally is taken asevidence that technology-assisted education is effective. A primary limitation of this research,however, is that it cannot explain why technology enables effective learning or when and how touse technology to achieve a specific learning outcome (Lockee, Burton, & Cross, 1999). Indeed,many scholars argue that effective technology-assisted learning has little to do with technology,but instead is critically dependent on how technology is adaptively and selectively applied toachieving particular learning objectives (Clark, 1994; Defense Technical Information Center,2000; Ehrmann, 1994; Firdyiwek, 1999; Hannafin & Land, 1997; Owston, 1997; Willis, 2000).This possibility could account for the fact that some comparison studies show mixed or evennegative effects of using educational technology. Understanding the why, when, and how ofeffective technology-assisted education is necessary for understanding the drivers of instructionaldelivery and diagnosing effectiveness shortfalls.

The impact of technology-assisted instruction beyond the educational setting [i.e.,Kirkpatrick's (1994) third and fourth levels of evaluation] has not been addressed by mostempirical studies. Relatively little has been done to measure behavior change systematically oncestudents enter or re-enter non-school settings. Organizational indicators of success influenced bytechnology-assisted education of the workforce also remain relatively unexplored (though seeCianciolo et al., 2006 for an exception). This gap likely is the result of the difficulty of obtainingjob performance data and organizational measures of success, although some educationalprogram evaluations focus exclusively on organizational indicators of success, such as return oninvestment and cost savings.

3

Effectiveness Conceptualized as Cost Savings

Proponents of technology-supported instruction, especially distance education, claim thatthe introduction of educational technology will produce a significant cost savings throughreduced faculty and administrative staff expenses, reduced materials production, storage, and/orshipping costs, and reduced travel and student housing expenses. Even though the fixed costsassociated with procuring instructional technology are greater, variable costs tend to be lowerwhen fewer instructors and printed course materials are required, for example if course durationis compressed (see, e.g., Whalen & Wright, 1999). Although the literature on cost accounting fortechnology-assisted education is quite diffuse, nearly all authors argue that when conductingsuch cost studies, all relevant expenses must be accounted for, including not simply the cost ofthe technology, but the cost to prepare, deliver, and support the course (Clark, 1994; Levin,1986; Morgan, 2000).

Relative to the amount of discussion on the cost of technology-supported education, fewpublished studies have been conducted that actually calculate the cost savings enabled bytechnology-supported instruction. One exception is the work of Phelps, Ashworth, and Hahn(1991), who compared the cost of delivering the U.S. Army Engineer Officer Advanced Coursevia asynchronous computer conferencing to the cost of delivering the same course in theschoolhouse. The cost categories they included were course production (based on estimated timevalues and hourly rates for courseware production), equipment, training, supplies, and operationsand support. Whalen and Wright (1999) calculated projected cost differences between fivedifferent course delivery methods (five technology-supported and one traditional), using adetailed methodology that included the full-range of fixed and variable costs associated withcourse production and delivery. Their approach enabled the replicable cost comparison ofdifferent applications of instructional technology and did not require readers to assume thatdifferent types of technology have equivalent implications for cost relative to traditionalinstruction.

Discussions and analyses of cost savings generally focus solely on models and methodsfor cost accounting and rarely also discuss educational benefits (Cukier, 1997; though see Phelpset al., 1991, for an exception). Cost accounting in education is an extremely complex topic andassessment of cost savings requires a different type of expertise than assessment of learningoutcomes, so perhaps for these reasons it is not coupled with educational measurement.However, assessing financial and learning impact within the same evaluation would enabledecision makers to make informed judgments about cost-benefit tradeoffs (Cukier, 1997). Theincreased costs associated with technology-based course delivery may be worth the expense ifeducational objectives and/or learning audiences previously beyond reach could be obtained.

Background - Evaluation Models Addressing Causes and Effects

Several scholars recognize that multiple factors, such as individual learner and/orinstructor readiness, technology usability, curriculum quality, and organizational support forcontinuous learning, play a role in the effectiveness of an educational or training solution (e.g.,Dean, Biner, & Coenen, 1996; Salas, Rhodenizer, & Bowers, 2000). There is informalrecognition or listing of these factors [e.g., Defense Technical Information Center (DTIC), 2000;

4

Willis, 2000] and there exists a handful of formal models for assessing these factors as theyrelate to the success of technology-assisted education (see, e.g., Belanger & Jordan, 2000;Ehrmann, 1994; Harrison, Seeman, Behm, Saba, Molise, & Williams, 1991).

For example, Harrison et al. (1991) identified, through a combination of literature reviewand interviews, four categories to assess when evaluating distance education: (1) instruction(e.g., student-instructor interaction); (2) management (e.g., technical support and planning); (3)telecommuting (largely the unique features of distance education); and (4) support (e.g.,organizational support for the program). Harrison et al. used these categories to construct an 89-item course evaluation survey, which was then validated in an experimental study. Scale scoresfor each of the four categories were moderately correlated, even in the context of satisfactory tohigh scale reliabilities, suggesting that the categories, while not unique, were distinguishable.However, the researchers only examined the possible factors influencing distance learning anddid not attempt to capture the learning outcomes associated with the course evaluated in thestudy.

Ehrmann (1994) described the planning effort behind the Flashlight Project, which wasintended to produce generalizable, targeted, and validated methods for evaluating technology-assisted higher education. In this planning effort, Ehrmann identified the targets for assessment,advocating a combined process-outcome approach. The outcomes he identified were learningeffectiveness, enrollment and retention, and cost control. Among the processes Ehrmannidentified were teaching-learning strategies especially relevant to technology-supportedinstruction and faculty roles and attitudes towards instructional technology. Ehrmann argued thatimprovements in learning outcomes, access, and costs on a departmental and/or institutionalscale required changes in teaching practices and learning environments, which in turn requiredchanges in institutional patterns of technology use.

The previously described research exploring the effectiveness of technology-assistededucation provides a firm foundation for developing metrics to assess the learning effectivenessand organizational impact of Army LLCs. One could conclude from this body of work that (a) itis important to use multiple levels of analysis to conceptualize educational outcomes; (b) costmetrics should capture the full cost of technology-supported instruction, not just hardware andsoftware expenses; (c) cost effectiveness metrics should enable the concurrent review of costsand benefits; and (d) factors leading to educational outcomes, such as organizational support andtechnology usability and access, should be measured in order to fully understand the learningeffectiveness of technology-assisted education.

What the previously described research cannot do is provide a comprehensive theory orframework for conceptualizing how Army LLCs achieves educational and organizational impactthrough the introduction of blended learning (i.e., combined face-to-face, individual,asynchronous collaborative and synchronous collaborative learning) solutions. An in-depthapplication of the cause-effect (or causal) approach to understanding educational impact mustoccur. Such an application would extend previous research by integrating multiple perspectiveson the drivers and indicators of blended learning effectiveness and by specifying the means bywhich blended learning has an impact on organizational measures of success through behaviorchange.

5

A Way Forward: The Logic Modeling Approach

Logic modeling is a tool for conceptualizing how a program transforms resources (e.g.,funding, personnel, etc.) into external results through activities, output, and multiple levels ofoutcomes. McLaughlin and Jordan (2004) describe the logic model as a "useful advanceorganizer for designing evaluation and performance measurement" (p. 7). A logic model makesexplicit the often tacitly understood functioning of a program, revealing the possibilities forevaluation and the targets for assessment and providing a systematic means to approach thecause-and-effect evaluation of program functioning. Building a logic model is analogous toconducting a terrain analysis; it reveals the areas that must be captured (i.e., assessed) in order toachieve strategic success (i.e., maximally informing decision making) with a program evaluation.

Logic models are especially useful for planning the effectiveness evaluation of programsnot easily associated with traditional return-on-investment metrics, but whose continued fundingdepends on demonstrations of impact (Cianciolo & Prevou, 2006; McLaughlin & Jordan, 2004).The use of logic models does not preclude a cost-effectiveness evaluation, but rather provides away to consider cost and benefits together as interrelated characteristics of the same initiative.Logic models also prevent the focus of evaluation from becoming too narrowed on a particularaspect of a program, such as outcomes--which cannot be used to diagnose problems--andactivities--which are easy to measure but meaningless when disassociated from outcomes. Thelogic model approach used in the present evaluation therefore represents an extension of thework reviewed above by (a) integrating multiple previous assessment approaches into a unified,systematic approach; (b) capturing multiple levels of outcome, to include organizational impact;and (c) assessing both cost- and learning-effectiveness in the same study.

BUILDING THE LOGIC MODEL FOR LLC EVALUATION

A variety of resources were used to build the logic model of LLC impact. Theconstruction process began with a review of the planning and communications documentationprepared by the directors of the three pilot lifelong learning initiatives. This documentationincluded the LLC master plan, operational requirements statement, use case summaries, draftmetrics for each lifelong learning initiative, and implementation progress presentations. Reviewof this documentation identified the core structure of the LLCs--their technical and staffingcomponents--as well as the envisioned impact of the LLCs on Army education and training,operational performance, and organizational success. Phone conversations, face-to-faceinterviews, and/or video teleconferences were conducted with the director of each of the threeinitiatives to gather their explanations of how the technology comprising the LLCs would lead tothe envisioned organizational impact through enhanced learning and behavior change. Inaddition, phone conversations and face-to-face discussions with key LLC personnel wereconducted in order to identify the activities of each and their expected contribution to LLCfunctioning. This component of the logic model construction process ensured a thorough,concrete understanding of the lifelong learning initiative as envisioned by the Army andimplemented by the individual pilot schools.

6

Concurrently, an extensive review of the scholarly and professional literature ontechnology-assisted learning was conducted, with a special emphasis on theory and best practicein blended learning. The literature review covered a broad range of topics, including (a) theanticipated behavioral outcomes of learning in a technology-supported environment; (b) thesocial impact of including technology, especially networked computers, in the learning process;and (c) the instructor, student, and organizational determinants of effective learning. A review ofthe military technical and professional literature was conducted to more fully understandconcretely such organizational outcomes of interest to the Army as mission readiness and cultureshift. This component of the logic model construction process fleshed out the informationgathered from the LLC planning and communications documentation and interviews with LLCleadership and staff. It helped to make explicit the link between technology and organizationalimpact that often is implicit in discussions of technology-assisted learning in the Army (e.g.,Freeman, 2003).

Finally, face-to-face interviews and focus groups were conducted with curriculumdevelopers, instructors, and students at the Command and General Staff School at FortLeavenworth. The intent of these conversations was to capture the modifications that must bemade to the general logic model in order to make it applicable to the Leavenworth lifelonglearning initiative in particular, which was assessed as part of the present investigation.

The LLC Logic Model

This section presents a detailed overview of the logic model developed for the presentresearch and depicted in Figure 1 below. The following description of each component of themodel--resources, activities, outputs, and outcomes--is coupled with the justification of itsinclusion in the model.

Resources

The resources briefly described below represent the categories of resources typicallyfound in the logic models of other programs (McLaughlin & Jordan, 2004). Of importance is thesupport these resources provide to executing the program activities. Tracking critical resourcesmakes it possible for evaluators to inform decisions regarding resource allocation within aprogram by linking resources to outcomes through activity and output.

Money. Best practice in cost accounting for higher education indicates that there are threebroad categories of expense: (1) personnel, supplies, and other costs directly attributable tocourse development and delivery (e.g., instructor salaries and benefits); (2) personnel, supplies,and other costs indirectly attributable to course development and delivery (e.g., salaries andbenefits for technical support personnel and administrative staff); and (3) costs associated withfacilities and equipment use (e.g., heating and electricity) (Hyatt, 1983; Middaugh, 2000). Allfinancial resources associated with producing, administering, and supporting a course should beaccounted for, especially for technology-assisted courses in which the temptation is to limit costaccounting to technology requirements (Clark, 1994; Levin, 1986).

7

Personnel. In addition to the LLC leadership and the technical staff, the LLC planningdocumentation (LLC Operational Requirements, 2005) lists several other key personnel typesinvolved in lifelong learning. These key personnel types include training developers, subjectmatter experts, instructors, course managers, and doctrine developers as well as others whobenefit from the program, that is, students and general purpose users. In the present investigaiton,the personnel of particular focus were the technical staff and the leadership, for their criticalrole in the online presence of the LLC, and the instructors and training developers (i.e.,curriculum developers and CBT/WBT courseware production teams), for their critical rolein providing

Resources Technology, personnel, leadership,facilities, needs assessment

Activities Use of resources by personnel tocreate a blended learning environment

Output - Learner access to and use of theOutput L n blended learning environment'sBlended Learning content and technology

Characteristics of the blended

Short-term Outcome - learning environment that differentiate

Enhanced Instruction/Learning it from instructor-focused learningI and that address known shortfalls

Intermediate-term Outcome - Readiness, time to competency and

Enhanced Performance lifelong learning orientation

ILong-term Outcome -

Organizational Excellence/Lifelong Learning Culture shf

Figure 1. Logic Model for Army LLC Impact

learning content. Most of the other personnel were excluded in the present research in an effortto balance the current and future instantiations of the LLC. These excluded personnel rolesrepresent envisioned users, so explicating their role in the initiative would have been somewhatspeculative and subject to change. Students and general purpose users were not included in

8

personnel because their primary role was considered to be more analogous to a consumer than aproducer in the program.

Technology, equipment, and supplies. The technology, equipment, and supplies for theLLC include the hardware, software, and network connections necessary to create and sustain theonline presence of the initiative, but potentially also go beyond the technical footprint to includeequipment for students or other users and office supplies for personnel.

Logic models may include additional resources in order to capture program inputs thatare not conventionally considered requirements but nevertheless are critical to program success(McLaughlin & Jordan, 2004). Chief among these resources is a needs assessment. The needsassessment states in concrete detail the purpose of the program--the problems it will solve,performance or behavioral shortfalls it will address, and/or the ways in which it will extendorganizational success. The needs assessment also explains why the program is the best solution.Importantly, the needs assessment provides the focus for program activities, aligning them withthe envisioned outcomes and expected impact of the initiative (Cianciolo et al., 2006).

Activities

The activities of the LLC's technical staff, curriculum developers, instructors, coursewareproduction teams, and leadership collectively produce the main output of the initiative--anytime,anywhere access to Army schoolhouse curriculum content. The tasks these types of personnelexecute were determined in part by review of the LLC planning documentation, interviews withLLC personnel, and, where necessary, literature review.

Technical staff The technical staff is the critical interface between LLC users and theeducational technology. As such, the technical staff first ensures that the system is accessible byperforming the setup, integration, customization, and management of LLC components andmigrating course content across LLC components (e.g., from Sharepoint to Bb). Secondly,the technical staff fosters the effective use of the educational technology by providing trainingto instructors, curriculum developers, and other users on the LLC components (see alsoDTIC, 2000). This training not only should include instruction on the basic system functions butalso on the standard operating procedures that have been developed in the schoolhouse toleverage the system capabilities. A third key function of the technical staff is to providetechnical support to students, faculty, curriculum developers, and other users, assisting withenrollment, usability problems, and other inquiries (e.g., password changes). In serving thisfunction, the technical staff also provides support for answering Field Army users'operational questions, either by contacting a relevant, local subject matter expert or byfacilitating a connection between users and more remote experts, such as through BCKSprofessional forums. The effectiveness of technical support in performing these functions is animportant determinant of the success of technology-assisted education (DTIC, 2000; Willis,2000)

Curriculum developers. The fundamental role of the curriculum developer is to populatethe LLC with engaging, effective, and relevant course content. In order to enhance teaching andlearning through the development of more relevant curriculum materials, developers must

9

leverage the capabilities of the LLCs to collaboratively generate course content, even whengeographically distributed. Such collaboration ensures that curriculum materials are integratedand build on one another to reinforce key learning objectives. Curriculum developers also mustupdate course content based on feedback from instructors and students, for example byleveraging access to Army Knowledge Management resources (e.g., BCKS, Center for ArmyLessons Learned). Importantly, early adopters of the LLCs' educational technology must leadthe development of standard operating procedures (SOPs) for leveraging the capabilities ofthe LLC components. As with any digital system with multiple users, shared awareness of howinformation is organized is critical to collaboration, and effective procedures for building thisshared awareness must come from the collaborators themselves. Early adopters also mustmentor late adopters on system functionalities to enhance course development (Willis,2000). Late adopters may be curriculum developers who are initially resistant to using the systemor who are newly hired and therefore unfamiliar with the collaborative technologies their peersare using. Both the presence of SOPs and formal mentorship among curriculum developersrepresent well-recognized and Army-endorsed knowledge management practices for enhancingperformance.

Instructors. Instructors ensure that students and other users of the LLC participate in aneducational experience that leverages the unique capabilities of blended learning systems. Forexample, instructors must deliver curriculum materials by posting them in the LLC.Although curriculum developers and the technical staff are responsible for creating andmigrating course content, instructor updates posted in the LLC ensure that content is (a)accessible to all users of the LLC (with the appropriate permissions); and (b) relieved of thelimitations associated with a lengthy curriculum development and review process. Other meansfor delivering curriculum updates, such as print or email, prevent the achievement of some keygoals of the system, such as cost-effectiveness and uniform access. Instructors may customizecourse content based on student feedback by leveraging the easy access to Army knowledgemanagement and other resources that the LLC provides. Instructors also should use the system toperform course administrative duties, which preserves classroom time for actual learning(Bourne, 1998). Instructors also may leverage system capabilities to evaluate student progressand report grades to school administrators. Creative use of the LLC components to evaluatestudent progress may involve such activities as asynchronous discussions and distributedpresentations and collective exercises (Bonk & Cummings, 1998). Finally, early adopters of theLLC components should lead the development of SOPs for leveraging the system capabilities(e.g., collaborative capabilities in Bb) and mentor late adopters on application functionalitiesto enhance course instruction. As with curriculum developers, late-adopting instructors may bethose who initially resist system use or who are new to the faculty. Such knowledge managementpractices would ensure that best practice is internalized and disseminated.

CBT/WBT courseware production team. The CBT/WBT courseware production team,when present, must ensure that CBT/WBT courseware delivered via the LLC is reliable,instructionally sound, efficiently delivered, and cost-effective. To do this, project teams mustbe maintained to perform CBT/WBT analysis, design, development, implementation,maintenance, and validation. The maintenance of these project teams ensures that specialistsare allocated to each component of the development process. That is, instructionaldesigners/analysts are assigned to designing and validating course content as well as overseeing

10

the course generation process. Programmers ensure the effective implementation and technicalsupport of the courseware and media specialists enable engaging multimedia presentation ofcourse content. The courseware production team also provides contractual, technical, andeducational/quality oversight of contractor-developed CBT/WBT. Where the demand fordistance-learning courseware is too great for the in-house team, it nevertheless must assist inmaintaining the quality and efficiency of the development process through consultation andsupport. Finally, the courseware production team must maintain a database of CBT/WBTtechnologies, capabilities, and techniques in order to leverage the latest knowledge and bestpractice in multimedia instruction. The presence of such a database ensures that all members ofthe team have access to the knowledge they need to make the courseware as effective aspossible.

Leadership. The LLC leadership is key to ensuring that all stakeholders are maximallyengaged and that through their coordinated activity the initiative reaches its goals. To facilitatethe coordinated activity of the other LLC personnel, the leadership must provide the vision thatserves as a shared objective for all involved. The vision should be based on a user andstakeholder needs assessment that maximally aligns the goals of the initiative with the interestsand needs of the people involved and with the Army's organizational mission (Cianciolo et al.,2006; DTIC, 2000; Thach, 1993). The leadership must then communicate the vision andmarket the LLC concept to stakeholders (LLC Master Plan, 2005; Willis, 2000). Stakeholdersinclude curriculum developers, instructors, courseware developers, technical staff, students, andthe operational Army because each of these individuals are affected by the introduction of a newlearning initiative and their buy-in and personal investment are critical determinants of itssuccess. To ensure the effective functioning of the LLC, the leadership must procure resourcesto maintain/update the LLC, oversee operations, prioritize limited resources across LLCfunctions, and initiate/organize the development of SOPs for leveraging the systemcapabilities. Although SOPs must ultimately come from the personnel who use the LLCcomponents, leadership is necessary to guarantee that the task of SOP development is taken onand that the key individuals are involved who can facilitate the process and ensure SOP adoption.

Output

24/7 uniform access. Explicit throughout the LLC Master Plan (March, 2005) is thevision that the teaching and learning environment will be improved via integrating resident anddistance-based training and education into a blended, seamless process that enables standardizedinstruction anytime, anywhere. As stated in the planning documentation, the main output of theLLC then is "relevant, standardized training and education, on demand, to the right Soldier,leader, or unit, at the teachable moment, regardless of location" (p. 1). LLC output may beconceptualized both as access to the system (i.e., the system truly is available 24/7 to allmembers of a diverse user population) and actual use of the system by students and other users.Use of an accessible system is necessary to achieve the intended educational and organizationalimpact of the initiative, and to link the activities of personnel to these outcomes.

Computer-/Web-based courseware for conducting proponent courses. Unique to the FortLeonard Wood LLC is the presence of a CBT/WBT courseware production team, whichcomprises instructional designers, programmers, and media specialists who collectively perform

11

CBT/WBT analysis, design, development, implementation, maintenance, and validation forinstructors. The output of this team should be courseware that is readily available to meetcourse needs. That is, the courseware should be reliably playable in Distance Learning XXIclassrooms, digital training facilities, and any other location by which distance education isaccessed. If the courseware is not accessible, it cannot achieve enhanced instruction and learning,no matter how educationally sound and engaging it is. The courseware also should feature thesame range in content that is found in the courses that are delivered via distance education.

Outcomes (Short-, Intermediate-, and Long-term)

LLCs are expected to have a wide set of outcomes, ranging from short-term impact on thelearning and education process, to long-term impact on organizational culture. Intermediate-termimpact on individual learners is believed to link short- and long-term outcomes as more learnersuse the LLCs more often to achieve lifelong learning. Each outcome component of the LLC logicmodel described below was defined and justified according to (1) explicitly stated goals for theLLC initiative; (2) implicitly recognized changes that the LLC could produce in learning andperformance; and (3) theoretical outcomes expected from blended learning environments.

Short-term Outcome - Improved teaching and learning environment. The LLC planningdocumentation (LLC Operational Requirements, December 2005) states that one high-level goalof the initiative is an "improved teaching and learning environment." The temptation to define"improved teaching and learning environment" in terms of the advanced technologicalcapabilities that constitute LLCs is reflected in some of the initial metrics drafted by LLCstakeholders, which target largely the technical functioning of the initiatives. However, definingimprovements in this way confounds technology with its intended educational effect. Thisprecludes the specification of the link between technology and intermediate-term LLC outcomes,such as improved student performance, enhanced readiness, and the like. Identifying how LLCsachieve improvements to the teaching and learning environment must answer the question "Whatnecessary changes to teaching and learning can be brought about by the implementation of ablended learning environment?" Metrics associated with this outcome must capture whether thechanges are indeed occurring.

Discussions with LLC directors and other stakeholders revealed that the LLCs couldenable several improvements to the teaching and learning environment. Chief among theseimprovements is enhanced relevance of training and educational content, that is, thecorrespondence of training and educational content to the demands of the operationalenvironment. LLCs are thought to enhance relevance primarily in two ways. First, LLCs connectthe institutional Army to the operational Army through centralized access to the BattleCommand Knowledge System Leader Network and numerous other online discussion forums.This connection is believed to reduce the time required to incorporate operational lessons learnedinto course content because input comes directly from the field rather than through a lengthyinstitutional review and vetting process. Second, the online presence of LLCs eliminates therequirement to replicate and ship training and educational materials to students or other learnerswho are not located at the schoolhouse. Replication and shipping can add a significant time delayto distributing instructional materials (sometimes up to a year), expanding the gap betweenlearning content and the reality of the operational environment. Increased correspondence

12

between training and educational content and the operational environment may also be achievedthrough the timing of instructional delivery. In some contexts, just-in-time instruction may bethought of as more relevant than 'just-in-case" instruction, although formal education must strikea balance between these endeavors.

Discussions with the director and staff of the Fort Leonard Wood LLC revealed that theirLLC specifically addresses the limited expertise instructors may have in producing coursewarefor computer-based training (CBT) and/or web-based training (WBT) for distance learning. TheFort Leonard Wood LLC is unique among the LLCs in having an in-house coursewareproduction team that comprises instructional designers, programmers, and media specialists whocollectively perform CBT/WBT analysis, design, development, implementation, maintenance,and validation for instructors. This courseware development team improves the teaching andlearning environment by enabling the delivery of more advanced CBT/WBT courseware fordistributed/distance learning that is educationally sound and compliant with Army trainingregulations. Specifically, educationally sound courseware should meet rigorous, independentlyderived criteria for content, instructional activities, performance assessment and feedback, andusability (Hays, Stout, and Ryan-Jones, 2005). Therefore, the delivery of CBT/WBT coursewareproduced by specialists may enhance the teaching and learning environment by obviatingindividual differences in instructor capability to produce engaging, effective, and usablecourseware.

Review of the scholarly literature indicated that there are several ways by whichintroducing blended learning to the instructional process may improve both teaching andlearning. The most commonly cited improvement is the emergence of a learning experience inwhich the instructor acts as a facilitator of adult learning, fostering knowledge-seeking andknowledge-generation in his or her students through wide-ranging access to resources, people,and information. As stated in Firdyiwek (1999), "the pedagogy of online instruction is based onthe effective use of electronic learning environments for the development of cognitive skillsthrough access to information, interactivity with tools, and communication" (p. 29). There isgeneral agreement in the literature (e.g., Abell, 2003; American Distance Education Consortium,2003; Bonk & Cummings, 1998; Bonk & Reynolds, 1997; Chickering & Gamson, 1987;Herrington, Herrington, Oliver, Stoney, & Willis, 2001; Mason, 1991; McLoughlin & Oliver,1999; University of Illinois, Urbana-Champaign, 1999) that the following instructor behaviorsreflect facilitation of adult learning and effective use of educational technology:

* Provides clear guidance on expectations and evaluation criteria* Provides positive and informative feedback* Provides challenges to students without overwhelming them0 Assigns group tasks and roles within group tasks0 Involves students in generating course material* Facilitates analysis and reflection through questioning0 Encourages students to use resources (including other experts) outside the classroom for

research and learning

These behaviors are applicable to any classroom, traditional or otherwise (e.g., Abell,2000), but they are especially important in distributed learning situations in which students lack

13

the social cues and interpersonal connections they receive in face-to-face settings (Hiltz, 1998;Kreijns, Kirschner, Jochems, 2003). Failure to leverage the technological capabilities present inblended learning solutions may reduce the educational effectiveness of instruction relative totraditional classrooms by creating confusion, frustration with the learning experience, and socialdistance.

For students learning synchronously in a classroom setting (traditional or virtual),enhanced instructor efficiency is another improvement to the teaching and learningenvironment enabled by blended learning solutions. A wide range of scholars and practitionersagree [e.g., Clark, 1994; Defense Technical Information Center (DTIC), 2000; Ehrmann, 1994;Firdyiwek, 1999; Hannafin & Land, 1997; Owston, 1997; Willis, 2000] that the long-rangeimpact of technology on learning is a function of the degree to which the technology is leveragedto support change in the learning process. Such change is partly enabled when instructors use theadministrative functions of instructional technologies to protect valuable classroom learning timefrom administrative interruptions (Bourne, 1998). Instructors enhance classroom efficiency byusing asynchronous collaborative tools to prepare students for class discussions, to makeannouncements, to collect homework, to administer exams, and so on. The "extra" availableclass time may then be used for more in-depth exploration or processing of ideas throughdiscussion or practical group exercises.

The presence of a learning community is considered a key hallmark of an effectivelyfacilitated adult classroom, especially one supported by blended learning technologies thatenable learners to connect and share easily anytime, from anywhere. In the context of blendedlearning, learning communities may be defined as networks of learners who are tied to oneanother through the exchange of intellectual, practical, and/or social support (Haythornthwaite,2002; Moller, 1998; Wellman, Carrington, & Hall, 1988). Learning communities characterizedby strong ties between a small number of individuals enhance the teaching and learningenvironment by (a) providing social support critical to retention and educational success,especially for distance learners (Haythornthwaite, 2005; Kreijns et al., 2003; Sanders &Burnside, 2001); and (b) transforming the learning environment into a vehicle for organizationalsocialization through the growth of shared experience and values (Chao, O'Leary-Kelly, Wolf,Klein, & Gardner, 1994; McMillan & Chavis, 1986). Large learning communities characterizedby a greater number of weak, diffuse ties--in effect communities of practice (Wenger,McDermott, & Snyder, 2002)--facilitate information sharing among diverse people and newknowledge development (Haythornthwaite, 2002; Wellman & Gulia, 1999). Learningcommunities are more easily formed when learners are within close proximity of one another,but ties that enable the exchange of support may be formed and maintained even at a distance(Sanders, 2002; Wellman & Gulia, 1999).

Intermediate-term outcome - Improved student performance. The LLC Master Plan(March, 2005) states that LLCs will "improve Soldier performance through resident training anddistance learning." Referring to the 2004 fiscal year training guidance given by the CommandingGeneral of the U.S. Army Training and Doctrine Command (TRADOC), the master plan impliesthat improved student performance will feature personal responsibility for learning and enhancedskill development. In theory, improved student performance results from the effective use ofeducational technology to revolutionize instruction and enhance the meaningfulness of course

14

content. Importantly, student performance in this research primarily is conceptualized in terms ofgeneral learner competencies that may be improved through the effective use of educationaltechnology. Conceptualizing student performance in this way makes the logic model moregeneralizable across LLCs while capturing aspects of performance that are most closely linked tothe changes in instruction brought about by LLCs.

The contemporary operating environment has placed greater demand on Soldiers andleaders to have a diverse, robust skill set that includes both individual and team-basedcapabilities. Diverse skill sets are required so that Army personnel can effectively carry out full-spectrum operations with daily changing mission requirements. Soldiers and leaders also mustacquire and retain robust behavioral repertoires that transfer to a variety of performanceenvironments, such as urban versus open terrain. They have precious few opportunities to trainas collectives in the unit setting and so must rely on institutional training to develop theindividual and team-based competences necessary for collective performance. Enhanced skilldevelopment therefore comprises a greater range in skill sets acquired by Soldiers and leaders,greater transfer of training to diverse performance environments, and improved collective taskexecution. Skill development may be enhanced via blended learning through more field-relevantinstructional content, more frequent and varied training opportunities, more productive learningtime due to reduced administration in the classroom, and more frequent opportunities toparticipate in group learning activities (e.g., Abell, n.d.; Bonk & Reynolds, 1997; Bourne, 1998).

Enhanced learner independence and responsibility follows from effective instructorfacilitation in the blended learning environment (McLoughlin & Oliver, 1999). Althoughgenerally not studied as its own subject, it may be conceptualized as one's sense of ownership ofone's professional development and learning process, indicated by establishing learning goals,engaging in independent research, preparing for learning opportunities (e.g., class discussions),and maintaining regular contact with learning resources. Although the development of learnerindependence and responsibility is an important outcome of any learning environment, theblended learning environment is believed to enhance these characteristics by making it easier forinstructors to promote increased student participation in facilitated group activities (e.g.,asynchronous conferencing), learning self-management (e.g., database and/or portfoliogeneration), and independent research (e.g., internet searches and/or participation in onlinecommunities of practice) (McLoughlin & Oliver, 1999).

The behaviors reflecting learner independence and responsibility are tightly coupled withother learner characteristics that also are influenced by instructor effectiveness, namelyenhanced learning self-efficacy (Abell, 2003; Gibson, 1996) and enhanced motivation (Abell,2000; Cornell & Martin, 1997). Learning self-efficacy within the context of blended learning isreflected in students' perception of their ability to lead their own learning process and to achievetheir learning goals using the information, people, and other resources available to them throughonline content and collaborative tools. One's learning self-efficacy may be negatively affectedby educational technology if the technology is not usable or otherwise introduces difficulty to thelearning process due, for example, to reduced performance feedback or social interaction (Abell,2003; Gibson, 1996; Haythornthwaite, 2005; Kreijns et al., 2003). In addition to instructoreffectiveness, the job-related applicability of the learning content is a well-recognized motivatorof adult learners (Abell, 2000; O'Donnell, 2005). To the extent that the blended learning

15

environment is leveraged to enhance the relevance of course content and the performance-environment representativeness of learning activities (e.g., through simulations and collaborativeactivities), learner motivation can be expected to increase (Cornell & Martin, 1997).

Enhanced higher-order thinking and enhanced reflective capability are long-recognized student-performance goals of any educational initiative (Dewey, 1933; Garrison,1991; Lipman, 1987; Nickerson, 1989). The introduction of blended learning technologies intothe learning process does not obviate these goals, but may enhance their achievement byincreasing the access students have to diverse perspectives and cognitive resources (Grotzer &Perkins, 2000; Haythornthwaite, 2002; McLoughlin & Oliver, 1999; Paul, 1987) and the timestudents have to engage in guided critical thinking, problem solving, and reflective practice inthe classroom (Bourne, 1998; Grotzer & Perkins, 2000). Enhanced higher-order thinking andreflective capability may be captured through special purpose measures that reflect criticalthinking and problem solving skills or that exercise the process of reflection.

Intermediate-term outcome - Improved individual and unit performance in the field.Highlighted in the LLC Master Plan (March, 2005) is the importance of supporting TRADOC'simperative to "provide Soldiers and leaders who can immediately contribute to unit readiness onthe first day they arrive in their unit" (p. 5). Similarly, the Army must deploy cohesive units thatcan immediately contribute to mission accomplishment the day they arrive in theater. Rapidreadiness is dependent in part on the retention and adaptation of skills and competencies as wellas the swift acquisition of new ones. A key aspect of individual and unit performance in the fieldthat can be influenced by LLCs, then, is just-in-time competency. Swift skill acquisition isenabled by anytime, anywhere access to one's proponent schoolhouse, a defining characteristicof the LLC. The retention and adaptation of skills and competencies that support just-in-timecompetency stems in part from the enhanced cognitive ability reflected in improved studentperformance, as described above, and in part from the adoption of a lifelong learning orientation,as described below.

Intermediate-term outcome - Adoption of lifelong learning orientation. The adoption of alifelong learning orientation is a signal outcome of the LLC initiative and a critical determinantof its perpetuation. The LLC Master Plan (March, 2005) states: "Soldiers must perceive learningas a continuous process that must be refreshed throughout their career." Lifelong learningorientation might therefore be conceptualized as one's distal motivation to engage inopportunities to learn. Distal motivation is one's intent to engage in a particular task, isreflected in the goals one sets for task performance, and determines how one allocates effortacross the range of possible tasks (Kanfer & Ackerman, 1989). A Soldier or leader who intendsto engage in continuous learning will set goals for participation in learning opportunities andprioritize their efforts to target lifelong learning. A person's distal motivation is a function ofthree subjective assessments: (1) the perceived utility of performance (i.e., the desirability/payoffof the chosen goal); (2) the perceived utility of effort (i.e., one's intrinsic need to achieve); and(3) the perceived relation between one's effort and performance (e.g., as indicated inperformance feedback with regard to goal achievement) (Kanfer, 1987). Put another way, peoplewill choose to engage in opportunities to learn if they feel that lifelong learning has certainbenefits, that goal achievement is worthy of effort in and of itself, and that engagement inspecific lifelong learning opportunities will result in goal achievement and payoff.

16

The relation between a blended learning initiative and intent to engage in opportunities tolearn is complicated. A blended learning initiative may not be as dominant as someorganizational characteristics (e.g., promotion policy) in driving the perceived payoff of lifelonglearning. In addition, external factors may play a larger role in determining people's intrinsicneed for achievement than characteristics of the learning environment. However, theeffectiveness of the LLCs may have a strong impact on Solders' and leaders' perception thatengaging in lifelong learning will help them to achieve their professional goals, as opposed tosome other activity. The integration of education and training initiatives on the one hand andpersonnel initiatives on the other may be the most effective way to facilitate the development oflifelong learning orientation (Moran, 2006).

Intermediate-term outcome - Enhanced mission readiness. The LLC OperationalRequirements (December, 2005) state that the LLCs will provide "improved training andeducation support" to the Field Commander and deployed Army. Implicit in this statement is thegoal of achieving enhanced mission readiness, an objective of particular interest to the Army.Readiness may be conceptualized in two ways: (1) as unit status [i.e., U.S. Department of theArmy (DA), 2006]; and (2) as individual and team characteristics that affect unit performance(e.g., McGonigle, Casper, Meiman, Cronin, Cronin, & Harris, 2005). LLCs may have an impacton both of these aspects of readiness over both the intermediate- and long-term.

Unit readiness, as captured by unit status reports, is a function of personnel numbers,equipment availability, equipment readiness, and training status (DA, 2006). LLCs, by enablinganytime, anywhere access to standardized training materials, may be expected to enhance unitstatus reporting--specifically the reason codes associated with sub-optimal personnel andtraining components of unit readiness. Personnel readiness is determined by comparing theavailable strength, the available military occupational specialty qualified (MOSQ) strength, andthe available senior grade strength to the required unit strength. Of particular relevance to LLCsthat provide MOS-qualification training is the available MOSQ personnel percentage componentof personnel readiness. The MOSQ personnel percentage is the ratio of available MOSQpersonnel to required MOSQ personnel. It is reduced when Soldiers cannot attend MOS-qualification training when they need it. A unit's training status is determined jointly by thecommander's estimate of the proportion of the mission-essential task list (METL) that has beentrained to standard and the number of training days required to reach standard on METL coretasks. Training readiness estimates may be influenced by a number of factors, includingpersonnel absences during training exercises because they do not have the required MOS-qualification training or absences due to attending MOS-qualification training at theschoolhouse. Reserve units experience great difficulty reaching optimal training status for thesereasons (Sortor, Lippiatt, Polich, & Crowley, 1994). By making MOS-qualification trainingavailable anytime and anywhere, LLCs may enhance unit status reporting by reducing thenumber of Soldiers who are not qualified for their duty positions or who must travel toparticipate in qualification training.

LLCs also should reduce the time to optimal training status (as reflected in unit statusreports) by allowing more people to complete MOS-qualification training in a timely fashion.This is because anytime, anywhere training enabled by the LLC blended learning environment is

17

not limited by the number of seats available in the schoolhouse (see also Freeman, 2003). Thosewho need MOS-qualification training can receive it just in time, rather than waiting until theschoolhouse has the space to seat them. Having MOS-qualified (MOSQ) personnel available forunit training should reduce the number of days required to reach standard on METL core tasksby improving training quality (Sortor et al., 1994).

Along these lines, LLCs may enhance MOSQ training status, that is, the number ofindividuals receiving MOS-qualification training relative to those who need it. For example,there are a great many more individuals requiring MOSQ and leader training in the reservecomponent than who are able to receive it through residential schooling (Winkler, Shanley,Crowley, Madison, Green, Polich, et al., 1996). LLCs may affect the proportion of individualsboth requiring MOS-qualification training and receiving it by making "virtual" training seatsavailable to anyone. The use of training seats available by the reserve component also isinefficient, but this is likely due to factors outside of the sphere of influence of LLCs (Brown,2002).

Arguments have been made against the adequacy of unit status reporting (e.g., James,2004; Moore, Stockfisch, Goldberg, Holroyd, Hildebrandt, 1991; Sortor et al., 1994; Winkler etal., 1996), in part because unit status reporting is subjective and because it does not capture all ofthe determinants of mission success (e.g., leadership quality, troop morale, and proportion oflow-density/high-demand DMOSQ personnel). In addition, unit status ratings are very difficultto change for a variety of reasons (Moore et al., 1991). To the extent that there is reducedvariance in unit status reports due to subjectivity or reduced external validity due to lack ofcontent validity, the utility of unit status reports as indicators of readiness (and LLC impact onreadiness) will be limited.

Behavioral theories of readiness partially address the challenges associated with unitstatus measures of readiness. The advantage of behavioral theories is that they lend themselveswell to using psychometrically sound measures of readiness. The disadvantage of such theories isthat the link between the identified readiness components and actual mission performance is notfully known'. Components of readiness in the psychological literature include unit cohesion,physical fitness, technical competence/job performance, organizational citizenship behavior(e.g., helping behaviors, punctuality, etc.), preparedness to deploy, and organizationalcommitment/retention (McGonigle et al., 2005). These components are in turn influenced byskill development, job satisfaction, family adaptation, perceived organizational support, andself/collective efficacy (McGonigle et al., 2005).

LLCs may improve readiness by enhancing (affective) organizational commitment.Affective organizational commitment is distinguished from other forms of commitment(normative and continuance) by its emotional characteristic (i.e., organizational members staybecause they want to) and its positive relation to organizational measures of success (e.g.,efficiency; McGonigle et al., 2005). LLCs may influence affective organizational commitmentby buttressing users' perceptions of organizational support. As described in Chao (2006), anyinteraction an individual has with the organization--whether through other members of theorganization, through organizational initiatives, or some other means--acts as an agent of

Note this same criticism has been leveled against unit status reports (e.g., Junor, 2005; Moore et al., 1991).

18

organizational socialization. To the extent that LLCs convey to users that the Army is doing itsbest to support Soldier and leader education (i.e., by meeting learners' needs), affectivecommitment may be increased (Burnam, Meredith, Sherbourne, Valdez, & Vernez, 1992; Gibson& Tremble, 2006). Conversely, if the LLC fails to meet learners' needs or presents a learningenvironment characterized by poor usability and/or significant technical difficulty, affectivecommitment may be decreased as learners become cynical about the Army's ability andmotivation to support their professional development (Burnam et al., 1992; Gibson & Tremble,2006).

Along these lines, LLCs may improve readiness through enhanced socialization inorganizational goals and values. In a five-year study of engineering, management, and otherprofessionals, Chao et al. (1994) demonstrated that study participants who changed organizationswere significantly less socialized in organizational goals and values prior to leaving theiremployers, suggesting a link between this form of socialization and organizational commitment.Moreover, socialization in organizational goals and values had the strongest relation among sixsocialization dimensions (people, politics, history, job proficiency, language, and organizationalgoals and values) to career involvement and job satisfaction and accounted for significantvariance in these variables above and beyond job tenure and organizational tenure. Througheffective instructor facilitation of distance learning and the development of learningcommunities, LLCs contribute to this socialization process. The findings of Sanders (2002) andWellman and Gulia (1999) indicate that distance between learners does not necessarily hamperthe development of social bonds and learning communities. Therefore, to the extent thatinstructor-student, student-student, and student-curriculum interaction using the LLCs reinforcesshared organizational goals and values, increases may be observed in job satisfaction,performance (i.e., job-related problems), and organizational commitment (see also Major,Kozlowski, Chao, & Gardner, 1995).

LLCs also may affect readiness through reductions in work-education-family conflict,however it is unknown how well they will do so. Adams, Jex, and Cunningham (2006) identifiedthree types of work-family conflict: (1) time-based (actual and perceived time available,including separations from family); (2) strain-based (work strain producing family strain); and(3) behavior-based (e.g., role conflict, family versus student). Although anytime, anywhereeducation and training reduces learner travel requirements, the length of time away from familyhas been found to be generally unassociated with various individual readiness factors, such asjob-related problems, lost duty time, absence from alerts or deployments, and organizationalcommitment (Burnam et al., 1992). However, conflicts between the demands of work, education,and family have been identified as determinants of drop out rates from distance education andreduced learning achievement (e.g., Haythornthwaite, 2005; Kreijns et al., 2003; Phelps et al.,1991; Sanders & Guyer, 2001). Research suggests that the negative impact of learning at adistance may be alleviated through the development of learning communities, perceptions thatthe learning environment represents organizational support for its members, and perceptions thatthe time spent on education is necessary (Burnam et al., 1992; Gibson & Tremble, 2006;Haythornthwaite, 2005). Anytime, anywhere education enabled by LLCs might reduce work-education-family conflict; it must not increase it.

19

Long-term outcome - Culture shift. The LLC Master Plan (March 2005) refers to thefollowing quote from the Army Chief of Staff Army Knowledge Management Strategic Plan:"Becoming a knowledge based organization involves more than technologies - it requires deepcultural shifts - from traditional practices to collaboration, teamwork, and innovation; frominformation hoarding to knowledge sharing; from traditional skills to Internet-Agecompetencies." This reference illustrates the primary long-term goal of the lifelong learninginitiative, facilitating this cultural shift through access to advanced, collaborative technologies.The means by which LLCs can have an impact on cultural shift is by enhancing collaborationorientation and fostering internalization of anytime, anywhere learning through repeatedindividual success with anytime, anywhere learning. The success of LLCs in accomplishingthese goals depends on how well the distributed learning environment is leveraged to create thesense of a broad community of learners of which everyone, including instructors, is an importantpart.

Sherry and Wilson (1997) argue that transformative communication, an alternative toinstructor-dominated instruction that is enabled by learning environments with internet access,promotes the internalization of anytime, anywhere learning. Students may find they havesomething to teach the instructor about the technology used in the course and also that they haveeasy access to outside experts (e.g., through websites and online discussion forums). Rather thanthe opposite, students may call the instructor's attention to these valuable learning resources.Through asynchronous and synchronous distributed collaboration, activity in the blendedlearning environment role-models a change in thinking about what a "classroom" is--fromconventional notions of a lecture hall or desk and whiteboard to home offices, cafes, and evenconversations around the water cooler. To the extent that students begin to see just aboutanything as an opportunity to learn, they have internalized anytime, anywhere learning. Alongsimilar lines, blended learning environments--especially those that connect to a wide communityof experts and peers--may enhance collaboration orientation by expanding learners' conceptionof the learner network and their understanding of their role in facilitating the collectiveknowledge acquisition of this network.

Long-term outcome - Enhanced educational cost-effectiveness. Although theestablishment of a blended learning environment is associated with significant capitalinvestment, it potentially presents a more cost-effective training and education solution over thelong-term due to reduced variable costs such as printing and shipping (Whalen & Wright, 1999).Yet, due to the significant expense of technology-assisted education it is not recommended forthe purposes of saving costs, but for expanding academic outreach (Willis, 2000). However, theincreased cost associated with technology procurement, staffing, and housing may be offset bygreater outreach, resulting in a lower cost per student than traditional alternatives. LLCs canachieve this enhanced cost-outreach by providing access to a greater number of students whomay participate in institutional education from home and deployed locations. Increased outreachalso would enhance throughput effectiveness relative to previous conditions as more peoplewho require training and education are able to receive it. LLCs situated at schoolhouses thatmaintain ranges, equipment, and simulations for training may offset costs associated withtechnology procurement by reducing the range equipment/supplies requirements to conducttraining. Finally, LLCs with in-house CBT/WBT courseware production teams, such as the FortLeonard Wood LLC, may enhance CBT/WBT courseware development cost-effectiveness

20

relative to working with outside contractors. Such cost savings were demonstrated in Phelps etal. (1991).

METRICS AND MEASURES FOR EVALUATING LLCS

Two-hundred and twenty-nine metrics and associated measures were developed to reflectthe above-described components of the LLC logic model. A complete listing of these metrics andmeasures is featured in Appendix A. The metric chart is organized according to the logic modelstructure discussed above, with metrics reflecting the status of the LLC on each of the modelcomponents, subcomponents, and elements (e.g., Activities - Technical Staff- Provide technicalsupport). Metrics were designed to enable sampling of users or courses, rather than requiringassessment of an LLC in its entirety. Because the LLCs feature multiple levels of blendedlearning (Graham, 2006) metrics in the chart are associated with a description of the type oflearning environment to which they apply as well as their corresponding measure. Specifying theapplicable learning environment should buttress the generalizability of the chart through anexamination of the level of blending and of the learning environments used in a particular LLC.The metric chart also depicts the pilot LLCs to which each metric applies.

The key design requirements for the metric measures were feasibility and utility. In orderto be used, the measures must not require a great deal of in-depth system analysis or complexdata analyses. Yet, the measures must provide useful information about the metrics that rule out,to the extent possible, alternative interpretations of metric data. Candidate measures featured inthe metrics and measures chart in Appendix A include surveys, interviews, archival datacollection (e.g., from Army personnel databases or schoolhouse financial reports), and some low-level system analysis.

The metrics and measures presented in Appendix A represent an initial attempt to specifythe assessment criteria for evaluating LLCs. Some modification to this chart may be necessary,given that not all metrics and measures were tested in the present investigation. Ideally, all of themetrics in the chart applicable to a particular LLC would be used in an evaluation. This may notbe feasible given time constraints or may not be necessary for well established LLCs.

APPLICATION OF THE LOGIC MODEL AND METRICS TO EVALUATING THEFORT LEAVENWORTH LIFELONG LEARNING INITIATIVE

The LLC logic model and associated metrics were used to conduct a formative evaluationof Fort Leavenworth's LLC, also called the Fort Leavenworth Lifelong Learning Initiative (LVNLLI). The complete description of this evaluation is presented in Appendix B. Data werecollected during the months of September and October 2006 using surveys and interviews,system analysis, and retrieval of archival financial and enrollment data. Data collection largelyfocused on the application of the LVN LLI to Intermediate-Level Education (ILE) conducted inresidence at Fort Leavenworth. Surveys therefore were administered to curriculum developers,instructors, and students directly involved with resident ILE (2006 February-start and 2006August-start classes). Some interviews also were conducted with ILE instructors at satellitecampuses and at Total Army School System (TASS) Battalion sites.

21

Overall, the evaluation results indicated that the LVN LLI is a cost-effective solution forenhancing the educational outreach of ILE curriculum materials. It currently costs approximately$163.14 more per student to deliver ILE using the LVN LLI but the increase in cost due totechnology procurement and associated personnel and facilities expenses is a vanishingly smallfraction of the total amount spent to deliver ILE. Moreover, the relative cost-per-student to theArmy will further decrease as the printing and shipping of course materials to remote sites iscompletely phased out over the next two years. For what amounts practically to the same amountof money, ILE delivered online using the LVN LLI is in the process of eliminating the one- tothree-year lag in curriculum content between the schoolhouse and Army reserve facilities. It alsoalready supports anytime, anywhere learning for ILE students in residence and at satellitecampuses, as reflected in system usage data.

Unfortunately, there was relatively little the LVN LLI could do to reduce the 6-10 monthlag between changes in the operational environment and revisions to the standardized ILEcurriculum due to the lengthy institutional curriculum review, revision, and vetting process.However, the LVN LLI may assist in circumventing this problem by making it easier forinstructors to supplement curriculum materials with up-to-date articles, professional discussions,and emerging doctrine. Twenty-nine percent of ILE instructors surveyed reported that theysupplemented the curriculum materials more frequently than in pre-LVN LLI conditions. Thisproportion is likely to increase as faculty become more facile with Blackboard. In addition,substantial proportions of students (>85%), instructors (74%), and course authors (80%) reportedthat the ILE course content was relevant to the jobs of ILE graduates.

There did appear to be some challenges to the full-scale adoption of the LVN LLI bycourse authors, instructors, and students in the schoolhouse. Although the rates of activeadoption (i.e., primary use of the system components) increased during the first-year pilot, theyremain below 100%. A combination of survey and interview data indicated that technicaldifficulty was not the main barrier to system use. First, a weighted average of student, courseauthor, and instructor survey responses indicated that the LVN LLI technical support wasaccessible (62%) and useful (66%). Moreover, the system workarounds they reported usinginvolved typically some other form of technology. For example, SharePoint was the mostfrequently cited alternative to Blackboard when posting or accessing curriculum materials.

The main barriers to system use appear to have been (a) ready availability of alternativesto the system that were more familiar and easier to access; (b) inconsistent or absent informationmanagement procedures necessary to make posted content easy to find; and (c) lack of instructorand course author involvement in and buy-in to the lifelong learning concept. As has been foundin previous evaluations of this kind (Cianciolo, Heiden, & Prevou, 2006), greater success hasbeen achieved in the technical implementation of the initiative than in the cultivation ofstakeholder enthusiasm and investment. When asked what they thought the Army's main reasonfor implementing Blackboard and Sharepoint was, a minority of students surveyed (33%)responded that the Army had made the decision in the best interests of leader education.Although the Army surely had multiple reasons for adopting the new instructional technologies--including enhancing leader education--actively using the LVN LLI to foster students' perceptionof organizational support for leaders would make it a more effective asset in enhancingorganizational commitment and cultural shift.

22

In any case, challenges to the active adoption of the LVN LLI did not hamper theacademic experience of ILE students and instructors in the first-year pilot. A very highproportion of surveyed students (79% on average) reported that ILE instructors generallydemonstrated the classroom facilitation behaviors recognized as critical to developing adultlearners. In addition, survey responses indicated that a majority of students (a) tookresponsibility for their own learning via independent study (70-98%); (b) reported high learningself-efficacy (88%); and (c) were motivated to learn the topics taught in ILE (91%). While fifty-eight percent of instructors reported that the LVN LLI did not have an impact on classroomefficiency a small minority of instructors (10%) reported that the LVN LLI enhanced theirclassroom efficiency. Although it may seem insignificant, this result is noteworthy given the factthat the introduction of instructional technology often is associated with increased workload forfaculty (Willis, 2000).

In order for the already successful LVN LLI to achieve optimal educational andorganizational impact, stronger beliefs in its purpose and higher active adoption rates of itscomponents should be achieved. The accomplishment of this goal requires fostering theinstructor buy-in necessary to learn and adopt unfamiliar technologies and to role-model theiruse for students. Improved buy-in may be developed through (a) engaging other schoolhousecomponents, especially faculty and staff development, in viewing and cultivating LVN LLI-assisted teaching as a critical instructor competency; (b) encouraging and shepherding theinvolvement of instructors and course authors in the development of information managementprocedures through needs assessment and iterative system design; and (c) spreading strategiccommunications that explain the purpose and goals of the system, that anticipate technicallimitations of the system (i.e., login requirements), and that immediately follow system outages.

CONCLUSION

The present research represents the first attempt to integrate the widely diffuse literatureon program evaluation, blended learning, organizational effectiveness, and military readiness inorder to conduct an evaluation of an Army technology-assisted learning initiative. The basis ofthe LLC assessment framework in theory makes it generalizable not only across current andfuture LLCs, but also across blended learning initiatives more generally, addressing a gap in thescholarly literature with regard to the effectiveness assessment of educational technology.Review of the assessment framework by key stakeholders from each of the pilot LLCs ensuredgeneralizability across the different initiatives. The assessment framework developed as part ofthis investigation proved to be a feasible and useful tool for capturing and representing thefunctioning of blended learning initiatives such as LLCs, from the acquisition and use ofresources to the achievement of organizational impact.

This research is not without its limitations, however. First, strategies for evaluatingeducational effectiveness have historically assumed that institutional goals for education arealigned both with student learning objectives and with those learning objectives that must be metto further society or organizational performance. The design of the LLC assessment frameworkis consistent with this assumption. That is, outcome measures reflecting the alignment (i.e.,content coverage) between what is taught via LLCs and what must be taught in order to advance

23

student's expected achievement and organizational excellence were not included in theframework. This alignment is critical to achieving organizational impact via an educationalinitiative (see e.g., Cianciolo et al., 2006), and should be included in a comprehensive picture ofLLC effectiveness. Second, the limited scope of the investigation prevented an explicit test of thevalidity of the assessment framework developed. Additional research should determine whetherthe framework (a) applies as expected to other LLCs, especially those with curricula that differfrom Intermediate-Level Education; (b) produces the same results using quasi-experimentalevaluation, where possible; and (c) accurately predicts the future status of an LLC based oninitial evaluation and implementation of recommendations.

The findings of the Fort Leavenworth evaluation indicated the importance of taking acausal approach where quasi-experimental design was not possible. Assessment of outcomesalone would have indicated that the initiative had achieved its goals but would have obscured thefact that a subset of these goals--teaching and learning effectiveness--was achieved largelyindependently of the use of blended learning technologies. A causal assessment framework suchas the one researched and developed in the present investigation verified whether the apparentsuccess of an LLC was truly driven by the activities and outputs generated by the initiative. Italso provided clear recommendations for bridging gaps between resources, personnel activities,system outputs, and outcomes in order to enhance impact. Future LLCs will achieve the greatestsuccess by engaging all stakeholders in the design and implementation process in order to winthe buy-in of instructors, other staff, and learners. The enhanced functioning of the initiativeresulting from stakeholder engagement will enable the LLC to serve as a powerful driver oflifelong learning and organizational change.

24

REFERENCES

Abell, M. (2000). Soldiers as distance learners: What army trainers need to know. Proceedings ofthe Interservice/Industry Training, Education, & Simulation Conference (I/ITSEC). Orlando, FL.

Abell, M. (2003). Deepening distributed learning: Motivating Soldiers to learn, grow, achieve.Proceedings of the Interservice/Industry Training, Education, & Simulation Conference(/ITSEC). Orlando, FL.

Adams, G. A., Jex, S. M., & Cunningham, C. J. L. (2006). Work-family conflict among militarypersonnel. In C. A. Castro, A. B. Adler, & T. W. Britt (Eds.), Military life: The psychology ofserving in peace and combat (Volume 3, pp. 169-192). Westport, CT: Praeger SecurityInternational.

American Distance Education Consortium. (2000). ADEC Guiding principles fordistance teaching and learning. Available: http://www.adec.edu/admin/papers/distanceteaching_principles.html.

Barry, M., & Runyan, G. (1995). A review of distance-learning studies in the military. AmericanJournal of Distance Education, 9(3), 37-47.

Belanger, F., & Jordan, D. H. (2000). Evaluation and implementation of distance learning:Technologies, tools, and techniques. Hershey, PA: Idea Group Publishing.

Bonk, C. J., & Cummings, J. A. (1998). A dozen recommendations for placing the student at thecentre of web-based learning. Educational Media International, 35(2), 82-89.Bralley, N. (2006).ILE: A new system for CGSC students. Army Logistician, 38(1).

Bonk, C. J., & Reynolds, T. H. (1997). Learner-centered web instruction for higher-orderthinking, teamwork, and apprenticeship. In B. H. Khan (Ed.), Web-based instruction (pp. 167-178). Englewood Cliffs: Educational Technology Publications.

Bourne, J. R. (1998). Net-learning: Strategies for on-campus and off-campus network-enabledlearning. Journal of Asynchronous Learning Networks, 2(2), 70-88.

Bralley, N., Danley, J., French, D., Soby, C., & Tiberi, P. (2003, July 18). UnderstandingIntermediate Level Education: How is it different from the former Command and General StaffOfficer Course? TRADOC News Service. Retrieved January 26, 2006 from the World Wide Web.Available at: http://www.monroe.armv.mil/pao/TNSarchives/July03/ILE.htm.

Brown, S. (2002). Army reserve training seat allocation model. Unpublished Master's thesis.Naval Postgraduate School, Monterey, CA.

Burnam, M. A., Meredith, L. S., Sherbourne, C. D., Valdez, R. B., & Vernez, G. (1992). Armyfamilies and Soldier readiness. Santa Monica, CA: RAND Corporation.

25

Champagne, M. V., & Wisher, R. A. (2000). Design considerations for distance learningevaluations. In K. Mantyla (Ed.), The 2000/2001 ASTD distance learning yearbook (pp. 261-286). New York: McGraw-Hill.

Chao, G. T. (2006). Organizational socialization: Leaders and developmental networks for workadjustment. White paper presented at the U.S. Army Science of Learning Workshop, Hampton,VA.

Chao, G. T., O'Leary-Kelly, A. M., Wolf, S., Klein, H. J., & Gardner, P. D. (1994).Organizational socialization: Its content and consequences. Journal of Applied Psychology,79(5), 730-743.

Cheung, D. (1998). Developing a student evaluation instrument for distance teaching. DistanceEducation, 19(1), 23-43.

Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduateeducation. Retrieved June 9, 2006 from the World Wide Web. Available athttp://honolulu.hawaii.edu/intranet/committees/FacDevCom/guidebk/teachtip/7princip.htm.

Cianciolo, A. T., Heiden, C., & Prevou, M. (2006). Assessing Army professionalforums: Metricsfor effectiveness and impact (Technical report in publication). Alexandria, VA: U.S. ArmyResearch Institute for the Behavioral and Social Sciences.

Cianciolo, A. T., & Prevou, M. (2006). Effects-based KAM metrics: Defining impact in terms ofoutcome instead of activity. Seminar presented at the e-Gov Conference. Washington, DC.

Clark, R. E. (1994). Assessment of distance learning technology. In E. L. Baker & H. F. O'Neil,Jr. (Eds.), Technology assessment in education and training (pp. 63-78). Hillsdale, NJ: Erlbaum.

Cornell, R., & Martin, B. L. (1997). The role of motivation in web-based instruction. In B. H.Kahn (Ed.), Web-based instruction (93-100). Englewood Cliffs, NJ: Educational TechnologyPublications.

Cukier, J. (1997). Cost-benefit analysis of telelearning: Developing a methodology framework.Distance Education, 18(1), 137-152.

Dean, R., Biner, P., & Coenen, M. (1996). Distance education effectiveness: A systems approachto assessing the effectiveness of distance education. Education at a Distance Journal, 10(3), J 17-J20.

Defense Technical Information Center. (2000). Distance learning failure factors. In K. Mantyla(Ed.), The 2000/2001 ASTD distance learning yearbook (pp. 49-50). New York: McGraw-Hill.

Dewey, J. (1933). How we think: A restatement of the relation of reflective thinking to theeducative process. Boston: Heath.

26

Ehrmann, S. (1994). Project "Flashlight"planning grant: Final report (Report No. HE030726).Washington, DC: Corporation for Public Broadcasting.

Firdyiwek, Y. (1999). Web-based courseware tools: Where is the pedagogy? EducationalTechnology, 39(1), 29-34.

Freeman, M. W. (2003). Distance learning in the U.S. Army: Meeting the readiness needs oftransformation. In M. G. Moore, & W. G. Anderson (Eds.), Handbook of distance education (pp.655-661). Mahwah, NJ: Erlbaum.

Garrison, D. R. (1991). Critical thinking and adult education: A conceptual model for developingcritical thinking in adult learners. International Journal of Lifelong Learners, 10(4), 287-303.

Gibson, C. C. (1996). Toward an understanding of academic self-concept in distance education.The American Journal of Distance Education, 10(1), 23-36.

Gibson, J. L., & Tremble, T. R. (2006). Influences of work-life support of officers'organizational commitment and negative work-family spillover (Research Note 2006-02).Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Gifford, L. J. (1998). Graduate students 'perceptions of time spent in taking a course by internetvs. taking a course in a regular classroom. Paper presented at the Mid-South EducationalResearch Association Annual Meeting. New Orleans, LA.

Graham, C. R. (2006). Blended learning systems: Definition, current trends, and futuredirections. In C. J. Bonk & C. R. Graham (Eds.), Handbook of blended learning: Globalperspectives, local designs (pp. 3-21). San Francisco: Pfeiffer Publishing.

Grotzer, T. A., & Perkins, D. N. (2000). Teaching intelligence. In R. J. Sternberg (Ed.),Handbook of intelligence (pp. 492-515). New York: Cambridge University Press.

Hannafin, M. J., & Land, S. M. (1997). The foundations and assumptions of technology-enhanced student-centered learning environment. Instructional Science, 25, 167-202.

Harrison, P. J., Seeman, B. J., Behm, R., Saba, F., Molise, G., & Williams, M. D. (1991).Development of a distance education assessment instrument. Educational Technology Research& Development, 39(4), 65-77.

Hays, R. T., Stout, R. J., & Ryan-Jones, D. L. (2005). Quality evaluation toolfor computer- andweb-delivered instruction (Technical Report No. 2005-002). Orlando, FL: Naval Air WarfareCenter Training Systems Division.

Haythornthwaite, C. (2002). Building social networks via computer networks: Creating andsustaining distributed learning communities. In K. A. Renninger & W. Shumar (Eds.), Buildingvirtual communities (pp. 159-190). New York: Cambridge University Press.

27

Haythornthwaite, C. (2005). Social network methods and measures for examining e-learning. E-learning seminar, University of Southampton. Retrieved August 17, 2006 from the World WideWeb. Available at:http://www.wun.ac.uk/elearning/seminars/seminars/seminar two/seminartwo.html.

Herrington, A., Herrington, J., Oliver, R., Stoney, S., & Willis, J. (2001). Quality guidelines foronline courses: The development of an instrument to audit online units. In G. Kennedy, M.Keppell, C. McNaught, & T. Petrovic (Eds.), Meeting at the crossroads: Proceedings ofASCILITE 2001 (pp. 263-270). Melbourne: The University of Melbourne.

Hiltz, S. R. (1990). Evaluating the virtual classroom. In L. M. Harasim (Ed.), Online education:Perspectives on a new environment (pp. 133-183). New York: Praeger.

Hiltz, S. R. (1998). Collaborative learning in asynchronous learning networks: Buildinglearning communities. Invited Address at WEB98, Orlando, Florida. Retrieved from the WorldWide Web on March 18, 2004. Available at:http://eies.njit.edu/-hiltz/collaborative learning in asynch.htm

Hyatt, J. A. (1983). A cost accounting handbook for colleges and universities. Washington, DC:National Association of College and University Business Officers.

James, G. B. (2004). Reserve component readiness assessment methodologies: Is there a betterway? Monograph of the School of Advanced Military Studies, U.S. Army Command andGeneral Staff College, Fort Leavenworth, Kansas.

Junor, L. J. (2005). The defense readiness reporting system: A new took for force management.Joint Force Quarterly, 39, 30-33.

Kanfer, R. (1987). Task-specific motivation: An integrative approach to issues of measurement,mechanisms, processes, and determinants. Journal of Social & Clinical Psychology, 5(2), 237-264.

Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: Anintegrative/aptitude-treatment interaction approach to skill acquisition. Journal of AppliedPsychology, 74(4), 657-690.

Keene, D., & Cary, J. (1992). Effectiveness of distance education approach to U.S. Army reservecomponent training. In M. G. Moore (Ed.) ACSDE Research Monograph No. 3 (pp. 97-103).University Park, PA: The Pennsylvania State University, American Center for the Study ofDistance Education.

Kirkpatrick, D. L. (1994). Evaluating training programs: The four levels. San Francisco: Berrett-Koehler.

28

Kreijns, K., Kirschner, P. A., Jochems, W. (2003). Identifying the pitfalls for social interaction incomputer-supported collaborative learning environments: A review of the research. Computersin Human Behavior, 19(3), 335-353.

Levin, H. M. (1986). The economics of computer-assisted instruction. Peabody Journal of

Education, 66, 52-66.

Lipman, M. (1987). Some thoughts on the foundations of reflective education. In J. B. Baron &

R. J. Sternberg (Eds.) Teaching thinking skills: Theory and practice (pp. 151-161). New York:W. H. Freeman.

LLC Executive Agent. LLC Master Plan. (March, 2005). Unpublished document.

LLC Executive Agent. LLC Operational Requirements. (December, 2005). Unpublisheddocument.

Lockee, B. B., Burton, J. K., & Cross, L. H. (1999). No comparison: Distance education finds a

new use for "No Significant Differences." Proceedings of selected research and development

papers presented at the National Convention of the Association for Educational Communicationsand Technology. Houston, TX.

Major, D. A., Kozlowski, S. W. J., Chao, G. T., & Gardner, P. D. (1995). A longitudinalinvestigation of newcomer expectations, early socialization outcomes, and the moderating effects

of role development factors. Journal of Applied Psychology, 80(3), 418-431.

Mason, R. (1991). Moderating educational computer conferencing. DEOSNEWS, 1(19), 1-11.

McGonigle, T. P., Casper, W. J., Meiman, E. P., Cronin, C. B., Cronin, B. E., & Harris, R. R.

(2005). The relationship between personnel support programs and readiness: A model to guidefuture research. Military Psychology, 17(1), 25-39.

McIsaac, M. S., Blocher, J. M., Mahes, V., & Vrasidas, C. (1999). Student and teacherperceptions of interaction online computer-mediated communication. Educational MediaInternational, 36(2), 121-131.

Middaugh, M. F. (Ed.). (2000). Analyzing costs in higher education: What institutionalresearchers need to know. San Francisco: Jossey-Bass.

McLaughlin, J. A., & Jordan, G. B. (2004). Using logic models. In J. S. Wholey, H. P. Hatry, &K. E. Newcomer (Eds.), Handbook ofpractical program evaluation (2nd ed, pp. 7-32). SanFrancisco: Jossey-Bass.

McLoughlin, C., & Oliver, R. (1999). Pedagogical roles and dynamics in telematicsenvironments. In M. Selinger & J. Pearson (Eds.), Telematics in education: Trends and issues(pp. 32-50). Amsterdam, The Netherlands: Pergamon.

29

McMillan, D. W., & Chavis, D. M. (1986). Sense of community: A definition and theory.Journal of Community Psychology, 14, 6-23.

Milam, J. (n.d.). Cost analysis of online courses. Retrieved May 30, 2006 from the World WideWeb. Available at h1ttp://www.airweb.org/links/reports/costanalysis.html.

Moller, L. (1998). Designing communities of learners for asynchronous distance education.Educational Technology Research & Development, 46(4), 115-122.

Moore, S. C., Stockfisch, J. A., Goldberg, M. S., Holroyd, S. M., Hildebrandt, G. G. (1991).Measuring military readiness and sustainability. Santa Monica, CA: RAND Corporation.

Moran, K. (2006). Sea Warrior and the revolution in training: The right person, right place,right skill, right time, best value. Presented at the U.S. Army Science of Learning Workshop.Hampton, VA.

Morgan, B. M. (2000). Is distance learning worth it? Helping to determine the costs of onlinecourses. Retrieved June 15, 2006 from the World Wide Web. Available athttp://www.marshall.edu/distance/distanceleaming.pdf.

Nickerson, R. S. (1989). On improving thinking through instruction. Review of Research inEducation, 15, 3-57.

O'Donnell, K. (2005). National household education surveys program 2003: Tabular summaryof adult education for work-related reasons 2002-03. Washington, DC: National Center forEducation Statistics.

Oliver, R., & McLoughlin, C. (1999). Curriculum and learning-resources issues arising from theuse of web-based course support systems. International Journal of EducationalTelecommunications, 5(4), 419-436.

Owston, R. D. (1997). The World Wide Web: A technology to enhance teaching and learning?Educational Researcher, 26(2), 27-33.

Paul, R. W. (1987). Dialogical thinking: Critical thought essential to the acquisition of rationalknowledge and passions. In J. B. Baron & R. J. Sternberg (Eds.) Teaching thinking skills: theoryand practice (pp. 127-148). New York: W. H. Freeman.

Phelps, R. H., Ashworth, Jr., R. L., & Hahn, H. A. (1991). Cost and effectiveness of home studyusing asynchronous conferencing for reserve component training (Technical Report No. 1602).Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Russell, T. L. (1999). The no significant difference phenomenon. Raleigh, NC: North CarolinaState University.

30

Salas, E., Rhodenizer, L., & Bowers, C. A. (2000). The design and delivery of crew resourcemanagement training: Exploiting available resources. Human Factors, 42(3), 490-511.

Sanders, W. R. (2002). Collective staff training in a virtual learning environment (ResearchReport No. 1788). Washington, DC: U.S. Army Research Institute of the Behavioral & SocialSciences.

Sanders, W. R., & Burnside, B. L. (2001). Assessment of initial delivery of the Armor Captains'Career Course (Distance Learning) (Research Report No. 1775). Alexandria, VA: U.S. ArmyResearch Institute for the Behavioral and Social Sciences.

Sanders, W. R., & Guyer, C. W. (2001). Commanders' Survey: Armor Captains' Career Course(Distance Learning) (Research Report No. 1771). Alexandria, VA: U.S. Army Research Institutefor the Behavioral and Social Sciences.

Sherry, L., & Wilson, B. (1997). Transformative communication as a stimulus to webinnovations. In B. H. Kahn (Ed.), Web-based instruction (67-73). Englewood Cliffs, NJ:Educational Technology Publications.

Sims, R., Dobbs, G., & Hand, T. (2002). Enhancing quality in online learning: Scaffoldingplanning and design through proactive evaluation. Distance Education, 23(2), 135-148.

Sortor, R. E., Lippiatt, T. F., Polich, J. M., & Crowley, J. C. (1994). Training readiness in theArmy reserve components. Santa Monica, CA: RAND.

Thach, L. (1993). Exploring the role of the deliverer in distance education. International Journalof Instructional Media, 20(4), 289-307.

University of Illinois Faculty Seminar. (1999). Teaching at an Internet distance: The pedagogyof online teaching and learning. Retrieved May 8, 2006 from the World Wide Web. Available at:http://www.vpaa.uillinois.edu/tid/report.

U.S. Army Training and Doctrine Command. (2004). The growth and future of lifelong learning.Retrieved February 18, 2006 from the World Wide Web. Available athttp://www.tradoc.anny.mil/pao/web specials/lifelong learning/lllgrowth.htm.

U.S. Department of the Army. (2006, March). Unit status reporting. Washington, DC.

Wellman, B., Carrington, P. J., & Hall, A. (1988). Networks as personal communities. In B.Wellman & S. D. Berkowitz (Eds.), Social structures: A network approach (pp. 130-184). NewYork: Cambridge University Press.

Wellman, B., & Gulia, M. (1999). Virtual communities as communities: Net surfers don't ridealone. In M. A. Smith & P. Kollock (Eds.), Communities in cyberspace (pp. 168-194). London:Routledge.

31

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities ofpractice.Boston, MA: Harvard Business School Press.

Whalen, T. & Wright, D. (1999). Methodology for cost-benefit analysis of Web-based tele-learning: Case study of the Bell Online Institute. The American Journal of Distance Education,13(1), 25-43.

Willis, B. (2000). Effective distance education planning: lessons learned. In K. Mantyla (Ed.),The 2000/2001 ASTD distance learning yearbook (pp. 43-48). New York: McGraw-Hill.

Winkler, J. D., Shanley, M. G., Crowley, J. C., Madison, R. A., Green, D., Polich, J. M., et al.(1996). Assessing the performance of the Army reserve components school system. SantaMonica, CA: RAND.

32

F-4

UU

odb0c~ -. :A

0 C

0d 0C) C-z r.. C

0 0j

2 0. 0.

w cl +.L W 0L -00Z e

C. Cj

> 2. > VC.ýCK : CO C . 0 C -

Ct 0-te 0. X -ýCU ( -ct

0 0t c

0U, U, 00 t ''

WD 0i Q >

r.II= &0> 00 c

f" 3- d4 .- - -

5 5 :.- Q E 0 QEd:3 0 C'

z,~ C'

-3 q).T

U0 di 0 c ~ C

4 ~.crj 0 ~ ~ ~ 0 I-cr, U

PC4 4.~ Q4.C

CCE

CL r.

Qx

03 m

C' aI-.o1 CO-C

*4~~~- - -4*I.. L) f)

w C 6)

ca v

C.) (1) V1~U> 0 t-4U

in cc Z~

>1 E C,

L) C.) - C .o 7: 0

.~C, M~- -0 m) V~ 0. o C) . .

0 00

C) > o u l

.0

-C ThC Iu' *4C."-o

0cc *EnO4 QL . 04 -c 00n -0 -U 0.- -

L) 0

a'n 0U >

0 U) ;>

Q o~ r.

0~~ r.wr._> q lca a0 CZcn- ",jo cnM c

0~

- d 0 o.v 0

0. J-

W- C, to 0g0 0 to ~ to c -

"o 0 0 0 o :14~ -~d I-

-z 00 C-'s-

72 > 0z a )C)'I 0 Q )c

OJ >d > In . o ""F-

0

- d 0

o E0 did-S

lei~

(D

0-~

0.~ ~ 0 0~

.2 u~ 2 U -

0 -0

00

0 ctj4-

4)u

0 .0 E0

C) Z: 0. v -

0 0-

*0

I- w- -') F! g- 4

- z w- -0 -- . 0

C),

wO :J~ 0 0. r.~' co 00o :;s r

co~ 00.4 r-A .

EnM- ,; Q cjz 0+. 00 (A co 0

rnV L) 3 CL

x X x

+ 0C

0 4

- 0U)-O~ 0 4 00 ~ ~ u -0 b

r. Z5 a)) 'C 0 .0 rl 0 0 g 0 , oV

co0 0 ¶' 00

CO4 U) ~ - ) CA 0

co 0 > o0

> (L

S3 'o 4) Q) a) a,) *

0 0) . .

(D)0 ~ 0 ~ 40, 0 CIS0 U 's - W- 0 f .. 2 -d fl > -0U)r

C)~ 0 u a

0 Lt

-, 0 w0El L.0

z 00 U)U z o

0' 0 B U~ 2 'A-. 2 co)* 0--

u))' 0 0 0 0 (4

0a 0

00 C')

C''s

W 40

04) UV_ 0__ ___ __ U__0 _ ___ _T 0 0 - C.

5-

18

::s. M 0c

0 cn

co 0)

t~ -' '04j=- tý 4

0D m 4 c 0 0N0

W

0 C 5- c '

- E C/) 0 ~ -0 m V) oE w 4

444 .

C-5- - 0l 0- -- cz0)-4 bz r,'R - ;-"

Ctt~0. 0. Crr 04Jz

uo ' ( "l) 10 l .) L

0d ce -o -4 >

Q)' Z4 4 - 0 V) 01 0 "0-C )5 -

0 ~ ~ ~ ~ -0w m "a. l

0. 0 0 0 '0 ~ -0 m0 m dc

"ay.. 'R5"L)~~ -0b

04 a) -C34 0.4,5 ou (D C. co

;: > t-ca c Q) (44m5 0 4$ c 0o 4~C 4C3 ' *j .- --

a) --( 4 ) 5 - .0 r. rt. "C co '5 0' CS U4 '

w 4)C.) V 0.) .-0

codo ~ 'oco 0 m 0

0) > co

W 0 0

5- C.) 0o ", >,c lW) 0-- 5-0 5-

co 00 ,.s.. a)0~ .0

0 o "2

ux

X X~

C) CýC', ~~I C)C 4

C:) C C) C

cC:., + 04.) ODG 4> U

-~ 0 0 s 0 odI-

& -0 .. D 0 0C

0 0 ~ -

m~3- U 0 Mb 0 0a

0 2d 2

0~- x2 00uýC 0*0. 010 u o 'd 0

.14 -- .1- v) 0 4 Q 0*-0. &Z - C-CJ 0 to~ 0 0 a t 0 t

v0)

0z 0)*9>0)0v 0)0> M

00 0.o d) 4 0U co cc 0

0D 0 0> C0 >

0 0) .) m '!c

03 QJ w 0

o_ oL~0 ____0__CO

0-4

o u

-C, -

,Q.03

~Ca. 00 UCf'' 0 vi'

0 0)

" z:0 0 C-0 '-y 0 -;

, - o C', Q

Z; - 0 V)- ~ 0

co ?ý a

C,0 > -c0l 0-1 0 ~ 4c0 *

aj 0 -0~-~~

I- Cz cl

"0C) -. :,=

C.~~0 "QO 20

C.. 0 L Q

-o0Z XC C) Q) o 0, Q)-~ a) a

C14 -Q Ei e 8 Z Cd) M 0000 nM M ) 04 tk

o ) w0o(: -D C',

0 C

oC~ 00 C,

M_ 0w~ Cd OCtý 0

Cl

UD~

4 Q

POCC)u

=,C

cl P

C)zu

0 o -

P) 4 -'a) C

0 0 0u~) C

U) 0~c 0 > Q 1. 0

"I U

cu ci z 1Wu 6. ,, Elz c- , )

E ~ 4'~~ C C)C c)c-~z PO 0 o ~ ~ .C C

C) L) cC)4 C

-' 0 0 0 0 4m cn = oct ~ .- ~ - - - - -0$

0 a) V3) u

A 0 C

C's 0t') C,3 t) U

b 4) - '0 WWC) 4"~ "0

0- (D4 Cu 0,C)-4 ;.0CJ 0 0 0'Ac

U) C) u-C In~u

"K 0 0 0 C

CK ~ 0 "0o Cd Cý -;3 , C . u.I

0 =-

CZ0)- w~Q~ 0-

a-ý >, - -w-

CuC . 0 ~C u- 0 C)En 43'0

O0-C OCd vO4-.a)C" Oz O C A

xx

o nzda) 4)U,) a)PP-

*0 00 '0 F IV)a CA V C/u co

7ý 0- C; - C 0 P ,0

'C>JC >C >7 >7 >70

a) C' a

-~. (1) ) d

0.M' 'CCl 0. WCC C', CC o

m. U) 0 ACE 0M0.) 0 M. 0 0C-4

a) -z m)a)--U U - Z --- .- -t C

C0 -. a)0t)ja -- *JG

~~Ca)LO 0' ~~

m~ UO ) C-cr C0

0 M C)r 0 C

v)~a CC:) L) In 0F

w~ Cj kn 0~ .,. '0 0 ~ a ' )~

bi)Ca u_0 0C -0 C)Ca 0 aý0.) ,r.4(L) or - C)C'

z ~ f ,~ ~ Cl Cn 2 :ý0 C3 C

--

) U)) CI U

V ) En U) U) U) mJ U

0) 0) L

00.)0

ID)

* U)U) U U) ) U U3~~~-..~~c CcC)C)C) . . .

0) 4) >.22 0 0d

00 0 0 0 0 -

o 4 0 0p 0 0 0

.d mU) U)Lt >1 0 0 0)d-

W 0 CO n a 0 00 0) ~0 ) c

""0 -U,'0 2E

-U) co U)0

u o

5~.) C CJ. '50-

0'Q0) ) >~

.)~~0.)0 co co 0~-

Wo 0-

0 0 R

v) V)) w-) V))

0. 0 a

C) ( (2 .2 i~i~ hi

ro

(2 (0 (L) a3 1-

Q)) Rhia

0 -E o 4Cl. =h 0 'h0Ch3 -02 o 7:1 zTC3

~~~- 0.) .- C',()~CUC

0 ~0u 0 0~ "0 -0 (D

E(l 6Q r

7a 73 C) 7) -zý cd En

hi) WC

CUj

0(2CULn404

CC

40 - '4

40

r. 0

42' tc)) o. C. C) .

C)0 -0 4. - 0

= (L v C

W a). C) z o . c - )

IV .0 w . >, o V, 4

>, C) ýGO C3 0.-. ')

04~ 4- s0 :: 3c

4 -. 'CU L.0 >>)o C

(1C)

cl 0...*-.4C

C.WDC. .'Us. C.

C.) . 2

clCX,

-e

0 cl 0

0)0 Z 0~

~~ >.EO 'n m En E

En c' ,o o E n cCZ Q) a c a a a

C Cr--.

0 0

- 03

- 4))

-0p t00EV" r

0) , - C C

4) (1 r

o - ~ co V~ u

> 0 C

0U 00 ~ 4)

C', CZ 0

O~u>

S0 0 -048

0l

al +

02 w m9z 0- ti - .i

Cd 0

w al '00~a 00 cn.-~6

0 0 "~~u -

Cd W

.u 0 (L 0 ýO

aj~ cri

0 ~-c~ S o~ i-

' 0 0 . 0 0 ~ - ' C do -

W ca Uco ~ ~ . 0 0 zýp-

- , ' 4 Z.2i0 ) a

Qu " 0ri) m )W -"

ctw -

OUUQ

ci

o d +

r r

u c cl U U -uo ý oDo

rA 0

*j 0 0 .C )

a- .0

Q0o Z ~ Z 00

0. 0 I

0 0co

-~x 0.

u 0

2. cja-'0n'0

oJ 0 0 C0',

to 0J 0.'O0,.,o

-0,0n

0 v9 Eo >' Z

C) o.J 00.

C) tb0 C, c ,S

on SC)c.55 _ )

,2 CCs____ __ ____ ___ I __ ___ ___ ____ ___ ____ ___ __0

r.mx 0 V)

CA 0

Q V~C, 81 0 0)

W r. V) 0,

cz- m ~ 0J C,3

00 Q c.mc

0 v 'V) In. - -U- ( Q E c n a

0

0~0~

0 0 to) to)

4) m

'0

,o < *--)

0 4)

0 ) p

m 0 00 >)

(~0 w) 'A r- ()cq 0 00* 0 uctl.

au) 04 t2 000~ tzU 0 0 0'

04

gcn 0 "4"z ýl u( 2

ýt 0

O CQ ± o +

V 0 l 4 t

;00 .- I-. Q

0o

0 0 o Z : 00

(0 .- - .$.,. ri 'o-'o 0

() tnCO Mr

L) 0

I- 4, r- m () .r

LO) V)

r -. = vi

El aa t u

0 0''4 M

o~ -a0 -x

4x, >,,;, n

V)g cc Eý 0 'o"

cd -C,

60

0.) -o Q, -

0 0-4 u0 U

(CO

ol co'

cl-

U0

o(/to 0'o00 C(3

00

CC c a

(L - En> as 0 E

ca0 3-0C) (1 (C.4 r4 -

3-~r 0C -o 80 U.~U 9w- o. 40.C 0 -P.

o0C

0CII CU

= 0

0/ 0~ -l 4

H

(00)

o+ + "

8- z. o

_ CKS M. w-t

ua V)Q)- co ca _ zC ,

0~~~~ ~ ~ -t / )E )o

z -

UU

U0 Ff Z

z ~o-0a~a.

bAYCOO

- 0

-O) Ca.

m 0&..0 . u > CJ C)

"(j E a) E Q0~ _ a)

C' C)

> 0)

caU

a) W, C) i> p U~

x X X

C0

0 0 r

as. 0

oc00 2 0~ 0 0D 0 .- D

Is -ol w CU 0 CUal 0 ,

CU 0- 0C C r. ' U 4Z~C 0C- -U co cbUz

4Z. ZS M CU 0cz

C.) C/) 0.-- Q-) C -2 c)

0

>U 0 u - 0S2 0 0 0 00

(on C/)

0 toQ 0 0 = 0t' - >-(

CZ--

C)C)En

'nae -6 ": .) -

40 .! r 9C

aýC)- 0 ja,0E

-L)

rA ~-

C)' _ UC~~ U ~~-~---OC0n Q~ I-*0 CC n~

x Xx

a) a)

w a) 3 d) 0) 0 ctia) CA m 00 0 )

w Cd0

-z -C; (Do. '

o Q)~ CI a) V ~ >

co 0a It A0 ma '

C/ m ' C/) a ) 0 cD V)~ -d

0 a) 2 a)

01 V) a) Z~

U),a E cnMW 0~C >- 0 LI 00 .2~CC a

a) a) a) a) 0 L a

o- a) 3- a) X3r::s 0 :3 C.- . 0 0 0 t

4)* < )U ~ ) C U S0 E-ý (L- r

- C aa))H -.V ) 0 W2 0 ) In 0*a)

u0 z ý o z o' 0 0 a

03 C- -- )

a) -0 0 co- 0 - 3-

>) U)> 0o -z

7---- PQ) 3 -

0" 14 'a

a) U)Q cCa 9 ) a

v - C

0 o 0 g4-0 0 z 0 O .cn0U

U.) Cc 4ua) 0) C

0d 0z v 0)ol o -' -rn A.. --

CIO "0 I D 0) V) V)

oO Z' 0' 0 '0 0 '- 1 o

U0 U0 0 a0)

0 0)c 0

0 0 0 0 0 0d

.2 Oo 00 .90 .90

00 00. 00 0) 0 00

0u 0U 0U 0U 0) )6- Q

0- 03 Cd0.cn)

0 Z 0)

~~~- 6p 0 ~ ~ 9-0 0Id -' - "0 -0 -a m a

L.-ý 0 C)COC)-4 04-0 0 0 a

)- (D)~- "8 " = - 4 r

340) 0 M)) 0 :3 V2 0), -C0

4 0 Wi

o to

-- 0. 0 0a) 0 ' 0

0 p > m 0 'do 8

u -l

0)0Q. 0

-~ I~

CIS- as 0

V~ 0

0 >

m- m~ 0o 0 04. d Q0 cz) f In 0)

C~Cl

C) 0.z

0 0

v Q))

M 9) V 2 0 =->0)U. CDU

0ý CO .0 0

0s

- - 4-ca

0- T' =>r vUJJf~ - - 4-. - - -

cu Q) 0 0 u-oi- 00 C

W

0000 0 0 0

22 0 0 0

u~L u. .0

00 0.0'-~~ c"13-

0 0 04 0 o 0

a) 0 E ) L) () t) ,C) (

~~c" 0o0 _ 0

0 mo U-0-0 .) 0 01-0 0 0 C 00U c

0) m) w

-2()~ 0) U)a- m> o0 b4o~ C)0

-) x) 0 0 004.) r."

ca0)

r -0

0 0 'ow .

00

co 0 0 W 0-0)

.0 u

U)

[-

5 )0

z)-

w ~ H-o - ,

4- v0)Q 0.)

0 00

0404 ga40~-0

0) 000 0 0

ow ~ Q 0 0. 0 0.

0 0

0 V00 0 - 0 0,-o -8-0"

ý- > ~ C) CO c) 0" CO CO

C. *a a>) m C) 0 1- C) .oc 0 4C)-o 0 0 0 0 0 01

as C/C

000 CO) 0) -~' )~j C )1-' 4-) 0)C ~ O~ 0C )~

o 0) MC lo U) CI,)) V 0c

ti o~0 -0

0 r-CO C4- *

CO T/ CO i

00 1 14 0 -C

o 00)OC "0 .2/ U.0)

I~00ýD Mo 8 *00n -C3

0 0 0

oD oj a)0

r- .6 - 7 A

o o 0 C

0 0

E ')) p -'nC

00V 0)

~r ri 'C ) C)

0O) 0 . 0 0 0

o .0 ~ 0 0

0 0z 0 0ci~

CI 00 $,O

0~ 0 0 0C (nCOh) 4

>E 0 0u 00ctl 0 rn 0

m0

0 Cn 0oco 0) -co 0 C6

CO U)~U 0 o)

Zp 0d-0W ;ow.

u--.

0 (

ci cC)

oo +* 44

C- -+ El~~L) .L) .

cf ) 0) U) V) CA

0) ~ o-0 -'0 C/)-0 -'0 En 0 '-'0 c

a~ 0 0

.;-) .j :-40D d -al

(L) 0)Q.) W0 a.) 0) a)0 0)))0

o~ý -0 -. .0 0)-*0

04 0

(,) 0 0)0C0) 0a :: E ,o U

?"0 0) o 0)U2 '0 ý: z bw0 "r a n En

Q 0 :J0C', .2 0 0 0-0) "IJ -0 t- ")~ -t0" 4

0.) t-0 00 0 r-. 0 o c

00 *'.. 0) 04ý *0

00E 0 P

0 0)0 0 a

~0_t :3 v 0

a)4 ) 0) a) 4.)

Q)4 QJ) a) Q)

ot..

00 0 0 0 0

U 0

*0 -c 0d- _0 t

0 -0 *0 *0 *0

= - .- U3w- /

o 0 -~0 ~ 0.) )Q -o- 4)

0 4 ~0 u0 u5

0 4)0o0 0 4

P~~ 0 .J2 0 >~~ 0 .0 "0 V ~') W- En a) P -

U) 'Ld 4. 4)3 -C q CL-ý 0 4)

a) co- C 0 w, -(1)~~1- 0o 01 Ci-Z '0 n C

04044 01 co t.! 4)4, "

as 0~ 0 )0mc

z ~ z~4. (/) 00Q>

o o :.-•

SBd

Im"

S.= •+ .= •..-. • o = • •.• •S,,.€

•-, o • o • • ,, • o > • •.• ,u •'"

=,• o 5 = o"• =;-t•= •'•Z, =;-t= =At-=0 • O0 • • t• t-- 0 n:• O• 0 "• O0

• •.• • • • •-• .. •.,• •-" • • •I ",•

S= ",• ,""

S= . ,• •.= =• '= .= • • • o=

.. -- • • m \o,..C IZ 0 o .'<, •--• 0 Cl o I• "

S.... .. . ...• • • • o o,•o•,•" ......... . .....

u

cc 0

o ozo co 0 m

u~ 0 0o

u

Cw0)0

H ocz cz M~

ci > r 4

C0- 0

0 ou

2~ Ei-4 o V -.

o o 0 0 >~~

o, Cc'

al0

0 0'.~

0(0

0 cz

cc 0.0

M co' >d C,Cd 0

0~ u

00

-d L 0 -q ý .-ZI.-

H al

00000~0 0 3C

CaO

CE- 0Q M )or- >- 0. 0-

t >

U 0 L-

00 0~-

.CIO U))

oo 0.

Q) cn (D

F- F-

U') -

0 0 0 0

cd h F-. P4 F4.

u C. C-. 0~--

£ý C ,

r-. as-~4

cc '-'4- - 0 - 0

0 - ~~- .0 - -

0 >0Ct & -

PtCt 0. u

C 0

13 W- CA

0Cct0 Q

oo 0

0 0

L) a)

It U) c)

'0 0

00 00 0 0.

-0~~ 0 101 01))t

o Uto.6 CZ 0 0 0. ()0 Q )4.0 0 0

-U 0 0 dc

as m cl %U - 0 0

c-N, t z 0 rý 00. rJ. -0.

0 co 0-~, 0 -0

&0. v0 0 -

bl 0

X

a) r. O

o u +~ +0 +

U ~Uý

z i~' 0no G

0 v~O 0 0 0

0l- 0 0 0

() 0f 0 006 0

rA r -r.

0 + 0

'Z - --ý C

0 0 0 u

&... 0. 0

0 0u_ z

a) o ~ -)

0 0o 0 0

> >~

.. 0 a)

coo

<U o

rjCj

4-. . I-.a) :14 r.= )

Z) 0) () 79)C)- 0 -4- a),ý z + +C) .a) 0 2 ")U coZ~ co as

- ~ ~ Q 64 ) )U

as 0

0 .2 m.

0 0)

0 04 04

C- 00 0 0

a) 0..U U )

cn 0 0

a) = 0N 0,

> 0 .oa 0

cu- -0 U)

o 0oz. o o

a> a)Ua)4UCaU)IC)

CV) c Cul

Q) 4)4-u

CD y

v~C 4 ~ 0*-6 . m'mu 0- co ~ C C) M

,4 , - L4C 2~C)) Cu, au

> Q4) m - C14 0 ''A 'n4 cd - C-S0C O

o 0 ) Cu ~~(CC)4 V) Eui~u

0 rAC ~ 2

CL

t- 4- U

Cu 4.

10 ILI40Y2 0

.- c c 0)C

ot

E C0 0

It 0~: 4- :-w

cu 5_.C _ > I

ci 0

rh m In CuWu 00~ 0A r0

0 0 0u Cu~ Cu u, .. u Q)

0 Cu

4) 11) _0Uc u

ux

t - . ,, .

Q) cz 4. c

(L) -0 0O ' o C" a

~o~a)Q~OaCc ( zC.O- C Cd

03 , 1: - 0 "d 0 m~

m) U. C5.'

0 0z0 C.0 (Z 0-) -

nCn

0 .00 .

0 o) 0p-o

cti cz E 0 "at

V~a 5 0a) a > e

0ý 0 a 0 C Z ' r j

ýJ Cf on 0 C'suV 2

1:

Q~ 00 coa)Q)c a)

0 0 )

;0

a) ) n

F-4~

ux

0480

C) Q - 0 2 -40 0 t .- U t " 0 •0 0

o . o 00 0 M M 6 0 - 0 c0 0 co.

C:J U), m U) 0 .: 0 ) 1

>. ia , .C.) o m a> ýGc

0 " 0 0 0

. m m CA

-gm 0 0 0

0 000

CL -

C)

C)

0.

W 0

4C0 0 U 0s

00

S• oco0

CU0 o u -00 2

(0 - 0 cos

C~~ C14) u- u. C

In Ca 0f 2 0 2 0ooU) 0 C' g-4 0 r.

C)-~~In C) Un -c

x X x

a.) C,

rn 0

o 0" C13 C--

> i3 Cj a.) .j2 .2 .

UA U .(ý4 ~ -

-o . ~ c Y C) -~ ~ ~ C ~ C~ Cj4C

o L 0 l 0 0 0. 0 0

.0 .0 0 0 0 0

u u o- 4 4

0. C 0 0 0 0

o 00 0V 0

-6 -CV a)

6. 0 (,j -. r.)

r. - H U0 0. -C) ;)

C) 0

o Cc, 0 9C.. O18 0. S n:t t

m -en~~~ 0) -

0 00~304 L) r )

on -d C .0 0.b

C) 4

2__4 0'4 u ,.r

>< x

ej X X

-M 0~

C)+ + + +

E.0 a) > q

~ cn

- 0

m zz

0 .0

Cu d

-.- --- " )Ow C, .ý Q~ * 2

= -- E 0 ca 0

0a 0 0 0z

C)Ccqj 0 0 0

- U))

0u X

C.) u *U*~- id cl ) 't - ~ CC.d J-) E

- D

EA ~ ~ C) C~~~- C )w cd rA r

InuO ~ a ~ - C C C~

LO U) c Eco v

74

APPENDIX BEVALUATION OF THE FORT LEAVENWORTH LIFELONG LEARNING INITIATIVE

Executive SummaryOverview

The Fort Leavenworth Lifelong Learning Initiative (LVN LLI) is a suite of technologies(Blackboard, Microsoft Sharepoint, and Macromedia Breeze) that enables, among other things,online posting of schoolhouse curriculum materials and collaboration (asynchronous andsynchronous) among technical staff, students, faculty, and curriculum developers. The primarygoal of this blended learning initiative is to provide current, standardized curriculum materials 24hours a day, 7 days a week, to students located in the schoolhouse, on satellite campuses, atArmy reserve facilities, and at personal sites. The LVN LLI also is intended to support two-waycollaboration between the field and institutional Army, enabling enhanced course relevance andgraduate reachback to the institution.

The purpose of the present evaluation was to capture the effectiveness of the LVN LLIwhen applied to delivering Intermediate-Level Education (ILE). Specifically, this formativeevaluation attempted to define and measure such program outcomes as enhancedteaching/learning environment, enhanced course relevance, improved student performance, andcost-effectiveness. Factors theoretically linked to these outcomes--chiefly system use and theactivities of instructors, curriculum developers, technical support, and program leadership--alsowere assessed in order to tell a complete story about the functioning of the initiative. Thisappendix presents the evaluation in detail, with the main findings summarized immediatelybelow.

Data Collection

Following the completion of a program logic model and metric development, data werecollected during the months of September and October 2006 using a variety of methods thatincluded surveys and interviews, system analysis, and retrieval of archival financial andenrollment data. Data collection largely focused on the application of the LVN LLI to ILEconducted in residence at Fort Leavenworth, as this was the focus prescribed by the projectstakeholders. Surveys therefore were administered to curriculum developers, instructors, andstudents directly involved with resident ILE (2006 February-start and 2006 August-start classes;see Appendices B-1, B-2, and B-3 for the surveys and a description of their administration).Some interviews also were conducted with ILE instructors at satellite campuses and at TotalArmy School System (TASS) Battalion sites.

Findings

Overall, the evaluation results indicate that the LVN LLI is a cost-effective solution forenhancing the educational outreach of ILE curriculum materials. It currently costs approximately$163.14 more per student to deliver ILE using the LVN LLI but the increase in cost due totechnology procurement and associated personnel and facilities expenses is a vanishingly smallfraction of the total amount spent to deliver ILE. Moreover, the relative cost-per-student to the

75

Army will further decrease as the printing and shipping of course materials to remote sites iscompletely phased out over the next two years. For what amounts practically to the same amountof money, ILE delivered online using the LVN LLI is in the process of eliminating the one- tothree-year lag in curriculum content between the schoolhouse and Army reserve facilities. It alsosupports anytime, anywhere learning for ILE students in residence and at satellite campuses, asreflected in system usage data.

Unfortunately, there is relatively little the LVN LLI can do to reduce the 6-10 month lagbetween changes in the operational environment and revisions to the standardized ILEcurriculum due to the lengthy institutional curriculum review, revision, and vetting process.However, the LVN LLI may assist in circumventing this problem by making it easier forinstructors to supplement curriculum materials with up-to-date articles, professional discussions,and emerging doctrine. Twenty-nine percent of ILE instructors surveyed reported that theysupplemented the curriculum materials more frequently than in pre-LVN LLI conditions. Thisproportion is likely to increase as faculty become more facile with Blackboard. In addition,substantial proportions of students (>85%), instructors (74%), and course authors (80%)surveyed reported that the ILE course content was relevant to the jobs of ILE graduates.

There do appear to be some challenges to the full-scale adoption of the LVN LLI bycourse authors, instructors, and students in the schoolhouse. Although the rates of activeadoption (i.e., primary use of the system components) increased during the first-year pilot, theyremain below 100% (between 5% and 40%, depending on the system component and purpose forwhich it was accessed). A combination of survey and interview data indicated that technicaldifficulty was not the main barrier to system use. First, a weighted average of student, courseauthor, and instructor survey responses indicated that the LVN LLI technical support wasaccessible (62%) and useful (66%). Moreover, the system workarounds they reported usinginvolved typically some other form of technology. For example, SharePoint was the mostfrequently cited alternative to Blackboard when posting or accessing curriculum materials.

The main barriers to system use appear to have been (a) ready availability of alternativesto the system that were more familiar and easier to access; (b) inconsistent or absent informationmanagement procedures necessary to make posted content easy to find; and (c) lack of instructorand course author involvement in and buy-in to the lifelong learning concept. As has been foundin previous evaluations of this kind (Cianciolo et al., 2006), greater success has been achieved inthe technical implementation of the initiative than in the cultivation of stakeholder enthusiasmand investment. When asked what they thought the Army's main reason for implementingBlackboard and Sharepoint was, a minority of students surveyed (33%) responded that the Armyhad made the decision in the best interests of leader education. Although the Army surely hadmultiple reasons for adopting the new instructional technologies--including enhancing leadereducation--actively using the LVN LLI to foster students' perception of organizational supportfor leaders would make it a more effective asset in enhancing organizational commitment andcultural shift.

In any case, challenges to the active adoption of the LVN LLI did not hamper theacademic experience of ILE students and instructors in the first-year pilot. A very highproportion of surveyed students (79% on average) reported that ILE instructors generally

76

demonstrated the classroom facilitation behaviors recognized as critical to developing adultlearners. In addition, survey responses indicated that a majority of students (a) tookresponsibility for their own learning via independent study (70-98%); (b) reported high learningself-efficacy (88%); and (c) were motivated to learn the topics taught in ILE (91%). Fifty-eightpercent of instructors reported that the LVN LLI did not have an impact on classroom efficiency,and a small minority of instructors (10%) reported that the LVN LLI enhanced their classroomefficiency. Although it may seem insignificant, this result is noteworthy given the fact that theintroduction of instructional technology often is associated with increased workload for faculty(Willis, 2000).

Conclusions and Recommendations

If nothing about the LVN LLI implementation were to change from this point forward,ILE instruction would continue successfully, and students--especially those at Army reservefacilities--would benefit from greater access to more relevant standardized curriculum, all atrelatively little cost to the Army. However, the findings of the present evaluation illustrate theimportance of taking a logic modeling approach to understanding the impact of lifelong learninginitiatives. Assessment of outcomes alone would have indicated that the LVN LLI had achievedits goals but would have obscured the fact that a subset of these goals--teaching and learningeffectiveness--was achieved independently of the use of the LVN LLI's components.

In order for the already successful LVN LLI to achieve optimal educational andorganizational impact, stronger beliefs in its purpose and higher active adoption rates of itscomponents should be achieved. The accomplishment of this goal requires fostering theinstructor buy-in necessary to learn and adopt unfamiliar technologies and to role-model theiruse for students. Improved buy-in may be developed through (a) engaging other schoolhousecomponents, especially faculty and staff development, in viewing and cultivating LVN LLI-assisted teaching as a critical instructor competency; (b) encouraging and shepherding theinvolvement of instructors and course authors in the development of information managementprocedures through needs assessment and iterative system design; and (c) spreading strategiccommunications that explain the purpose and goals of the system, that anticipate technicallimitations of the system (i.e., login requirements), and that immediately follow system outages.

77

Overview of the Fort Leavenworth Lifelong Learning Initiative

The Fort Leavenworth Lifelong Learning Initiative (LVN LLI) was designed to supportthe educational mission of the U.S. Army Command and General Staff College (CGSC). TheCGSC is one of the major subordinate organizations comprising the Combined Arms Centerheadquartered at Fort Leavenworth, and provides formal institutional education to field gradeand higher officers through five component schools. Figure B-1 shows the organizational chartof the CGSC, adapted from CGSC Circular 350-1. The components of the CGSC directlyrelevant to the present evaluation are shown in orange, and their connection with the LVN LLIwill be explained in further detail below.

S Commandant

[Deputy Commandant

SAssistant Deputy CGSC Sergeant

Commandant F; Major (Acting)

SCenter for Army U.S. Air Force US.Nv US.MinCopTransformation Chair edrhp ][1-[ USýv tU..Mrn op

ýDe~an ;f Academics Chief of StaffII

Digital Leader Graduate Degree Personnel andDevelopment Center Programs Administration CGSC OperationsI I -I IAcademic Outreach Accreditation Logistics CGSC Plans

Program

Quality Assurance Office Faculty & Staff M PDevelpmn Resource Management Public AffairsI v pment

Registrar Curriculum Development Directorate of Reserve Directorate ofFComponent Progs Educational Technology

I ICommand & General chool of Advanced School of Command School of Advanced Army Management

Staff School Mili Studies f Preparation Distnce Leamin Staff College

Figure B-1. CGSC Organizational Chart

The Components of the L VN LLI

Technology procurement and integration for the LVN LLI began in 2004, and at the timeof this evaluation the initiative comprised three major components: Blackboard, MicrosoftSharePoint, and Macromedia Breeze. Each of these components, described below, was selectedto provide a unique capability in supporting the development of curriculum materials, thedelivery of course content, the collaboration among faculty, staff, and students, and the storageof curriculum materials for reachback from graduates in the field.

78

Blackboard. The Blackboard Academic Suite is a web-based platform designed toorganize and host a wide variety of file types typically distributed in educational settings,including (but not limited to) digital documents, presentations, and videos. Blackboard alsoprovides a means for faculty to communicate with students via posting announcements andsending group emails and to conduct such classroom administrative tasks as administeringexams, collecting assignments, and recording grades. Students may also communicate with eachother using Blackboard's discussion board. Blackboard is especially well suited for conductingasynchronous collaborative distance-learning courses, such as those provided by manyuniversities, and for conducting blended-learning courses in which one component involvesasynchronous distance learning. Online posting of course content also reduces the time andmoney involved in replicating and shipping curriculum materials to multiple class sites, a benefitof particular relevance to the CGSC.

Microsoft SharePoint. A combination of Microsoft Windows server technology andaccess privileges, SharePoint enables the intranet-based storage and management of large,complex file collections. SharePoint users with the appropriate permissions and access to anorganization's domain can create websites ranging in complexity to host and organize content,post announcements, create and maintain message boards, and send emails, among other things.SharePoint is intended to support asynchronous collaboration among curriculum developers bymaking working draft files and reference information available in a way that is easy to locate andunderstand. SharePoint also is intended to support collaboration among students, to distributegeneral reference materials (e.g., emerging doctrine) and to connect students administratively tothe schoolhouse.

Macromedia Breeze. Breeze is a web-conferencing platform that enables synchronouscollaboration through such capabilities as document sharing, voiceover IP, multiple-personvideo, and an electronic whiteboard. Breeze was included in the LVN LLI in large part to enableadvanced user technical support and to enable distributed academic activities, such as instructoror course author meetings and virtual classroom presentations and group exercises. At the timeof this research, Breeze was in the preliminary stages of implementation in the LVN LLI, so it isnot addressed further in this report.

Implementation, maintenance, and oversight of the LVN LLI are conducted by the CGSCDirectorate of Educational Technology (DOET; highlighted in orange in Figure B-1 above). TheDOET comprises 18 staff members whose efforts are devoted to providing the informationtechnology necessary to support the CGSC's educational mission, which includes classroom andoffice networking and automation in addition to the LVN LLI. With the Lewis and ClarkBuilding--which will house CGSS and much of the CGSC administration and support, includingthe DOET--under construction, the DOET also is heavily involved in planning, procuring, andoverseeing the incorporation of advanced information technology into over 365,000 square feetof brand new classrooms, offices, and auditorium space. Eight additional staff members, on-sitecontractors, are devoted entirely to the LVN LLI, providing technical support, technologyintegration, and user training.

79

The L VN LLI and Officer Education

Shown at the bottom of Figure B-i (above) are the five schools comprising the CGSC.Highlighted in orange to indicate their direct relevance to the present evaluation are theCommand and General Staff School (CGSS) and the School of Advanced Distance Learning(SADL). Together, these two schools deliver the common core or Intermediate-Level Education(ILE) component of the Command and General Staff Officer's Course (CGSOC). The CGSSdelivers ILE to students in residence at Fort Leavenworth and to students attending satellitecampuses located at Forts Belvoir, Gordon, and Lee. The SADL develops, distributes, andadministers classroom-based ILE to students attending instruction via Total Army SchoolSystem (TASS) Battalion sites and self-contained, web/CD-based ILE instruction to non-residentstudents completing their coursework in other locations. The present evaluation focused on theadministration of ILE using the LVN LLI because at the time the study was conducted, ILE wasthe major course supported by the initiative. The CGSC is in the process of implementing othercourses online, such as those administered by the School of Advanced Military Studies.

The ILE curriculum predates the LVN LLI by approximately two years. First piloted in2003, the purpose of ILE was to make third tier officer education available to all Army majors ina manner best suited to their career field (Bralley, 2006; Bralley, Danley, French, Soby, &Tiberi, 2003). Prior to ILE, 50% of Army majors were selected to attend the 10-month CGSOCat Fort Leavenworth. The selected majors came from a variety of career fields, however asubstantial portion of the CGSOC curriculum was relevant only to a subset of these officers.Majors in the Operations Career Field benefited from the complete curriculum, but majors inother career fields (e.g., the Institutional Support Career Field) found only the common corecomponent to be relevant to their educational needs. Enabled by the satellite campuses andadvances in distance learning technology, ILE is an efficient and cost-effective solution forensuring educational outreach. Majors in the Operations Career Field attend the full 10-monthcourse at Fort Leavenworth, while majors in the other career fields complete only the CGSOCcommon core curriculum (i.e., ILE), lasting approximately sixteen weeks, at one of the satellitecampuses or via distance education (Bralley, 2006; Bralley et al., 2003).

As shown in Figure B-2 below (adapted from CGSC Circular 350-1), the ILE curriculumis taught in two parallel sequences of blocks of instruction punctuated by periodic visits by guestspeakers to CGSC. All ILE students, regardless of their course location, receive this same basiccurriculum, although the structure and duration of the course differs somewhat for distancelearners, and guest lectures must be attended remotely or asynchronously by students located off-post.

Standardized curriculum materials for ILE are developed by teams of lesson authorsgrouped according to block of instruction. Block authors lead a team of lesson authors, ensuringthat the content developed is accurate and coherent. Curriculum revisions made following anadministration of ILE go through a rigorous development process that begins withrecommendations gathered from a post-instructional conference. A curriculum design reviewinvolving the college leadership precedes the revision process then, prior to distribution tostudents, revised ILE curriculum materials undergo an institutional review process wherebyaccuracy is further checked and instructional effectiveness examined.

80

Cl00 C200 C300 C400 C999Foundatios Strategic Studies "- Operational T t Sui End of Course

Foundations Tactical Studies Practical Exercise

1,100Fl 00HIO0 LIO0 Force

History Leadership Management

Guest Speakers

Figure B-2. ILE Curriculum

ILE students in residence at Fort Leavenworth and at satellite campuses are divided intosections of approximately 72 students, which are in turn divided into four staff groupscomprising approximately 18 students. Each section is taught by 12 instructors coming fromeach of five departments within CGSS: Tactics (4 instructors), Joint and MultinationalOperations (4 instructors), Logistics and Resource Operations (2 instructors), Military History (1instructor), and Leadership (1 instructor). Staff groups contained in a section have one instructorfrom each department such that a section's four tactics instructors each have responsibility for asingle staff group whereas a section's single military history instructor has responsibility for allfour staff groups, and so on.

Producing and Delivering ILE with the L VN LLI - Use Cases

ILE Curriculum Development. SharePoint is used in the LVN LLI to store and manageILE curriculum content during the development process. Block authors are encouraged to useSharePoint to coordinate with each other and with lesson authors so that content is notunnecessarily duplicated and so that general educational themes (e.g., critical thinking) commonacross blocks may be reinforced in topic-relevant ways. Block authors have permissions to createand modify SharePoint websites to coordinate their team's efforts.

Prior to the implementation of the LVN LLI, ILE curriculum materials underdevelopment were stored on an internal network and block authors created file systems toorganize content. Replacing the internal network with SharePoint enables block authors andlesson authors to work together using a central information management resource even whenthey are geographically distributed on- and off-post. The SharePoint site also links curriculumdevelopers to a variety of knowledge management resources, including the Battle CommandKnowledge System discussion forums and Army Knowledge Online (AKO), so that informationdirectly from the field is readily available and can be used to enhance the currency and relevanceof course content.

ILE Instruction. The intended use for Blackboard is to deliver standardized ILEcurriculum materials to students located on- and off-post. Secured by AKO authentication,content posted on Blackboard is accessible anytime, anywhere by students and faculty with the

81

appropriate permissions. Because the curriculum development and review process creates a gapof several (up to 10) months between the initiation of the process and content delivery, ILEinstructors are encouraged to supplement the formal curriculum with material they identify asmore relevant or current. Instructors supplementing the formal curriculum may locate materialsusing readily available links to Army knowledge management resources. They are directed topost their content on Blackboard and to alert students to new materials. Instructors also areencouraged to use the administrative functions present in Blackboard, particularly the gradebook, to facilitate classroom activity.

Prior to the implementation of the LVN LLI, instructors distributed curriculum materialsvia an internal network and via print. Access to course content therefore was restricted to peopleon post who could access computers on the network, unless substantial printing expenses wereincurred to facilitate home study. Remote ILE students also received curriculum materials inprinted form and, more recently, via compact disc. Posting course content in Blackboard insteadof on an internal network enables access to course content from anywhere. Web-based access tocourse content not only enables the delivery of standardized course content across modes ofdelivery (resident, satellite, TASS Battalion, etc.), but eliminates the need for printing andshipping to remote sites. Eliminating printing and shipping simultaneously reduces cost andenhances curriculum currency for students off-post because the lead time to reproduce anddistribute course materials is significantly reduced.

Student collaboration. In addition to supporting collaboration among curriculumdevelopers, SharePoint also is used to support ILE student collaboration and connection to theschoolhouse. The student SharePoint site features such resources as reference libraries and links,academic calendars, and announcements from a variety of sources, including the deputycommandant and the CGSC student division. Students with the appropriate permissions maycreate shared workspaces on the site to post their contributions to class discussions, groupprojects, and/or practical exercises. Enabling collaboration within a protected domain facilitatesinformation assurance in an environment where a large number of people may post content.Unlike in SharePoint, students do not have permissions to post content to Blackboard so thataccess controls may promote information assurance.

Metrics and Measures Used to Evaluate the Leavenworth LLI

The purpose of the evaluation was to capture the learning effectiveness of the LVN LLI.The goal of this evaluation was to represent the cost-benefit tradeoffs of the emerging initiativein a way that would be easy for decision makers to understand and use. Specifically, thisformative evaluation attempted to define and measure such outcomes as enhancedteaching/learning environment, enhanced course relevance, improved student performance, andcost-effectiveness. Factors theoretically linked to these outcomes, chiefly system use and theactivities of instructors, curriculum developers, technical support, and leadership, also wereassessed in order to tell a complete story about the functioning of the initiative.

Of the 229 metrics listed in the metric and measures chart (see Appendix A, above), 17were selected by the CGSC DOET as being particularly relevant to capturing the costs andbenefits of the LVN LLI. Eight of these metrics were activity metrics, three were output metrics,

82

and six were outcome metrics. No resource metrics were selected. Additional resource, activity,and outcome metrics from the metrics and measures chart were used to supplement the 17selected because they (a) would provide useful diagnostic and outcome information; (b) werenecessary for capturing cost-effectiveness; and (c) were feasible to capture in the time available.

Evaluation data were collected in September and October of 2006 via a combination ofinterviews, surveys, and retrieval of archival financial and other data. Data collection largelyfocused on the application of the LVN LLI to resident ILE, as this was the focus prescribed bythe project stakeholders. Surveys therefore were administered to curriculum developers,instructors, and students directly involved with resident ILE (2006 February-start and 2006August-start classes 2; see Appendices B-1, B-2, and B-3 for a complete description of thesurveys and their administration). All metrics used in the present evaluation and their associatedmeasures are described below.

Outcome - Course Relevance

Six metrics capturing the relevance of the ILE curriculum materials (formal and informal)were measured. These metrics were selected in order to explore a key expected outcome of theLVN LLI -- that course content would be more relevant because (a) curriculum developerswould have easier access to the field Army and to each other, enabling more frequent updates tostandardized curriculum materials; (b) instructors would have more flexibility to insertsupplemental materials into the curriculum in real time, enabling more frequent updates; and (c)placing content online would eliminate replication and shipping timelines. A detailed descriptionof each metric is provided below.

1. Frequency of curriculum developer updates to main curriculum relative to the pre-L VN LLI situation. As described previously, formal decisions to update the main, standardizedILE curriculum follow a post-instruction conference and curriculum design review. Courseauthors then implement these decisions, ideally completing the production and editorial processbefore the next cycle of the course. Course authors were asked to report via survey howfrequently they updated the main ILE curriculum (relative to course cycles) based on currentevents or feedback in past years, then were asked to report the impact that SharePoint has had onthe frequency of such updates. Eight of 9 course authors who reported on curriculum updatingindicated that in the past they infrequently or never updated the curriculum. Seven of the 9course authors also indicated that SharePoint had not had an impact on the frequency ofupdating--that the duration of the curriculum development and review process, rather thantechnical factors determined the frequency of revisions.

2. Curriculum developer perceptions of main curriculum relevance. Course authors weresurveyed to capture their perceptions of the relevance of the standardized curriculum materials.A single question asked how current or relevant they felt the curriculum was to the jobs of ILE

2 The first full-scale pilot of the LVN LLI with resident students was conducted with the 2006 February-start ILE

class with follow-on implementation executed for the 2006 August-start class. Some initial piloting was done withthe 2005 August-start ILE class, but this pilot effort was a scaled-down version of the 2006 February-start pilot andalso was completed prior to the beginning of the present evaluation.

83

graduates. Of the 10 curriculum developers who responded to this question, 8 indicated that thecurriculum content was either totally relevant or more relevant than in previous years.

3. Frequency of instructor updates to ongoing course materials. As described previously,the standardized ILE curriculum posted on Blackboard in the LVN LLI can be up to 10 monthsbehind changes in the contemporary operating environment. To enhance the currency of coursematerials, instructors may use Blackboard to distribute news articles, professional literature, oremerging doctrine, among other things, just in time to replace older readings or support classdiscussion. The frequency of such updates was assessed by asking instructors how often theymade them: daily, weekly, every couple of weeks, monthly, and never. Of 69 instructors whoresponded to this survey question, 64% reported updating daily or weekly.

4. Frequency of instructor updates to ongoing course materials relative to the pre-L VNLLI situation. This metric was captured by asking instructors to indicate how frequently theysupplemented the standardized curriculum during the 2006 February- and/or 2006 August-startclasses, taught using the LVN LLI, relative to previous ILE classes for which an internal networkwas used to distribute course content. Of the 63 instructors who responded to this surveyquestion, 29% (N = 18) reported making updates somewhat or a lot more frequently than inprevious situations. Fifty-six percent (35 of 63) reported that their frequency of updates wasapproximately the same.

5. Instructor perceptions of course relevance (main and supplemental content).Instructors were surveyed for their perceptions of the relevance of the standardized andsupplemental curriculum materials. A single question asked how current or relevant instructorsfelt the main and supplemental curriculum was to the jobs of ILE graduates. Of the 65 instructorswho responded to this question 42% (N = 27) stated that the curriculum was totally relevant ordefinitely more relevant than in the past. Thirty-two percent (21 of 65) stated that the curriculumwas relevant but that emphasis is rightly placed on education, rather than mission preparation.

6. Student perceptions of course relevance (main and supplemental content). Studentswere surveyed to capture their perceptions of the relevance of the main and supplementalcurriculum materials. Two questions asked how current or relevant students felt the main andsupplemental curriculum was to the jobs they expected to assume after graduation. Of the 251students who responded to these questions, 73% (N= 184) felt the main curriculum was eithermore relevant than irrelevant or totally relevant to the jobs they expect to have after graduating.An additional 16% of respondents (N = 40) indicated that the formal curriculum was relevant,given the fact that institutional education was supposed to be "just-in-case," not "just-in-time."Sixty-six percent of surveyed students (N = 165) felt that the supplemental content was eithermore relevant than irrelevant or totally relevant. An additional 22% of respondents (N = 54)indicated that the supplemental content was relevant, given its academic purpose.

Summary. Ideally, the opportunity to update standardized ILE curriculum materialswould occur before each course administration. Updating the curriculum this frequently wouldenable the institution to deliver education as adaptively as possible to changes in the operationalenvironment. There is significant difficulty associated with updating the curriculum this often,namely the length of the curriculum review, revision, vetting, and printing process relative to the

84

number of overlapping course administrations. Simply put, the ILE course administration cycleoutpaces the curriculum development cycle. The result is that standardized ILE curriculummaterials delivered to August-start resident students are 6 to 10 months behind the changes to thecurrent operating environment. Non-resident course administrations (i.e., at TASS Battalions) areeven further behind--up to three additional years--in part due to the time required to print orreplicate and ship course materials throughout the U.S. and internationally.

Unfortunately, the LVN LLI cannot change the speed of the formal curriculum review,revision, vetting, and printing process. That said, substantial proportions of students (>85%),course authors (80%), and instructors (74%) surveyed reported that the ILE course content wasrelevant to the jobs of ILE graduates. It is unknown how pre-LVN LLI ratings of ILE courserelevance would compare to those obtained in the present evaluation. However, 29% ofinstructors surveyed reported that they supplemented the curriculum materials more frequentlythan in pre-LVN LLI conditions, and 56% reported making approximately the same number ofupdates in both conditions. Importantly, when fully implemented and adopted, the LVN LLI willeliminate the one-year discrepancy in ILE content currency between TASS Battalion sites andthe CGSS due to the replication and shipping timelines required to produce and distribute coursecontent remotely.

Outcome - Instructors as Facilitators ofAdult Learning

Three metrics capturing the degree to which ILE instructors leveraged the LVN LLIblended learning environment (i.e., the combination of face-to-face and asynchronouscollaborative learning settings) to facilitate adult learning were measured. Proponents oftechnology-assisted learning claim that technology can revolutionize the way education isconducted, but instructors must leverage the available capabilities to achieve an impact onstudent learning and lifelong learning orientation (e.g., Clark, 1994; Ehrmann, 1994). Thepurpose of the selected metrics was to explore whether the presence of the LVN LLI(specifically, Blackboard) has had an impact on how instruction and learning is accomplished inILE. A detailed description of each metric is provided below.

1. Percent of students who reported that they were encouraged to contribute materials tothe course curriculum. Students were asked to report via survey whether any of their instructorsencouraged them to contribute to the course content in their respective areas of expertise. Therange in possible answers included "no encouragement," "no discouragement," and "activeencouragement." Of the 251 students who responded to the survey question, 74% (N-= 186)indicated that instructors actively encouraged contributions. Sixty-eight percent of February-startstudents (54 of 80) reported active encouragement, whereas 78% of August-start students (131 of169) made the same report.

2. Percent of students who reported that instructors encouraged them (or demonstratedhow) to use resources other than him/herselffor class work or other purposes. Students surveyedwere asked to report whether any of their instructors encouraged them to consider learningresources in addition to him- or herself. Respondents could select "no encouragement," "nodiscouragement," and "active encouragement." Of the 250 students who responded to the surveyquestion, 65% (N = 163) indicated that instructors actively encouraged the exploration of

85

alternative learning resources. Roughly equal proportions of February-start and August-startstudents reported active encouragement [67% (53 of 79) and 65% (109 of 169), respectively].

3. Quality of real-time instructor facilitation behaviors in the classroom. This metric wascaptured through a five-part survey question in which students were asked to report the relativenumber of instructors who engaged in five classroom facilitation behaviors. These behaviorsincluded: (1) provided clear guidance regarding expectations and performance evaluationcriteria; (2) provided constructive and informative performance feedback; (3) presented learningchallenges that were difficult but not overwhelming; (4) assigned group tasks that had well-defined tasks and roles for each group member; and (5) engaged students in asking questions andreflecting on class materials. Table B-I below summarizes student responses to this question.

Table B-1. Student Reporting of Instructor Classroom Facilitation BehaviorFacilitation Behavior N respondents % Reporting at least "more did than didn't"

All Feb-start Aug-startGuidance 250 86% 78% 90%Feedback 249 79% 71% 83%Challenge 247 79% 71% 82%Group 250 79% 68% 84%Questions 250 89% 81% 92%

As shown in the table above, a substantial proportion of students reported the presence ofseveral instructor facilitation behaviors in the classroom. Notably, this proportion was higher for2006 August-start students on all behaviors, suggesting that instructors developed as facilitatorsof adult learning between the two cycles of the course. This interpretation should be viewed withcaution, however, because August-start students had only recently begun classes when surveyedwhereas the February-start students were close to graduating. The data collected cannot rule outthe possibility that the difference shown in the table above was due to such group differences aslevel of fatigue or habituation to the course and/or reliance on short-term versus long-termmemory to make judgments.

Summary. A very high proportion of surveyed students (79% on average) reported thatILE instructors generally demonstrated the classroom facilitation behaviors recognized as criticalto developing adult learners. It is unknown whether this proportion would be the same forstudents who took ILE prior to the launch of the LVN LLI. However, the pattern of respondingsuggests that ILE instructors improved their facilitation skills between the 2006 February- and2006 August-start course cycles. On all but one of the facilitation behaviors sampled, a greaterproportion of August-start students reported a higher frequency of instructor facilitation.Although alternative explanations for this pattern cannot be ruled out, the finding could indicatethat adoption of the LVN LLI is having an effect on instructional quality. A more in-depthevaluatuion of classroom facilitation behavior involving actual classroom observation acrosscourse cycles is warranted.

86

Outcome - Presence of Learning Community

The emergence of a community of learners is believed to be both an outcome ofsuccessful blended learning and a key determinant of lifelong learning orientation. Lifelonglearners rely on the support of the learning community to acquire information and test ideas,among other things (Haythornthwaite, 2002; Wellman et al., 1988). Seven metrics weredeveloped to capture learning community, but only one was measured in the present evaluationdue to time constraints and emphasis on other metrics of more immediate interest to projectstakeholders. This metric was selected because it directly addresses the use of a LVN LLIcomponent (SharePoint) in enabling community support for learners.

1. Presence (Yes/No) of a student leader using SharePoint or other means tocollaboratively organize student efforts (due date lists, class calendar, etc.). Students weresurveyed to determine whether there was a student leader in their staff group who usedSharePoint to coordinate student efforts. Of the 250 students who reported on student leaders,57% (N = 144) indicated that their staff group had a student leader (appointed or emergent) whoused SharePoint to coordinate. Thirty-six percent (N = 89) indicated that there was a studentleader, but that he or she did not use SharePoint to coordinate. The proportion of student leadersusing SharePoint was greater for August-start students than for February-start students [64%(101 of 159) and 58% (42 of 72), respectively].

Outcome - Student Responsibility for Own Learning

Four metrics were selected to capture the degree to which ILE students had takenresponsibility for their own learning process. Instruction designed to facilitate the adult learningprocess, enabled by a blended learning environment with easy access to web-based resources andexpert communities, should lead to increased resourcefulness on the part of students as theybegin to see themselves as active agents in their own learning (McLoughlin & Oliver, 1999;Sherry & Wilson, 1997) The purpose of the selected metrics was to explore whether studentsleveraged the LVN LLI to conduct independent evaluation in ILE. A detailed description of eachmetric is provided below.

1. Percent of registered students who used Battle Command Knowledge System (BCKS)Leader Network to supplement course materials, complete assignments, or for own research.This metric was captured by asking students to indicate via survey how important BCKS was tothem when searching for current information to complete class assignments or conductindependent research. Students were asked to indicate the importance of the BCKS LeaderNetwork relative to other KM resources. Seventy-five students (30%, 75 of 249) reported thatthey did not search for current information. Of the remaining 174 students who responded to thissurvey question, 30% (N = 52) indicated that they used BCKS as much or more than other(Army and non-Army) KM resources.

2. Percent of registered students who used other Army KM resources (including SMEsand other forums) to supplement course materials, complete assignments, or for own research.Similar to the previous metric, students were asked to report on the relative importance of ArmyKM resources (excluding BCKS, e.g., AKO, Center for Army Lessons Learned, etc.) when

87

completing class assignments or conducting independent research. Twenty-seven students (11%,27 of 250) reported that they did not search for current information. Of the remaining 239students, 60% (N = 144) indicated that they used Army KM resources as much or more thanBCKS or non-Army resources.

3. Percent of registered students who used non-Army resources to supplement coursematerials, complete assignments, or for own research. As above, this metric was captured usinga single survey question in which students were asked to indicate the importance of non-AmyKM resources relative to Army KM resources (including BCKS) when searching for currentinformnation to conduct academic activity. Five students (2%, 5 of 251) reported that they did notsearch for current information. Of the remaining 246 students, 96% (N= 236) indicated that theyused non-Army KM resources as much or more than Army KM resources.

4. Percent of registered students who accessed a particular Blackboard page prior towhen the corresponding class/lesson was held. A single survey question was used to capture thismetric. Students were asked to report how frequently (never, sometimes, often, always, and N/A)they accessed course content in Blackboard prior to a class meeting in order to prepare. Of the249 students who responded to this survey question, 9% (N = 23) indicated that their staff groupdid not have content posted on Blackboard, so they did not access it prior to class. Of theremaining 226 students, 55% reported that they often or always used access to Blackboard toprepare for class.

Summary. A fairly high proportion of students (between 70% and 98%) reported usingknowledge management (KM) resources to search for current information to conduct academicactivity, indicating that students are taking charge of their own learning. Most students whoconducted searches (82%) used non-Army resources alone or more than Army resources,however, so it is unclear whether the LNV LLI in specific has had an impact on studentresourcefulness. Fifty-five percent of students whose course materials were posted in Blackboard(9% claimed that their course materials were not posted in Blackboard) reported that theyaccessed the LVN LLI prior to class in order to prepare. This finding suggests that access fromoutside the schoolhouse is being used to support student performance in the absence of printedmaterials distributed by the college.

Outcome - Learning Self-Efficacy

Learning self-efficacy is believed to be an important outcome of any learning experience,regardless of whether it is technology-supported (e.g., Abell, 2003). The assumption behindassessing learning self-efficacy in the present evaluation was that high levels of learning self-efficacy would reflect the successful application of technology to the learning environment. Inother words, they would reflect that the introduction of technology did not interfere with self-efficacy and that the technology could support levels of self-efficacy that are consistent with anexternal criterion. Learning self-efficacy has only one corresponding metric in the metric chart(see Appendix A), which was assessed in the present evaluation.

1. Percent of students who reported that they were capable of leading their own learningprocess. Students were asked via survey to report how capable they felt of leading their own

88

learning. Students had the option of indicating whether they had the opportunity to lead theirown learning or if students should lead their own learning. Respondents having the opinion thatstudents should be able to lead their own learning had the option of indicating how well theybelieved they accomplished their learning goals. Of the 250 students who responded to thissurvey question, only one felt that students should not lead their own learning. Of the remaining249 students, 88% (N= 222) indicated that they had accomplished (February-start) or wereaccomplishing (August-start) most or all of their learning goals during ILE. This high proportionof students, which was largely the same for the February- and August-starts, suggests that theinstructional technologies comprising the LVN LLI are not preventing high levels of learningself-efficacy, although it is unknown to what degree the technologies may help foster thisorientation or to what degree students goals' for their own learning match those of theeducational institution.

Outcome - Motivation to Learn

As with learning self-efficacy, motivation to learn is a critical outcome of any learningsituation, especially for adult learners for whom education is not compulsory (Abell, 2003).Technology supported learning environments should not reduce motivation to learn, and mayeven enhance motivation, if the technology is easy to access and well-designed. Learningtechnology also may affect motivation indirectly through increased alignment of course contentwith operational demands (i.e., enhanced course relevance). There is only one metric in thecompleted metric chart that is associated with motivation to learn (see Appendix A), and it wasassessed in the present evaluation.

1. Percent of students who reported that they were motivated to learn subjects addressedby the courses they took. Students were surveyed to assess their motivation to learn the subjectstaught in ILE. A single question allowed students to indicate their level of motivation as afunction of content relevance and instructor attitudes. Of the 251 students who reported theirmotivation to learn, 91% (N = 228) indicated that they felt generally (69%, N = 172) or highly(22%, N = 56) motivated (i.e., that course content spoke to their interests and that instructorswere engaging). Although it is impossible to tell from these data what positive impact the LVNLLI has on ILE students' motivation, it is clear that a large proportion of ILE students surveyedwere motivated to learn. This finding suggests that a negative impact probably did not occur.

Outcome - Enhanced Instructional Efficiency

Of the three metrics for instructional efficiency listed in the metric chart, only one wasselected for the present evaluation due to time limitations and the irrelevance of one of themetrics to the LVN LLI. The selected metric was chosen over the other relevant metric becauseit more directly reflects the impact of the educational technology introduced via the LVN LLI.Specifically, the LVN LLI may enhance classroom efficiency by reducing the amount ofteaching time spent doing administrative tasks. Blackboard enables instructors to makeannouncements, distribute handouts, collect homework, and administer exams, among otheradministrative tasks, all outside of the classroom setting.

89

1. Estimated reduction in time spent in classroom doing administrative tasks, includingannouncements, handouts, and testing. In a single survey question, instructors were asked toestimate how much classroom time spent on administrative tasks was reduced (relative to pre-LVN LLI situations) by using Blackboard. Of the 69 instructors who responded to this question,10% (N = 7) indicated that using Blackboard to conduct administrative tasks reduced theirclassroom administration time and 58% (N = 40) reported no change in classroom administrationtime.

Outcome - Enhanced Organizational Commitment

Affective organizational commitment is a component of individual readiness andperceptions of organizational support are a critical determinant of this commitment that can beaffected directly by organizational initiatives (McGonigle et al., 2005). Of the five metricsdeveloped to capture the effect of Lifelong Learning Centers on perceptions of organizationalsupport, only one was selected for measurement in order to manage the scope of the surveyadministered to ILE students and to focus the evaluation on topics of immediate interest tostakeholders.

1. Percent of users who reported feeling that the L VNLLI reflected the Army's intent tosupport the warfighter. Currently comprising the vast majority of users of the LVN LLI, ILEstudents were surveyed to assess the effect of the LVN LLI on perceptions of organizationalsupport. Students were asked what they thought to be the main reason why the Army decided toadminister ILE using Blackboard and SharePoint. Students could identify financial, political,educational, unknown, or other motives for the change. Student suggestions for other motives forthe change were reviewed for their alignment with the given motive categories, and were in somecases re-assigned if a close match was determined. For example, responses indicating that thereason for Blackboard and SharePoint was that the Army wanted to advance educationaltechnology were recoded as educational in nature. Similarly, if a response referred to makingpaper projects digital, it was recoded as financial in nature. Responses suggesting that the Armyhad decided to use Blackboard and SharePoint to keep up with civilian universities or to movethe Army distance learning program along were not recoded.

Of the 251 students who reported their opinion regarding the Army's motives, 33%indicated that they believed the Army's motives were educational in nature and in the bestinterests of leader education. Twenty-five percent of February-start students (20 of 80) selectededucational motives, whereas 37% of August-start students (63 of 169) selected the same. Thisfinding suggests that the LVN LLI currently does not serve to reinforce the majority of students'perceptions organizational support. For this reason, it may have limited impact on affectiveorganizational commitment. The proportion of students for which the LVN LLI may influenceperceptions of organizational commitment appears to be increasing, however, alternativeexplanations for this trend (i.e., the point in the academic program at which students weresurveyed) cannot be ruled out.

90

Outcome - Cost-Effectiveness

Of the six cost-effectiveness metrics presented in the metric and measures chart, one wasselected for the present evaluation--a comparison of the cost per student to deliver ILE usingtraditional means versus the LVN LLI. The comparison cases used to build this metric were thefirst full pilot year of the LVN LLI (February 2006 to present, including all student types) andthe two pilot years of the ILE curriculum change from the traditional CGSOC (2003-2004 and2004-2005, all student types).

An analysis of the cost per student of a blended learning initiative must reflect the costsassociated with distributing course materials as well as those costs associated with producing thecurriculum content and providing instruction and course administration (e.g., enrollment, recordkeeping, etc.; see Cukier, 1997; Levin, 1986). This approach is followed regardless of the meansused for accessing course materials (i.e., classroom, video teleconference, web-based contentrepositories, etc.). A cost accounting approach that goes beyond instructional technology isnecessary because a complex of interrelated expenses comprises the total cost of education,including (a) direct personnel, supplies, and other costs, including printing and shipping ofcourse materials; (b) indirect support costs, such as facilities and maintenance, administrativestaff, faculty development, etc.; and (c) opportunity costs. Indeed, the cost of technology tosupport a blended learning initiative is a relatively minor expense compared to the otherexpenses involved in education (Levin, 1986).

The fiscal promise of blended learning--and the LVN LLI in particular--is that it willreduce printing and shipping costs for course materials (and perhaps eventually travel andhousing costs for students), although a corresponding increase in technology-related costs also isexpected. In order to understand the cost-effectiveness of the LVN LLI in the context ofproviding ILE to all Army majors, the financial tradeoff involved in placing course materials onthe web must be considered as a proportion of bigger picture expenses. Presentingcomprehensive cost data not only allows decision makers to keep educational technology costs inperspective but also allows decision makers to determine what future efforts should have thegreatest impact on the cost of education.

Cost per student for administering ILE in the L VNLLI vs. using traditional means. Todevelop a robust cost per student comparison metric, best practices in cost accounting for highereducation and technology-supported learning were applied (Cukier, 1997; Hyatt, 1983; Levin,1986; Middaugh, 2000; Milam, n.d.). Briefly, these cost accounting methods require capturingthe direct and indirect costs allocated to education delivery. Direct costs include personnelsalaries and benefits, supplies, travel, and other directly allocable expenditures, such as contractsand printing and shipping costs. Indirect costs are broken down into annual, variableexpenditures (i.e., costs associated with institutional support, such as facilities maintenance andfaculty and staff development) and annual use charges associated with fixed expenditures (i.e.,buildings and information technology).

Ideally, all costs, no matter how seemingly trivial, would be accounted for whendeveloping the cost per student metric. There was limited time to collect financial data in thepresent effort however, so decisions were made to manage the scope of the work while

91

producing a reasonable reflection of (a) change in cost per student across mode of contentdelivery; and (b) change in cost per student as a proportion of total cost. The criteria forincluding/excluding cost data from the analysis were relevance to project sponsors, immediacyof involvement in ILE delivery, presence in CGSC financial reporting, and data availability.

A summary of inclusions and exclusions is presented in Table B-2 below. Direct costsassociated with staffing and supplying the TASS Battalions were excluded because these costsare incurred by organizations outside of the CGSC (e.g., the Army Training Support Center) andbecause reproduction and shipping costs lag approximately one year behind the implementation

Table B-2. Inclusions and Exclusions in the Present Cost per Student MetricCategory Item Included Excluded

Direct Costs Personnel SADL personnel and staff CGSS faculty (resident and(salaries, benefits, located at Fort Leavenworth satellite) and admin staffetc.) to deliver (requested data not madeILE available); Faculty and admin

staff located at the TASSBattalions

Supplies CGSS (resident and satellite) Supplies for TASS Battalionsand SADL supplies

Other CGSS printing and reproduction SADL printing and reproductionand shipping to satellite and shipping to TASS Battalionslocations; DOET contract costs and individuals (incurred by theassociated directly with the LVN Army Training Support Center)LLI (hardware, software,supplies, personnel, and travel)

Indirect Institutional DOET and QAO direct costs FSDO direct costs; CGSCCosts Support - CGSC (personnel, supplies, and other executive staff (e.g.,

costs, allocated by estimated commandant) and all otherpercent personnel time spent on institutional support offices inCGSS and ILE) CGSC (e.g., academic outreach,

library, etc.)Institutional None All (e.g., housing, food service,Support - Fort post security, etc.)LeavenworthFacilities Replacement costs for Replacement costs for

information technology, information technology,classrooms, offices, etc., for classrooms, offices, etc., forCGSS presence in Bell Hall CGSS presence in Eisenhower(represented as annual use Hall, for SADL (located incharges, allocated by assignable Eisenhower Hall), for satellitesquare feet of the New Lewis & locations, and for TASSClark Building) Battalions

Opportunity All (i.e., cost of None AllCosts activities not

conducted becauseresources wereallocated to ILE)

92

of a new initiative3. Direct costs for CGSS personnel were excluded because the requested datawere not made available for investigation and in any case would have spuriously increased theLVN LLI cost per student metric due to cost-of-living raises. Selected indirect costs associatedwith institutional support (within the CGSC and across Fort Leavenworth) were excludedbecause they were not of primary interest to the project sponsors and required installation-widecoordination efforts that exceeded the scope of this short-term research. Indirect facilities costsassociated with the CGSC presence outside of Bell Hall (soon to be moved to the new Lewis &Clark Building)--i.e., Eisenhower Hall, satellite campus facilities, and TASS Battalion facilities--were excluded because replacement cost data were not readily available and required significantcoordination beyond the scope of the research. Finally, opportunity costs were not consideredbecause they would be speculative at best.

The overall impact of these exclusions on the cost per student metric is that (a) cost perstudent for both traditional and LVN LLI delivery will be significantly underestimated; and (b)the relative change in cost per student across mode of content delivery will be significantlyoverestimated. In addition, because cost-of-living escalation could not be accounted for, thechange in cost per student is slightly higher than it would be if such escalation was addressed.

The Faculty and Staff Development Office (FSDO) was considered for inclusion as anindirect cost because it supports the educational effectiveness of the LVN LLI throughdeveloping and certifying ILE instructors. Changes in the demand for instructors or in thedemand for specialized instructor skill sets could have an impact on the staffing requirementsand personnel allocation of the FSDO. Similarly, the Quality Assurance Office (QAO) wasconsidered as an indirect cost because the staffing or personnel allocation of the QAO can beaffected by changes introduced by the LVN LLI. The QAO provides annual and periodiceducational program evaluations for the CGSC, and the introduction of new technology producesa requirement to evaluate that technology, perhaps independently of the course to which thetechnology has been applied. FSDO costs were excluded from the present cost metric becausethere has not yet been a change in FSDO staffing requirements due to the LVN LLI. FSDOpersonnel expenses over the past several years associated solely with cost of living increaseswould spuriously increase the LVN LLI cost-per-student relative to previous conditions.

The widely recognized challenges to collecting cost per student metric data include (a)differences between the financial reporting methods of the institution and the financial reportingrequirements of cost accounting; (b) determining the basis for allocating indirect costs; (c)quantifying opportunity costs; and (d) accounting for all known costs associated with educationdelivery (e.g., Middaugh, 2000). Consistent with reports of other program evaluations in highereducation, all of these challenges were encountered in the present evaluation. The cost perstudent comparison metric therefore must be considered an estimate serving illustrative purposesrather than a hard number reflecting actual resource expenditure.

3 Discussion with staff at the Army Training Support Center indicated that approximately $200K in curriculumproduction and shipping will be saved by placing ILE on the web. The initiation of these savings will not beobserved until 2008 and will not be sudden, but placing ILE on the web eventually will result in an annual savingsof $200K.

93

Overall, the abbreviated cost per student for delivering ILE using the LVN LLI relative tothe abbreviated cost of delivering the ILE pilot curriculum using traditional means is greater by afactor of 1.2 . It cost $163.14 more per student in the first year to deliver ILE using the LVN LLIinitiative than using traditional means averaged over two years. This increase is trivial comparedto the overall cost of providing ILE, which is significantly underestimated due to the excludedcosts described above. Had these costs been included, the relative cost of delivering ILE via theLVN LLI would have been even lower.

Selected Output Metrics - Actual System Usage

Three metrics capturing the actual usage of the LVN LLI were measured in an attempt tounderstand better the outcome metrics presented above. Several of these outcomes, such as highproportions of student learning self-efficacy, self-leadership, and motivation, cannot be linkeddefinitively to the LVN LLI as a causal factor because the analogous data were not collectedprior to its implementation. In other words, there was nothing to compare the present outcomedata to, so it is unknown whether it represents a change. Usage data should illuminate thepotential impact of the LVN LLI on the outcomes measured because use of the system isnecessary to achieve impact. Use data also reveals relative demand on the system, which mayhave implications for technical support staffing decisions or for identifying the geographiclocation of most users. The three usage metrics described below were selected in order to explorewhether or not the educational technology made available to curriculum developers, instructors,and students was used to enhance the learning experience. A detailed description of each metricis provided below.

1. Pattern of access as a function of time of day. Because ILE course content is stored inBlackboard, and student access to Blackboard can reflect direct interaction with curriculummaterials, patterns in student hits to the Blackboard document repository were examined.Specifically, the shape of the frequency distribution of hits as a function of time of day (hour byhour) was analyzed. This analysis was conducted using data collected automatically byBlackboard for resident and satellite ILE students.

Between June 20 and September 21, 2006, Blackboard had received 48,652 hits. TheBlackboard data received to conduct further analysis provided only a summary of the frequencydata (i.e., total number of hits, a histogram plot of the hits as a function of time of day, and alisting of the number of hits per time of day and their corresponding percentage), so arepresentative sample of these hits (487, or 1%, preserving the relative frequency of hits as afunction of time of day) was used to plot the distribution of hits and test the normality of thisdistribution, as shown in Figure B-3.

A Kolmogorov-Smirnov test of normality was applied to the sample dataset, and wassignificant (p = .044), suggesting that the distribution of hits a function of time of day is non-normal. The Kolmogorov-Smirnov test is known to be sensitive to large sample sizes, however,and other data (i.e., skewness = -.019; kurtosis = -.335) indicate that the deviation fromnormality is slight. Visual inspection of the data reveals that students primarily access the

4 The exact estimates and calculations used to derive this metric can be obtained by contacting the report author, Dr.Anna T. Cianciolo (acianciolo(cpresearch.net).

94

Blackboard document repository during normal business hours, with reduced access at othertimes, but that there is a spike in activity after the dinner hour, between 7 and 10pm.

Together, these findings represent an expected pattern of access given that the studentsaccessing the Blackboard content repository (i.e., students in residence at Fort Leavenworth andat satellite locations) are in adjacent time zones (i.e., Central and Eastern). The spike in accessafter the dinner hour suggests that students are using the system to conduct educational taskswhile at home or otherwise away from the classroom setting. A more horizontal distributionshould be expected in a complete sample of user activity that captures reach back from widelydistributed locations, such as Hawaii, Korea, Europe, and the Middle East.

50

40

o'30

20

10-

U--

20

0 5 10 15 20

Time of Day

Figure B-3, Pattern of Access to Blackboard Content Repository as a Function of Time of Day

2. Frequency of access to system during normal business hours vs. off-hours. The samedata were used to compute this metric as were used to compute the previous metric. Businesshours were defined as 7amn to 5pm, Central Standard Time, to account for classes beginning at8am in the Eastern time zone. Approximately 69% of student access occurred between 7amn and5pm, and 3 1 % between 5pm and 7am. Of the off-hours hits, approximately 91 % occurredbetween 6amn and 7amn and between 5pm and midnight. Again, this pattern is expected given thestudent population using the system. A greater equivalence between the percentages of on- andoff-hours hits would be more likely if the hits of all types of users (i.e., graduates usingBlackboard for reachback) were tracked.

95

3. Percent of registered users who actively used the system. This metric was captured bycombining selected survey responses given by curriculum developers, instructors, and students.Specifically, a set of four yoked questions was present in the surveys administered to the threeuser types (one question each in the curriculum developer and instructor surveys, two questionsin the student survey), and the responses to these questions were examined collectively. In theirrespective survey, each type of user responded to a question in which they were asked theprimary means by which they carried out educational tasks. For example, curriculum developerswere asked what means they used most often to share course content during the authoringprocess. Instructors were asked the means they used most often to deliver supplemental coursecontent. Students were asked the primary means by which they accessed course content and

5coordinated with other students .

Respondents indicating that they used a means other than the LVN LLI components"most often" or "primarily" to conduct educational activities were not considered active users ofthe system. In sum, forty percent of course authors (4 of 10) used SharePoint to collaborate oncurriculum development, 30% of instructors (21 of 69) used Blackboard to distribute coursematerials, 37% students (92 of 251) accessed Blackboard to retrieve course materials, and 5% ofstudents (13 of 251) used SharePoint to coordinate with one another on group projects or otheractivities. For instructors (N= 69), a greater proportion actively used Blackboard to teachAugust-start than to teach February-start ILE (34% and 24%, respectively). In contrast, a smallerproportion of August-start students reported actively using Blackboard than February-startstudents (28% and 55%, respectively). The reason for the inconsistent pattern of thesedifferences is unknown. The relatively small proportion of students using SharePoint tocoordinate with one another likely is due to the fact that (a) the students surveyed were co-located and preferred face-to-face means of collaboration (38%); and (b) login requirementsmade using SharePoint from home more difficult to use than email (50% reported using email,and survey responses indicated that login requirements were a detractor).

Summary. System (i.e., Blackboard knowledge repository) access data indicate that theLVN LLI is available and used around the clock, with access patterns reflecting those expected ifusers are students primarily located within the same or adjacent time zones and if these studentusers access the system from home to conduct their studies. Peak access activity occurred duringbusiness hours and in the early morning and immediate post-dinner hours.

In the schoolhouse, the LVN LLI components (i.e., Blackboard and SharePoint) appear tobe competing with other alternatives, and even with each other, for usage. That is, only 30% ofsurveyed instructors reported using Blackboard as their primary means for distributing coursecontent, with SharePoint being the most frequently reported alternative (39% of instructors).Students surveyed reasonably reported a similar pattern of use; 37% reported using Blackboardmost often to access course content, citing SharePoint most frequently as the alternative (33%).Only 5% of surveyed students reported using SharePoint to collaborate with other students, withemail and face-to-face meetings most frequently cited as alternatives (50% and 38%,respectively). Comments on student and instructor surveys indicated that there was some

5 It was assumed that 100% of students used Blackboard to access main, or standardized, curriculum content becauseit was not available using other means.

96

confusion as to why there were two systems which have so many overlapping features. Whenforced to make a choice, people appear to use the most readily available or familiar means toconduct their academic tasks.

Selected Activity Metrics - Technical Staff

Four metrics capturing the activity of technical staff were measured. These metrics wereselected in order to explore the degree to which technical staff activity could be expected tosupport curriculum developer and instructor use of the software comprising the LVN LLI (seeDTIC, 2000; Willis, 2000). According to the logic model developed as part of this research,effective use of the software is a critical determinant of the learning effectiveness of the system.A detailed description of each metric is provided below.

1. Percent of instructors and curriculum developers trained to technical criterion. Theoriginal plan for capturing this metric was to (a) audit Blackboard and SharePoint instruction; (b)develop a knowledge or performance measure to capture student achievement of the learningobjectives; and (c) determine the percentage of students performing successfully on the specialpurpose measure. After auditing the Blackboard and SharePoint instruction, however, it wasdetermined that a knowledge or performance measure would not be necessary to capture studentachievement and that all students achieve the technical learning objectives upon completion ofthe instruction.

On-site contractors in the CGSC DOET provide basic Blackboard and SharePointinstruction on every working Friday and more advanced instruction on an as-needed basis forindividuals or groups with special requests. DOET began delivering Blackboard and SharePointinstruction in July of 2006, prior to which the instruction was conducted by the CGSC FSDO.DOET assumed responsibility for conducting the training so that technical instruction could bedelivered by staff dedicated to understanding and maintaining currency with the technology.Participants in the instruction include curriculum developers (i.e., block and lesson authors),instructors, and ILE students (including international students) who have voluntarily sought itout. Basic SharePoint instruction is required for ILE students who are appointed to serve as theinformation management officers for their staff groups. Learner groups in the Blackboard andSharePoint instruction are usually small, averaging approximately four students, and most of theinstruction is conducted on a one-on-one basis.

Both Blackboard and SharePoint basic instruction are accomplished via a combination ofPowerPoint presentation and practical exercises. Class time usually lasts approximately one-hour. The Blackboard instruction has three technical learning objectives: (1) Add anannouncement; (2) Add course content; and (3) Add users to a staff group. The SharePointinstruction has five technical learning objectives: (1) Create a library (document or picture); (2)Add a survey; (3) Add a custom list; (4) Create a view; and (5) Set permissions. Students do notleave the class until they have accomplished these learning objectives.

Two survey questions were developed to address participation in and retention of theBlackboard and SharePoint instruction. Both curriculum developers and instructors were askedwhether they had participated in Blackboard or Sharepoint instruction and whether they had to

97

relearn anything they had been taught during the instruction in order to perform their duties. Of 9course authors and 67 instructors who indicated that they had taken Blackboard or SharePointinstruction, 53% (37 of 67 instructors; 3 of 9 course authors) reported having to relearn theinstructional content.

It is unknown which of these respondents participated in instruction provided by theDOET versus the FSDO. Because all but one of the 10 course authors surveyed had authored forboth February- and August-start ILE, it is unlikely that they participated in DOET SharePointinstruction. However, among the instructors, 76% (N = 13) of those who taught only theFebruary-start reported relearning Blackboard instruction whereas only 44% (N= 17) of theAugust-start reported relearning. This trend suggests that instruction provided by the DOET,which began in July of 2006, may be producing greater retention.

2. Percent of instructors and curriculum developers trained to criterion on contentmanagement/delivery procedures. Interviews with the DOET technical support as well ascurriculum developers and instructors indicated that formal information-management proceduresfor Blackboard or SharePoint do not exist. Instead, some control measures were built into thedesign of the software applications by the DOET technical staff. Beyond these control measures,curriculum developers and instructors were given the authority to create informal information-management procedures in collaboration with their working groups (and with the DOETtechnical staff, if desired). For these reasons, instructors and curriculum developers do notreceive training on content management/delivery procedures.

3. Accessibility, usability, and utility of help documentation for instructor and curriculumdeveloper self-development. The accessibility, usability (i.e., readability), and utility of helpdocumentation was assessed using three separate survey questions in which curriculumdevelopers and instructors reported their perceptions. Instructors reported on the accessibility,usability, and utility of Blackboard help documentation, which includes an electronic version ofthe Blackboard user manual and additional online help for basic functions developed in-house bythe DOET. Curriculum developers reported on the accessibility, usability, and utility ofSharePoint help documentation, which includes an electronic version of the SharePoint usermanual.

Of the 69 instructors who responded to the survey, 66 reported on the accessibility andusefulness of the Blackboard help documentation, and 68 reported on its readability. Asubstantial proportion of respondents for each question indicated that they had never looked foror used the help documentation (44%, 51%, and 57%, respectively). Of those who looked forhelp documentation, 58% (21 of 36) reported that they could usually or always find the topicsthey were looking for. Thirty-nine percent of respondents (12 of 31) who used the Blackboarddocumentation found that it either partially or completely met their information needs, and 41%of respondents (12 of 29) reported that the Blackboard documentation was understandable.

Of the 10 curriculum developers who responded to the survey, 10 reported on theaccessibility of the SharePoint help documentation, 9 on its usefulness, and 8 on its readability.Most course authors reported either having not looked or not used the documentation. The singlecourse author who reported looking for documentation reported usually being able to find it. Of

98

the 3 course authors who reported using the SharePoint help documentation, none of them foundit useful. The one person who reported on the readability of the help documentation (7 of 8reported not using it) indicated that they generally had no trouble with readability. It is unknownwhy more course authors reported having used the help documentation than having looked for it.It also is unknown why the number of authors reporting on the utility of the help documentationwas higher than the number reporting having used it.

4. Perceived accessibility/responsiveness and utility of technical support. Students,instructors, and course authors were surveyed regarding their perceptions of the accessibility andthe utility of the technical support provided by the DOET. The responses of these different usersare summarized in Table B-3 below.

Table B-3. Perceived Accessibility and Utility of Technical Support - Survey ResponsesUser Type % Looked for TS % Accessible % Used TS % Good Experience

Course 80% (8 of 10) 88% (7 of 8) 80% (8 of 10) 88% (7 of 8)AuthorInstructor 71% (49 of 69) 69% (34 of 49) 64% (44 of 69) 73% (32 of 44)Student 71% (178 of 250) 58% (104 of 178) 63% (156 of 247) 63% (98 of 156)Average 71% (235 of 329) 62% (145 of 235) 64% (208 of 326) 66% (137 of 208)Note, TS = Technical Support.

As shown in Table B-3, the majority of course authors, instructors, and students soughtand used the technical support provided by the DOET. Importantly, the majority of people foundthe technical support they were looking for. The proportion of satisfied users was larger forAugust-start students and instructors than for February-start students and instructors (81% goodexperience versus 67% good experience for instructors, and 67% good experience versus 54%good experience for students.) All but one of the course authors had written content for bothFebruary- and August-start ILE.

Summary. The technical staff appears to provide adequate, effective support to LVN LLIusers. It provides basic training on the functions of the LVN LLI components in which the largemajority of instructors and course authors surveyed (>90%) took part. A minority of courseauthors (33%) and August-start instructors surveyed (44%) reported that they had to relearnsome of the instructional material, but it is unclear how many of these people went through thetraining provided by the DOET versus the FSDO. Although formal, institutional SOPs forinformation management in the LVN LLI do not exist, the DOET technical staff assists teachingteams and curriculum development teams in creating their own procedures. Overall, students,instructors, and curriculum developers reported having a good experience with finding and usingtechnical support (62% and 66%, respectively). The reported frequency of good experiences washigher for August-start survey participants (70%) than for February-start participants (56%). Incontrast, the Blackboard and SharePoint help documentation appeared to be of limited utility.The small majority of instructors (58%) could find Blackboard help documentation, but only39% found it useful or readable (41%). The large majority of course authors surveyed did notlook for or use SharePoint help documentation.

Selected Activity Metrics - Curriculum Developers

99

Ten metrics capturing the activity of curriculum developers (i.e., block authors and lessonauthors) were measured. These metrics were selected in order to explore the degree to whichcurriculum developers were actually using SharePoint to collaborate. These metrics also weredesigned to capture how frequently course authors were leveraging resources made available inthe LVN LLI to enhance course content. According to the logic model, effective use ofSharePoint and other LVN LLI resources by curriculum developers is an important determinantof the efficiency and relevance of ILE course content. A detailed description of each metric isprovided below.

1. Perceived ease of using SharePoint to share documents and other materials used ingenerating course content. A single survey question was used to assess curriculum developers'perceptions of the ease of using SharePoint. Of the 9 course authors who responded to thissurvey question, 8 (89%) indicated that it was easy or fairly easy to post and locate curriculumcontent in SharePoint.

2. Percent of curriculum developers who reported being able to use SharePoint outside ofthe schoolhouse setting. A single survey question was used to assess curriculum developers'experiences accessing and using SharePoint from home or some other location other than theiroffice at Fort Leavenworth. Of the 7 course authors who responded to this survey question, 5(71 %) reported having no real problems.

3. Percent of curriculum developers using SharePoint to store, share, and revise coursecontent materials. Curriculum developers' use of SharePoint to store, share, and revisecurriculum materials was captured using a survey question in which respondents selected theprimary means by which they distributed course content to other developers. Respondentsselecting an option other than SharePoint were not considered to be using SharePoint to store,share, and revise because they were actively involved in using some other means. Of the 10curriculum developers who responded to the survey, 4 (40%) reported using SharePoint as theprimary means for distributing course content materials to others.

4. Percent of curriculum developers who reported using means to collaborate otherthan/in addition to SharePoint (e.g., email, thumb drives, etc.). The percentage of curriculumdevelopers using means in addition to/other than SharePoint to collaborate was assessed with thesame survey question described above. Of the 10 curriculum developers who responded to thesurvey question, 6 of 10 (60%) reported using something other than SharePoint as the primarymeans for distributing course content materials to others. The most frequently cited alternativeswere the internal network (2 of 10) and email (3 of 10).

5. Percent of curriculum developers using Battle Command Knowledge System (BCKS)Leader Network to enhance course content. Curriculum developers were asked in a single surveyquestion how important the discussion forums in the BCKS Leader Network were for them whenconducting searches to enhance course content. Respondents answering that the Leader Networkwas equally or more important relative to other resources were assumed to be using actively theLeader Network. Of the 7 course authors who reported searching for current information, all ofthem reported actively using the Leader Network for conducting knowledge searches.

100

6. Percent of curriculum developers using other Army Knowledge Management (KM)

resources to enhance course content. In the same survey question described above, curriculumdevelopers were asked how important other Army KM resources, such as Army KnowledgeOnline or the Center for Army Lessons Learned, were for them when conducting knowledgesearches. Respondents answering that Army KM resources other than BCKS were equally ormore important relative to other resources were considered to be using actively Army KM

resources. Of the 9 course authors who reported searching for current information, 7 (78%)reported actively using Army KM resources for conducting knowledge searches.

7. Percent of curriculum developers using non-Army resources to enhance coursecontent. The same survey question to assess the above two metrics was used to assess this metric.Curriculum developers were asked to indicate how important non-Army resources, such as

Google or Early Bird, were for them when conducting knowledge searches to enhance theircourse content. Of the 10 course authors who reported searching for current information, 2 (20%)reported actively using Army KM resources for conducting knowledge searches.

8. Presence offormal and/or informal standard operating procedures (SOPs)for

leveraging the capabilities of the software applications, including content organization (Yes/No).The presence of formal and/or informal standard operating procedures was assessed byinterviewing the technical support staff and surveying and interviewing curriculum developers.The interviews indicated that SOPs did not exist formally outside of the structure of the

SharePoint site for curriculum developers. Informal SOP development sometimes wasaccomplished when working groups of course authors requested assistance from the technicalstaff in leveraging SharePoint's advanced capabilities. When surveyed about their participationin SOP development, 7 of 9 course authors responded that there were no SOPs.

9. Percent of early adopters involved in SOP development. Curriculum developers weresurveyed to assess how many were involved in SOP development. Early adopters were notdistinguished from late adopters in the present research because all curriculum developers usingSharePoint in the pilot phase and assisting in SOP development were considered early adopters.Of the 9 curriculum developers who reported on SOP development, 2 reported that they had beeninvolved in such development but that the SOPs they worked on did not cover everything andwere not always followed.

10. Presence of a formal and/or informal mentorship system for assisting late adopters

leverage software capabilities (Yes/No). The presence of a formal and/or informal mentorshipsystem was assessed by interviewing the technical support staff and surveying curriculumdevelopers. Both sources of information indicated that there was not a formal mentorshipprogram. Informal mentorship in the form of advanced SharePoint and Blackboard classes, wasprovided by the DOET technical support staff on an as-needed basis. Individual developers orworking groups of developers could request advanced training to meet their specific

requirements. Course authors who reported on mentoring via survey indicated that help withSharePoint by peers was largely informal (4 of 10 respondents reported asking others for help)but that occasionally more formal mentoring occurred (3 of 10 respondents reported havinggroup discussions or regularly helping others).

101

Summary. Although the response rate for course author surveys was 67%, it must benoted that only 15 course authors were invited to participate in the survey, so several of thecurriculum developer metrics are based on the reports of only 10 people. That said, the activeadoption rate of SharePoint was 40% among course authors. Course authors do not appear tohave had much difficulty using or accessing SharePoint. Eighty-nine percent felt SharePoint waseasy or fairly easy to use and 71% reported no significant problems accessing SharePoint fromhome. Instead, adoption rates lower than criterion (i.e., 100%) appear to be linked to the readyavailability of flexible alternatives for sharing documents, such as the college internal networkand email.

Course authors occasionally helped one another to use SharePoint, but there does notappear to be a consistent formal codification of what was learned. Most course authors stated thatthere were no SOPs for posting content in SharePoint, although 2 of 9 claimed to have beeninvolved in the development of SOPs. The most common mode of seeking help was to ask apeer, rather than to develop a collective knowledge base.

Army KM resources appear to have been most important to curriculum developers whenauthoring course content. The large majority of course authors stated that BCKS Leader Network(7 of 7) and other Army KM resources (7 of 9) were equally or more important than non-Armyresources, such as Google and Early Bird. This finding suggests that curriculum developers aremaking use of the available connection to the operational Army, despite the fact that they cannotaccelerate the curriculum production process in order to respond adaptively to what they find.

Selected Activity Metrics - Instructors

Eleven metrics capturing the activity of instructors were measured. These metrics wereselected in order to explore the degree to which instructors were actually using Blackboard todistribute course-related materials. These metrics also were designed to capture how frequentlyinstructors were leveraging resources made available in the LVN LLI to supplement the formalcurriculum. According to the logic model, effective use of Blackboard and other LVN LLIresources by instructors is an important determinant of the relevance of ILE course content(Clark, 1994; Ehnnann, 1994; Owston, 1997). A detailed description of each metric is providedbelow.

1. Percent of instructors who used means other than Blackboard to deliver coursecontent. This metric was assessed with a single survey question in which instructors were askedto report the means they used most often to post course content. Instructors were directed to postall course content to students via Blackboard, so a selection other than Blackboard in response tothis question was intended both to reflect system adoption and to identify the preferredalternatives to system use. Of the 69 instructors who responded to this survey question, 70%reported using something other than Blackboard most often when posting course content. Themost frequently sited alternative means for posting content was SharePoint (39%, 27 of 69).Other alternatives sited included printed materials (17%), email (9%), and the internal network(4%).

102

Of the 17 respondents who taught only February-start ILE, 76% (13 of 17) reported usingmeans other than Blackboard to post content most often, whereas 66% (27 of 41) of the 2006August-start respondents reported the same. This trend suggests increasing adoption ofBlackboard, but that adoption levels remain somewhat low. For both sets of respondents,SharePoint was the most frequently cited alternative to posting in Blackboard (53% and 37%,respectively).

2. Percent of instructors who used BCKS Leader Network to post supplemental coursecontent in Blackboard. Instructors were asked in a single survey question how frequently theyaccessed discussion forums in the BCKS Leader Network to conduct information searches toenhance course content. Respondents answering that they accessed the Leader Network daily orweekly were considered to be using actively the Leader Network. Of the 65 instructors whoresponded to this question, 12% (N = 8) responded that they did not search for supplementalinformation. Of the remaining 57 respondents, 4% (N = 2) reported actively (weekly only) usingthe Leader Network for conducting knowledge searches. Seventy percent reported never using it.

3. Percent of instructors who used other Army KM resources to post supplemental coursecontent into Blackboard. Similar to the survey question above, instructors were asked howfrequently they used other Army KM resources to conduct knowledge searches. Respondentsanswering that they used Army KM resources other than BCKS daily or weekly were consideredto be using actively Army KM resources. Of the 67 instructors who responded to this question9% (N = 6) indicated that they did not search for supplemental information. Twenty-six percentof the remaining respondents (16 of 61) reported actively (daily and weekly) using Army KMresources for conducting knowledge searches. Twenty-eight (17 of 61) percent reported neverusing Army KM resources.

4. Percent of instructors who used non-Army resources to post supplemental coursecontent into Blackboard. The same survey question to assess the above two metrics was used toassess this metric. Instructors were asked to indicate how important non-Army resources, such asGoogle or Early Bird, were for them when conducting knowledge searches to enhance theircourse content. Of the 69 instructors who responded to this question, 7% (5 of 69) indicated thatthey did not search for supplemental information. Seventy-eight percent (50 of 64) reportedactively (daily and weekly) using non-Army resources for conducting knowledge searches.

5. Perceived ease of updating course content within the L VN LLIframework. Instructorswere surveyed to determine their perceptions of how easy it was to use Blackboard to distributesupplements to the formal ILE curriculum. Of the 60 instructors who reported having posted inBlackboard, 58% (N= 35) indicated that that it was fairly easy or easy to post content and 42%reported some or great difficulty. Sixty-four percent of the February-start instructors who postedin Blackboard (9 of 14) reported that it was easy or fairly easy and 64% of the August-startinstructors (23 of 36) reported the same. Thirty-six percent of both samples reported some orgreat difficulty.

6. Percent of instructors who used Blackboard to provide "read ahead" notes for eachclass. In a single survey question, instructors were asked to identify which of severaladministrative tasks they used Blackboard to conduct. The list of administrative tasks included

103

(a) making announcements; (b) delivering handouts; (c) coordinating class activity or groupprojects; (d) administering multiple-choice exams; (e) receiving course products (e.g., termpapers, presentations, etc.); (f) recording grades; and (g) none of the above. Of the 69 instructorswho responded to the survey, 52% (N = 36) reported using Blackboard to provide "read ahead"notes (i.e., to deliver handouts) for each class. Forty-nine percent (N= 34) used Blackboard tomake announcements, 26% (N 18) used it to coordinate class activity or group projects, and12% (N = 8) used it to receive course assignments. Approximately one-third of the sample (N=23) reported not using Blackboard for any administrative tasks.

7. Percent of instructors who used Blackboard to administer course exams. As indicatedin instructor responses to the question described above, 10% (N 7) used Blackboard toadminister course exams. One likely reason for this low percentage is that many of the methodsused to assess student performance in ILE do not involve multiple-choice exams.

8. Percent of instructors using Blackboard to report student grades to administrators.Ten percent of instructors (7 of 69) reported using Blackboard to report grades. Currently,Blackboard is not the official means by which grades are delivered to the college administration.Rather, a different means--the legacy data entry and storage system used by the administration,which is not interoperable with Blackboard--is used. It is likely for this reason that instructorsreported a low adoption rate for the Blackboard grade book. The instructors interviewed did notsee a benefit to recording the same grades twice.

9. Presence offormal and/or informal SOPs for leveraging the capabilities of thesoftware applications for course administration and facilitation (Yes/No). The presence offormal and/or informal standard operating procedures was assessed by interviewing the technicalsupport staff and surveying and interviewing instructors. The interviews indicated that SOPs didnot exist formally outside of the structure of the Blackboard site for instructors. Informal SOPdevelopment sometimes was accomplished when teaching teams requested assistance from thetechnical staff in leveraging Blackboard's advanced capabilities. When surveyed aboutparticipating in the SOP development process, 52% (34 of 66) of instructors stated that therewere no SOPs.

10. Percent of early adopters involved in SOP development. Instructors were surveyed toassess how many were involved in SOP development. Early adopters were not distinguishedfrom late adopters in the present research because all instructors voluntarily and actively usingBlackboard in the LLC pilot and assisting in SOP development were considered early adopters.Of the 66 instructors who responded to this question, 33% (N= 22) reported being involved inthe development process.

11. Presence of a formal and/or informal mentorship system for assisting late adoptersleverage software capabilities (Yes/No). The presence of a formal and/or informal mentorshipsystem for instructors was assessed by interviewing the technical support staff and surveyinginstructors. Both sources of information indicated that there was no formal mentorship program.As described previously, informal mentorship in the form of advanced Blackboard classes, wasprovided by the DOET technical support staff on an as-needed basis. Individual instructors orteaching teams could request advanced training to meet their specific requirements. Forty percent

104

(N = 27) of the 69 instructors who responded to the survey indicated that they were occasionallyor regularly involved in some kind of informal group activity (e.g., discussion groups) whoseintent was to share knowledge about leveraging Blackboard's capabilities.

Summary. Overall, the instructor activity metrics revealed that instructors have been slowto adopt Blackboard both for posting course content and for conducting administrative tasks.Although adoption rates improved from 24% (February-start) to 34% (August-start), a greaterlevel of adoption will be necessary to optimize the impact Blackboard could have on teachingand learning. The metric results do not suggest that technical difficulty is the main barrier toBlackboard use, but rather that instructors have several alternatives readily available (andperhaps more familiar) to them for conducting academic tasks. For instance, instructors mostfrequently reported using SharePoint as their primary alternative to Blackboard for postingsupplemental course content (39%).

There appeared to be interest among instructors to help one another in using Blackboard,but formal codification of what was learned seemed to exist in pockets. For example, about asmany instructors stated that there were no standard operating procedures for posting content inBlackboard as there were instructors reporting that they had participated in the development ofjust such procedures. Moreover, the most common mode of seeking help was to ask a peer,rather than to develop a collective knowledge base.

Of the knowledge resources instructors rely on to supplement curriculum materials, non-Army resources, such as Google and Early Bird, by far have the greatest frequency of access(78% weekly or daily access), relative to Army KM resources (26% weekly or daily access) andthe BCKS Leader Network in specific (4% weekly or daily access). This finding suggests adisconnect between the operational Army and the institutional Army that should be addressedbefore the full impact of the LVN LLI on course relevance can be achieved. However, the abilityto repair this disconnect is beyond the scope of the LVN LLI architects, and the majority ofstakeholders feel that course content is already relevant.

Selected Activity Metrics - Leadership

Six metrics capturing the activity of the LVN LLI leadership were measured. Thesemetrics were selected in order to explore the efficacy of the leadership in overseeing theinitiative, procuring resources, and generating vision in collaboration with stakeholders (see, e.g.,DTIC, 2000; Willis, 2000). According to the logic model, effective leadership is an importantdeterminant of stakeholder adoption of instructional technology, which is critical for achievingeducational impact. A detailed description of each metric is provided below.

1. Stakeholder perceptions of inclusion in the decision-making process. This metric wascaptured by surveying curriculum developers and instructors, asking them if they hadparticipated in the decision making that led to the adoption of SharePoint and Blackboard as areplacement for the system they were used to using. Seven out of 10 (70%) of the curriculumdevelopers indicated that they did not provide input to the decision making process. None of thethree course authors who did report providing input were sure if their input had been used. Of the67 instructors who reported on their involvement in the decision making, 82% (55 of 67) percent

105

indicated that they did not provide input and none of those who reported providing input weresure whether their input had been used.

2. Percent of curriculum developers who understood the "commander's intent "for theLLC. A single survey question addressed this metric. Curriculum developers were asked toindicate their understanding of the LLC concept and were given a series of options from which toselect the statement that best reflected their understanding. Seven of the 10 course authors whoresponded to this survey question selected a definition that captured at least some aspect of theintent of the LLC concept. Three of 10 selected a definition that closely resembled statements inthe LLC planning documentation.

3. Percent of instructors who understood the "commander's intent "for the LLC. In aseparate survey, instructors were asked the same question as the curriculum developers. Sixty-two percent (N = 42) of the 67 instructors who responded to this survey question selected adefinition that captured at least some aspect of the intent of the LLC concept. Forty-eight percent(N = 33) selected a definition that closely resembled statements made in the LLC planningdocumentation.

4. Adequacy offunding for the LL. This metric was captured by an interview with theDOET leadership in which the leadership was asked about the adequacy of LLI funding and thechallenge areas in procuring funding for the LLI. The DOET leadership indicated that there wasno difficulty securing funding to meet the costs of the LLI and that the current funding levelswere adequate for meeting LLI needs.

5. Adequacy ofpersonnelfor the L VN LLL This metric was captured partially by aninterview with the DOET leadership in which the leadership was asked about the adequacy ofpersonnel for supporting the LVN LLI and, if inadequate, what additional personnel must behired to effectively support the LVN LLI. The DOET leadership indicated that current technicalstaff levels were sufficient for meeting LVN LLI needs. It also was communicated, however, thatdown time in the availability of the LVN LLI to students, faculty, and curriculum developers wasa result of preventable mistakes by technical personnel. For example, a week-long outage ofBlackboard was attributed to the failure of the technical staff to follow common procedures forbacking up database contents. A 45-minute outage of SharePoint was attributed to the failure ofthe network administrator to maintain the appropriate access permissions to the FortLeavenworth domain.

The DOET leadership emphasized that although the Web presence of the LVN LLI hasachieved very high levels of stability, even minor downtime can have a substantial impact onuser trust in the LVN LLI's ability to meet instructional needs and consequent use of the system.Activities reported as necessary to foster user trust after system downtime included explainingthe cause of the downtime and the estimated time to solve system problems. Methods for sharingthis explanation included telephone, email, and face-to-face meetings.

Technical staff asked about the adequacy of the personnel supporting the LVN LLIindicated that there were fewer personnel available than required for the start-up phase of theinitiative (8 personnel at the time of this reserarch--as opposed to 10--with 3 of the eight

106

personnel hired on in the summer of 2006). Competing demands on limited staff time may

account for the occasional errors leading to system down time. However, the technical staff

indicated that the current number of personnel would be sufficient to run the LVN LLI when it

has achieved a steady state and that staffing decisions must balance present and future needs.

6. Adequacy of technology for the L VNLLI. This metric was captured by an interviewwith the DOET leadership in which the leadership was asked about the adequacy of technologyfor supporting the LVN LLI and, if inadequate, what additional technology must be procured toeffectively support the LVN LLI. The DOET leadership indicated that the technology currentlyavailable to the DOET was adequate for meeting LVN LLI needs.

Summary. The LVN LLI leadership has successfully procured resources to ensure theadequate functioning of the technical aspects of the initiative. This emphasis on technologicalfunctioning is consistent with the mission of the DOET, which was assigned to lead theimplementation of the initiative. There appear to be, however, some less technical leadershiptasks, arguably outside of the scope of the DOET, that have not been addressed and likely haveimplications for system adoption. These interpersonal leadership tasks include communicatingthe educational vision of the LVN LLI and winning the stakeholder buy-in. A large majority(81 %; 62 of 77) of instructors and course authors reported that they did not provide input to thedecision-making process that has created new work demands for them. In addition, there isevidence that nearly half (53%, 41 of 77) of the surveyed stakeholders' (i.e., course authors andinstructors) were unaware of the intent of TRADOC's lifelong learning initiative.

Conclusions and Recommendations

Results Summary

Overall, the evaluation results indicate that the LVN LLI is a cost-effective solution forenhancing the educational outreach of ILE curriculum materials. For what amounts practically tothe same amount of money, ILE delivered via the LVN LLI will eliminate the one-year lag incurriculum content between the schoolhouse and Army reserve facilities and already supportsanytime, anywhere learning for ILE students in residence and at satellite campuses. There isrelatively little the LVN LLI can do to reduce the 6-10 month lag between changes in theoperational environment and revisions to the standardized ILE curriculum, but it is assisting incircumventing this problem by making it easier for instructors to supplement curriculummaterials. The majority of students, instructors, and course authors surveyed reported that coursecontent was relevant to the jobs of ILE graduates.

The active adoption rate of the LVN LLI by course authors, instructors, and students inthe schoolhouse remains well below 100%. Survey and interview data indicated that technicaldifficulty was not the main barrier to system use. Rather, the main barriers to system use appearto have been (a) ready availability of alternatives to the system that were more familiar andeasier to access; (b) inconsistent or absent information management procedures necessary tomake posted content easy to find; and (c) lack of instructor and course author involvement withand buy-in to the lifelong learning concept. As has been found in previous evaluations of this

107

kind (Cianciolo et al., 2006), greater success has been achieved in the technical implementationof the initiative than in the cultivation of stakeholder enthusiasm and investment.

Limited adoption of the LVN LLI did not appear to hamper the academic experience ofILE students and instructors in the first-year pilot. Large proportions of students reportedeffective classroom facilitation behaviors by instructors, high levels of learning self-efficacy andmotivation, and active involvement in their own learning through independent study. Themajority of instructors indicated that the LVN LLI did not negatively impact classroomefficiency, and a small proportion of instructors reported that the LVN LLI enhanced theirclassroom efficiency.

Recommendations

It would be feasible to recommend that no changes be made to the LVN LLIimplementation. If nothing about the LVN LLI implementation were to change from this pointforward, ILE instruction would continue successfully, and students--especially those at Armyreserve facilities--would benefit from greater access to more relevant standardized curriculum,all at relatively little cost to the Army. Moreover, adoption rates likely would increase on theirown [albeit slowly and probably not to criterion (i.e., 100%)], as indicated by the generallyupward trend in adoption rates occurring during the first-year pilot.

That said, the findings of the present evaluation illustrate the importance of taking a logicmodeling approach to understanding the impact of lifelong learning initiatives. Assessment ofthe LVN LLI's outcomes alone would have indicated that the initiative had achieved its goals butwould have obscured the fact that a subset of these goals--teaching and learning effectiveness--was achieved independently of the use of the LVN LLI's components. Put another way, the LVNLLI has yet to have an impact on enhancing the ILE teaching and learning environment becausethe unique capabilities of blended learning are not yet being leveraged. Comments made bysurveyed students who had used Blackboard as part of their civilian education support this pointwith observations that civilian educators were more effective at leveraging Blackboard'scapabilities to create a true blended learning environment. In addition, survey responses indicatethat neither a majority of students (42%, 104 of 250) nor of instructors (26%, 16 of 61) was yetconvinced that Blackboard would enhance ILE education. A typical "no significant difference"between traditional and technology-assisted learning may be found in this case (see Russell,1999), but this finding indicates that the blended learning initiative is not living up to itscapability to transform the classroom and engender a cultural shift toward a lifelong learningorientation.

In order for the already successful LVN LLI to achieve optimal educational andorganizational impact, stronger beliefs in its purpose and higher active adoption rates of itscomponents must be achieved. The accomplishment of this goal requires fostering the instructorand course author buy-in necessary to learn and adopt unfamiliar technologies and to role-modeltheir use for students. Improved buy-in may be developed through (a) engaging otherschoolhouse components in viewing and cultivating LVN LLI-assisted teaching as a criticalinstructor competency; (b) encouraging and shepherding the involvement of instructors andcourse authors in the development of information management procedures; and (c) spreading

108

strategic communications that explain the purpose and goals of the system, that anticipatetechnical limitations of the system (i.e., login requirements), and that immediately follow systemoutages.

Other Schoolhouse Components. As described previously, the CGSC DOET isresponsible for the implementation, maintenance, and oversight of the LVN LLI. However, avisual inspection of the CGSC organizational chart reveals that other components of theschoolhouse must be involved if a culture of acceptance and engagement in blended learning isto be infused throughout the college. Chief among these components is the Faculty & StaffDevelopment Office, which could assist in developing instructors' ability to visualize andtransform the learning experience using Blackboard, even involving Blackboard in the facultydevelopment process itself. Similar assistance could be provided to course authors usingSharePoint. Importantly, successfully delivering blended learning must not be viewed as a matterof technology alone, but of effectively integrating educational need, instructional strategy, andadvanced technologies (Clark, 1994; DTIC, 2000; Firdyiwek, 1999; Willis, 2000).

Information Management. The vast majority of recommendations from surveyed studentswas that the schoolhouse choose either Blackboard or SharePoint for posting content andeliminate entirely the system not selected. In both interviews and survey comments, studentsmentioned being confused by instructors' inconsistent use of two systems with overlappingcapabilities and were unable to find content as a result. This confusion appears to have stemmedfrom ineffective or absent information management procedures on the part of faculty whoinstead of using the required system (Blackboard) as the primary means for posting content useda more familiar alternative (SharePoint). Faculty and course authors who communicatedreluctance to use either system via interview listed inadequate information management as alimiting factor.

The DOET technical staff supports the development of instructor and course authorfacility with the LVN LLI technologies, even assisting in the development of informationmanagement strategies within teaching and authoring teams and modifying the LVN LLIinterface to address known information management challenges. However, the technical staff isnot responsible for developing school-wide information management procedures, and thisresponsibility does not appear to be held elsewhere. Instructors and course authors informallyhelp each other, but there is no widely-recognized standard for collaboration. A formalassessment of the information-sharing strategies currently used by faculty and course authorsshould be conducted to understand user needs and to reveal best practices that can be broadlyapplied. Instructors, curriculum developers, and even students should be actively involved in theprocess of articulating need and should participate in system interface design revisions. Ideally,leadership of this needs assessment and design process would be executed by an integrated teamof technologists, faculty and staff developers, department chairs, instructor and course authorteam leaders, and student representatives (e.g., staff group information officers).

Strategic Communications. Strategic communications follow system outages, but do notappear to occur as anticipatory events that could shape perceptions and advance organizationalgoals. For example, the CGSC Circular 350-1, which describes the CGSC, its components, andILE, does not mention the LVN LLI or TRADOC's lifelong learning initiative in general. A

109

paragraph or two in the circular explaining the technologies and their intended use would helpresident students to understand what the expected benefit is of the system, to view themselves aspart of a larger ILE student body that benefits from outreach, and perhaps to see that the LVNLLI implementation has been made in the best interests of leader education. Currently, aminority of students see the initiative as motivated by the desire to improve education, but a shiftin this perspective is necessary to empower the initiative to enhance organizational commitmentand individual readiness.

Strategic communications that anticipate technical limitations also may be helpful infostering user support and understanding when difficulties occur. Such communications mayinclude advertising Blackboard and SharePoint instruction for students and explaining the sourceof multiple login requirements and the reason for two overlapping systems.

Concluding Comments

Ultimately, decisions regarding the future of an initiative are values based (Cukier, 1997).This evaluation report cannot provide values. It is merely intended to support the values-baseddecision making of others with objective data. The objective data indicate that the LVN LLI istechnically well implemented, cost-effective, and enhances educational outreach. The objectivedata also indicate that some investment is required to strengthen the initiative's impact onteaching and learning. This representation of the LVN LLI places it squarely within thepopulation of technology-assisted learning initiatives working through their first-year pilot andwithin the population of maturing Army programs designed to support human functioning withtechnological means. The close coupling of the LVN LLI technologies with the needs andexpectations of people who use them will transform the already successful initiative into a potenteducational and organizational multiplier.

110

APPENDIX B- 1CURRICULUM DEVELOPER SURVEY

The curriculum developer survey was administered online for 7 days in collaborationwith the CGSC Quality Assurance Office. Curriculum developers who authored course contentfor either the 2006 February-start ILE class or the 2006 August-start class (or both) received aninvitation via email to participate in the survey. Ten curriculum developers responded to thesurvey, representing a 67% overall response rate. Of these 10 individuals, 0 had authored onlythe February-start ILE curriculum, 1 had authored only the August-start ILE curriculum, and 9had authored curriculum materials for both classes. Due to the small number of respondents ineach category, the data were collapsed across all course authors for each question. A completereproduction of the survey is shown below.

SharePoint and Resident ILE - Course Author Experience

The 2006 February-start and 2006 August-start CGSOC classes are serving as test casesfor using SharePoint (instead of the g: drive) as the primary means to post and share coursecurriculum materials during the Course Authoring process. Working within this pilot program,you have probably experienced the inevitable challenges associated with transitioning from onetechnology to another. Likely, there also have been some unanticipated benefits of the change.

The purpose of this brief, 21-item survey is to capture selected aspects of the CourseAuthor experience using SharePoint to post and share common core curriculum materials. Thesurvey has 3 parts relating to:

1. Technical Support;2. Curriculum Development; and3. Participation in the Transition Process.

The goal of this survey is to take the "pulse" of the transition effort -- to identify thestrengths and weakness of the technical support provided to Course Authors as well as the actuallevel of Course Author use of SharePoint. The survey also is intended to reveal the level ofparticipation Course Authors perceive they have had in the process of transitioning from the g:drive to SharePoint.

This survey is completely anonymous and voluntary. Your responses are in no wayassociated with your name and you may choose, without consequence, not to participate.

As you fill out this survey, consider your experiences in developing curriculum materialsfor the 2006 February-start or the 2006 August-start ILE CC. Select the response that bestdescribes your experiences associated with each statement. Also, there are optional open endedquestions at the end of each section to add comments.

If you have any questions concerning the content of this survey, contact Dr. AnnaCianciolo, 217-621-3918. If you experience technical difficulties with it, contact Mr. DouglasLoa, 913-684-7275.

111

This Survey has been approved by QAO and the Survey Control number is 06-080

Section I - Technical Support

1. How accessible did you find the technical support to be?

()What tech support? I didn't know there was tech support.(I knew how to find the tech support team, but actually getting a hold of someone was

unpredictable; on more than one occasion I had to wait longer than I would have liked.()I wouldn't know; I knew there was a tech support team, but I never tried to contact

anybody.(I generally didn't have trouble finding or connecting with someone in tech support; if I

ever had to wait, it wasn't a big deal.()Finding and connecting with tech support was no problem at all; I always got

somebody when I needed them.

2. How useful did you find the technical support to be?

()Not at all useful; I had to find other ways to fix my problems."()They solved technical problems ok, but if the tech support was better, the kinds of

problems I had wouldn't exist.(I wouldn't know; I never asked for tech support.()I had a pretty good experience; the system worked well enough and when problems

did come up, they were usually fixed the first time around.()They're the best; the system worked great and whenever I needed technical assistance,

it was provided quickly, effectively, and professionally.

3. How accessible did you find the SharePoint help documentation to be?

()What help documentation? I didn't know there was help documentation.(I think there was supposed to be help documentation, but I sure couldn't find it.()I wouldn't know; I never looked for help documentation.(I could usually find help documentation on the topics I needed to know about.(I could always find the topic I was looking for in the help documentation.

4. How useful did you find the SharePoint help documentation to be?

()Not at all useful; I don't even bother looking anymore.(It was ok for the topics it handled, but it usually didn't handle the topics I needed help

in.(I wouldn't know; I never used the help documentation.(It generally provided useful information, but I don't think it handled everything.(It always met my information needs.

112

5. How usable (readable) did you find the SharePoint help documentation to be?

()The help documentation was mostly gibberish written by someone who knows thesystem but not how to write.

(It was ok; I usually had to read it a couple of times over to get what it was saying, buteventually I understood it.

()I wouldn't know; I never used the help documentation.(I usually didn't have trouble understanding the help documentation, but I'm used to

reading technical writing.()The help documentation was easy to understand; even my grandmother would get it!

6. Have you participated in SharePoint instruction?

()Yes()No - I already know how to use SharePoint()No - I didn't know there was SharePoint instruction()No - I didn't have time

6a. Did you have to re-learn anything you were taught in order to perform instructional tasks?

()Yes()No

Additional Comments regarding the Technical Support (optional):

Section II - Curriculum Development

7. How knowledgeable do you feel are with using SharePoint?

your students

()Very knowledgeable()Knowledgeable()Somewhat knowledgeable( ) Very little()Not at all

you

()Very knowledgeable()Knowledgeable()Somewhat knowledgeable() Very little()Not at all

113

8. When you distributed curriculum materials to other Course Authors, what means did you usemost often?

(SharePoint()g: drive()Distributed via email()Distributed via memory stick/thumb drive()Printed materials()Other (e.g., AKO personal webpage) [

9. How easy was it to post and locate curriculum materials in SharePoint?

(It was very difficult; I mostly used other means to share resources with CourseAuthors.

()Let's just say it wasn't the g drive; it limited my options for sharing information withthe people who I felt needed it.

()No idea; I never posted in SharePoint.()Fairly easy, but the g: drive is more flexible.()Piece of cake; I had no problems at all.

For the following question, consider your experience in past years writing ILE common corecurriculum materials.

10. How frequently were you able to update/revise course content based on new developments inthe field, instructor or student feedback, etc.?

()Regularly; prior to each course administration.()Semi-regularly; prior to most administrations of the course, but not all.(Infrequently; not often enough to do much good.()Never; revisions are so out of synch with course administrations that I don't even try

anymore.)N/A; I've only worked on the core curriculum once.

For the following question, consider your experience writing ILE common core curriculummaterials since the introduction of SharePoint.

11. How has using SharePoint changed the frequency with which you update/revise coursematerials?

()Frequency greatly decreased; Time that was spent modifying course content is nowspent trying to find and/or store files on SharePoint.

()Frequency somewhat decreased; SharePoint makes it harder for Course Authors toshare the resources that would enhance or synchronize course content.

()Not at all; Delays in revising course content have nothing to do with technology - theyare administrative in nature.

()Frequency somewhat increased; SharePoint makes it easier for Course Authors toshare the resources that would enhance or synchronize course content.

114

()Frequency greatly increased; One-stop access to professional forums, Armyknowledge repositories, and other Course Authors has revolutionized the way we dobusiness.

(N/A; I've only worked on the core curriculum once.

Additional Comments regarding Curriculum Development.(optional):

12. The SharePoint Learning System enhances your students ILE education.

()Strongly Agree()Agree()Undecided()Disagree()Strongly Disagree

13. How easy was it to access SharePoint from outside the schoolhouse?

(I tried to work from home (or away), but never could get access.(I could occasionally get access to SharePoint, but not often enough to make working

from outside the schoolhouse a viable option.(I wouldn't know; I never tried to access SharePoint from outside the schoolhouse.

I was able to get access to SharePoint from outside the schoolhouse when I needed it,but I don't see being able to work from outside the schoolhouse as a big advantage.

()Very easy; I had no problem with access and it enabled me to get valuable work donewhen I otherwise would not have been able to do it.

14. How current or relevant did you feel the core curriculum you developed was to the upcomingjob demands of the ILE students?

()Not at all relevant()It was no less relevant than one can expect given the rapidly changing operational

environment and restrictions on using FOUO content in the classroom)It covered relevant topics, but remember our focus is on education, not missionpreparation

()It was definitely more relevant than it used to be, but we're not there yet()Totally relevant; Graduates will be able to arrive in their units completely prepared to

fulfill the requirements of their duty position

15. When you searched for current information to write curriculum materials, how would yourate the importance of the following resources:

Professional forums in the Battle Command Knowledge System (BCKS) Leader Network

()Extremely Important()Somewhat important()Equal Importance()Not too important

115

()Not important at all

()Didn't search for current information

Army knowledge repositories (e.g., AKO, CALL, etc.)

()Extremely Important()Somewhat important()Equal Importance()Not too important()Not important at all()Didn't search for current information

Non-Army resources (e.g., Google, Early Bird, etc.)

()Extremely Important()Somewhat important()Equal Importance()Not too important()Not important at all()Didn't search for current information

Section III - Participation in the Transition Process

16. Did you assist in developing standard operating procedures within your department and/orblock for posting and organizing content in SharePoint?

()What standard operating procedures? There are none to my knowledge.()I was asked to help with developing standard operating procedures but didn't have

time to get involved; I'm not sure if they were ever completed.(I believe we have standard operating procedures, but they came from the tech support

people, not the instructors.(I was involved in developing the standard operating procedures, but they don't cover

everything and aren't always followed.(I participated fully in the development of standard operating procedures; they are

widely available and followed.

17. Do you feel that there are enough faculty to teach ILE, given the additional classes enabledby delivering courseware on the web?

(Yes()No

18. Did you participate in formal or informal mentoring among Course Authors on usingSharePoint to share curriculum materials (either as a mentor or a "mentee")?

()There was no mentoring--formal or informal--among Course Authors.

116

(I believe Course Authors did help each other with SharePoint, but I was neverinvolved.

()Sometimes if I needed help, I would ask another Course Author/Sometimes ifsomeone needed help, they would ask me.

()Every once in a while, a few of us would have a conversation about how usingSharePoint could change the way we teach.I was regularly involved in helping others leverage SharePoint's capabilities.

19. Which of the following classes did you author for?

February Start()Yes()No

August Start()Yes()No

20. SharePoint is one component of the Fort Leavenworth Lifelong Learning Center (LLC).What is your understanding of what this LLC is?

()Never heard of it.(It's a new learning initiative pushed down by higher headquarters.()It's a way to conduct resident and satellite instruction using the same course materials

so all majors can get the same core intermediate-level education.()It's an online repository of curriculum materials meant to support anytime/anywhere

access to the institution.()It's a networked, collaborative environment designed to link the operating and

generating forces through common access to course materials, emerging doctrine, anddiscussion forums.

21. Did you participate in the decision-making process that led to the requirement to useSharePoint instead of the g drive?

(I had no idea they were considering a change; soliciting Course Author input wouldhave been helpful.

(I wasn't included in the decision making, but I don't feel I needed to be.I was made aware that a change was being considered as an alternative to the g drive,

but I didn't provide any input to the decision-making process.()Somebody asked me what I felt about using something other than the g: drive, but I'm

not sure my input really made a difference one way or the other.(I had full visibility on the decision-making process and felt my input was received and

considered.

22. What would you like the CGSC leadership to know about any aspect of the SharePointSystem?

117

This completes the survey.

Again, thank you for participating in this survey. Your time and responses are valued.

118

APPENDIX B-2INSTRUCTOR SURVEY

The instructor survey was administered online for 9 days in collaboration with the CGSCQuality Assurance Office. Instructors who taught either the 2006 February-start ILE class or the2006 August-start class (or both) received an invitation via email to participate in the survey.Sixty-nine of 201 instructors responded to the survey, representing a 34% overall response rate.Of these 69 individuals, 25% (N = 17) had taught only for the February-start ILE class, 59% (N--41) had taught only for the August-start ILE class, and 13% had taught both classes (N= 6). Tworespondents (3%) reported having taught neither class, but had used Blackboard for teachingother components of the CGSOC.

Blackboard, SharePoint and Resident ILE - Instructor Experience

The 2006 February-start and 2006 August-start CGSOC classes are serving as test casesfor using Blackboard (instead of the g: drive) as the primary means of distributing curriculummaterials. Working within this pilot program, you have probably experienced the inevitablechallenges associated with transitioning from one technology to another. Likely, there also havebeen some unanticipated benefits of the change.

The purpose of this brief, 23-item survey is to capture selected aspects of the instructorexperience using Blackboard to deliver the ILE CC curriculum and to conduct classroomeducation. The survey has 4 parts relating to:

1. Technical Support;2. Online Course Administration;3. Classroom Course Administration; and4. Participation in the Transition Process.

The goal of this survey is to take the "pulse" of the transition effort -- to identify thestrengths and weakness of the technical support provided to instructors as well as the actual levelof instructor use of Blackboard. The survey also is intended to reveal the level of participationinstructors perceive they have had in the process of transitioning from the g: drive to Blackboard.

This survey is completely anonymous and voluntary. Your responses are in no wayassociated with your name and you may choose, without consequence, not to participate.

As you fill out this survey, consider your experiences in delivering instruction for the2006 February-start or the 2006 August-start ILE CC. Select the response that best describesyour experiences associated with each statement. Also, there are optional open ended questionsat the end of each section to add comments.

If you have any questions concerning the content of this survey, contact Dr. AnnaCianciolo, 217-621-3918. If you experience technical difficulties with it, contact Mr. DouglasLoa, 913-684-7275.

119

This Survey has been approved by QAO and the Survey Control number is 06-078

Section I - Technical Support

1. How accessible did you find the technical support to be?

()What tech support? I didn't know there was tech support.(I knew how to find the tech support team, but actually getting a hold of someone was

unpredictable; on more than one occasion I had to wait longer than I would have liked.()I wouldn't know; I knew there was a tech support team, but I never tried to contact

anybody.(I generally didn't have trouble finding or connecting with someone in tech support; if I

ever had to wait, it wasn't a big deal.()Finding and connecting with tech support was no problem at all; I always got

somebody when I needed them.

2. How useful did you find the technical support to be?

()Not at all useful; I had to find other ways to fix my problems."()They solved technical problems ok, but if the tech support was better, the kinds of

problems I had wouldn't exist.()I wouldn't know; I never asked for tech support.(I had a pretty good experience; the system worked well enough and when problems

did come up, they were usually fixed the first time around.()They're the best; the system worked great and whenever I needed technical assistance,

it was provided quickly, effectively, and professionally.

3. How accessible did you find the Blackboard help documentation to be?

()What help documentation? I didn't know there was help documentation.(I think there was supposed to be help documentation, but I sure couldn't find it.()I wouldn't know; I never looked for help documentation.()I could usually find help documentation on the topics I needed to know about.(I could always find the topic I was looking for in the help documentation.

4. How useful did you find the Blackboard help documentation to be?

()Not at all useful; I don't even bother looking anymore.(It was ok for the topics it handled, but it usually didn't handle the topics I needed help

in.(I wouldn't know; I never used the help documentation.(It generally provided useful information, but I don't think it handled everything.(It always met my information needs.

120

5. How usable (readable) did you find the Blackboard help documentation to be?

()The help documentation was mostly gibberish written by someone who knows thesystem but not how to write.

(It was ok; I usually had to read it a couple of times over to get what it was saying, buteventually I understood it.

(I wouldn't know; I never used the help documentation.(I usually didn't have trouble understanding the help documentation, but I'm used to

reading technical writing.()The help documentation was easy to understand; even my grandmother would get it!

6. Have you participated in Blackboard instruction?

()Yes()No - I already know how to use Blackboard()No - I didn't know there was Blackboard instruction()No - I didn't have time

6a. Did you have to re-learn anything you were taught in order to perform instructional tasks?

()Yes()No

Additional Comments regarding the Technical Support (optional):

Section II - Online Course Administration

7. How knowledgeable do you feel are with using Blackboard?

your students

()Very knowledgeable()Knowledgeable()Somewhat knowledgeable( ) Very little( ) Not at all

you

()Very knowledgeable()Knowledgeable()Somewhat knowledgeable()Very little()Not at all

121

8. When you distributed supplemental curriculum materials, what means did you use most often?

()Blackboard()SharePoint()g: drive()Distributed via email()Distributed via memory stick/thumb drive()Printed materials()Other (e.g., AKO personal webpage) [

9. How easy was it to post supplemental curriculum materials in Blackboard?

()It was very difficult; I mostly used other means to share resources with my students.()Let's just say it wasn't the g drive; it limited my options for sharing information with

the people who I felt needed it.()No idea; I never posted in Blackboard.()Fairly easy, but the g: drive is more flexible.()Piece of cake; I had no problems at all.

10. How frequently did you supplement the core curriculum materials based on current events(for example, by distributing a relevant news story or emerging doctrine)?

()Daily(Weekly()Every couple of weeks()Monthly()Never

11. How frequently did you supplement the core curriculum materials relative to past ILEcourses when you used the g: drive for posting course content?

()A lot more frequently()Somewhat more frequently()Pretty much the same()Somewhat less frequently()A lot less frequently(N/A - I don't supplement curriculum materials(N/A - This is my first time teaching the course

12. Which of the following administrative tasks did you use Blackboard to conduct (check allthat apply)?

()Making announcements (e.g., for changes to readings)()Delivering handouts (e.g., supplemental read-aheads)()Coordinating class activity or group projects()Administering (multiple-choice) exams()Receiving course products (e.g., term papers, presentations, etc.)

122

()Recording grades()None of the above

13. When you searched for current information to supplement the curriculum materials, howoften did you use:

Professional forums in the Battle Command Knowledge System (BCKS) Leader Network

()Daily()Weekly()Every couple of weeks()Monthly()Never()Didn't search for supplemental information

Army knowledge repositories (e.g., AKO, CALL, etc.)

()Daily()Weekly()Every couple of weeks()Monthly()Never()Didn't search for supplemental information

Non-Army resources (e.g., Google, Early Bird, etc.)

()Daily()Weekly()Every couple of weeks()Monthly()Never()Didn't search for supplemental information

Additional Comments regarding the Online Course Administration (optional)

Section III - Course Administration

14. The Blackboard Learning System enhances your students ILE education.

()Strongly Agree()Agree()Undecided()Disagree()Strongly Disagree

123

15. How much would you estimate that doing administrative tasks in Blackboard reduced thetime you had to spend on administration in class?

(I actually had to spend more class time on admin because students often had problemswith the system.

()No real reduction in in-class admin time.()Some reduction in in-class admin time; probably there will be a greater reduction

when I learn more about how to use the system.()Reduced the time to zero; I did not have to do admin tasks in class, so we devoted

class time entirely to learning.

16. How current or relevant did you feel the core curriculum (with supplements) was to theupcoming job demands of your students?

()Not at all relevant(It was no less relevant than one can expect given the rapidly changing operational

environment and restrictions on using FOUO content in the classroom(It covered relevant topics, but remember our focus is on education, not mission

preparation()It was definitely more relevant than it used to be, but we're not there yet()Totally relevant; Graduates will be able to arrive in their units completely prepared to

fulfill the requirements of their duty position

Additional Comments regarding the Course Administration (optional):

Section IV - Participation in the Transition Process

17. Did you assist in developing standard operating procedures within your teachingteam/section for posting, organizing, and sharing (e.g., across staff groups) content inBlackboard?

()What standard operating procedures? There are none to my knowledge.I was asked to help with developing standard operating procedures but didn't have

time to get involved; I'm not sure if they were ever completed.)I believe we have standard operating procedures, but they came from the tech supportpeople, not the instructors.I was involved in developing the standard operating procedures, but they don't covereverything and aren't always followed.

(I participated fully in the development of standard operating procedures; they arewidely available and followed.

18. Do you feel that there are enough faculty to teach ILE, given the additional classes enabledby delivering courseware on the web?

()Yes()No

124

19. Did you participate in formal or informal mentoring among instructors on using Blackboardfor instruction (either as a mentor or a "mentee")?

()There was no mentoring--formal or informal--among instructors.(I believe instructors did help each other with Blackboard, but I was never involved.()Sometimes if I needed help, I would ask another instructor/Sometimes if someone

needed help, they would ask me.()Every once in a while, a few of us would have a conversation about how using

Blackboard could change the way we teach.(I was regularly involved in helping others leverage Blackboard's capabilities.

20. Which of the following classes did you teach in?

February Start()Yes()No

August Start

()Yes()No

21. Blackboard is one component of the Fort Leavenworth Lifelong Learning Center (LLC).What is your understanding of what this LLC is?

()Never heard of it.()It's a new learning initiative pushed down by higher headquarters.(It's a way to conduct resident and satellite instruction using the same course materials

so all majors can get the same core intermediate-level education.()It's an online repository of curriculum materials meant to support anytime/anywhere

access to the institution.()It's a networked, collaborative environment designed to link the operating and

generating forces through common access to course materials, emerging doctrine, anddiscussion forums.

22. Did you participate in the decision-making process that led to the requirement to useBlackboard instead of the g drive?

(I had no idea they were considering a change; soliciting instructor input would havebeen helpful.

()I wasn't included in the decision making, but I don't feel I needed to be.(I was made aware that a change was being considered as an alternative to the g drive,

but I didn't provide any input to the decision-making process.()Somebody asked me what I felt about using something other than the g: drive, but I'm

not sure my input really made a difference one way or the other.(I had full visibility on the decision-making process and felt my input was received and

considered.

125

23. What would you like the CGSC leadership to know about any aspect of the Blackboard

System?

This completes the survey.

Again, thank you for participating in this survey. Your time and responses are valued.

126

APPENDIX B-3

STUDENT SURVEY

The student survey was administered online for approximately three weeks incollaboration with the CGSC Quality Assurance Office. Students in the 2006 February-start ILEclass and the 2006 August-start class received an invitation via email to participate in the survey.Two-hundred fifty-one students responded to the survey, representing a 30% overall responserate. A complete reproduction of the survey is shown below.

Blackboard, SharePoint and Resident ILE - Student Experience

Your resident CGSOC ILE class is among the first to use Blackboard as the primarymeans of distributing curriculum materials and SharePoint as the primary means of courseadministration. As students during this pilot, you have probably experienced the inevitablechallenges associated with transitioning from one technology to another. Likely, there also havebeen some unanticipated benefits of the change. The purpose of this brief, 22-item survey is tocapture selected aspects of the ILE CC student experience relating to:

1. Course content;2. Course Administration; and3. The Overall Learning Experience.

The goal of this survey is to take the "pulse" of the pilot effort -- to identify howBlackboard and SharePoint are used by students and to understand the impact of thesetechnologies on course effectiveness and learning.

This survey is completely anonymous and voluntary. Your responses are in no wayassociated with your name and you may choose, without consequence, not to participate.

As you fill out this survey, consider your experiences during ILE CC. Select the responsethat best describes your experiences associated with each statement. There is an open endedcomment box at the end of the each section; please feel free to add comments.

If you have any questions concerning the content of this survey, contact Dr. AnnaCianciolo, 217-621-3918. If you experience technical difficulties with it, contact Mr. DouglasLoa, 913-684-7275.

This Survey has been approved by QAO and the Survey Control number is 06-077

127

Blackboard, SharePoint, and Resident Intermediate-Level Education (ILE):A Survey of the Student Experience

Section I - Course Content

1. How relevant was the common core curriculum to the job you expect to have after yougraduate?

()Not at all relevant.()More irrelevant than relevant; some topics addressed my needs and interests, but I

won't be able to apply directly what I learned.()No less relevant than any other academic experience; school learning is supposed to

be "just in case" not "just in time."()More relevant than irrelevant; topics generally addressed my needs and interests, and

with some effort I should be able to apply what I learned.()Totally relevant; I will be fully prepared for my next job.

2. How relevant were the supplemental materials (e.g., readings, video interviews, etc.) yourinstructors provided to the job you expect to have after you graduate?

()Not at all relevant; they just didn't cover topics of interest to me.()More irrelevant than relevant; they covered relevant topics but at too abstract a level

to be useful.()No less relevant than one can expect in the academic environment and with

restrictions on FOUO content sharing.()More relevant than irrelevant; there were some really useful resources, but I could

have used more of them.()Totally relevant; I know I'll want to reach back from the field to ILE courses further

down the road so I can stay current.

3. Did any of your instructors encourage you to contribute content to the course materials?

()No; nobody asked for student input. It was assumed that instructors do the teaching,and students do the learning.

()Sort of; more often than not, instructors did not ask for input, but we were encouragedto contribute if we offered.

()Yes; instructors generally recognized students' specific areas of expertise and activelyfound ways to fold our input into readings and/or classroom discussion.

4. Did any of your instructors encourage you (or demonstrate how) to find resources in additionto him or herself to learn more about a topic discussed in class?

()No; instructors generally saw themselves as the leading authority on the topic theywere teaching.

()Sort of; instructors didn't discourage independent research, but it also wasn't importantfor doing well in class.

128

()Yes; instructors actively encouraged us to do our own research by using professionalforums, AKO, CALL, or other resources, and showed people where to find things ifthey didn't already know.

5. When you searched for current information to complete class assignments or to conductindependent research, how would you rate the importance of the following resources:

Professional forums in the Battle Command Knowledge System (BCKS) Leader Network

()Extremely important - This was the only type of resource I used()Somewhat important - I used this type of resource more than others()Equal importance - I used this type of resource as much as I did the others()Not too important - I used other types of resources more than this one()Not at all important - I never used this type of resource()Didn't search for current information

Army knowledge repositories (e.g., AKO, CALL, etc.)

()Extremely important - This was the only type of resource I used()Somewhat important - I used this type of resource more than others()Equal importance - I used this type of resource as much as I did the others()Not too important - I used other types of resources more than this one()Not at all important - I never used this type of resource()Didn't search for current information

Non-Army resources (e.g., Google, Early Bird, etc.)

()Extremely important - This was the only type of resource I used()Somewhat important - I used this type of resource more than others()Equal importance - I used this type of resource as much as I did the others()Not too important - I used other types of resources more than this one()Not at all important - I never used this type of resource()Didn't search for current information

Additional Comments regarding the Course Content (optional):

Section II - Course Administration

6. What was the primary means you used to access course materials?

()Blackboard(j) SharePoint()g: drive()Email()Memory stick/thumb drive()Printed documents from the instructor()Other[ ]

129

7. What was the primary means you used to coordinate with other students?

()Blackboard()SharePoint()Email()Phone()Face-to-face meetings()Other [

8. How often did you access Blackboard prior to a class meeting in order to prepare by readingor reviewing lecture materials ahead of time?

()Never; our class content wasn't posted in Blackboard/there was always a better waythan Blackboard to get to course content.

()Sometimes; if the course content was actually on Blackboard and I had the time.()Often; course content was usually on Blackboard and I tried to make time to get to it

before class.()Always; I could count on content being on Blackboard and actively used access to it to

prepare.(N/A; I didn't prepare for class by reading ahead or reviewing lecture materials.

9. Was there someone among the students in your staff group who used SharePoint to coordinatestudent efforts (e.g., group assignments, due date lists, class calendar, etc.)?

()No; we all pretty much fended for ourselves when trying to succeed in class.()No; we had a student leader, but he/she didn't use SharePoint to coordinate our efforts.()Yes; we had a staff group student leader appointed by the instructor to help out.()Yes; a staff group student leader emerged within the first few weeks of class.

How difficult was it to access from outside the schoolhouse?

10. Blackboard()Extremely difficult; I stopped trying altogether.()Somewhat difficult; I only did it when I absolutely had to.(I wouldn't know; I didn't try to access it from outside the schoolhouse.()Not too difficult; I had no more difficulty with accessing this than one would expect()Extremely easy; I never had any problem accessing this from outside the schoolhouse

11. SharePoint()Extremely difficult; I stopped trying altogether.()Somewhat difficult; I only did it when I absolutely had to.(I wouldn't know; I didn't try to access it from outside the schoolhouse.()Not too difficult; I had no more difficulty with accessing this than one would expect()Extremely easy; I never had any problem accessing this from outside the schoolhouse

130

What would you do to make the system easier to access?

12. How often did your common core instructors do the following:

Provide clear guidance regarding their expectations and performance evaluation criteria?

()Every instructor did this()More instructors did this than didn't do this()About as many instructors did this as didn't do this()Fewer instructors did this than didn't do this()No instructor did this

Provide constructive and informative feedback on your performance?

()Every instructor did this()More instructors did this than didn't do this()About as many instructors did this as didn't do this()Fewer instructors did this than didn't do this()No instructor did this

Present learning challenges that were difficult but not overwhelming?

()Every instructor did this()More instructors did this than didn't do this()About as many instructors did this as didn't do this()Fewer instructors did this than didn't do this()No instructor did this

Assign group tasks that had well-defined tasks and roles for each group member?

()Every instructor did this()More instructors did this than didn't do this()About as many instructors did this as didn't do this()Fewer instructors did this than didn't do this()No instructor did this

Engage students in asking questions and reflecting on class materials?

()Every instructor did this()More instructors did this than didn't do this()About as many instructors did this as didn't do this()Fewer instructors did this than didn't do this()No instructor did this

131

13. During the common core course, how successful did you feel in leading your own learningprocess?

()Fully capable; I had all of the tools and access I needed to accomplish my learninggoals.

()Somewhat capable; I felt I was successful in reaching many of my learning goals, butI sometimes lacked the time to accomplish everything I wanted to.

()Not sure; I was never really given the chance to lead my own learning.()Not very capable; I don't feel I had the kind of access to learning resources that I

needed to succeed.()Not at all capable; I spent so much time just trying to figure out Blackboard and

SharePoint, that I had little opportunity to take charge.(N/A; I don't feel students should lead their own learning - that's why they have

instructors.

14. How accessible did you find the technical support to be?

()What tech support? I didn't know there was tech support.)I knew how to find the tech support team, but actually getting a hold of someone was

unpredictable; on more than one occasion I had to wait longer than I would have liked.)I wouldn't know; I knew there was a tech support team, but I never tried to contact

anybody.()I generally didn't have trouble finding or connecting with someone in tech support; if I

ever had to wait, it wasn't a big deal.()Finding and connecting with tech support was no problem at all; I always got

somebody when I needed them.

15. How useful did you find the technical support to be?

()Not at all useful; I had to find other ways to fix my problems."()They solved technical problems ok, but if the tech support was better, the kinds of

problems I had wouldn't exist.(I wouldn't know; I never asked for tech support.()I had a pretty good experience; the system worked well enough and when problems

did come up, they were usually fixed the first time around.()They're the best; the system worked great and whenever I needed technical assistance,

it was provided quickly, effectively, and professionally.

Additional Comments regarding the Course Administration (optional):

132

Section III - Overall Learning Experience

16. How motivated did you feel to learn the subjects in the common core course?

()Highly motivated; the course content spoke to my interests and/or the instructors wereengaging.

()Generally motivated; more often than not, the course content spoke to my interestsand/or the instructors were engaging.

()Generally not motivated; the course content usually did not speak to my interestsand/or the instructors were not engaging.

()Barely motivated; the course content was irrelevant and/or instructors seemed poorlyqualified or burned out.

17. What do you think is the primary reason that ILE courses are being administered viaBlackboard and SharePoint?

()The Army wants to save money.()The Army leadership likes Blackboard and SharePoint and decided everyone has to

use it.()The Army thinks using Blackboard and SharePoint is in the best interests of leader

education.(I couldn't say how the decision to use Blackboard and SharePoint was made.(Other [ ]

18. The instruction I've received on Blackboard supports my use of Blackboard in ILE, so far.

()Strongly Agree()Agree()Undecided()Disagree

()Strongly Disagree

19. When I use Blackboard, connecting to the internet has not been a recurring problem.

()Strongly Agree()Agree()Undecided()Disagree()Strongly Disagree

20. The Blackboard Learning System will enhance my ILE education.

()Strongly Agree()Agree()Undecided()Disagree()Strongly Disagree

133

21. My ILE instructors are knowledgeable on the Blackboard Learning System.(Choose one}(Strongly Agree()Agree()Undecided()Disagree(Strongly Disagree

22. What would you like the CGSC leadership to know about any aspect of the Blackboard orSharePoint System(s)?

The following questions are strictly for demographic information only.

23. What CGSC Class are you in?

()AY 007-001()AY 006-002()AY 006-002 (Pilot)

24. My CGSS Section is

() 1()2( ) 3 (PILOT)()4()5()7()809()10() 11

()12()13()15()16()18()19()20

Staff Group()A()B()C()D

134

This completes the survey.

If you would like more information regarding the Lifelong Learning Center, copy and paste thelink below into the address bar on the next page.

http://www.tradoc.army.mil/pao/Web-specials/lifelongjlearning/intro.htm

Again, thank you for participating in this survey. Your time and responses are valued.

135

136

APPENDIX C

ACRONYMS

AKO Army Knowledge Online

BCKS Battle Command Knowledge System

CBT computer-based trainingCGSC Command and General Staff CollegeCGSOC Command and General Staff Officers' CourseCGSS Command and General Staff School

DA Department of the ArmyDMOSQ Duty military occupational specialty qualifiedDOET Directorate of Educational Technology

FSDO Faculty and Staff Development Office

ILE Intermediate-Level Education

KM knowledge management

LLC Lifelong Learning CenterLVN LLI Leavenworth Lifelong Learning Initiative

METL Mission essential task list

MOSQ Military occupational specialty qualified

QAO Quality Assurance Office

SADL School of Advanced Distance LearningSOP Standard Operating Procedure

TASS Total Army School SystemTRADOC U.S. Army Training & Doctrine Command

WBT Web-based training

137