Project Falcon Base: A Freshman Introduction to Engineering Using Problem Based Learning
A. George Havener, D. Neal Barlow
Department of Aeronautics
United States Air Force Academy Abstract This paper is a summary report on an experimental freshman-engineering course conducted at the United States Air Force Academy (USAFA) during the period August 1996 – May 1999. The purpose of the course, Engr 110Z, Project Falcon Base: An Introduction to Engineering, was to develop foundation skills in problem solving, independent learning, teamwork and communication, while concurrently introducing basic principles of engineering to a group of freshman cadets. Problem Based Learning (PBL)1,2 was the pedagogy used to engage the cadets in a motivational project; design a plan to deploy a manned mission to Mars. Twelve specially designed workshops were used to provide instruction on problem solving skills needed by the cadets to complete the project. A variety of assessment processes were used to evalua te the effectiveness of the course. Among the findings, the final data show that without follow-on PBL experiences in the remaining academic program, the problem solving skills initiated in the freshman course are of little value to the students. Additional data to include comparisons in academic performance between the cadets that took Engr 110Z and those who did not are also reported. The paper concludes by presenting 10 recommendations intended for other educators desiring to use PBL in engineering courses. Introduction Based on the seven educational outcomes for cadets at the United States Air Force Academy (Table 1)3, an experimental freshman engineering coursed was designed and conducted for three years. Taught to approximately 40 cadets per term, the experiment sought to determine how problem solving skills could be learned and practiced by freshmen along with learning introductory principles of Aerospace, Civil, Electrical, and Mechanical Engineering. Of the seven educational outcomes in Table 1, problem solving, teamwork and developing good communication skills were priorities.
Table 1. USAFA Educational Outcomes
1. Officers who can frame and resolve ill-defined problems. 2. Officers who are intellectually curious. 3. Officers who can communicate effectively. 4. Officers who possess a breadth of integrated fundamental knowledge in the basic
sciences and engineering, and in the social, political and military sciences. 5. Officers who can work effectively with others. 6. Officers who are independent learners. 7. Officers who can apply their knowledge and skills to the military profession.
Except for final assessment findings, Engr 110Z is documented elsewhere4-7, so only the main features are repeated here. Using PBL and the Mars-Mission project, each class-section became a project team, selected a team manager, and then identified task leaders to head sub-groups. The sub-groups focused on separate tasks such as: travel to Mars (orbital mechanics), energy requirements (thermodynamics), living on Mars (civil, electrical and mechanical engineering), and physiological and psychology issues (ethics, sociological, health and safety). The team manager was responsible for organization, maintaining schedules, and the written and oral t eam reports that were the products of the team. The task teams were responsible for educating the entire project team on the fundamentals of their respective task. Why the Mars Mission PBL-Problem – USAF graduates are Air Force Officers who, during their careers, participate in and have responsibility for a variety of systems -programs. Such programs often present interdisciplinary ill-defined problems requiring expertise in engineering, economics, politics, sociology, medicine, psychology, and law. So in addition to device-oriented problems, USAFA cadets need school-experiences working on multifaceted interdisciplinary situations. The ill-defined aspects of a manned mission to Mars presents the cadets a challenging and timely interdisciplinary problem requiring them to interact with instructors from various engineering and social science departments as well as external agencies such as NASA. Student Selection – For the Classes of 1999, 2000, and 2001, the registrar randomly selected 20 cadets from each freshman class to fill two sections per semester. The random selection excluded cadets who were enrolled in remedial courses in English or mathematics, excluded intercollegiate athletes, and ensured that 10-12 percent of each class were females, nominally the same percentage for the Air Force and USAFA cadet wing. Skills Development – Table 2 shows the specific skills and knowledge introduced to the cadets during the first quarter of the course. Experience2 shows that students learn and develop problem solving skills best through a three-step process: (1) Introduction: knowledge of the skill or tool is introduced in a traditional lecture manner. (2) Bridging: understanding the skill begins by using it, generally on a familiar situation. (3) Application: confidence and competence with the skill are strengthened by extending the use to a new situation. In Engr 110Z, Mini-Workshops4,5 and structured homework assignments were used to accomplish Steps (1) and (2). For Step (3), strengthening the skill was inherent in applying it to the Mars mission project. Mini-Workshops – Each mini-workshop was designed around an Assignment Sheet such as the example shown in Fig 1. The particular Assignment Sheet in Fig. 1 is for Skill-2, Concept Maps. Each Assignment Sheets contained five parts: (1) Discussion: the skill or tool plus its use are briefly described. (2) Engr-110Z Goals: the educational objective(s) of the assignment are identified. (3) Outcomes: the cadets’ capability sought by completing the assignment successfully are identified and discussed in class. (4) Task: a specific exercise is given for practice thereby initiating utility with the skill or tool. (5) References: sources for additional information and guidance are listed. The task defined on each Assignment Sheet is graded and returned to the student with a feedback sheet. The feedback sheet is the instructor’s evaluation of the student’s performance with regard P
to the stated outcomes. Direct correspondence between assigned requirements and feedback evaluation is crucial for each student to see his/her level of performance as well as to benefit
Table 2. Problem Solving Skills, Tools Introduced in Engr-110Z
Skill or Tool Description 1. Mars Facts
Basic Scientific Knowledge on Mars as Known from Viking, Mariner and Pathfinder NASA Probes
2. Concept Maps
Concept mapping architecture and design: Concept mapping is used to convey information, graphically display brainstorming, and help frame ill-defined problems.
3. Project Planning
Project planning is an essential step in problem solving. Students learn to formulate problem statements, task lists, resources available and needed, and timelines. Project planning is the executable format of problem solving.
4. Research
Research includes use of the WWW and the traditional sources of libraries, teachers, and practitioners.
5. Figures & Charts
Students learn how to convey information using proper formats for figures and charts. This skill is a fundamental communication skill.
6. Modeling
Foundations in modeling based on principles of mathematics, physics, and engineering.
7. Ill-defined Problems
Students learning to recognize ill-defined problems from those that are deterministic. Framing involves identifying unknown information. An important ingredient is providing for timely analysis of information and updating problem solving approaches.
8. Teamwork
Teamwork involves leadership and elements of project planning. The students learn that their team is no stronger than its weakest link, and that cooperative participation of all members is required to complete a project effectively.
9. Technical Reports
Foundations in technical report writing. An effort coordinated with the English Department.
10. Oral Briefings
Foundations in preparing and presenting information in oral presentations.
11. Computer Nets
A Tool: Foundations in using the USAFANet and the WWW.
12. PowerPoint
A Tool: Slide and figure preparation using MS-PowerPoint.
13. Spreadsheets
A Tool: Foundations in using MS-Excel to create and use spreadsheets to catalog data and to make calculations.
from the instructor’s suggestions for improvement. We have found that the feedback sheet improves student performance because it unambiguously matches requirements with evaluation. To be effective, we also found that the graded feedback sheets need to be returned to the students in the class period immediately following workshop. We also found that student attitudes and performance improved by including compliments and “things-to-work-on” statements on the feedback sheets. Findings Throughout the experiment, a variety of assessment measures were used to determine how well the objectives of Engr 110Z were obtained, and how well the relatively new educational pedagogy of Engr 110Z had impacted the intellectual development of the selected-cadets. Standard measures such as examinations, written and oral final project reports, mini-workshop
exercises, and peer evaluations were used to determine student performance. In addition, the Small Group Instructional Diagnostic8, focus groups9, surveys to include multi-dimensional scaling (an attitudes-toward-learning questionnaire), personal interviews, and faculty evaluations were part of the assessment measures. Demographics- Prior to implementing Engr 110Z, a concern was to avoid having Engr 110Z become a recruiting mechanism for the engineering programs. The data in Table 3 show that 58 of the 202 cadets that took Engr 110Z selected an engineering major. For these three cadet-classes, 637 cadets out of the total population of 2801 cadets majored in engineering, nominally 23%. Of the 58 Engr 110Z cadets who majored in an engineering program, statistically 50 of them would have selected an engineering program even if they had not taken Engr 110Z (25% of 202). So perhaps Engr 110Z was a deciding factor for about 8 cadets, a small number that suggests Engr 110Z did not substantially recruit cadets into engineering programs at USAFA.
Table 3. Engr 110Z Demographics
Class Populations Class Size 110Z Cadets
Engr Majors Engr 110Z Grad Who Majored Engr
1999 989 59 (6%) 206 (21%) 18 (31% of 59) 2000 982 88 (9%) 230 (23%) 24 (29% of 88) 2001 830 55 (7%) 201 (24%) 16 (29% of 55) Totals 2801 202 (7%) 637 (23%) 58 (29% of 202) Performance Estimates Based on Cumulative GPA Scores - The question relative to Fig. 2 is: did Engr 110Z impact learning skills which in turn led to better academic grades? The center column for each class in Fig. 2 pertains to all cadets in that class who did not take Engr 110Z and who were not considered to be at risk as a freshman. Cadets in the freshman class are considered to be at risk if their entry-level examination scores indicate they need remediation in mathematics and English. No at risk cadets took Engr 110Z, so the meaningful comparison is between the left and center columns for each class. The right-hand column presents the data for each class minus the cadets who took Engr 110Z. Figure 2 shows that for the 1999 and 2001 classes, the GPA scores are slightly higher, but the differences are insignificant. For the class of 2000, the cadet GPA for Engr 110Z are significantly below the class score, the causes being unidentifiable from these results alone. Considering the average GPA for all three classes (3.01 for Engr 110Z, 3.03 or the others), apparently Engr 110Z had no affect on improving grades over the 4-year academic program. Performance for Problem Solving – All cadets at USAFA take a capstone engineering design course (Engr 410) in the senior year. Engr 410 is designed to provide teams of nominally 20 cadets per section an opportunity to use their knowledge and skills to design, build, test and deliver a product that meets customer’s needs according to a sta tement of work (SOW). Most often, the SOW pertains to a device that would be a benefit to a handicapped person. Engr 410 also introduces cadets to the principles of the Air Force Acquisition System. For the class of 2000, a special section of this course was arranged to have only Engr 110Z grads, cadets who as
freshman took Engr 110Z. Another section that contained no Engr 110Z grads was the control group. Both classes had the same instructor, and same SOW. Table 4 shows the results of the instructor’s ratings for categories based on the original Engr 110Z educational outcomes. The numerical results (5 high, 1 low) indicate little distinction between the two teams. Moreover, the instructors comments were, “While differences in leadership styles and class personalities were observable, the overall team performances were the same. Distinctions in GPA performance (in Engr 410) seem not to be a good basis to claim effects created by Engr 110Z. Distinctions are more likely the result of cadet attitudes, project difficulty, Project Manager leadership, and instructor guidance.”
Table 4 Instructor Ratings for Educational Outcomes in Capstone Design Course
Category Engr 110Z Grads Control Group Problem Solving
- Framing - Organization - Critical Thinking
4 4 4
4 3 3
Intellectual Curiosity 3 4 Research 4 5 Teamwork 4 3 Communication
- Oral - Written - User’s Manual
4 3 5
4 3
4+ Average Value 3.9 3.7
Attitudes Towards Learning- The Behavioral Science Department at USAFA conducted a survey of the Class of 2000 to assess entry-level attitudes towards learning. The survey was given twice in the first semester, in August 1997, and again in December 1997. Figure 3 shows the results obtained for four of the 34 questions asked on the survey. The responses showed an increase in the learning attitudes for the cadets enrolled in Engr 110Z, and a decrease for all the other cadets. A plausible explanation for this result is that the cadets in Engr 110Z were being challenged by the Mars project and their PBL experience project in a favorably different intellectual manner than their peers were being challenged in other courses. We speculate that upon entering the Academy, freshman cadets look forward to academic experiences that are distinctly different from their high school courses. Engr 110Z provided that different experience. At the onset in August, the newness of the Mars project and the PBL classroom were intimidating, so the attitude scores of the Engr 110Z cadets were slightly lower. By the end of the course, the “looking back” perspective allowed these cadets to see how much they had learned and accomplished, as a result, their attitudes about learning were high at this time. Among the numerous cadet comments collected upon the completion of the course, the following student comments are evidence of this argument: “ This course more or less goes against everything most of us learned in high school. Usually high school kids get the right answer and if you don’t get it, you’re wrong. It was simply that
way. There’s one way to start and one way to finish an assignment, and there’s no different paths to choose. I don’t think the real world is that way. I think the real world is full of ill-defined problems, and therefore, I think the course was beneficial...” “See, if you’d have asked me during the term [Would you take Engr 110Z again?], while we were doing it, I would have said, NO WAY! But now that you can look back and see what you have learned from it, it's amazing how much you come out of it with. If I knew [then] what I know now, 100 percent, I’d take it again!” On Critical Thinking Skills Development – Several questions on the end-of-course critiques, given in class at the end of every semester were designed to assess the student’s perspective on how Engr 110Z fostered critical thinking. Figure 4 shows a summary of the responses for the Class of 2000. From these responses, we conclude that the cadets believe that Engr 110Z fostered critical thinking. These results are compared to the responses for all other freshman courses, typically, English, History, Chemistry, Foreign Language, and mathematics (calculus). Multi-Dimensional Scaling Diagnostic – An educational psychologist in the USAFA Behavioral Science Department acquired data for a listing of paired engineering terms, and used the data to develop a Multi-Dimensional Scaling (MDS) Map. This diagnostic shows trends in intellectual development based on understanding the meaning of engineering terms. The raw data (7 high, 1 low) are the respondent’s choices for the relative strength of association between word pairs on a list. In the present application, the word pairs were combinations involving the terms: energy, momentum, stress, current, power, mass, kinetic energy, angular velocity, and acceleration. For example, each respondent decides on the relative association between energy-momentum, energy-stress, energy-power, and so on, for all word-pair combinations, word order being unimportant. The cadets in the Class of 2000 took this diagnostic twice, once at the start of Engr 110Z, and again at the end of the course. Several members of the faculty also took the diagnostic to locate faculty consensus points for each term on the MDS Map, Fig 5. The MDS MAP does not show right or wrong answers. Rather, it shows spatial locations based on statistical evaluation of the responses for each paired-term for the members of the sample group. Figure 5 shows that the cadet’s spatial location for the terms, energy and stress, approached the faculty location at the end of the course. Similar trends are observed for the other terms stated above. These findings suggest that the students’ initial interpretation of these terms had become more consistent with the faculty’s interpretations by course completion. Upon analyzing these data, the educational psychologist stated, “I believe these data suggest that Engr 110Z had a strong positive effect on the [cadets’] understanding of the concepts analyzed in this assessment. The MDS solutions to the paired concept-associations show a strong movement towards the faculty’s concept space as a result of taking Engr 110Z.” Focus Groups – Numerous Focus Group sessions were conducted over the duration of the three year-experiment. In each focus group, a small number of cadets (typically 5) met with an educational psychologist to ascertain issues affecting their learning. Overall the cadets were uncomfortable by not having a course text or a course syllabus (a course syllabus was develop and used in the final 3 semesters), they struggled with teamwork because as freshman cadets, personal freedom is restricted at USAFA, and they felt that they were not learning enough
content. However, the Focus Groups revealed strong positive attitudes toward careers in terest, preparation, and progress on attaining the USAFA Educational Outcomes (Table 5).
Table 5 Focus Group Questions: All Engr 110Z Sections
Topics
Score (1-strongly disagree, 5- strongly agree)
Engr 110Z has prepared me for other USAFA Engineering Courses 4.2 Engr 110Z has prepared me for other USAFA courses 4.3 Engr 110Z has helped to prepare me for my Air Force Career 4.9 Engr 110Z has helped me attain the USAFA Educations Outcomes (Table 1) Framing and Resolving Ill-Defined Problems 5 Intellectual Curiosity 4.4 Teamwork 4.8 Communication Skills 5 Independent Learning 4.1 Breadth of Fundamental Integrated Knowledge 4.3 Being a Military Professional 4.2 Conclusions This paper is the last in a series of papers on the Project Falcon Base course4-7. The initial results reported in the earlier papers presented optimism on improving the Academy’s ability to better meet its critical thinking educational outcomes. This optimism is now tempered by the realization gather from the assessment data that a single freshman experience can not have a lasting effect on achieving such outcomes. Below are 10 comments intended to be recommendations to other educators considering PBL courses in their engineering programs. (1) PBL was the right pedagogy for Engr 110Z because it genuinely engaged the cadets in active, ownership-type learning that above all else allowed them to develop foundations for framing and resolving ill-defined problems, the primary educational outcome for Engr 110Z. (2) Educational gains acquired from an initial PBL course must be strengthened with a series of carefully planned follow-on PBL experiences; otherwise, the initial gains will be discarded by students as they progress through the balance of their academic program. (3) Designing a PBL course involves consideration for content and process. Here, we define content to mean learning information, and process to mean using information to frame and resolve ill-defined problems. In PBL courses, the emphasis is on process, so the amount of content covered could be relatively low. In fact, much of the content to be learned should be inherently part of the process. In PBL courses, the students need to learn on their own from conscientious engagement in the process instead of being taught content by the instructor in classroom lectures. Thus, PBL problems must be carefully selected on the basis of wanting students to obtain specific educational outcomes. (4) PBL courses inherently involve an attitudinal experience called, the grieving process1, which applies to students and instructors alike. The distinction we make here is that the instructors
must know about the grieving process, and they must be equipped and effective in advancing the students through it. This requirement is especially important for the initial PBL experience because if done incorrectly or ignored, the initial PBL experience will likely be an intellectual ordeal for the students, clearly an undesirable situation. (5) Along with designing PBL courses, an instructor training program needs to accompany the design and implementation of PBL courses. In the PBL classroom, the instructor is a guide to and a resource for the students. The instructor is not the central source of information or the lead authority in the course. Replacing “guide” for “authoritative lecturer” is a challenge that must be learned and practiced by PBL instructors because the natural tendency, especially for a new PBL instructor, is to “step-in” and “take-over.” When this happens, the instructor undermines and most likely destroys the PBL learning experience for the students. As described in (4) above, PBL instructors must learn how to proficiently and constructively cope with the grieving process. (6) PBL relies heavily on Socratic teaching in a structured academic environment. In this environment, TIME IS THE ENEMY. Course content and duration are in conflict. No doubt if students had the luxury to learn through contemplation and discussion with a master, much like Socrates’ students did, the process of self-discovery would be optimized. If time were not a constraint, this method would undoubtedly prove to be the most effective teaching method available. Unfortunately, courses are bound by time intervals in which a minimum amount of academic material must be covered. For the PBL instructor, the challenge is to not give up the Socratic approach to cover a lot of content, but to create balance between content and process according to a set of well defined educational outcomes. PBL can not be used reasonably well in all engineering courses in a curriculum; it is however extremely effective in achieving educational outcomes that pertain to problem solving, critical thinking, and teamwork. (7) While PBL focuses on process and limits the amount of content that can be introduced , the data from the Multi-Dimensional Scaling Diagnostic indicate that course outcomes can effectively include engineering fundamentals. If expectations and resources are correctly identified in the course structure, students can operate as independent learners in developing knowledge of fundamentals. (8) A situation that we call, “instructor comfort,” is another challenge. The teacher-centered classroom is the comfort zone for most instructors, primarily because it is structured, organized, reactively easy, and controlled by the instructor. Little of this items are true for the PBL classroom. Thus, PBL instructors must be schooled in course objectives, educational outcomes, and the PBL paradigm. Moreover, conducting a PBL classroom requires leadership lesson plans without which will help prevent instructors from slipping back into the teacher-centered pedagogy. We have found that instructors new to PBL do best whe n they audit a PBL course prior to becoming the PBL instructor. Likewise, our experience has shown that failure to use an effective PBL instructor training program results in frustration for both the students and the instructor. (9) Conducting PBL classes requires more time and effort than conducting traditional classes. As such, the overall workload for faculty engaged in PBL classes must be reduced; otherwise the PBL instructor will become stressed and subsequently be ineffective in this paradigm.
(10) Lastly, students ratings of PBL courses and PBL instructors can be lower than those for faculty in traditional courses, particularly in freshman PBL courses. Until students understand the PBL paradigm, and actually experience it, they are apt to think the course is badly designed and poorly conducted. Thus, school administrators and the PBL instructors both need to understand that low student ratings are likely, and as such, the PBL instructors must not feel threatened professionally by their institution because of such ratings. References 1. Woods, D.R., Problem-based Learning: How to Gain the Most from PBL , ISBN 0-9698725-0-X, The McMaster University, Hamilton, ON, L8S 4L8, 1994: email, Hocker@Boostore,services,mcmaster, Canada. 2. Woods, D.R., et al., “Developing Problem Solving Skills: The McMaster Problem Solving Program,” Journal of Engineering Education, ASEE, Vol. 86, No.2, April, 1997. 3. “United States Air Force Academy Educational Outcomes and Their Assessment: An Integrated Approach,” Educational Outcomes Assessment Working Group, A Presentation at The American Association of Higher Education, 10th Annual Conference on Assessment and Quality, Boston, MA, June 13, 1995. 4. Havener, A.G. and Dull, C., “An Information Resource Web-Page for A Freshman Problem Based Learning Engineering Course,” Session 1253, ASEE Annual Conference & Exposition, Seattle, WA, June 28, 1998. 5. Havener, A.G. and Van Trueren, K., “Teaching Introductory Engineering: A Problem Based Learning Experience,” 35th Annual Rocky Mountain Bio-Engineering Symposium, Copper Mountain, CO, April 17-19, 1998. 6. Barlow, D.N., Havener, A.G., Kouri, J., Van Trueren, K, and Smith, M.L., “Project Falcon Base: A Freshman PBL Engineering Experience,” AIAA Paper 96-5531, AIAA World Aviation Congress, Anaheim, CA, October, 1996. 7. Barlow, D.N., Havener, A.G., Kouri, Marlino, M.R., and Smith, M.L., “Project Falcon Base: A Freshman PBL Engineering Experience,” Session 2653, ASEE Annual Conference & Exposition, Washington, D.C., June 23-26, 1996. 8. Millis, Barbara, et. al, “Small Group Instructional Diagnostics,” Internal Document, Center for Educational Excellence, United States Air Force Academy, CO, Jun 1998. 9. Millis, Barbara, et. al, “ Focus Groups for Educational Course Design,” Internal Document, Center for Educational Excellence, United States Air Force Academy, CO, Jul 1999.
