Sharad Oberoi and Susan FingerCarnegie Mellon University

Tracking Design Knowledge in Engineering Student Projects around

Course Milestones

Collaborative learning in design

• Goal– Develop tools that allow instructors to monitor

the progress of project-based courses and assess student interactions outside the classroom

Collaborative learning in design

• Assertions– Most learning in design classes takes place in

team meetings and in individual activities undertaken to help meet team goals

– Prior research has shown that noun phrases can be used as surrogates for design concepts

– For successful design teams, the number of distinct noun phrases expands at the initial stages of the project and contracted as the project progresses

Mabogunje, A., Measuring Conceptual Design Performance in Mechanical Engineering: A Question Based Approach, Doctoral Dissertation, Stanford, CA: Stanford University, 1997.

Outline

• Setting– Engineering design capstone course– Ongoing project to understand collaborative

learning by student design teams

Student design projects

• Characteristics– Students bring together knowledge from

different sources:• collaborate among themselves, • share design knowledge• negotiate with each other, faculty members

and the client, in order to create engineering artifacts.

– Students reuse previous knowledge and create new knowledge within the context of the problem.

Capturing in-process data

• For 7 years, RPCS has used the Kiva for team collaboration– Light-weight collaboration tool– Combines functions of e-mail and bboards– Widely accepted and liked by student teams; it

feels likes chat and meets their needs– Each year’s Kiva has hundreds of threads and

thousands of posts and filesWe have 7 years of data of all the team

conversations and files that would normally go through email or chat

Kiva collaboration tool

• Takes advantage of students’ willingness to send email, use IM, post on newsgroups, send text messages

• Allows students to cross-post and to post to teams in which they are not members

• Design goal: Create an interface that students perceive to be equivalent to their preferred communication modes; that is:

make it feel like chat

Design education testbed

• RPCS: Rapid prototyping of computer systems class of Spring 2008– Interdisciplinary, capstone design course– Class of 25 students taught by 2 instructors, 1 TA– Project on the design of a virtual coach called

Guru for new power wheelchair users– The aim was to monitor wheelchair users and

assist the clinician in assessing if any revisions were required in the prescriptions

– Major teams in the class were: Chair, Clinician, Infrastructure and Sensors

RPCS 2008 Timeline

• Phase 1: Requirements exploration and definition, benchmarking, and a rough conceptual prototype

• Phase 2: Creation of a functional prototype that satisfies most of the feasibility constraints

• Phase 3: System integration and more design iterations to make final refinements

Approach

• Discussion posts and attachments of the class were converted to text

• Noun phrases were extracted using the Stanford part-of-speech tagger

• Number of distinct noun phrases was counted for each group

Approach

• Advantages of using noun phrases– Noun phrases act as an objective design

document metric for predicting design process performance

– The pattern of distinct noun phrases at every stage of a design artifact can reveal insights about the state of the project

Approach

• Three analyses were done to attribute nouns to students– First analysis: Includes only the documentation

posted to the four main teams in the project– Second analysis: Includes only the documentation

made by members of the teams regardless of the thread in which it was posted

– Third analysis: Compares the time-based trends of noun phrases in the editing/presentation and design categories within the documentation

• Transformation of design concepts across phases was tracked

Results: First Analysis

• Extracts noun phrases only from the documentation posted to the four main teams in the project

• The peaks decrease with time• Instructor documents included in the

analysis, so design contributions made by the individual teams are not adequately reflected in their vocabulary

Results: First Analysis

Aggregated noun phrase counts by week

Mid-term break

Phase-1 Phase-2 Phase-3

Results: First Analysis

Distinct noun phrase counts by week

Results: Second Analysis

• The activities of the team members were tracked throughout the semester, instead of analyzing only the material posted in the teams’ threads

• This process attributes only the noun phrases used by members of a team to the team’s knowledge

• Peaks of distinct noun phrases expanded at the initial stages of the project and contracted as the project progressed

Results: Second Analysis

Aggregated noun phrase counts by week

Results: Second Analysis

Distinct noun phrase counts by week

Results: Third Analysis

• The editing/presentation and design activities were tracked separately to understand the ex ternalization done by students around project milestones

• The noun phrases in the editing/presentation documents vis-à-vis the design documents and discussions were compared

• Student design activities were found to not necessarily follow the class deadlines, unlike those including editing of the project reports or presentations

Results: Third Analysis

Noun phrase counts in design activities by week

Results: Third Analysis

Noun phrase counts in editing tasks by week

Transformation of Design Concepts across Phases

• Some design concepts are introduced by teams in project-based courses but wane with time, i.e., they are either abandoned or transformed into a related concept in the final design

• Reflects how student understanding about the artifact is evolving through the project

Transformation of Design Concepts across Phases

Teams Phase 1 Phase 2 Phase 3

Sensors different tilt

sensors, tilting angle

accelerometer, general tilt,

tilt measurements, tilt range,

tilt recline feet elevation,

tilt recommendation, tilt sensors,

tilt, wheelchair tilting

angle

accelerometer, nac tilt recline, pressure tilt,

tilt data, tilt measurements,

tilt position, tilt range,

tilt recommendation, tilt sensors,

wheelchair tilting angle

Noun phrases used by the Sensors Team to refer to the tilt of the wheelchair

Conclusions

• The paradigm of using noun phrases from student documentation to assess design team progress has been accepted for the past two decades.

• This research takes the paradigm forward and shows how noun phrases in a design project can help to measure how the class has performed from the perspective of prior research models of expan sion and contraction of vocabulary for successful projects.

• The paper also shows what activities the students are engaged in outside the instructors’ purview

Conclusions

• When the instructors have access to such information in real-time, they can actively monitor the progress being made by the teams

• They can also intervene in a timely manner in case they detect a potential teachable moment.

Questions?