Understanding Early Stage Design Processes
Maria YangRobert N. Noyce Career Development Assistant ProfessorEngineering Systems Division and Dept. of Mechanical EngineeringNovember 25, 2009
ESD.83 Doctoral Seminar in Engineering Systems
2
Early stage of design for products and systems High-impact phase within design and development
process Good design process leads to good design outcome Challenge: Early stage of design fluid, ambiguous,
difficult to measure Goal: Understand (and measure) informal design
activities through their outputs
3
General approach Descriptive rather than predictive Many models about what goes on in design that are
based on intuition and experience Is that what really happens? Instrument the design process (Leifer)
What are artifacts of design process? How to capture their evolution? (Yang)
4
Engineering vs. psychology approach NSF creativity workshops (with Frey) Psychologists use controlled studies (Paulus)
Pros – Min. confounding factors, individuals Cons – Short exercises (not realistic design), need
many participants (500 psych students – ltd domain knowledge)
Engineers coarser grain (Leifer, Agogino) Pros – More complex design activities, longer
projects, groups Cons – Confounding factors such as group dynamics
5
Who to observe Ideal: Real world projects, access to process and
project data Do you think this is always possible? Often, companies do not like to open themselves to
scrutiny Confidentiality Embedded (Owens)
Students in the classroom Novices Assessment inherent part of education process
6
Design Preferences Design consists of 2 distinguishing activities
Idea generation (synthesis) Idea selection
Idea selection assumes set of preferences Formal design synthesis approaches require formal
weightings for a preferences (Antonsson) Populate design space given sets of preference weightings
Reality – Design preferences given informally (“I like this better than
that” vs. “weight1 = .2, weight2 = .3”) Design preferences often aggregate of group opinion. Not
easy to do explicitly.
7
8
Metrics for design process
Research
Early stage design processDesigners
Design problem Design outcome
Design data
Clarify Generate Select
9
Metrics for design process
Research
Early stage design processDesigners
Design problem Design outcome
Design data
Clarify Generate Select
Sketching and Concepts (Yang 09; Yang 03)
Prototyping (Yang 04; Yang 05)
Design Information Retrieval (Yang, et al 05; Yang and Cutkosky 97, 98; Wood, Yang, et al 98; Yang, et al 98)
Designers and teams (Yang & Jin 07, 08)
Sketching Skill (Yang & Cham 07; Cham & Yang 05)
Preference Information (Ji, et al 07; Yang & Ji 07)
Users & needs (Lai, et al 09)
Sketching skill
11
Sketching in Design Sketches capture & communicate
[Ullman 90; Verstijnen 98; McKim 80; Schön & Wiggins 92]
Sketching process linked with design cognition [Nagai & Noguchi 03; Suwa & Tversky 97; Goel 95]
Sketching is “dialogue” [Cross 99; Shah, et al 01; Goldschmidt 91; Tovey, et al 03]
If sketching is language of design, is sketching proficiency linked to design process or performance? [Yang & Cham 07; Cham & Yang 05]
12
Research questions What is the nature of sketching skill in design?
Is drawing a generic ability? How are different drawing skills related? Research in mental imagery [Kosslyn 84; Kosslyn 94]
1. Comprehensive, generic “trait” 2. Task-based skill3. Somewhere between 1) and 2)
Hypothesis Sketching ability similar to (3)
13
Research questions How is sketching ability linked to fluency?
Hypothesis: Those who draw better also draw more
How is skill related to design outcome? Hypothesis: Can sketching skill serve as an
indicator of outcome?
14
Related work For conceptual design,
sketching preserves ambiguity [Goel 95; Kavakli, et al 98]
Sketch classification Function [Ullman 90;
Ferguson 92; van der Lugt 05; Goel 95]
Elements [McGown 98; Rodgers 00]
Sketching and outcome Teams who sketch vs.
those who don’t [Schütze 03]
3D sketching & outcome [Song & Agogino 04]
What about sketching skill?
15
Survey to assess drawing skill(do try this at home)
1. In 3 minutes, draw a bicycle with as much detail as possible.
2. Hold out the items given to you in your non-dominant hand (left-hand for right-handed persons). In 3 minutes, make a drawing of your hand and the items [two small candy bars].
3. Visualize and draw the following in 2 minutes: A rectangular box that is open at the top. Inside the box is a rubber ball. The front of the box has a large button, and each side of the box has a large “X” painted on it.
16
Survey goals & assessment Engineering sketches may utilize many elements1. Bike task - Mechanical recall
Recall and sketch familiar mechanical object Structure, function (“Look like a bike? Could you ride it?”)
2. Hand task - Drawing facility Realistic, well-composed drawings from a still Proportions, realism (“Does this look like a hand?”)
3. Box task - Novel visualization Visualize specific features Proportions, 3D perspective, realism
17
Drawing Facility Task
Novel Visualization Task
Mechanical Recall Task
Level 1 Level 3 Level 5
18
Design outcomes Sketch fluency
Paper design logbooks; relatively objective Perspective drawings; more skill required
Grades for class and for final project Rankings by external judges Spearman Correlations
19
Results: Types of sketching skill Possible results
1. Comprehensive skill: Strong correlations between tasks
2. Task-based skill: No correlation3. Skill lies between the two: Range of correlations
Results suggest option 3
Correlation between sketch tasks. N = 32, Rs >= 0.296 for = 0.10.
0
0.05
0.1
0.15
0.2
0.25
0.3
Bike task and Handtask
Bike task and Boxtask
Hand task and Boxtask
Corr
elat
ion
coef
ficie
nt, R
s
20
Sketching ability and fluency Total: Drawing “well” correlates positively 3D: Bike task correlates negatively Drawing skill vs. other means of visualization?
N = 32, Rs >= 0.296 for = 0.10
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
Bike task Hand task Box task
Corr
elat
ion
Coef
ficie
nt, R
s
Total sketches
3D sketches only
21
Sketching and Design Outcome Sketch fluency: Positive but no sig. correlation Sketching skill: No clear trends Design process depends on many skills/factors Project type, outcome measures More studies needed
N = 33, Rs >= 0.291 for = 0.10 N = 32, Rs >= 0.296 for = 0.10
0
0.05
0.1
0.15
0.2
0.25
0.3
Project grade Class grade Avg. ranking
Cor
rela
tion
coef
ficie
nt, R
s
Total sketches
3D sketches only
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
Project grade Class grade Avg. ranking
Corr
elat
ion
coef
ficie
nt, R
s
Bike task
Hand task
Box task
22
Conclusions1. Is sketching ability generic?
Sketching skills not created equal Possible reason: Different cognitive skills required
(gearhead and artist)2. Is sketching ability linked to sketch fluency?
Hand and box task correlate, but not bike Sketch fluency (partly) influenced by how much a
designer can design without drawing. Possible reasons
Mechanical recall = visualization in head Common complaint: “I don’t need to keep a logbook”
23
Conclusions3. Sketch ability linked to design performance?
No relationship between sketch tasks and outcome “Good” sketchers did not necessarily do well (or vice
versa) Possible reasons
Engineering design complex, requires many skills; sketching is only one
Sketching may be behavioral rather than a necessary element of design activity (doodler)
Extraction of preferential probabilities from design team discussion
25
Overview Making choices is one key activity in design Designers express "design preferences" by assigning
priorities to a set of possible choices Assigning preferences can be complex for a team
Elicitation of preferences from a group (surveys, voting) Aggregation of preferences among the group
[Mark 02]
26
Research Questions How can preferential probabilities of a design team be
extracted? (Ji, et al 07; Yang & Ji 07) Obtained implicitly, not explicitly How to address aggregation?
Do preferential probabilities evolve over time? Way to describe how a team selects alternatives throughout the
design process How does extracted information compare with that obtained
explicitly? Consistent with preference information captured via surveys?
Preferential probability: Likelihood one alternative will be selected as “most preferred” over all others
27
Approach Extract preferential probabilities from transcripts of
design team discussion Design alternatives known a priori Assume preference-related information embedded
No formal aggregation of individual information Simple collection of words
Assumptions What designers think in one time interval relates to what
they thought in the previous interval Designers tend to speak positively about the design
alternative they prefer more and negatively about those they prefer less
28
Related Work Preference Extraction
Surveys: The lottery method [Hazelrigg, 99; Otto & Antonsson, 93] Pair-wise comparison: AHP [Saaty 00], fuzzy outranking [Wang 97] Multi-criteria overall aggregation function using MoI [Scott & Antonsson 98] Conjoint Analysis [Green 90] and Discrete choice analysis [Hensher &
Johnson 81; Ben-Akiva & Lerman 85] Collaborative filtering [Kohrs & Merialdo 00]
Group Preference Aggregation Cardinal utility functions for accumulating group preferences [Keeney 76] Structured pair-wise comparison chart [Dym, Wood & Scott 02] Aggregation with equal weights [Bask & Saaty 93] Aggregation with unequal weights [Jabeur, et al. 99; See & Lewis 05] Arrow’s Theorem: no guarantee of consistency in a group [Arrow 70, 86]
Design Process Evolution Surveys [Brockman 96], Coding of design journals [Jain & Sobek 06] Team cohesion analysis (“Story telling”) [Song, et al 03]
29
Models Preference Transition Model (PTM): relationship
between preferences in 2 consecutive time intervals
Utterance-Preference Model (UPM): relationship between preferences and utterances in one interval
i+1 n i m
p P( =a | =a )= 1-p N-1
when n mwhen n mp p
ìï =ïïíï ¹ïïîMost-preferred alternative in i+1 Probability designers won’t change most-
preferred alternative
0 1p£ £
Alternative uttered in time interval i
Most-preferred alternative in interval i
Probability designer will utter their most-preferred alternative
i n i m
q P( =a | =a )= 1-q n N-1
when n mwhen me p
ìï =ïïíï ¹ïïî1 1qN < £
30
Case Study 1: Large Scale Space System Design Highly concurrent, real-world design team working
on concept stage of space system architecture 17 experienced scientists and engineers; range of
disciplines Focused on group of 4 working on single subsystem Three 3-hour sessions of discussion ~28,000 words Primary team member talked nearly 85% of the
time Two component selection problems
[http://history.nasa.gov]
31
Case study 1: Results from Large Scale Space System Design
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12
Time Interval (in 10 minutes)
Mos
t-Pre
ferr
ed P
roba
bilit
y
Alternative a1
Alternative a2
Alternative a3
[Ji, Yang & Honda, 2007, ASME IDETC 2007]
a2 and a3 alternate with each other as most-preferred choice
Alternative a1 is least preferred
View of how probability of preference changes over time for a re-design problem
32
Case Study 2: Coffee Maker Design Small engineering team with 3 graduate students
Name/ID Glass carafe Stainless-steel carafe Plastic carafe
Photo
Description Glass with warming plate
thermal-insulated stainless-steel
thermal-insulated plastic (inside glass)
Cost $10.00 $20.00 $15.00
Footprint size Big Small Small
Fragility Fragile Strong Fragile inside
Durability Durable Durable Less durable
Heat retention Good Satisfactory Good
Weight Light Heavy Light
Portability Not portable Portable Portable
Easy to clean Easy to clean Not easy to clean Not easy to clean
Style Moderately attractive Very attractive Not attractive
Capacity 2 - 6 cups 2 - 6 cups 2 - 6 cups
Spout Does not dribble Dribbles after pouring Dribbles after pouring
Can tell how much coffee is left
Yes No No
33
Case study 2: Preferential Probability Results From Transcript Analysis
00. 10. 20. 30. 40. 50. 60. 70. 80. 9
1
12: 00 22: 45 32: 30 41: 22 50: 20Ti me (mm:ss)
Pref
eren
tial
Pro
babi
lity
gl asssteelpl ast i c
34
Case study 2: Comparison of Preferential Probabilities from Transcripts and Surveys
00. 20. 40. 60. 8
1
12:00 22:45 32: 30 41: 22 50: 20Ti me (mm: ss)
Pref
eren
tial
Pro
babi
lity
PPSPPT
00. 20. 40. 60. 8
1
12: 00 22: 45 32: 30 41: 22 50: 20Ti me ( mm: ss)
Pref
eren
tial
Pro
babi
lity
PPSPPT
Stainless Steel Carafe
Plastic Carafe
00. 20. 40. 60. 8
1
12: 00 22: 45 32: 30 41: 22 50: 20Ti me (mm: ss)
Pref
eren
tial
Pro
babi
lity
PPSPPT
Glass Carafe
Measure Possible Ranges
Results
L1 norm [0, 16] 1.84L2 norm [0, 4] 0.543Cosine similarity [0, +1] 0.974Pearson product-moment correlation coefficient
[-1, +1] 0.956p-value:
7.67E-9
Spearman’s rho [-1, +1] 0.833p-value:
6.56E-5
35
Conclusions Approach capable of extracting preferential
probabilities Preferential probabilities extracted from transcripts
changed over the course of the design process In this work, preference-related information
extracted from the transcripts was consistent over time with those from surveys
Future work
37
Integrated view of design activities Design thinking manifests itself in different forms at
different points of the design process What are these forms? How do they collectively evolve over time? What is their relationship to outcome?
Sketches Prototypes
Text
Sketches Prototypes
TextTime
38
System modeling Formulate better system level models to improve
system design and reliability Consider emergent properties: nonlinear, complex
interactions between subsystems Draws on existing subsystem models and empirical
system data Allows prediction of future states, balancing of design
trade-offs System model
Thermal-hydraulic
subsystem
Structuressubsystem
Controlssubsystem
Other subsystems…
Complex interactions among subsystems
39
Modeling the language of design Understand how designers express preference in
natural language Linguistically and mathematically model preference as
expressed in engineering design texts Advance basic knowledge of the “language of design” Challenge: Model uncertainties in preference and
convert into mathematical models applied to formal design decision-making
Recommended for NSF AwardLanguage of RiskPreference, Choice and , Uncertainty &
Preference
Mathematical Models
Engineering Design
Validation
40
Teaching ESD.40 Product Design & Development 2.009 Product Engineering Processes IAP 2.97 Design-A-Palooza (new, mostly ugrad)
Focus on defining problems
41
Acknowledgments Thoughtful support of MIT Engineering Systems
Division and Department of Mechanical Engineering 2006 NSF CAREER Award DMI-0547629 NASA Cooperative Agreement NNA04CL15A