1
Empirical Studies of Design Ideation: Empirical Studies of Design Ideation: Alignment of Design Experiments with Lab ExperimentsAlignment of Design Experiments with Lab Experiments
JamiJ. ShahNoe Vargas-Hernandez
Mechanical and Aerospace EngineeringArizona State University, Tempe, AZ
ASME 2003 International Conference on Design Theory and MethodologySeptember 5, 2003, Chicago Il
NSF Grant Number: DMI-0115447
Steve M. Smith David R. Gerkens
Department of PsychologyTexas A&M University, College Station, TX
Muqi WulanDepartment of Industrial and Manufacturing Systems Engineering
Beijing University of Aeronautics and Astronautics, Beijing, China
2
INTRODUCTIONINTRODUCTION
There are many methods for design synthesisHow useful are these? Which ones are better? There isn’t much empirical data on specific effectiveness of Idea Generation (IG) methods
MOTIVATION
For many years, Psychologists and Designologists have studied IG. Their experiments have different “ecological” validity (i.e. realism captured)
•Lab Experiments done by Psychologists
Low level – use few and simpler variables
•Design Experiments done by Engineers
High level – use more and complex variables
PSYCHOLOGY VS ENGINEERING
3
INTRODUCTIONINTRODUCTION
If our experiments at different levels have the same results, we can establish a connection and we can run more of the simpler Lab Experiments
More Experiments means more empirical data on specific IG methods
Our alignment approach is still WIP and is part of a bigger project
Our ultimate objective is to develop a theoretical model of design ideation
Such a model would help us better understand IG methods
SCOPE
At this time we do not consider human variables as experiment variables (e.g. experience, creativity of designers) Our focus is on Intuitive IG methods
OBJECTIVE
4
INTRODUCTIONINTRODUCTION
Figure 1. Classification of Idea Generation Methods (Shah et al., 2000)
INTUITIVE
LOGICAL
FORMAL IG
METHODS
DESIGN IDEA GENERATION METHODS
INVERSION
FORWARD STEPS
MORPHOLOGICAL ANALYSIS BRAINSTORMING K-J METHOD
CHECKLISTS RANDOM STIMULI P-M-I METHOD
METHOD 6-3-5 C-SKETCH (Collaborative Sketching) GALLERY METHOD
AFFINITY METHOD STORYBOARDING FISHBONE
SYNECTICS
GERMINAL
TRANSFORMATIONAL
PROGRESSIVE
ORGANIZATIONAL
ANALYTICAL
HYBRID
DESIGN CATALOGS OF PHYSICAL EFFECTS OF SOLUTIONS HISTORY
BASED TRIZ
SIT
5
PRELIMINARY RESEARCHPRELIMINARY RESEARCH
For several years we have been doing experimental studies to define the effectiveness of ideation
PAST EXPERIMENTS
We used two distinct approaches to conducting experiments:
•Direct Method – IG methods are studied as a whole
•Indirect Method – Ideation Components are studied
And two distinct approaches to assessing the effectiveness:
•Process – Assess the ideation process (e.g. protocol studies)
•Outcome – Assess the ideas produced (e.g. sketches)
6
PRELIMINARY RESEARCHPRELIMINARY RESEARCH
IDEATION COMPONENTSIdeation Components are mechanisms that are believed to intrinsically promote IG or to help designers overcome mental blocks.
Examples of Blocks:•Being Judgmental•Emphasis on Quality•Lack of Motivation•Having a tight grip on problem specs.•Rigid Problem Representation•Design Fixation•Imposing Fictitious Constraints
Examples of Ideation Components:•Provocative Stimuli•Deferred Judgment•Flexible Representation•Frame of Reference Shifting•Incubation•Example Exposure
These are “common”: Known in Engineering Design Research and acknowledged by Cognitive Psychology
7
PRELIMINARY RESEARCHPRELIMINARY RESEARCH
Table 1. Effectiveness Measures for Idea Generation Outcome.
MEASURE DESCRIPTIONQuantity How many ideas were generated n
Quality How close it comes to meeting the design specifications
Novelty How unusual and unexpected an idea is as compared to other ideas
Variety How well the design space was explored
OUTCOME EFFECTIVENESSFour measures were defined in our previous projects (Shah, Kulkarni, and Vargas-Hernandez, 2000).
k
n
kjk
m
jj pSfM
1
11
1
10x1jk
jkjkjk T
CTS
nbSfM kk
k
m
jj /
4
113
m
jjk
kjk
m
jj fnpSfM
1
2
112 *
8
RESEARCH APPROACHRESEARCH APPROACH
FUNDAMENTAL ISSUES
LAB EXPERIMENTS (Done A&M Psychologists)
•Focus on “atomic” cognitive processes•Little similarity between the condition of these experiments and design concept generation in the real world.
DESIGN EXPERIMENTS(Done by ASU Engineers)
•Simulate real world better•Incorporate more and complex variables•Require prohibitive number of experiments•Unable to explain the performance of methods under different conditions.
How can we compare results from experiments at different levels?
RESEARCH APPROACHAlignment: Agree on the Ideation Components to study and the Effectiveness Metrics for assessment
How can we align these two?
9
RESEARCH APPROACHRESEARCH APPROACH
RESEARCH APPROACH
How can we compare results from experiments at different levels?
Alignment:Ideation Components Effectiveness Metrics Lab Experiments
Done byPsychologists
Design Experiments Done
By Engineers
Comparison
Results
Results
More Lab Exp.
Better Understanding of IG
Theoretical Model of
Design Ideation
Figure 2. Research Strategy
10
RESEARCH APPROACHRESEARCH APPROACH
The number of components identified is more than a dozen. Because of limited resources and the prohibitive number of experiments required to study all possible interactions, only the most relevant were selected.
IDEATION COMPONENTS
COMPONENT DESCRIPTIONProvocative Stimuli Excite ideas by exposing the subject to a concept ideaSuspend Judgment Postpone reaching decisions or making conclusions of an ideaFlexible Representation Unconstraint the manner in which ideas are representedFrame of Reference Shifting
Change in the basic set of ideas on which other ideas are interpreted
Incubation Period of time that elapses generation of ideas for a problemExample Exposure Excite ideas by exposing the subject to a model idea
Table 2. Selected Components
11
RESEARCH APPROACHRESEARCH APPROACH
Two levels were considered for each of the selected components. Although more levels could be defined, it is recommended to run experiments initially with few levels.
DESIGN OF EXPERIMENTS (DOE)
FACTORSRUN A B C
1 0 0 02 1 0 03 0 1 04 1 1 05 0 0 16 1 0 17 0 1 18 1 1 1
Full Factorial Experiments
A Frame of Reference Shifting
B Incubation
C Example Exposure
RUN LEVEL1 02 1
Table 3. Experiments in a 23 Full-Factorial Design
Table 4. Simple Comparative Experiments
All 6 Ideation Components were tested at ASU and TAMU simultaneously
Simple Comparative Experiments
•Provocative Stimuli
•Suspend Judgment
•Flexible Representation
12
Experiment VariablesSubjectsUndergraduate Engineering students. Comparable expertise/knowledge between subjects is assumed. Task One design problem was used for all experiments. The objective was to design a device for throwing a ping-pong ball the farthest distance. A list of allowed materials was given; this to improve the quality of sketches.Idea RecordingSubjects were asked to generate ideas individually using sketchesNuisance VariablesSimilar environmental settings procured for each run (classroom, noise, light, etc.)
DESIGN EXPERIMENTS DONE BY ASU ENGINEERS
RESULTSRESULTS
13
RESULTSRESULTSDESIGN EXPERIMENTS DONE BY ASU ENGINEERS
Figure 3. Sample Sketches from Design Experiments
Boat Wheels Attached
Catapult Cannon Hammer
Airplane
HIGH NOVELTY SET
LOW NOVELTY SET
14
RESULTSRESULTS
LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS
Experiment VariablesSubjectsUndergraduate Psychology students. Comparable expertise/knowledge between subjects is assumed. Task •Listing members of large taxonomic categories, sense impression categories, and ad-hoc categories.•Divergent thinking, unusual uses of common objects.Idea RecordingSubjects were asked to generate ideas individually using text for member listing tasks and sketches for divergent thinking tasksNuisance VariablesSimilar environmental settings procured for each run (classroom, noise, light, etc.)
15
RESULTSRESULTS
Figure 4. Sample Sketches from Lab Experiments
LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS
16
Table 5. Lab Experiments Done by TAMU Psychologists: Mean Ideation Effectiveness Scores
RESULTSRESULTS
CONDITION QUANTITY VARIETY QUALITY NOVELTYControl Group 1.51 1.20 1.47 0.00282Incubation Group 1.93 1.60 2.37 0.00373
Table 6. Design Experiments Done by ASU Engineers:Mean Ideation Effectiveness Scores
CONDITION QUANTITY VARIETY QUALITY NOVELTY
Control Group 4.86 2.81 6.15 4.71Incubation Group 5.11 6.24 7.31 6.76
IDEATION COMPONENT: INCUBATION
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
HIGH
17
RESULTSRESULTSIDEATION COMPONENT: INCUBATION
Table 7. Two Sample t-test
CONDITION METRIC t0 P-value
LAB EXPERIMENTS BY TAMUPSYCHOLOGISTS
QUANTITY 1.31 0.19
QUALITY 1.85 0.07
NOVELTY 3.16 0.002
DESIGN EXPERIMENTS BY ASU ENGINEERS
QUANTITY 1.10 0.14QUALITY 5.86 0NOVELTY 12.38 0
18
IDEATION COMPONENT: INCUBATION
RESULTSRESULTS
Table 8. Correlation Between Lab and Design Experiments
METRIC CONDITIONLEVEL
LAB DESIGNQUANTITY Control 1.51 4.86
Incubation 19.3 5.11Correlation 1.00
VARIETY Control 1.20 2.81Incubation 1.60 6.24
Correlation 1.00QUALITY Control 1.47 6.15
Incubation 2.37 7.31Correlation 1.00
NOVELTY Control 0.00282 4.71Incubation 0.00373 6.76
Correlation 1.00
19
CONCLUSIONSCONCLUSIONS
•Based on the results from (TAMU Psychologists) Lab and (ASU Engineers) Design Experiments, Incubation increases the effectiveness of ideas generated. Results correlate at both levels and show a satisfactory confidence level.
•Incubation’s positive impact on Design Ideation is substantiated by concrete Engineering evidence (from ASU Design Experiment results) and has a theoretical basis (from TAMU Lab Experiment results).
IDEATION COMPONENT: INCUBATION
20
CONCLUSIONSCONCLUSIONS
Connection Proven
Experiments on other components have been completed at ASU and TAMU.
•More experiments needed to prove connection•The alignment procedure provides a framework for comparison between both levels. Results for Incubation exemplify how the alignment works.
OVERALL
Run more of the simpler Lab Experiments and less of the more complex Design Experiments
More empirical data
Better understanding of Ideation
Theoretical Model of Design Ideation.
21
FURTHER CONCLUSIONSFURTHER CONCLUSIONS
•According to our results we found Frame of Reference Shifting (FORS), Incubation (I), and Example Exposure (EE) to have similar main effects.
•Interaction effects weren't that clear, probably because some components are much alike (specially FORS and EE) and hence aren't independent.
•This generates a question: Maybe these Ideation Components have the same effect on ideation ?
•Two or more Ideation Components sharing the same effect could belong to the same higher level Ideation Principle
22
FUTURE WORKFUTURE WORKIDEATION PRINCIPLES
Table 9. Comparison of Ideation Principles with Cognitive ComponentsPRINCIPLE COMPONENTABSTRACTION Abstractions
IncubationAnalogies-Metaphors
JUDGMENT Suspend JudgmentBreak RulesImpose Constraints
RANDOM Frame of Reference ShiftingPERSISTANCE Provocative Stimuli
Example ExposureRandom Connections
COMBINATIONS CombinationsFLEXIBLE REPRESENTATION
Flexible Representation
23
FUTURE WORKFUTURE WORKIDEATION PRINCIPLES
•Refine Ideation Principles and its Implementations (Ideation Components)
•Run more exercises, collect and analyze more data to prove/disprove our theory about Ideation Principles
•We still continue experimenting on Ideation Components, but with a better understanding of Ideation Principles, experiments can be better targeted (e.g. less redundant) and more efficient.
24
REFERENCESREFERENCES1. Altshuller, G., 1984, Creativity as an Exact Science, Gordon and Breach, New York. 2. Dennehy, E.B., Bulow, P., Wong, F., Smith, S.M., and Aronoff, J.B. (April, 1992). A test of cognitive fixation
in brainstorming groups. Paper presented at the meeting of the Eastern Psych. Association, Boston, MA.
3. Dodds, R.A., and Smith, S.M., 1999, Fixation. In M.A. Runco & Pritzker (Eds.) Encyclopedia of Creativity, San Diego, CA: Academic Press.
4. Ericsson, K., and Simon, H., 1984- “Protocol Analysis - verbal reports as data”, MIT Press.5. Finke, R.A., Ward, T.B., and Smith, S.M., 1992, Creative Cognition: Theory, Research, and Applications,
Cambridge, MA: MIT Press. 6. Hale, C., “Analysis of the Engineer Design Process in an Industrial Context”, Grant Hill Pubs, Cambridge,
1987.7. Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Design Studies, Vol. 12, pp. 3-11.8. Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London.9. Langley, P., and Jones, R., 1988, “Computational model of scientific insight,” in R. J. Sternberg, ed., The
nature of creativity – contemporary psychological perspectives, Cambridge University Press, NY. 10. McKoy, F., 2000, “Experimental Evaluation of Engineering Design Representations for Idea Generation”,
MS Thesis, Arizona State University, Tempe, AZ.11. Mckoy, F., Vargas-Hernandez, N., Summers, J. D., Shah, J., 2001, “Experimental Evaluation of
Engineering Design Representation on Effectiveness of Idea Generation”, Proceedings, ASME Design Theory and Methodology Conference, Pittsburgh, PA.
12. Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London.13. Martindale, C., 1995, “Creativity and Connectionism,” in S. M. Smith et al., eds., The creative cognition
approach, MIT Press, Cambridge, MA. 14. Schön, D., 1991,“Teaching and learning as a design transaction”, in Research in Design Thinking Delft
Press, 1991.
25
REFERENCESREFERENCES15. Schwartz, B.L. and Smith, S.M., 1997, The retrieval of related information influences tip-of-the-tongue
states. Journal of Memory & Language, 36, 68-86. 16. Shah, J., 1998, “Experimental Investigation of Collaborative Techniques for Progressive Idea
Generation,” Proceedings, ASME Design Theory and Methodology Conference, Atlanta, GA.17. Shah, J., Kulkarni, S., Vargas-Hernandez, N., 2000, “Guidelines for Experimental Evaluation of Idea
Generation Methods in Conceptual Design”, Journal of Mechanical Design, vol. 122, no. 4, pp. 377-384. 18. Shah, J. J., Vargas-Hernandez N., Summers, J. D., Kulkarni, S., 2001, “Evaluation of Collaborative
Sketching as an Idea Generation Technique for Engineering Design”, Journal of Creative Behavior, 35:3, pp.1-31.
19. Smith, D. K., Paradice, D. B., and Smith, S. M. (2000). Prepare your mind for creativity. Communications of the Association for Computing Machinery , 43, 110-116.
20. Smith, S. M., 1995, “Creative Cognition: Demystifying Creativity,” in C. N. Hedley et al., eds., Thinking and literacy – the mind at work, Lawrence Erlbaum Associates, Hillsdale, NJ.
21. Smith, S. M. and Blankenship, S. E., 1991, “Incubation and the persistence of fixation in problem solving”, American Journal of Psychology, 104, 61-87.
22. Smith, S. M., Carr, J. A., and Tindell, D. R., 1993, April, Fixation and incubation in word fragment completion. Paper presented at the meeting of the Midwestern Psychological Association, Chicago, IL.
23. Smith, S. M., and Vela, E., 1991. Incubated reminiscence effects. Memory & Cognition, 19 (2), 168-176. 24. Smith, S. M., Sifonis, C. M., and Tindell, D. R., 1998, Hints do not evoke solutions via passive spreading
activation. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society, Madison, WI. 25. Vargas-Hernandez, N. and Shah, J. J., 2002, Inventory of Creativity Exercises: 1995-2002, Tech. Report
ASU/DAL/IG/02-1, Arizona State University. 26. Wallas, G., 1926, “The Art of Thought”, Harcourt, New York.