ASEE CIEC Conference

Phoenix, AZ, February 6-10, 2013

Dr. Norali Pernalete, Dr. Cordelia

Ontiveros and Dr. Phil Rosenkrantz,

California State Polytechnic University, Pomona Thomas Cossio, The Boeing Company

Industry Involvement

in Developing Soft

Skills for Students in

the College of

Engineering at Cal

Poly Pomona

Cal Poly Pomona

+

In Memoriam

Tom Cossio

Director, Quality

Integration

The Boeing Company

1962-2013

Cal Poly Pomona

2

Cal Poly Pomona

One of the largest engineering programs in California

One of 24 California State University campuses

Over 5000 engineering students, seven departments,

11 undergraduate degrees, 5 masters degrees

Highly ranked among non-research universities

Tournament of Roses Parade float co-built each year

with Cal Poly San Luis Obispo

Cal Poly Pomona

3

Outline of Presentation - 1

Senior project symposium used for assessment

Added Symposium Survey in 2008 to get higher

level assessment data

First Year Results

Three year initiative for

Communications

Teamwork

Problem Solving

Cal Poly Pomona

4

Outline of Presentation - 2

Second Year Results

Five Year Results and Trends

Overall

By department

By evaluator classification

Conclusion and Future Plans:

What worked and what didn’t work

Future plans & Ideas

Broader Goals

Cal Poly Pomona

5

Projects

Symposium Day

Each Spring approximately 75 industry friends attend Project Symposium Day and evaluate over 200 senior projects.

Historically each program uses their own rubrics and processes for evaluation and assessment.

In 2008 the College of Engineering Assessment Committee conducted a post-symposium survey of industry representatives to assess seven outcomes from a college-wide perspective.

Projects Symposium URL: http://www.csupomona.edu/~engineering/events/symposium/index.htm

Cal Poly Pomona

Excerpt from Typical Project

Presentation Assessment Rubric

Cal Poly Pomona

7

Outcomes

4 Exemplary

(Professional Level

Quality)

3 Proficient (Good

Quality for Graduating

Senior)

2 Marginal (Acceptable

Level Quality for

Graduating Senior, but

room for improvement)

1 Unacceptable (Quality

level is not acceptable)

Ability to apply

advanced material

in the discipline

(UD)

Clearly demonstrates

highly professional

understanding of

material included in

upper division courses.

Adequately demonstates

understanding of upper

division course material,

but not necessarily at a

fully professional level.

Display of professional

level understanding of

upper division material is

somewhat uncertain or

raises doubts in the

listeners' minds about the

authority of the

presentation

Presentation fails to

demonstrate that the

speaker(s) understand or

appropriately use

knowledge of the discipline

Ability to identify

problems and

determine their

root causes.

Clearly demonstrates a

structured and

appropriate problem

solving approach was

used for the project.

Creative thinking and

solid reasoning was

clearly demonstrated.

Adequate approach to

problem solving but may

have lacked rigor or

thoroughness in several

aspects.

Problem solving

methodology addressed

and used, but approach

not formulated as well as

it could have been.

Problem solving approach

or methodology missing or

inadequate. Does not reflect

a systematic approach to

problem solving.

On-line survey sent via email to industry

representatives that attended the symposium

43 responses in 2008

44 responses in 2009

48 responses in 2010

50 responses in 2011

32 responses in 2012

Approximately a 2/3 response

Project Symposium Survey

Cal Poly Pomona

8

SurveyMonkey Survey Designed

and Sent to Symposium Attendees

Cal Poly Pomona

9

2008 Results - Number of Industry

Representative Visits by Department

0

2

4

6

8

10

12

14

Aerospace Chemical Civil Electrical &Computer

EngineeringTechnology

Mechanical Industrial &Manufacturing

Cal Poly Pomona

10

Qualities Rated on the Survey

Knowledge - Ability to apply knowledge of math, science, and

engineering

Conduct Experiments - Ability to design and conduct experiments, as

well as to analyze and interpret data

Design a System - Ability to design a system to meet desired needs

Multidisciplinary Teams - Ability to function on multi-disciplinary teams

Solve Engineering Problems - Ability to identify, formulate, and solve

engineering problems

Communications - Ability to communicate effectively

Use Engineering Tools - Ability to use the techniques, skills, and modern

engineering tools necessary for engineering practice

ABET a-e, g, k

Cal Poly Pomona

11

Average importance for engineering

graduates? (5 = very important)

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

knowledge designexperiments

design multi-disciplinaryteams

solveengineering

problems

communications use engineeringtools

2008 Results

Cal Poly Pomona

12

Effectiveness of the Cal Poly

Pomona educational experience?

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

knowledge designexperiments

design multi-disciplinaryteams

solve engineeringproblems

communications use engineeringtools

2008 Results

Cal Poly Pomona

13

Gap Analysis Comparison (gap = effectiveness – importance)

-0.70

-0.60

-0.50

-0.40

-0.30

-0.20

-0.10

0.00

0.10

0.20

app

lykn

ow

led

ge

desig

ne

xp

erim

en

ts

desig

n a

syste

m

mu

lti-d

iscip

lina

ryte

am

s

so

lve

eng

ine

erin

gp

rob

lem

s

co

mm

un

icate

use e

ngin

eerin

gto

ols

2008 Gap

Cal Poly Pomona

14

Analysis

Gap analysis showed three

areas needing greatest

improvement:

Communications

Ability to solve engineering

problems

Multidisciplinary teamwork

College adopted a three year

plan to address these three

focus areas

Cal Poly Pomona

Three-Year College Plan

2008-2009: Communications

2009-2010: Multi-disciplinary Teamwork

2010-2011: Problem Solving

Annual

Themes

Adopted in

Fall 2008

Cal Poly Pomona

2008-2009: Communications

The ability to communicate orally, visually, and in

written form

Includes critical thinking and comprehension skills as

well as interpersonal communication skills

Leads up to the teamwork theme in 2009-2010.

Cal Poly Pomona

17

2009-2010: Multi-disciplinary

Teamwork

The ability to perform as an effective team leader or

team member in multidisciplinary situations.

This theme could align very well with the teamwork

approaches found throughout the business world today

and embodied in various quality management systems

and strategies (e.g., Six Sigma Quality, AS 9100, ISO

9001:2004, etc.).

There are several “High Performance Team” models that

could be considered and adopted under this theme.

Cal Poly Pomona

18

2010-2011: Problem Solving

Almost every engineer will need problem solving

skills.

These skills include:

Problem identification

Prioritizing

Root cause analysis

Problem diagnosis

Creative and innovative problem solving

Implementation

Cal Poly Pomona

19

Communications

Program

2008-2009

“Professor for a Day” panel discussion

plus class speakers.

Engineering Futures presentation on

communications by ΤβΠ National Office

Liberal Arts and Science faculty attended

Senior Project Presentations. STEM

Committee formed to work on joint

projects.

Three workshops for faculty led by invited

experts from inside and outside the COE.

Pilot project requiring presenters to

provide a one-page abstract usable for

immediate assessment of writing skills .

Pilot project using English M.A. students

to coach students preparing

presentations.

Cal Poly Pomona

Second

Assessment

Cycle 2009

Same assessment process

followed at Project Symposium

Day in 2009

Program changes:

Added Showcase

presentations

Change in makeup of

industry visitors

Changes may confound

results

Cal Poly Pomona

Number of Industry Representative

Visits by Department

0

2

4

6

8

10

12

14

16

18

Aerospace Chemical Civil Electrical &Computer

EngineeringTechnology

Mechanical Industrial &Manufacturing

08 Count

09 Count

Cal Poly Pomona

22

Average importance for engineering

graduates? (5 = very important)

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

knowledge designexperiments

design multi-disciplinaryteams

solve engineeringproblems

communications use engineeringtools

08 Q3 Avg

09 Q3 Avg

Cal Poly Pomona

23

Analysis

Results show consistency in

responses from year-to-year

Same three areas most

important:

Communications

Engineering problem solving

Multidisciplinary teamwork

Cal Poly Pomona

Effectiveness of the Cal Poly

Pomona educational experience?

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

knowledge designexperiments

design multi-disciplinaryteams

solve engineeringproblems

communications use engineeringtools

08 Q4 Avg

09 Q4 Avg

Cal Poly Pomona

25

Analysis

Averages dropped in four areas:

Ability to design experiments

and analyze data

Ability to design a system

Communications

Ability to use engineering tools

Averages increased in three

areas:

Knowledge

Multi-disciplinary teams

Problem solving

Communications dropped

the most of all the

outcomes assessed

Cal Poly Pomona

Gap Analysis Comparison

-1.20

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

apply

kn

ow

ledge

desig

n e

xp

erim

ents

de

sig

n a

syste

m

multi-d

iscip

lina

ryte

am

s

solv

e e

ngin

ee

ring

pro

ble

ms

com

munic

ate

use e

ng

ineerin

gto

ols

08 Gap

09 Gap

Cal Poly Pomona

27

Analysis

The gaps were smaller in all

areas except:

Ability to Design

Experiments and Analyze

Data

Communications

Ability to use engineering

tools

Three focus areas still have

largest gap

Communications continues

to be one of our areas in

need of improvement

Cal Poly Pomona

Q6-How willing would you be to refer someone to

the Cal Poly Pomona College of Engineering?

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 NotWilling

2 3 4 5 6 7 8 9 10Very

Willing

08 Response

09 Response

Cal Poly Pomona

29

9-10: Promoters – (-12%) People who are

selling your programs through word-of-mouth

7-8: Passives – (+10%) People who are

satisfied, but have reservations about

recommending the program to others. Usually not

satisfied with one or more aspects.

1-6: Detractors – (+2%) People who are

probably saying negative things about the program.

Cal Poly Pomona

Question 6 Implications 30

Analysis

Slight decrease in scores

using the scale given

Written comments show strong

support and confirm concerns

about communications

Overall analysis

Communication skills are

growing in importance

Valuable program but:

Did not reach enough

students or faculty

Need to continue and

expand

Cal Poly Pomona

2009-2010 Program

Multi-disciplinary Teamwork

Initial Observations

Team projects are used in many classes but very few

teamwork skills are taught

Many students are learning negative teamwork skills

as the result of being on dysfunctional teams and

witnessing traits like slacking or dominating.

Lack of resources for instructors and students

Difficult to evaluate team member performance

FYE is not enough. More reinforcement at sophomore

level.

Cal Poly Pomona

32

2009-2010 Program

Multi-disciplinary Teamwork

Develop a program with the following goals:

Develop resources for faculty such as on-line and

face-to-face workshop and on-line resources that can

be used in class

Provide resources for students

Collaboration with other colleges to help our students

in GE and support courses

Results:

Held one workshop for faculty

One very well attended presentation by alumni

Cal Poly Pomona

33

2010-2011 Program

Problem Solving

Faculty were too busy preparing for

ABET to put in proper effort on this

program

Ultimate goal is to prepare

resources for faculty and students

Held one very well attended panel

discussion by alumni for students

Resources developed but deployed

on a limited basis

Cal Poly Pomona

34

Five Year Overall Results for

Importance

Cal Poly Pomona

35

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

2008

2009

2010

2011

2012

Overall Five Year Results for

Effectiveness

Cal Poly Pomona

36

0.000.501.001.502.002.503.003.504.004.505.00

2008

2009

2010

2011

2012

Overall Gap Analysis

Cal Poly Pomona

37

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sign

a s

yste

m

Team

s

Solv

e p

rob

lem

s

Co

mm

un

icat

ion

s

Use

en

gin

ee

rin

gto

ols

2008

2009

2010

2011

2012

Question 6 – Five Year Results How willing would you be to refer someone to the Cal

Poly Pomona College of Engineering?

Cal Poly Pomona

38

0%

10%

20%

30%

40%

50%

60%

1 NotWilling

2 3 4 5 6 7 8 9 10Very

Willing

2008

2009

2010

2011

2012

Department Level and

Position Level Results

Cal Poly Pomona 39

Industrial & Manufacturing

Engineering Gap Analysis

Cal Poly Pomona

40

-1.40

-1.20

-1.00

-0.80

-0.60

-0.40

-0.20

0.00

0.20

0.40

0.60

Ap

ply

kn

ow

led

ge

Co

nd

uct

exp

eri

me

nts

De

sign

a s

yste

m

Team

s

Solv

e p

rob

lem

s

Co

mm

un

icat

ion

s

Use

en

gin

eeri

ng

too

ls

2008

2009

2010

2011

2012

Interpretation of

Industrial &

Manufacturing

Engineering

Results

Gap analysis was presented to

the IME Department Industrial

Advisory Council Members in

November 2012.

They were asked if the result

were valid. The unanimous

response was that the results

were loud and clear…

Communications and

teamwork are important for

industrial and manufacturing

engineers and we need to do

a better job.

Cal Poly Pomona

Aerospace Engineering Gap

Analysis

Cal Poly Pomona

42

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

Apply

know

ledge

Co

nd

uct

exp

erim

ents

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Aerospace

Engineering

Results

Aerospace Engineering

Presentations were not

traditional senior projects—

rather were class design

projects which probably

confounded the results

No noticeable improvement

over time

Communications had the

largest overall gap

First three outcomes were

large gaps compared to other

majors

Cal Poly Pomona

Chemical Engineering Gap Analysis

Cal Poly Pomona

44

-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

Apply

know

ledge

Co

nd

uct

exp

erim

en

ts

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Chemical

Engineering

Results

Problem Solving and

Communications were the

biggest gaps

All areas showed improvement

over time

Probably some of the smaller

gaps among all the

departments

Cal Poly Pomona

Civil Engineering Gap Analysis

Cal Poly Pomona

46

-1.5

-1.0

-0.5

0.0

0.5

1.0

Apply

know

ledge

Co

nd

uct

exp

erim

en

ts

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Civil Engineering

Results

Based on Large Team Projects

Communications showed the

biggest gap

2011 was an anomaly

compared to other years

No real trends in the gap

results over time

Civil was very proactive in

improving communications --

need to question effectiveness

of efforts

Cal Poly Pomona

Electrical & Computer Engineering

Gap Analysis

Cal Poly Pomona

48

-1.5

-1.0

-0.5

0.0

0.5

1.0

Apply

know

ledge

Co

nd

uct

exp

erim

en

ts

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Electrical &

Computer

Engineering

Results

Communications was the

largest gap category

Program had some of the

smallest gaps in the college

Mixed as to improving or

regressing – two categories

showed improvement over

time and the rest were

scattered

Cal Poly Pomona

Mechanical Engineering Gap

Analysis

Cal Poly Pomona

50

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

Apply

know

ledge

Co

nd

uct

exp

erim

en

ts

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Mechanical

Engineering

Results

Communications the largest

gap area

Some areas seem to have

small gaps

2012 showed gaps in areas

that had been mostly even or

free of gaps

Cal Poly Pomona

Engineering Technology Gap

Analysis

Cal Poly Pomona

52

-1.4

-1.2

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

Apply

know

ledge

Co

nd

uct

exp

erim

en

ts

Desig

n a

syste

m

Te

am

s

So

lve

pro

ble

ms

Co

mm

un

ication

s

Use e

ngin

eering

tools

2008

2009

2010

2011

2012

Interpretation of

Engineering

Technology

Results

Communications shows the

largest gap

No real improvement trends

over time

Cal Poly Pomona

Gap Analysis Based on Evaluator

Position – Categorized by Evaluator

Cal Poly Pomona

54

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

En

gin

ee

r

Su

perv

isor

Mid

dle

Up

pe

r

Execu

tive

Ow

ner

Applyknowledge

Conductexperiments

Design asystem

Teams

Solveproblems

Communications

Useengineeringtools

Interpretation of

Results by

Position

Engineers were about half of the respondents overall and showed gaps in all areas with communications, problem solving and teamwork being the largest three

Supervisors were distinctly different than all other categories showing no gaps

Gaps became progressively larger as the position of the management evaluator went up in rank

Expectations vary based on perspective…this is worthy of interpretation by various stakeholders

Cal Poly Pomona

What Worked

Obtained data that was otherwise

being ignored

Data was useful at the college

and department level

Some Assessment Coordinators

used the feedback to guide

department improvement efforts

for outcomes

Communicated with the College

of Letters, Arts & Social Sciences

(CLASS) at a meaningful level

Showcase and Alumni

Presentations were very effective

Cal Poly Pomona

What Didn’t

Work

Results did not reach faculty as effectively as needed for buy-in and continuous improvement

Low participation among faculty in communications workshops

We did not follow through effectively with CLASS faculty to sustain interaction and effect changes

Most departments did not review results with Department Advisory Councils to interpret the results and plan for action

Cal Poly Pomona

Future Plans

Continue assessment program.

Keep providing resources for three focus areas that will

attract faculty buy-in. Expand reach of programs for both

faculty and students with on-line resources that are

reusable.

Tie into First Year Experience (FYE) programs

Strengthen collaboration with Liberal Arts and Science

departments

Take a long view and be persistent

Cal Poly Pomona

58

Ideas for Improving

Communications Program

Expand and increase seminars for faculty

Adopt a default “style guide” to use throughout the

COE (Default guide would be used unless instructor

or department specifies otherwise).

College-wide student competition judged by industry

with prizes for the best presentation, written report,

and research paper.

Expanded used of the Showcase in FYE courses.

Cal Poly Pomona

59

Other Ideas Choose a book about

communications to use

throughout the curriculum (at the

college or department level).

Use in engineering classes to

help make it real to the students.

Cal Poly Pomona

60

Broader Goals

Establish communications, teamwork, problem

solving, and other “soft skill” outcomes threads

across undergraduate curriculum:

Multi-year

Interdepartmental—across the college of

engineering

Cross-disciplinary—across the campus

Requires collaboration with general education

suppliers

Cal Poly Pomona

61