1

Natural Sciences Assessment

At Kennedy-King CollegeFall 2014

DRAFT - FOR DISCUSSION ONLY

Prepared by Robert Rollings, spring 2015

2DRAFT - FOR DISCUSSION ONLY

In fall 2014, Kennedy-King College administered an assessment of natural sciences learning to 1,370 student respondents in a wide

sample of classrooms in various subjects.

The test design progressed from questions requiring basic reading comprehension of science passages to questions demanding

critical thinking and application of concepts.

The assessment designers hypothesized that students would perform well with basic comprehension and poorly with critical

application.

With these results, we test that hypothesis.

3DRAFT - FOR DISCUSSION ONLY

Black Asia

nWhite

Hispanic

Middle Eastern

Native Americ

an

Other Ethnicit

y

No Selecti

onMixe

d

85.0%

1.5% 2.9% 6.5%0.4% 1.2% 4.1% 2.8% 3.4%

% Respondents by Race/Ethnicity

<181%

18-2233%

23-3031%

31-4018%

41-5010%

51-657%

>650%

% Respondents by Age Group

Female67%

Male32%

Other1%

% of Respondents by Gender Identity

Full-Time, 79%

Part-Time, 21%

% Respondents by Full Time/Part Time

The sample was demographically typical of KKC, except for a disproportionate representation of full-time students.

4DRAFT - FOR DISCUSSION ONLY

Biology32%

Chemistry9%

Communication9%

Humanities15%

Math7%

Microbio3%

Physical Sciences7%

Professional5%

Social Science12%

Class in which Survey was Taken

Biology Chemistry Physical Science Other Science

55.2%

19.6% 19.3%24.0%

Subjects in which Respondents have Taken at Least One Course

The assessment was administered in a broad array of classes, but in a disproportionate number of biology classes.

Thus we see the disproportionate number of respondents who report having taken a class in biology. We might expect higher overall performance on biology questions than if the survey were evenly distributed.

5DRAFT - FOR DISCUSSION ONLY

Students broadly agree with the value of science but are more likely to agree strongly that it is important for their careers than that they find it personally interesting.

Science is important to my career

I find the natural sciences personally interesting.

Understanding science and scientific methods helps in making consumer and life decisions.

5%

4%

3%

17%

16%

12%

32%

53%

54%

47%

27%

32%

Respondent Agreement with the Value of the Sciences

Strongly Disagree Disagree Agree Strongly Agree

6DRAFT - FOR DISCUSSION ONLY

025%

129%

224%

3+22%

Number of Math Courses Taken by Respondents

026%

138%

220%

3+17%

Number of Science Courses Taken by Respondents

No65%

Yes35%

Percent of Respondents who have taken college-level science courses

outside Kennedy-KingAbout a quarter of respondents have not taken a science class at KKC, and only 20.8% of the sample had taken no college science courses at KKC or elsewhere. 37% of the sample had taken more than one course in the sciences.

If we have been achieving our SLOs, an overwhelming majority of respondents should have mastery of scientific method and basic concepts.

7DRAFT - FOR DISCUSSION ONLY

Q1 Q2 Q3 Q4 Q5

57%

77%

56% 56% 57%

% Correct Responses for Section 1

Q6 Q7 Q8 Q9 Q10

31%19% 21%

37% 28%

% Correct Responses for Section 2

QIII-11 QIII-12 QIII-13 QIII-14 QIII-15

67%57%

18% 21% 25%

% Correct Responses for Section 3

8DRAFT - FOR DISCUSSION ONLY

Questions 1-5 measured ability to describe the scientific method and hypothesis testing. Most students performed well on this section.

Q1 Q2 Q3 Q4 Q5

57%

77%

56% 56% 57%

% Correct Responses for Section ISample Size: 1370

is just a theory

is a strict set of rules and procedures that lead to inarguable fact.

is based on continued observation and experiment.

is a framework for proving an argument you know to be true.

BLANK

11.6%

21.9%

57.4%

5.1%

4.0%

Responses to Q1, "The Scientific Method..."

People who believe that science leads to inarguable facts (or worse, is “just a theory”) may be unlikely to engage critical thinking when engaging scientific findings, and still less likely to perform science themselves.

Oddly, having taken a KKC science course makes no significant difference in correct response rates to this question!

However, a large minority of students show confusion about the basic nature of scientific inquiry.

9DRAFT - FOR DISCUSSION ONLY

Q6 Q7 Q8 Q9 Q10

31%19% 21%

37%28%

% Correct Responses for Section IISample Size: 1370

Questions 6-10 measured conceptual mastery of precision, accuracy, and significant digits, then on application of a basic formula (Density=Mass/Volume) to measurements.

Most students could not read volume measurements from these figures correctly, nor could they distinguish between precision and accuracy.

This is foundational language in any quantitative work in science.

Respondents performed poorly on this section. KKC students, including those who have taken science classes, answered correctly concerning significant figures (Q7) at levels worse than if they had responded to the question randomly.

10DRAFT - FOR DISCUSSION ONLY

QIII-11 QIII-12 QIII-13 QIII-14 QIII-15

67%57%

18% 21% 25%

% Correct Responses for Section IIISample Size: 479

Students had a choice between section III, concerning atoms, and section IV, concerning DNA.

Section III began with two questions that required reading comprehension and proceeded to three questions that required application of concepts from the reading.

Section IV measured reading comprehension throughout.

The performance gap between questions that measured comprehension versus application is visible and profound.

These results appear to confirm the hypothesis that students have learned how to read and recall basic findings of science but learn far less how to apply concepts and use critical problem-solving skills.

QIV-11 QIV-12 QIV-13 QIV-14 QIV-15

43%52%

69% 62%52%

% Correct Responses for Section IVSample Size: 603

11DRAFT - FOR DISCUSSION ONLY

0 1 2 3 4 5 6 7 8 9 10

4.2% 3.6%

6.9%

14.4%

22.0%19.7%

16.6%

8.8%

3.1%

0.6% 0.1%

Percent of Respondents by their Number of Correct Responses, Sections I-II

Only a handful of students answered every question correctly, whereas here we see another way of expressing that most students got the first two or three questions per section correctly, and then the harder questions requiring critical application incorrectly.

The overall average of correct responses for Sections I-II was 4.40 out of 10 possible.

The standard deviation is 1.9.

12DRAFT - FOR DISCUSSION ONLY

Overall Male Female Black Non-Black

4.4 4.334.48 4.41 4.36

Average Number of Correct Responses on Sections I & II

How well did various groups perform relative to each other?

Gender & Race matter a little—but not much.

4.40 is the overall average of questions 1-10 answered correctly.These graphs will all be set within boundaries of 3.5 to 5.5 so that they can easily be compared against each other.

Full-Time Part-Time FT/PT Left Blank

4.47 4.44

4.08

Average Number of Correct Responses on Sections I & II Part-time and Full-time students did

about equally well. Students who were careless or couldn’t be bothered to fill in that blank performed worse than others.

Note that women performed slightly better, breaking a stereotype concerning the natural sciences.

13DRAFT - FOR DISCUSSION ONLY

Strongly Agree that Science is important to a Career

Strongly Agree that Science is personally Interesting

Strongly Agree that Science helps with Life & Consumer

Decisions

4.75 4.72 4.67

Average Number of Correct Responses on Sections I & II

Strong agreement with the importance of science correlated with higher overall performance.

Traditional College Age (18-22)

Ages 23-40 Age 41 or Over

4.704.53

3.64

Average Number of Correct Responses on Sections I & II

Age mattered a surprising amount. Students over 41 fared far worse on the assessment.

14DRAFT - FOR DISCUSSION ONLY

Taken Chemistry Taken Biology Taken Physical Science

Taken Any Other Science

Taken No KKC Science

5.05

4.54.36

4.21 4.28

Average Number of Correct Responses on Sections I & II

4.28 4.36 4.374.9 4.68

Average Number of Correct Responses on Sections I & II

Performance could also be divided between students who had done a lot of science, or science at anywhere other than KKC, and everyone else.

Possibly it should raise eyebrows that students who have taken 3 or more science classes still only averaged about half the questions correct. Critical application skills are not being learned.

However, students appear to learn the skills relevant to this assessment more in chemistry classes than in other sciences at KKC.

15DRAFT - FOR DISCUSSION ONLY

CONCLUSIONSStudents understand how to report back scientific findings,but they have weaknesses in measurement, application, and critical thinking.How does it matter in your class if students can engage empirical critical thinking skills?Why do students not acquire these skills more frequently?How can we help them learn these skills?

Strong agreement with the value of science correlates with better performance on the assessment.Are there ways to increase student perceptions of the value of science?

Students aged above 40 performed significantly worse on the assessment.Are older students out of practice with these kinds of problems?Are different teaching methods depending on generation appropriate?

Students who had taken any class in chemistry performed significantly better on the assessment. Are several fundamental skills being included more in the chemistry curriculum than in other core science courses? Or could there be a confounding variable involved?