A First Look at Retention

Following changes in a two-semester Business Statistics Sequence at Duquesne University

Amy L. Phelps, phelpsa@duq.edu

Speaking About Teaching Business Statistics eConference September 2012

The Palumbo Business School graduates about 300 undergraduate students a year. The two-semester statistics sequence is required of all business students.

Duquesne is located on a secluded, private 48-acre campus in the heart of Pittsburgh.

Duquesne is one of America’s leading Catholic institutions offering liberal and professional education since 1878.

Private Urban Campus

www.business.duq.edu

Palumbo School of Business

4

AACSB Assurance of Learning Standards,

From AACSB White Paper, 2007: AOL focus is the Program (not student,

faculty, course…) “By measuring learning the school can

◦evaluate its students’ success◦use the measures to plan improvement

efforts◦provide feedback and guidance for individual students”

5

AACSB Assurance of Learning Standards

Prescribed process (AACSB, 2007):1. Definition of learning goals and objectives2. Alignment of curricula with adopted goals3. Identification of instruments and measures

to assess learning4. Collection, analyzing, and dissemination of

assessment information5. Using assessment information for

continuous improvement…

What is a ‘Program’

1. The undergraduate Business curriculum◦ Eg. The Business Core courses◦ Align with mission statement of the School of Business

2. An undergraduate Business major ◦ Eg. The economics major◦ Align with the mission of the Economics major

3. A collective set of courses with a common area of study◦ Eg. An assessment of the two required Stat courses◦ Align with specific learning objectives of the course

Statistical reform movement leading to GAISE◦ Copious assessment ff first course in stats

CAOS exam (DelMas) ◦ Statistical literacy, pre/post, first course◦ Sisto (JSM, 2006) and Phelps (JSM, 2007) applied the

CAOS to second semester business statistics sig improvement in scores

Retention ff gap in time◦ Tintle et al (SERJ, 2012)

First course, pre/post/4mos lag Compared traditional to randomized-based curriculum

Related Literature

CAOS Exam (DelMas) tracked pre/post◦ Hovered around 50% with stat II sig higher

CCPE: Duquesne exam aligned with LO◦ Stat I – sig ↑ year 3◦ Stat II – hovered in low 40%

ETS: Commercially marketed national exam◦ 6 Prob and Stat ques: 12% to 97% students

answered correctly Statistics Retention Exam created by myself

in alignment of school and program goals

Our History of Bstat Assess

18 MC questions◦ Concepts more than 70% answered correctly

The normal distn is defined by mean and sd identified correct interpretation of CI for mean identified correct interpretation of margin of error, p identified correct scatterplot for given reg equation correctly calculated joint prod from contingency table

◦ Concepts less than 50% answered correctly Use median when data are skewed When data are skewed right, mean > median As n increases, the length of CI decreases identifying which histogram has larger sd

First Stat retention assessmentAdministered Fall2009-Spring2011

2 short-answer, problem solving◦ Two-sample t-test

~1/3 rejected a non-sig result ~1/4 gave incorrect/inconsistent reasoning ~1/4 inconsistent written conclusion based on

whether they accepted or rejected the hypothesis◦ Simple linear regression

~20% identified sig lin pred with correct reasoning Wide variations interpreting R2 Much difficulty using Excel coefficients to create the

least squares equation

Written short answers

We launched a revised curriculum attempting to reduce theory and increase application using real-world examples

~Half the faculty used the traditional curriculum, half used the revised curriculum

Observed changes in retention exam scores administered Fall/Spring 2011-2012

Proposed curriculum changesAdministered in 2010-11

Business Statistics I◦ Descriptive statistics

(not including least squares regression)◦ Probability, Probability distributions, Expected values,

random variables etc…◦ Normal distribution, Central Limit Theorem◦ Sampling distribution for mean◦ Confidence interval and intro to hypothesis testing

Business Statistics II ◦ Review one-sample ttest, two-sample ttest, paired ttest,

Chi-square tests, SLR, MLR, ANOVA◦ While technology was used, it was not formally taught,

hand calculations were still assessed on written exams

Traditional curriculum

Business Statistics I◦ Descriptive statistics including least squares equ◦ Probability Rules presented through applications of

contingency table◦ Normal distribution, CLT, Sampling distributions for

means and proportions◦ One and two sample inference◦ Chi-square test of independenceIntention to prepare students for broad applications and case studies using statistical modeling More emphasis on written interpretations through lab exercises and teaching technology (Excel, StatCrunch)

Revised Curriculum

Business Statistics II◦ The second course emphasizing modeling through

applications of cases◦ One and two sample inference is revisited with

applications focused on using independent variable(s) to model/predict dependent variable

◦ SLR/MLR, ANOVA, > 2x2 table chi-square are introduced as cases and datasets become more advanced and modeling makes sense.

◦ Introduction to AnalyticsBigger and more real-life datasets, emphasizing problem-solving and communication of results.Bi-weekly lab written reports.

Revised Curriculum

Our thought was to give them the tools in the first semester

Foster understanding in the second semester through review, applications and cases

Emphasize written and verbal communication

Deeper understanding

Ability to thoughtfully advance through the scientific method and communicate results.

Conceptually

Ideally, Sophomores take the two course sequence and then in their Junior year they take Global Economics either in the Fall or Spring

Ideally, they take the retention exam following a one (summer) or two semester break (summer plus fall).

The retention exam

We compared student performance on the retention exam between those taking the stats II between the traditional and revised

The 2011-12 retention exam was reduced to 12 MC, the two-sample t-test and SLR word problems remained the same.

Data from the Fall (2011), Spring (2012) and Fall 2012 retention exams were combined

Methods

Curriculum n Score% MC % Open %

Traditional 187 43.2 54.0 33.9

Revised 123 59.8 65.0 55.9

Difference 16.6 11 22

p-value <0.0001 <0.0001 <0.0001

Results

Still collecting and analyzing MC

◦ Not much change◦ Maybe in identifying the use of median when data

are skewed: from 25% 38◦ Identifying stat sig slope from Excel output

from 50% 57% Short answer

◦ Still working on that but◦ very few inconsistent conclusion in 2 sample ttest◦ fewer stating not sig predictor b/c R^2 is low

Changes in specific questions

Regression analysis including semester lag (lag = 1-6 semesters) between stats II and retention exam

Pred Score = 0.668 – 0.192(Curr) – 0.02(Sem)

Adj R2 = 20.0%

Accounting for Semester Lag

1. Establish the assessment plan Define learning goals (6, linked to mission stmt)

identify outcome methods implement methods

2. Implement & manage the assessment plan

Methods Results analysis curr changes

3. Periodically update the assessment plan

Current Plan Diagram