Sloan bl workshop04192010final

For more information:“Next Generation Course Redesign” in the Nov/Dec issue of Change

Next Generation Course Redesign (2010) Peter Lang Publishing

“The Promise of Blended Learning,” AA&CU News

Goals for todayShare why we thought that

it was important to have a serious look at how we teach undergraduates at UNT

Provide a brief overview of the blended learning approach

Describe how the Next Generation Course Redesign™ Project works at UNT

Demonstrate two NGen courses

Challenge you to think about the optimum blend of pedagogies for an instructional unit

Why use a blend of pedagogies in the redesign?

The “Perfect Storm”Bad News

High DFW rates Demographics (higher and more diverse

enrollments)Financial (tuition cannot keep exceeding CPI)Accountability

Good NewsKnowledge of learningEmergence of digital tools

What Do We Know About College Students and How They Learn?

Brain ResearchThe role of neural networksEvery network can be traced back to an experiential

referentOur students have “grown up digital”

Comfortable in a multimedia information-ubiquitous environment

Social networking is second natureCognitive development is an important goal

Most entering freshmen are at Kegan’s 3rd stageCollege experience must encourage a “discipline of

inquiry”

We know that, if we provide an active learning experience that allows students to engage with the content, each other, and instructors, they can and will think critically and develop cognitively.

Goals of the UNT NGen ProjectTo improve student learning outcomes in

large enrollment undergraduate coursesTo have a university-wide impact through the

establishment of a Community of Practice and the creation of redesign that is both sustainable and replicable

(Translation) Goals:Students think, work hard, like what they are

doing, get good grades that mean something, and graduate

Doesn’t cost more and uses less spaceFaculty have fun

UNT’s Transformational QEP Goal

Next Generation Course Redesign™ is a Process…Faculty teams redesign 4-6 courses each year

(two-year commitment)“Choreographed” Process

RetreatsMonthly meetings with teams of faculty and

staffInstitution-wide forumsEnd-of-pilot meetingsEnd-of-project meetings

Next Generation Course Redesign™The redesign

occurs within an interdisciplinary Community of Practice

NGen Faculty Fellows have the option of being designated Senior Faculty Fellows

Creating a Culture of Reflection

Promulgate the philosophy of “never ending redesign”

Schedule small group meetings with the majority of time devoted to reflection and discussion

Provide opportunities for ongoing virtual discussion

Promote assessment-driven change

Next Generation Course Redesign™ also has products. NGen Courses have:Clearly articulated

student learning outcomes that include outcomes representing higher level learning.

Test items that match the student learning outcomes.

An assessment plan that shows how the test items map to the outcomes.

Common Assessment Across All NGen CoursesPreference for Course

Format Survey (NGen vs. Traditional)

Attitude Toward Subject (pre/post)

LEP Survey of Cognitive Development (pre/post)

Raw score distributions

(The Blend) NGen Courses have:Large group

lectures (0-30% of contact hours)

Small group experiential learning (30-60% of contact hours)

Media-rich interactive online environment (30-50% of contact hours) These are the “building blocks” for an

NGen course.

In NGen, Lectures are best used to:Create interest and motivation and provide

assurance that the students can be successfulClarify and expand upon, rather than deliver,

contentModel the acquisition of knowledge in the

field, e.g., “How does a historian/chemist/sociologist approach a research question?

Present the critical lower level concepts to provide scaffolding for higher level concepts

In NGen, Experiential Learning Activities are best used to:Introduce an emotional component (brain-

based learning)Analyze, evaluate, and synthesizePresent and defend newly-acquired

hypothesesProvide collaborative, cooperative, and

academic controversy activities that encourage thinking critically from multiple perspectives

Dilemma!If I use experiential learning in my class, how

am I going to “cover” all of the material?

In NGen, Online Learning Activities are best use to:Acquire lower level learning to free up time

for in-class experiential learningChunk content to overcome working memory

limitsProvide low-stakes assessments such as

quizzes for practice and confidence buildingProvide psychomotor experiencesProvide concrete experiences that are guided

and efficient

Next Generation Course Redesign™UNT offers 19 NGen Courses

Art History, Biology I, Computer Applications, U.S. History I & II, Human Development, World Literature, Principles of Language Study, Business Communications, Occupational Health, Music Appreciation, Developmental Math/College Algebra, Intro to Communications, Modernism and the Visual Arts, Organic Chemistry, American Government I, World Literature Since the Renaissance, and Sociology of Disasters

With five more in development (Motor Development, Survey of Mathematics, Global Marketing Concepts, American Government II, and Individuals and Society)

RFP has been issued to select up to eight courses to start the redesign process in the fall

Nicole Dash, PhDAssociate Professor, SociologyDirector, UNT Core Academy

Why Redesign? Large Enrollment No Textbooks 90% Lecture Students struggling to connect what they

learned with real life situations

Redesigned Model 10% Lecture: First two class periods

only 50% Online: 9 Online Modules 10% Online Activities and Projects 30% Small Group Experiential

Learning Activities

What the class looks like to a student:

Lecture:First two face to face classes only

Introduces how the course will runDiscusses the schedule and who attends class

whenFocuses on explaining online components Introduces subject

Disaster photos and impact discussions Basic definitions More complicated theoretical issues

Online 9 online modulesExamsDiscussionsGroup Sign UpsWiki Project

Experiential LearningSmall GroupsUse information learned online

Develop a Warning MessageStratified Monopoly/Disaster Scenario ExerciseMock City Commission Meeting

Back to Monopoly… How does the Monopoly game relate to SLOs?

Assesses two SLOs: 6.2 Investigate how social structures (such as race,

class, and gender) generates disaster vulnerability and privilege

6.5 Assess your own social position and resulting vulnerability and privilege

Ties the results of the Monopoly game to a disaster scenario

Requires students to step out of their own shoes first, and then apply that to their own lives

Preliminary FindingsStarting to Analyze Data:

Majority preferred Next-Gen Model to traditional model 57% Overall preferred Next-Gen to F2F 61% of Successful Students preferred Next Gen 64% of Unsuccessful Students preferred F2F All students who attended class and engaged in

activities passed the classUnsuccessful students tended to consistently

miss projects and assignments

Next Generation Course Redesign™: Example

ENGL 2210 World Literature I

Dr. Tracey Gau

NGen Senior Faculty Fellow

QEP: The N-Gen Course Model

A rigorous outcome-based assessment plan

Ongoing redesign based on assessment results

A specific mix of instructional strategies, including:Small-group Experiential ActivitiesMedia rich, Interactive Online ExercisesLarge-group Lectures

A Sample N-Gen Course: World Literature

Course Assessment Blueprint

Outcomes based assessment plan

Coordinates SLOs, instruction, and assessment items

Ensures that what is measured is what is valued

Shows types of assessment instruments used to measure each outcome

©2008 T.M.Gau Dr. Tracey Gau Assessment Plan for N-Gen ENGL2210: World Literature I

Test Items and Difficulty Level

Domain Course Learning Goals(Institutional) General Student Learning Outcomes

(Departmental)Specific Lesson Learning Outcomes

(Classroom)

Low

Med

Hi Tot

CR

World Literature -- Ancient through Renais-sance

1. Demon-strate an awareness and recogni-tion of the scope and variety of works of literature

1.1 Recall and recognize the historical sequence of major literary figures, texts, and movements within the Ancient, Middle, and Renaissance periods

1.1.1 Recognize that the history of representative epics are oral or written compositions

6 2 8

1.2 Identify conventional literary genres, elements, and devices

1.2.1 Identify major characters and figures that appear in more than one text

6 2 8

1.3 Employ discipline specific vocabulary in order to recognize the relationship between form and content

1.3.1 Apply genre characteristics to representative texts

4 4 8 1

1.4 Relate literary or cultural concepts, principles, terms, strategies, and styles to a range of literature

1.4.1 Identify and connect literary or cultural concepts as they directly relate to representative texts

5 3 8

World Literature Test Plan (Blueprint)

All course materials © 2009, Dr. T.M. Gau


Test Items and Difficulty Level

Domain Course Learning Goals

(Institutional)General Student Learning Outcomes

(Departmental)Specific Lesson Learning Outcomes

(Classroom)

Low

Med

Hi Tot

CR


2. Read critically and analytically

2.1 Analyze, evaluate, interpret, synthesize representative texts from the Eastern and Western traditions and relate them to their literary and cultural contexts

2.1.1 Relate cultural qualities of a hero to a representative character

3 2 2 7 1

2.2 Compare and contrast major literary figures, their situations, decisions

2.2.1 Compare the Eastern depiction of fate, virtue, and heroism to the Western depiction in representative texts

2 3 2 7 1

2.3 Connect various periods, texts, authors, and characters

2.3.1 Differentiate between concepts, such as absolutism and relativism, using examples from representative texts

3 2 2 7 1

2.4 Evaluate the ideas presented in a text, their implications, and their relationship to ideas beyond the text

2.4.1 Evaluate characters’ decisions and actions in the context of their various cultures and worldviews

2 2 3 7 1

World Literature Test Plan cont’d (Goal 2)


Test Items and Difficulty LevelDomain Course Learning Goals

(Institutional) General Student Learning Outcomes(Departmental)

Specific Lesson Learning Outcomes

(Classroom)

Low

Med

Hi

Tot

CR


3. Construct informed, organized and coherent written responses to literary texts

3.1 Formulate a central interpretive idea about the texts;3.2 Develop ideas logically and coherently with adequate supporting textual examples;3.3 Present ideas clearly and concisely;3.4 Observe the standard conventions of formatting, citation, grammar, and punctuation

Written Composition(See Constructed Response

Rubric)

4. Actively discuss ideas with others

4.1 Present (in discussion posts or small groups) information or conclusions that help other students summarize, synthesize, and integrate the conceptual material

4.2 Establish a relevancy between literature and how it affects one’s life, personal values, ethical behavior, aesthetic judgment, and problem solving

4.2.1 Relate readings to life and academic experiences4.2.2 Make interdisciplinary connections, such as with history, philosophy, art, rhetoric, business, politics

TOTAL EXAM ITEMS 31

20

9 60

5

World Literature Test Plan continued (Goals 3 and 4)

Structure of N-Gen N-Gen World LitWorld Lit


SLOs begin every lesson

Cognitive Categoriesand

Difficulty Levels

Low: Literal and Factual Knowledge• Recall, identify, understand

Med: Conceptual and Procedural Knowledge• Interpretive and Inferential• Apply, analyze, interpret

High: Meta-cognitive Knowledge• Evaluate, construct, create


Formative Assessment:Online Activities


Formative Assessment:Tools of Mastery


Formative Assessment:Games of MasteryAll course materials © 2009, Dr. T.M. Gau

Formative Assessment:Self Tests


Interaction and Collaboration: Debate


Summative Assessment: Constructed Responses




Best Practices: Assessments Assignments that encourage students to employ critical

thinking strategies

Rubrics or performance criteria that are made available to the students

Scaffolding activities that promote higher cognitive development

Providing students with ample opportunities for self assessment

All course materials © 2009, Dr. T.M. Gau 48

Summative Assessment:Multiple Choice Items

Offer high degree of reliability & validityVary in difficulty level:

Low (literal and factual) Medium (interpretive and analytical) High (metacognitive)

Correspond to Assessment Plan and SLOs Course Goal General SLO Specific SLO

Use Item Analysis to improve test items Conform to item-writing guidelines

Match test items to cognitive level of Learning Outcome Use Prompts to Address High(er)-Level Cognitive Category Use Specific Items Shells in Writing Multiple Choice Item Sets


SLO 2.1 According to Machiavelli, what two animals must a ruler imitate and what meaning does the analogy represent? A. Fox and Tiger; in addition to being cunning, a ruler must protect himself from traps. B. Fox and Cougar, in addition to being cunning, a ruler must be able to act alone. C. Fox and Lion; in addition to being cunning, a ruler must use force as well as laws to rule.

Rate The Item

(LOW .93/.90)

1. Low2. Medium3. High

SLO 2.1 Which piece of evidence that Iago presents to Othello has the GREATEST effect of conjuring up doubt about Desdemona’s fidelity? A. Cassio’s erotic dream B. Cassio’s drunken brawl with Roderigo C. The handkerchief he sees Cassio give to Bianca


Rate The Item

(MEDIUM .93/.90)

SLO 2.1 What is the BEST interpretation of Iago’s advice to Roderigo to “put money in thy purse”? A. Invest your time, money, and effort in joining Iago in revenge against Othello B. The marriage between Desdemona and Othello is frail and can easily be broken C. Because of woman’s changeable nature, Desdemona will tire of Othello and seek a new lover


Rate The Item

(HIGH .63/.64)

SLO 2.1 Which one of the following is the MOST COMPLETE description of what Iago represents? A. Vice Figure B. Communal Evil C. Parasitic Flatterer


Rate The Item

(MEDIUM .89/.83)

Item Analysis: Why it’s necessaryto validate m/c items

Item analysis of summative selected-response (MC) and constructed (written) response tests due at end of each semester

Flawed multiple-choice items are up to 15 percentage points more difficult than standard items (items that conform to the standard principles) when testing the same content,

with median passing rates that are about 3.5 percentage points lower for flawed items as compared to standard items

(Downing, 2005).

Item Prop. Disc. Point Prop. Endorsing PointNo. Correct Index Biser. Alt. Total Low High Biser. 1 .61 .51 .42 A .17 .30 .05 -.27

B .08 .15 .02 -.19 C .61 .35 .86 .42 * D .14 .19 .06 -.14 2 .21 .18 .18 A .07 .09 .05 -.08 B .22 .26 .18 -.09 C .49 .49 .45 -.02 D .21 .14 .32 .18 * 4 .73 .41 .39 A .73 .50 .91 .39 * B .02 .06 .00 -.16 C .12 .23 . 04 -.24 D .13 .21 .05 -.19 5 .82 .38 .43 A .02 .05 .01 -.15 B .82 .59 .96 .43 * C .05 .09 .01 -.16 D .10 .26 .02 -.33 6 18 .05 .07 A .18 .15 .20 .07 * B .10 .18 .04 -.22 CHECK A was specified C .41 .45 .33 -.08 D works better D .31 .20 .43 .19

CONSTRUCTED RESPONSE STUDENT SCORES

St ClarityStructure Logic

ReferEvid Sub tot

Grammar Sent Struct Dev sub tot

1 3 2 2 1 8 0 3 2 72 3 3 3 3 12 2 2 2 63 4 3 2 3 12 3 2 3 84 3 3 2 2 10 2 3 2 75 2 2 1 1 6 1 1 2 46 2 2 3 3 10 2 2 2 67 4 3 3 3 13 3 2 3 88 4 4 4 3 15 3 3 3 99 3 4 2 2 11 3 3 2 8

…..…..38 2 3 2 2 9 2 3 2 739 1 2 1 1 5 1 1 2 440 2 3 3 3 11 2 2 2 6

M 2.65 2.75 2.28 2.38 10.05 2.13 2.23 2.2 6.60Mean for content = 10 = 63% Mean for conventions = 6.6 = 55%

Do the average scores for each element make sense in terms of the intent of the outcomes they are measuring? (validity evidence)

Do the student responses appear to reflect what was taught in terms of the outcomes they are measuring? (validity evidence)

Did the prompt and the question seem to work? That is, did it produce a high degree of response?

Assessment System Model

Instruction Assessment Course Design Test Development

VALIDITY EVIDENCE

Learning Goals and Outcomes

Scores that are valid for theirintended interpretation

and use

SURVEY 1: Survey of Student Attitude Toward Subject of the

Course Survey designed to gather information on how the student feels about the subject

matter of the course the student is beginning, finishing, or has just finished

Administered Pre and Post

21 items on a five point scale that are specific to the “subject”- not teacher effectiveness- not course effectiveness

Inference = improved attitude correlates with N-Gen format and course redesign

13 items positive

8 items negative .

Specific Results for World Lit:Student Attitude Toward Subject

Items that increased:+.190 This subject is worth knowing+.254 Knowing this subject makes me more employable+.638 I know a lot about this subject+.317 This subject is useful to my everyday life

Items that decreased:-.079 This is a difficult subject for me-.095 Learning this subject requires a lot of hard work-.079 This subject is difficult to understand-.143 This is a complicated subject-.032 I will have no application of this subject in my profession-.032 I am scared by this subject


SURVEY 2: Preference for Course Format:

N-Gen or FTF

This survey consists of one survey question with two answer choices (N-Gen and FTF) with a short written response in which the student explains the reason(s) for his/her choice.

Instructor records an S (Successful = A,B,C) or U (Unsuccessful = D,F,W,I) after end of course. S and U can be typed into the online student- response data file at end of the course.

Two weeks prior to final exam, spring semester year

Example of student preference for course format (Ngen vs FTF)

If you were to start this course over again, would you prefer a traditional face-to-face format, or would you prefer the N-Gen redesign format you are experiencing? Please tell why.

CoursN=282

PrefN-gen

PrefFTF

TotalNumb Succ

Total Numb Un-succ

Succprefer N-Gen

Succprefer FTF

Un-succs prefer N-Gen

Un-succ prefer FTF

Count 172 110 201 81 149 52 51 30

% 61% 39% 71% 29% 74% 37% 63% 37%

Categories of ResponsesFormat Category Typical CommentN-Gen Pace I liked that I could do most of the work at my own pace when I had time to do it. If I wanted

to go a little slower, I could.

Flexibility

This course allowed me to work out my hectic schedule. Being able to submit quizzes and stuff online made my life easier.

Learning

I like to learn from a bunch of different sources at once and this course really allowed me the chance to do that. You got stuff from online sources and you got some face to face interaction and I think I ended up learning more in this course than I would have otherwise.

Practice I always liked the fact that you could go back and take the quizzes over and over again until you got them right. I really feel like that helped me a lot in the class.

FTF Manage I prefer a traditional face-to-face lecture because I would often forget about online assignments and I think my grade suffered. I need more structure in my courses so, I need to come to campus more often!

Learning I seem to absorb the information better in a traditional class format when I'm taking notes during a lecture. It just suits my learning style better.

People I can't seem to learn without a teacher lecturing to me and me taking notes. Maybe it something about the interaction in the communication. I guess I just like being around people.

Technical

I don't trust submitting my quizzes online. The internet didn't seem to be reliable. Sometimes the website would be slow or wouldn't work at all. SO frustrating!

Final Score Distribution

To allow a comparison of means from year to year and pre N-Gen to N-Gen

Due 1 week after end of each semester year 1 & 2

Test for statistical significance

Success Rates: World Literature I

70%63%

69% 68% 65% 68% 71%

84%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

ENGL 2210 Large Enrollment ClassesSuccess Percentages Across Semesters

Fall 04 Spg 05 Fall 05 Spg 06 Fall 06 Spg 07 Fall 07 Spg 08


Departmental AdvantagesAddresses problems of course drift, inconsistent learning

experiences, and inefficient use of faculty/course deliveryIncreases the quality of the department’s offeringsProvides a basic framework that meets university,

departmental, and course objectivesOffers a pedagogical training for new instructorsImplements best practices of both face-to-face and online

teaching and learningPromotes higher-level learning in introductory coursesMeets assessment standards of reliability and validityAllocates more time for faculty to work on other projects

All course materials © 2009 Dr. T.M. Gau

Vision and Objectives after RedesignSustainability and ReplicationOther UNT instructors teaching the courseAdoption of courseware by outside universities

1. World Literature I: Ancient through Renaissance

2. World Literature II: Since the Renaissance

Designer: Lead touchpoint resource Provides training to instructors Updates and maintains course Responds to feedback for course improvements

Cost: $90 per student (course fees)

All course materials © 2009 Dr. T.M. Gau

AssignmentSee handout with goals and SLO’s

Select goal and break into groups of 4-6Brainstorm combination of the three

pedagogies to reach the SLO’sReport

Discussion

Dr. Niki [email protected]

Dr. Brenda [email protected]

Dr. Tracey [email protected]

Dr. Phil [email protected]

For more information:Nov/Dec issue of Change

Next Generation Course Redesign™ Forthcoming from Peter Lang Publishing

Email to join Course Redesign NING

Date post:	09-Dec-2014
Category:	Education
Upload:	university-of-miami
View:	424 times
Download:	1 times

Sloan bl workshop04192010final

Education