Developing a Comprehensive Learning Outcomes Assessment Strategy for

Adult Learners

Cindy MacMillan& Susan Davenport

2

Agenda

• Introductions

• Characteristics of Thomas Edison State College

• Challenges in an adult learning environment

3

Introductions

• Susan Davenport, Dean of Heavin School of Arts and Sciences

• Cynthia MacMillan, Director of Outcomes Assessment and Accreditation

• Audience– Who’s here?– Where you are from?– Why this session?

4

Why Learning Assessment?

• Is it really all about accountability?

• Accreditors• Students• Mission• You

5

What are the challenges assessing learning in an adult environment?

• Heterogeneous population with different needs and skills

• Significant learning that the student brings to the learning environment

• Access to students for assessment

6

Where to Start• Phase One – Define Learning Outcomes at Institutional and

Programmatic Levels

• Phase Two-- Operationalize the learning outcomes

• Phase Three – Implementation of assessment (finalize assessment matrix)

• Phase Four – Data collection and Analysis

• Phase Five -- Recommendations and transformations made in response to Phase Four data

7

Where to start! (cont.)

8

Phase OneDefine Learning Outcomes at Institutional and

Programmatic Levels

• Buy In• Strength in diversity• Define and Describe Goals and Process

– Institutional, Programmatic, Course

9

Thomas Edison State College Institutional Outcomes

1. Information LiteracyThe Information literate graduate will demonstrate the ability required to recognize when information is needed and has the ability to identify, locate, evaluate and use effectively and appropriately the information needed.

2. Technological CompetencyGraduates will be able to solve problems and perform tasks using appropriate technological tools.

3. Responsible Ethical LeadershipGraduates will be able to articulate in their own words the ethical principles they have discovered and encountered in their academic discipline or profession. Graduates will recognize and articulate actual ethical dilemmas witnessed or experienced.

4. Scientific Reasoning Thomas Edison State College graduates will be able to analyze issues facing citizens by distinguishing scientific approaches from other ways to frame problems and issues.

5. Diversity/Global LiteracyThe Thomas Edison State College graduate will demonstrate an understanding of globally diverse perspectives, cultures, values, events, trends and issues.

10

Thomas Edison State College Institutional Outcomes (contd.)

6. Quantitative Reasoning/LiteracyThomas Edison State College graduates will be quantitatively literate. They will be able to understand and apply mathematical concepts and skills to inform personal and professional decision-making.

7. Written and Oral CommunicationThomas Edison State College Graduates will be able to write and speak with proficiency.

8. Critical Analysis & ReasoningThomas Edison State College graduates will be able to demonstrate critical thinking and problem solving skills and are able to analyze content, discover meaning or significance, draw conclusions, and make assessments.

9. Life Long LearningGraduates will demonstrate a commitment to lifelong learning.

11

Phase TwoOperationalize the Learning Outcomes

• Indirect versus direct measures

• Quantitative versus qualitative measures

• Triangulation

• Low versus High Stakes Assessment

• Baselines and Benchmarks

12

Phase Two Continued

Methods we are using – • Standardized Assessment – ETS Proficiency

Profile, SAILS• Graduate Survey, Adult Learner Inventory• Capstones• Embedded Analysis• Historical Data Review• Survey• ETS Major Field Test• E-Portfolios

13

Example: Assessment Matrix for Program Level Learning Outcomes

Bachelor of Science in Human Services outcomes:

Program Element which addresses this outcome

Measures (indicate direct or indirect) Benchmark/Target(acceptable and ideal if appropriate)

Interpret and critically analyze the research in the professional track

Course in Professional track

final project/paper A 70% earn a grade of B+I 80% earn a grade of B+ 

SOS 492 Research Methods

final research paper A 70% earn grade of B+ I 80% earn grade of B+ 

  ETS Proficiency Profile

Std. Test Critical Thinking A 60% achieve proficient or marginally proficientI 80% achieve proficient or marginally proficient

Apply theory to professional practice Capstone final paper A 70% earn grade of B+I 80% earn grade of B+

Apply knowledge of the specific skills, techniques, and agencies necessary to serve client populations

PSY 331 Intro Counseling

final exam A 70% earn grade of B+I 80% earn grade of B+

Capstone final paper A 70% earn grade of B+I 80% earn grade of B+

  Graduate Survey Item 7-a A 80% respond positivelyI 90% respond positively

Apply knowledge of cultural diversity as it relates to the field of Human Services. 

SOC 322 Cultural Diversity in the US

final exam A 70% earn grade of B+I 80% earn grade of B+

Capstone final paper A 70% earn grade of B+I 80% earn grade of B+

Apply theories of management as it relates to Human Services

MAN 301 Principles of Management

final exam A 70% earn grade of B+I 80% earn grade of B+

Capstone final paper A 70% earn grade of B+I 80% earn grade of B+

14

Phase Three

Implementation of Assessment

• Be realistic in scope• Not everything needs to be assessed each year• Be pragmatic in tying plan to funds• Benchmarking

– Internal– External

• Rubric elements•

Information Literacy Data Collection Sample

15

Example: Data Collection for Human Services Degree

16

Bachelor of Science in Human Services outcomes:

Benchmark/Target

2010 results

Interpret and critically analyze the research in the professional track

70% earn a score of 3 or above on a 4 point scale on rubric by reviewed independent consultant 

Above target

Apply theory to professional practice 70% earn a score of 3 or above on a 4 point scale on rubric by reviewed independent consultant 

Below target

Apply knowledge of the specific skills, techniques, and agencies necessary to serve client populations

70% earn a score of 3 or above on a 4 point scale on rubric by reviewed independent consultant 

Below target

Apply knowledge of cultural diversity as it relates to the field of Human Services. 

70% earn a score of 3 or above on a 4 point scale on rubric by reviewed independent consultant 

Below target

Apply theories of management as it relates to Human Services

70% earn a score of 3 or above on a 4 point scale on rubric by reviewed independent consultant 

Above target

Phase Four and Five– Analysis and recommendations

17

Information Literacy - SAILS Recommendations

• Continue to evaluate student learning outcomes in the area of Information Literacy using the SAILS.

• Embedded SAILS in an upper level course for general populations.

• Identify and implement additional methods to assess the outcome area not assessed by SAILS.

• Include the SAILS in a few more courses to ensure that all degree programs of the College are included in this informational literacy assessment.

• Target the SAILS administrations to 250-275 students per year in order to facilitate longitudinal analysis in the future.

18

Information Literacy – ENC-102 Recommendations

• Expand the rubric elements to include one on grammar and spelling.

• Expand the data collection efforts in ENC-102 to include one section per semester.

• Expand the roll of the lead mentor to include collecting reliability data.

• Expand the use of Waypoint in other schools, courses and assignments.

• Explore viable options to communicate details of Waypoint use with mentors.

• Explore ways to minimize the use of instructional design resources.

• Seek efficiencies such as sharing rubric elements across schools.

19

20

Human Services – Analysis and recommendations

Analysis of Human Services results led to the following curricular changes:

1. Modify the program description to more accurately describe the program and its objectives.

2. Modify the program outcomes to align with objectives in program by including diversity and theories of management.

3. Modify the general education requirements to 60 credits to align with the other degree offerings within the College. Require a three credit Ethics course under Humanities.

21

Phase Four and Five– Analysis and recommendations (contd.)

4. Modify the Core Requirements. Based on guidelines from national accrediting bodies (i.e. The National Human Services Organization and the Council for Standards in Human Services Education), the core requirements were replaced with the following areas: Theoretical Foundation, Intervention, Client Populations, Research and Mgmt. This improves the quality of student learning outcomes and ensures that the program objectives are appropriate and measurable.

5. Modify the Practicum to a capstone course and objectives to make the practicum an effective instrument for assessing the degree outcomes.

6. Change the grading of the capstone from Credit/No Credit to a letter

grade. This will ensure a more accurate assessment of the outcomes of the course. A benchmark of “B+” was recommended.

22

Your best practices, lesson learned and questions

Please share your feedback . . .

•

•

•

Thank you for your time and interest!