Department of Psychology & Human Ecology School of Education and Behavioral Sciences Masters of...

Department of Psychology & Human Ecology

School of Education and Behavioral Sciences

Masters of Science in Behavioral Sciences

421601610

1Program Quality Improvement Report 2009-2010

MSBS Program Objectives & Student-Learning Objectives

1. To foster the ability to synthesize and critically analyze information.

1. Students will synthesize and analyze information critically.

2. To promote research skills in the behavioral sciences.2. Students will demonstrate basic research skills in the

behavioral sciences.3. To promote basic knowledge of theories, methods, and

findings in the behavioral sciences.3. Students will demonstrate knowledge of theories,

methods, and findings in the behavioral sciences.4. To promote competence in the application of basic

knowledge within the field of the behavioral sciences.4. Students will apply basic knowledge within the field

of the behavioral sciences.


Holistic Alignment of Program Objectives to University, School, Departmental, & Program

MissionThe MSBS Program is dedicated to providing a

quality education for a diverse and dynamic student body. As a consequence, our program is designed to “serve that segment of the population employed in or interested in a broad spectrum of professions in the behavioral sciences.” Our students are educated to be … and have proven to be a dynamic force in the behavioral sciences in our community.

Program Quality Improvement Report 2009-2010 3


MissionThe MSBS Program is committed to fostering a student-

centered academic environment that combines innovative classroom teaching with experiential learning. We have listened to our students and have restructured our program to meet their career and educational goals. We have provided them an education anchored by dynamic professionals actively working in the field of the behavioral sciences and have collaborated with the behavioral sciences community to ensure an experiential learning environment.



MissionThe MSBS Program is dedicated to preparing

students for professional success, responsible citizenship, life-long learning, and meaningful contributions to a rapidly changing world. We have structured our program to prepare students for successful professional licensure in the behavioral sciences with a strong ethical underpinning which will ensure significant and dynamic contributions to a rapidly changing community.



MissionWith a strong emphasis on the study of

psychological cultural diversity and a zeal for a strong work ethic, the MSBS Program is a driving force in the cultural life and economic development of our community. Focused on promoting theoretical & methodological cultural competencies, our program challenges our students to be a driving force for compassionate care in our community.


Measures of Learning Outcomes

1. Direct Measures:a. Locally developed tests

i. Within course testsii. End of program comprehensive examination

b. Literature reviews scored by rubrics2. Indirect Measures:

a. Graduates’ exit interviewsb. Employer/supervisor evaluation of practicum

experience.


Measures of Learning and Service Outcomes

3. Strategies for addressing shortfalls in student learning prior to graduation:

a. A Student Data Base has been created:

i. Demographic Informationii. Undergraduate performanceiii. Educational/Career goalsiv. Recruitment & Retention

Informationv. Advisement Informationvi. Academic Tracking Information


Measures of Learning and Service Outcomes

b. Using the Student Data Base to facilitate identification of shortfalls, online learning modules will be developed in order to help student remediation.

i. Synthesis & Analysisii. Research Skillsiii. Theories, methods & findingsiv. Applications


ACTION PLAN 09-10 PQIRObjective 1: Students will synthesize and

analyze information critically1. Create embedded questions in Research Methods related

to synthesis and critical analysis.A. Develop validity and reliability for embedded questions.B. Action Taken: ABANDONED

2. Create rubric to assess synthesis and critical analysis in Research Methods final/applied project.A. Develop validity and reliability for the rubric.B. Action Taken: MODIFICATION: Create a rubric to assess

synthesis and critical analysis in within the first 18 credit hours and then again in the last 18 credit hours.



analyze information critically

3. Assess synthesis and critical analysis in first course post 21 credit hours.

A. Action Taken: 2.B.

4. Continue to indirectly assess synthesis and critical analysis in the exit interview process.

A. Action Taken: Exit interview data collection on synthesis and critical analysis continues.








ACTION PLAN 09-10 PQIR Obj. 2: Students will demonstrate basic research skills in the behavioral sciences.

1. Operationally Define Research Skills.A. Action Taken: Operational Definition of Research Skills.

1. Students will demonstrate knowledge in research ethics.

2. Students will analyze and synthesize information in the existing professional literature.

3. Students will demonstrate the ability to form a logical and testable hypothesis.



1. Operationally Define Research Skills.A. Action Taken: Operational Definition of Research

Skills4. Students will demonstrate the ability to develop

research designs appropriate to test the hypothesis.5. Students will demonstrate knowledge of statistics.

a. Students will demonstrate the ability to select the appropriate statistics for data analysis.

b. Students will demonstrate the ability to analyze data and interpret results.

6. Students will demonstrate the ability to present findings.



2. Develop embedded questions in Research Methods, Ethics and Statistics in order to directly measure the students knowledge of research skills.

1. Develop reliability and validity for embedded questions.

2. Action Taken: Embedded questions within three main courses were developed in Spring 2010 and data collection will begin Fall 2010.

1. Action Taken: Question were disseminated to topic matter experts to determine Content Validity. Experts were asked to sort questions by subject content. Crombach Alpha coefficient were performed and the results indicate a r = .92



3. Develop a rubric to evaluate the students ability to apply Research Skills in an end of course project.

A. Develop reliability and validity for applied Research Skills rubric.

4. Action Taken: Initial development of the rubric began Spring 2010. Developmental work continues.

A. Action Taken: Pending finalization of rubric.



4. Assess research skills in the behavioral sciences in first course post 21 credit hours.

A. Action Take: AMENDENEDAssess research skills in the behavioral sciences

within the first 18 credit hours. New programmatic changes have been submitted for Regent consideration.

5. Continue to indirectly assess synthesis and critical analysis in the exit interview process.

A. Action Taken: Continues to be assessed.Program Quality Improvement Report

2009-2010 18

Learning Objective 1Outcome Data

Students will synthesis and analyze information critically.

1. Rubric Measure - Direct Measure2. Exit Interview Questions pertaining to synthesis and

analysis – Indirect Measure.


Learning Objective 1: Students will synthesize and analyze information critically

PROGRAM OUTCOME

CURRICULUM AREA OR TARGET AUDIENCE

MEASUREMENTS OF STUDENT LEARNING

Measurements

Methods used to determine validity of measurement instruments

Methods used to determine reliability of measurements

Schedule for measurements

Learning Objective 1

Career & Lifestyle Development

Locally developed rubric for synthesis & critical analysis.

Criterion Validity was determined by requesting topic matter experts perform a card sort exercise by categories and descriptions . r = .89

Interater Reliability was performed.r = .68

Every semester so every student will be measured.


Assignment

• With the exception of Graduate Statistics, every graduate course has literary assignment meant to test the students’ ability to synthesize and analyze information. These assignments all include a review of the professional literature with an emphases on the students’ ability to pull information from different sources to create a cohesive analysis of the information.


Learning Obj. 1 – Direct MeasureRubric Outcome

CATEGORIES:1. Introduction & Summary2. Thesis3. Organization4. Supporting Information5. Evidence & Examples6. Paragraphs7. Transitions8. Conclusion9. Mechanics



1 = Below Standard2 = Approaching Standard

3 = Meets Standard4 = Above Standard

has not met programmatic expectations

Spring 2010 – Synthesis & Critical Analysis RubricMean & Standard Deviations

n = 23Intro & Summ

Thesis Organ Supp Info

Evid & Examp

Parag Trans Concl Mech Total

1.87 2.17 2.26 3.57 2.43 1.83 1.74 1.87 1.61 2.15

0.81 0.83 0.86 0.51 0.66 0.83 0.69 0.76 0.58 0.24



1. Trend Analysis:

First semester to use this measure. No trend analysis.

2. Action Plan:A. Continue to collect Interrater Analysis data.

a. Timeline: Spring 2011



2. Action Plan:B. Intervention to improve performance:

a. Determine the source of the problem. Is it a poor measure, poor instructor preparation, or poor student performance?

b. Give students the rubric with syllabus.i. Timeline: Each semester

c. Develop an online module to aid in understanding and usage of the dimensions of Synthesis & Critical Analysis.i. Timeline: Start development Spg 2011; Finished by Spg

2012


Learning Objective 1: Students will synthesize and analyze information critically

PROGRAM OUTCOME


MEASUREMENTS OF STUDENT LEARNING OR SERVICE OUTCOME

Measurements





Graduating students

Graduate Exit Interview (indirect)

No validity measure available.

No reliability measure available.

Every semester so every student will be measured.


Learning Objective 1: Indirect MeasureExit Interview Results 2009-2010

1. Please rate how well the program prepared you to synthesize information on a scale of 1 (low) to 7 (high)?a. n = 38b. Mean = 5.72c. Standard Deviation = 0.92

2. Please rat how well the program prepared you to critically analyze information on a scale from 1 (low) to 7 (high)?a. N = 38b. Mean = 5.78c. Standard Deviation = 1.01


Learning Obj. 1: Indirect MeasureExit Interview


Learning Obj. 1: Indirect MeasureExit Interview

Action Plan:1. Continue to collect data on the Exit Interview.2. It is expected that the corrective plan for

improving performance on the Rubric measure will improve on Exit Interview reporting.


Learning Objective 2Outcome Data

Students will demonstrate basic research skills in the behavioral sciences.1. Direct Measure: Research Proposal graded by Rubric. 2. Indirect Measure: Exit Interview Questions pertaining to attainment of research skills in the behavioral sciences.


Learning Objective 2: Students will demonstrate research skills in the behavioral sciences.

Rubric Outcome – Direct Measure

PROGRAM OUTCOME



Measurements




Learning Objective

Ethics & Professional Development

Graduate Statistics

Research Methods

Locally developed rubric for research skills will be used to determine how well a student has demonstrated research skills in a research proposal assignment.

Criterion Validity was determined by requesting topic matter experts perform a card sort exercise by categories and descriptions . r = .92

None available Every semester so every student will be measured.


Learning Obj. 2 – Direct MeasureRubric Measure

1. Students will demonstrate knowledge in research ethics.

2. Students will analyze and synthesize information in the existing professional literature.

3. Students will demonstrate the ability to form a logical and testable hypothesis.



4. Students will demonstrate the ability to develop research designs appropriate to test the hypothesis.

5. Students will demonstrate knowledge of statistics.a. Students will demonstrate the ability to select the

appropriate statistics for data analysis.b. Students will demonstrate the ability to analyze data

and interpret results.

6. Students will demonstrate the ability to present findings.



1 = Below Standard3 = Approaching Standard

4 = Meets Standard5 = Above Standard

Does not meet standard

Research SkillsMeansn = 10

Ethics RevLiture

Hypo Design ApproStat

InterResult

Present Total

3.70 4.24 3.15 3.62 4.40 4.24 4.00 3.91



1. Trend Analysis:

First semester to use this measure. No trend analysis.

2. Action Plan:A. Continue Development of Rubric:

a. At this point the rubric is just a likert scale without descriptions.b. Timeline: Sprg 2011c. Needs to create a statistics & ethics rubric.

d. Needs to develop specific description of each standard.B. Collect Interrater Reliability data.

a. Timeline: Each semester for graduating students.



2. Action Plan:C. Shortfalls in student performance.

1. Determine the source of the problem.a. Poor rubric?b. Poor rater training?c. Poor student performance?


Learning Objective 2: Students will demonstrate research skills in the behavioral sciences.

Exit Interview – Indirect Measure

PROGRAM OUTCOME



Measurements




Learning Objective

Graduating Seniors

Locally developed questions on Exit Interview pertaining to program’s ability to prepare students to do research.

None available None available Every semester so every student will be measured.


Learning Obj. 2 – Indirect MeasureExit Interview Outcome

Rate how well the program promoted competence in research skills on a scale from 1 (low) to 7 (high) ?n = 38Mean = 5.89Standard Deviation = 1.02


Learning Obj. 2: Indirect MeasureExit Interview Outcome



Action Plan:1. Continue to collect data on the Exit Interview.2. It is expected that the corrective plan for

improving performance on the Rubric measure will improve rating on Exit Interview reporting.



Students will apply basic knowledge within the field of the behavioral sciences.1. Rubric Outcome – Direct Measure2. Exit Interview Question pertaining to the application of knowledge within the field of the behavioral sciences– Indirect Measure


Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences.

Rubric Outcome – Direct Measure

PROGRAM OUTCOME



Measurements





Practicum Rubric used by the Oklahoma State Department of Health Evaluation of Supervised Experience.

None available None available. Every semester so every student will be measured.



Areas rated:• Individual counseling• Group counseling• Marital, Family, Couples counseling• Child counseling• Assessment/diagnosis• Treatment planning• Makes appropriate referrals



Continued:• Consults with other professionals• Conducts research• Knows licensing law and rules• Conforms to Rules of Professional Conduct• Uses disclosure statement• Is prompt/current on paperwork and records• Cares for own mental health



Continued:• Utilizes supervision sessions effectively• Maintains professional boundaries• Stays within limits of competence level• Keeps current with professional literature



Levels of appraisal:0 = No observation1 = Needs Improvement2 = Acceptable3 = Above average skillNote: “No observations” were not used in

calculation of proficiency of application



Application of knowledge within the field of the behavioral sciences.n = 7

Ind Grp M&F Child Dx Tx Ref Conslt Res

Fall2009

2.10 No Obs

2.00 2.33 2.20 2.00 2.14 2.30 2.50

Sprg2010

2.50 No Obs

2.33 2.67 2.60 2.50 2.57 2.50 2.50



Application of knowledge within the field of the behavioral sciences.n = 7

Law PC Discl PPW MH Sup Bound Comp PLit

Fall2009

2.20 3.00 3.00 2.70 2.20 1.70 3.00 2.50 1.50

Sprg2010

2.30 3.00 3.00 2.70 2.20 2.10 3.00 2.00 2.00



Action Plan:1. Continue to collect data this upcoming year.2. Attempt to collect state-wide data to use as a

comparison group.3. Make the rubric more detailed in order to

help practicum supervisors further understand the standard.

4. Use supervisor rating to make curriculum changes.


Learning Objective 4: Students will apply basic knowledge within the field of the behavioral sciences.

Exit Interview – Indirect Measure

PROGRAM OUTCOME



Measurements




Learning Objective

Graduating Seniors

Locally developed Exit Interview questions pertaining to the programs ability to prepare students in professional competency.

None available None available Every semester so every student will be measured.



Rate how well the program prepared you and promoted a sense of professional competence on a scale from 1 (low) to 7 (high)

n = 38Mean = 5.97Standard Deviation = 1.02





Action Plan:1. Continue to collect data on the exit interview.2. Investigate ways of establishing validity and

reliability on the exit interview measure.


Priorities for Academic Year 2010-2011

1. Develop direct measure outcome criteria for Learning Objective 3: Students will demonstrate knowledge in theories, methods and findings in the behavioral sciences. Timeline: Collect data Spg 2011

a. Will use Advanced Psychology of Personality as the measurement course.

b. Will develop definition of “theories, methods, & findings” in the behavioral sciences.

c. Will develop embedded question to represent knowledge of theories, methods, and findings.



2. We have fallen short of our target performance in several areas on the direct measure of learning objective 1.

a. Remediation:i. Hand out rubric to students so they are aware of our

expectations.ii. Create handouts (with examples) on synthesis &

critical analysis.iii. Create an online learning module to help students

with synthesis & critical analysis.



3. Research the use of the Counselor Preparation Comprehensive Examination (CPCE) as a pre-post measure of program effectiveness.

a. Nationally Normed Examination (nearly 300 universities)

b. Breaks data down in all eight counselor proficiency areas.

c. Low cost: $45 per student

Program Quality Improvement Report

2009-2010 56

Program Enrollment/Graduation Trend


Program Credit Hour Production Trend


Published information on graduates

59

Academic Year 09-10 Number of Graduates Working In Discipline Entered doctoral program

Summer 2009 6 5 0

Fall 2009 7 6 0

Spring 2010 15 12 1

Total 28 23 1

Program Quality Improvement Report 2009-2010

Conclusions

1. Overall the MSBS program is doing well. Credit hour enrollment and credit hour production are at all-time highs. Graduation rates are consistent with expectations. It takes students approximately 3.67 years to complete licensure educational requirements. Graduations rates average approximately 26% of the student body. This percentage falls only one percentage point below expectations.


Conclusions

2. Over 80% of our graduating seniors are working within the field of the behavioral sciences.

3. New programmatic changes will create a more predicable curriculum structure for the student. This, and other remediation changes, should increase graduation and retention rates.


Date post:	18-Dec-2015
Category:	Documents
Upload:	lenard-tate
View:	213 times
Download:	0 times

Department of Psychology & Human Ecology School of Education and Behavioral Sciences Masters of...

Documents