Engaging the Student Voice to Enhance Student Services

Dr Hamish Coatescoatesh@acer.edu.au

November 2006

Focus of the presentation

• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?

Quality data for management

• The quality of ‘education’ data underpins the cogency of many strategic and operational decisions

• Senior managers require timely access to informative and robust data

• But often the wrong data is collected• But much data management within Australian

higher education institutions is ad hoc We need to think strategically about what data

is required for institutional management We need efficient systems for analysing and

interpreting large amounts of complex data

Identifying existing data

• Does the institution have central data management protocols?

• What data exists centrally, and within faculties, departments, units, subjects…?

• What is the nature and limitations of existing data?

• What is the strategic or operational relevance of the data?

• How have the data and collections been developed?

Reviewing data needs

• What is required to bring disparate data together – initially, and in routine ways?

• How is this data best analysed and reported, and who should do this?

• What are the relevant points of reference – norms or benchmarks/criteria?

• What ‘gaps’ exist in the data collection?• What redundancies exist, and what are the

legacy issues?

Focus of the presentation

• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?

A summary of existing data…Inputs Processes Outcomes

Institutions Capital resourcesInstitutional

reputation

Institutional contextsLearning systems

GrowthReputation

Staff Teaching qualifications

Teaching experienceTeaching staff

numbersExpenditure per

studentStaff/student ratios

Teaching processes Teaching experienceTeaching resources

Students Entry scoresStudent diversityDemand/selectivityStudy access/equity

Engagement processes

Retention and attrition

Course completionsGraduate and generic Engagement

outcomesLearning outcomesGraduate

destinations

Internal ‘student support’ data

0 10 20 30 40 50 60 70 80 90

Internal satisfaction surveys

Equity success indicators

Usage statistics for student services

Internal reviews

Programs offered in flexible mode

CEQ / PREQ scales - support services

E-learning Help Desk response times

Size of library collection

Expenditure on services

Range of student services

Online student support / enrolment

Per cent of institutions (2004)

External ‘student support’ data

CEQ Learning Resources Scale (8/38 institutions)• The library resources were appropriate for my needs• The study materials were clear and concise• It was made clear what resources were available to help me learn• Course materials were relevant and up to date• Where it was used, the information technology in teaching and learning

was effective

CEQ Student Support Scale (18/38 institutions)• I was able to access information technology resources when I needed

them• Relevant learning resources were accessible when I needed them• Health, welfare and counselling services met my requirements• The library services were readily accessible• I was satisfied with the course and careers advice provided

Data use by institutions

0 10 20 30 40 50 60 70 80 90

Strategic planning and management

Reporting eg annual report, DEST profiles, AUQA

Support budget process

Course/curriculum reviews/development

Quality enhancement

Academic Organisational Unit reviews or evaluation

Benchmarking

Marketing/Promotion

Staff performance management

Distribute teaching funds

Reward/incentive strategies

Per cent

Focus of the presentation

• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?

Exploring ‘student engagement’…• Why take the ‘engagement approach’?• How can we factor ‘insights’ on student

engagement into quality management?• Why isn’t student engagement information

implicit in quality educational management?• Who defines how students should interact with

universities?• Who benefits from evidence-based

understandings of engagement?• What value could engagement insights add to

higher education?

Inputs Processes Outcomes

Institutions Capital resourcesInstitutional reputation

Institutional contextsLearning systems

GrowthReputation

Staff Teaching qualificationsTeaching experienceTeaching staff numbersExpenditure per studentStaff/student ratios

Teaching processes Teaching experienceTeaching resources

Students Entry scoresStudent diversityDemand/selectivityStudy access/equity

Engagement processesRetention and attrition

Course completionsGraduate and generic Engagement outcomesLearning outcomesGraduate destinations

Educational indicators

The idea of ‘student engagement’• Individuals learn and develop through behavioral,

cognitive and affective involvement with key educational and social practices

• People learn when staff and institutions provide supports likely to encourage involvement

• A specific idea which encompasses a broad range of relevant academic and non-academic phenomena

• Measures of engagement provide an index of whether students are engaging with university in ways likely to stimulate development and generate high quality learning

• Student engagement is not ‘satisfaction’, not ‘retention’, not just/primarily the evaluation of pedagogy, not an attitudinal/opinion survey, and not ‘student experience’

Engagement as a development

• Developed from interpretations of individual learning and development as constructive participation in university communities

– Astin’s ‘involvement principle’– Pace’s ‘quality of effort’– Tinto’s work on integration and departure

• Based on the identification of activities and conditions linked empirically with effective learning

– Chickering & Gamson’s ‘seven principles’– identification of process factors by Pace, Pascarella

& Terenzini, Astin, and Ewell

Engagement as a reaction

• Engagement offers a valid and consistent measure of the quality of education

• Limitations of alternative means of evaluating the quality of university education

– institutional resources and reputations– measures of teaching quality and teacher

qualifications– student input, progression and outcomes

The significance of engagement• Reflects students’ intrinsic involvement with

education• Offers a generalisable indirect measure of

educational outcomes• A direct measure of involvement in key processes• Focuses quality considerations on student learning

and development• Reflects the wide range of educationally meaningful

interactions that students have with their universities• A coincident measure of student activity that can be

used to evaluate and manage the quality of higher education

Typical qualities of student engagement

• Constructive teaching• Supportive learning environment• Teacher approachability• Student and staff interaction• Academic challenge• Active learning• Complementary activities• Collaborative work• Beyond class collaboration• Online forms of engagement

Styles of engagement

Possible measurement approaches

• One-to-one interaction

• Direct naturalistic observation

• Administrative data

• Time or activity diaries

• Assessment

• Interviews and focus groups

• Questionnaires and surveys

Student experience instruments

• College Student Experiences Questionnaire (CSEQ)• College Student Expectations Questionnaire

(CSXQ)• College Student Survey (CSS)• Freshman Survey (FS)• Your First College Year (YFCY)• National Student Survey (NSS)• First Year Experience Questionnaire (FYEQ)• Course Experience Questionnaire (CEQ)• Postgraduate Research Experience Questionnaire

(PREQ)

Student engagement instruments

• College Student Report (CSR)• Student Engagement Questionnaire (SEQ)• (CEQuery)

Levels of engagement

0.00 1.00 2.00 3.00

Constructive teaching

Supportive learning environment

Teacher approachability

Academic challenge

Active learning

Student/staff interaction

Complementary activities

Collaborative work

Beyond-class collaboration

Never Rarely Sometimes Often0.00 1.00 2.00 3.00

Constructive teaching

Supportive learning environment

Teacher approachability

Academic challenge

Active learning

Student/staff interaction

Complementary activities

Collaborative work

Beyond-class collaboration

Never Rarely Sometimes Often

0

1

2

3

OEOA

LOA

R OT OC OSI

OCS CT

SLE TA AC AL SSI

CA CW BCC

English not first language

English first language

often

some-times

rarely

never

Using ‘engagement’ data

• Guidance for prospective students on expectations, experiences and supports at an institution

• Data on resource use, management and educational value

• Information for internal and external quality enhancement activities

• Insight for graduate employers on individuals’ involvement with their work

• Feedback for teaching academics to help them manage and improve pedagogy and learning

• Advice for current students on course quality and learning practices, retention and involvement

Focus of the presentation

• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?

CEQuery context• Graduate Careers Australia conducts an annual survey of

graduates using the Graduate Destinations Survey (GDS) and the Course Experience Survey (CEQ) – these are sent out together

• These surveys capture a range of demographic variables linked to responses on graduate outcomes (GDS) and satisfaction (CEQ)

• The CEQ also has sections for written comments under the headings “Best aspects” and “Needs improvement” – these are analysed by CEQuery

• This enables the student’s qualitative comments to be linked with demographic and field of education variables, satisfaction scales and graduate outcomes

The opportunity and need

• Need not just to gain but retain students

• Rapid developments in I.T.

• 750,000 CEQ comments - unanalysed

• Listen to the ‘voice’ of students and not just rely on quantitative measures

What is CEQuery?• IT-enabled analysis of large amounts of

qualitative data on HE learning & teaching• Wide range of custom analyses against any of

the CEQ student demographic categories or items

• Results in tabular, graphical or digital modes• Allows user to test the veracity of the coding by

showing the comments that made up the count• Five domains and 26 subdomains

CEQuery coding domains

• Assessment

• Course design

• Outcomes

• Staff

• Support

CEQuery subdomains

• Assessment– Expectations– Feedback– Marking– Relevance– Standards

• Course design– Flexibility– Learning methods– Practice-theory links– Relevance– Structure

CEQuery subdomains

• Outcomes– Further learning– Intellectual– Interpersonal– Personal– Knowledge/skills– Work application

• Staff– Accessibility– Practical experience– Quality and attitude– Teaching skills

CEQuery subdomains

• Support– Infrastructure– Learning resources– Library– Social affinity– Student administration

– Student Services– Course administration

Hit Rates: Domain Level

Staff, Teaching Skills, Needs Improvement

• Some lecturers provided poor lectures and in some subjects the questions in the final exam were nothing like the ones we had been taught to answer.

• Affordable technology needs to be better -utilized as teaching aids by staff who know that teaching is more than standing up the front droning on

• Lecturers need to be brought in line now traditional teaching methods are becoming redundant to tutorials. They need to learn how to actively involve us, get us working creatively, experimenting – less chalk and talk.

• The teaching staff should give more constructive feedback and make learning more interactive during classes.

• The methods employed by lecturers are boring they need to participate more with audience to encourage participation in discussions and enhance interest.

Validity of the results

• CEQuery comments hit rate: 80%– Of the remaining 20% not hit in CEQuery, half are

not meaningful (cannot be coded)

• Coding of comments to sub-domains by CEQuery checked by multiple raters– Accuracy approx 90%

• CEQuery dictionary of words and phrases has been refined and updated, and custom dictionaries have been produced

A review of CEQuery

• Geoff Scott (2005). Accessing the Student Voice. DEST: Canberra.

• Analysed comments made by 95,000 graduates between 2001-4 at 14 universities

• Key insights into CEQuery use, correlates of response, key aspects of engagement, ‘best aspects’ and ‘improvement areas’

• Findings validated through a series of national workshops

Subdomain BA count NI count BA/NI (QUALITY)

Outcomes – personal (OP) 1668 88 19.0

Outcomes – further learning (OF) 347 24 14.5

Outcomes – intellectual (OI) 7197 541 13.3

Outcomes – unspecified (OU) 1126 173 6.5

Support – unspecified (SU) 277 54 5.1

Outcomes – interpersonal (OIP) 2455 595 4.1

Outcomes – knowledge/skills (OK) 8037 2746 2.9

Staff – accessibility (SA) 12748 10611 1.2

Course - Practical–theory links (CP) 9157 7658 1.2

Outcomes – work application (OW) 4715 5248 0.9

Assessment - unspecified 179 251 0.7

Support – learning resources (SR) 3970 6006 0.7

Course design – relevance (CR) 6335 9658 0.7

Staff – teaching skills 5548 9969 0.6

Support – infrastructure/environment (SI) 3423 6353 0.5

Support – library (SL) 1018 1933 0.5

Support – student services 784 1808 0.4

Assessment – standards (AS) 1873 5449 0.3

Support – student administration (SAd) 1078 4095 0.3

Course design – structure (CS) 3579 15668 0.2

Assessment – marking (AM) 386 2045 0.2

Assessment – expectations (AE) 308 1794 0.2

Assessment – feedback (AF) 316 2792 0.1

Perceptions of quality by subdomian

Subdomain BA count NI count TL count (IMPORTANCE)

Course – methods (CM) 22231 18338 40569

Staff – quality (SQ) 17417 13512 30929

Staff – accessibility (SA) 12748 10611 23359

Course design– flexibility (CF) 12754 10579 23333

Course design – structure (CS) 3579 15668 19247

Course - Practical–theory links (CP) 9157 7658 16815

Course design – relevance (CR) 6335 9658 15993

Staff – teaching skills 5548 9969 15517

Support – social affinity (SSA) 7249 3683 10932

Outcomes – knowledge/skills (OK) 8037 2746 10783

Support – student administration (SAd) 1078 4095 5173

Assessment – relevance (AR) 2537 1400 3937

Assessment – feedback (AF) 316 2792 3108

Outcomes – interpersonal (OIP) 2455 595 3050

Support – library (SL) 1018 1933 2951

Support – student services 784 1808 2592

Outcomes – personal (OP) 1668 88 1756

Course design – unspecified (CU) 1079 574 1653

Outcomes – unspecified (OU) 1126 173 1299

Staff – unspecified (SU) 347 334 681

Assessment - unspecified 179 251 430

Outcomes – further learning (OF) 347 24 371

Support – unspecified (SU) 277 54 331

Importance perceptions by subdomian

Clusters of the best learning methods

Cluster Hits

Face to face 11693

Simulations and labs 1357

Independent study and negotiated learning 3572

Practice-oriented and ‘real world’ 9329

ICT Supported 836

Clustered best learning activities in Business

Face to Face Lecture, group project, tutorial, class-

work exercise, discussion and sharing ideas, seminar and individual presentation, workshop, debate, 1:1 mentor, conference, symposium, forum, exhibition, peer assistance, group dynamics exercises, critique of student production, buzz group

Independent Study Completing assignments, thesis, self

teaching materials, self-directed study, project report writing, learning contract

Simulations & Labs Mock trial, role play, hypothetical,

games, experiments, labs, simulation

Practice oriented/real world Clinical placement, practicum,

practical experience, PLT, Coop Ed, work experience, hands on practice, field study/site visit, practical work,camps, real world problems, guest practitioners, professional mentor, design studio, artistic production, study abroad, case study

ICT-enabled learning Online search, web-based learning,

email, phone, teleconference, CD, DVD, tele-tutorial, radio, TV, digital image access, computer assisted assessment, simulations, computer generated demonstrations of processes, interactive programs

Most common best aspects by field of education

Science & Built Environment Team/group project, assignments, field

study/site visit, hands on practice, lecture, class exercises, laboratory work, practical work, practical experience

HealthClinical placement, practical experience, lecture, hands on practice, assignments, tutorial, class exercises, group project work, labs, practicum, work experience

EducationPracticum, practical experiences, assignments, hands on practice, lecture, tutorial, class discussion, class exercises, team/group project work

Management & Commerce Team/group project, assignments,

lecture, class exercises,, seminar – individual presentation, tutorial, discussion, case study, real world problems to solve, work experience

Society, Culture & Creative ArtsAssignments, class exercises, lecture, tutorial, group project, class discussion, hands on practice, practical experience, seminar – individual presentation, practical work

CEQ scales and CEQuery domains• All five domains significantly associated with all CEQ

scales• Best aspects comments predict higher CEQ scale

scores and needs improvement comments lower CEQ scale scores

• Strongest correlation with the Good Teaching Scale, Overall Satisfaction Item (OSI) and Clear Goals and Standards Scale

• For example: given OSI score, odds of positive rather than a negative comment given OSI score: Assessment 1.22, Course Design 1.34, Outcomes 1.40, Staff 1.21, Support 1.21 (eg: positive ‘support’ comment 21% more likely for each step up 5-point CEQ scale)

Assess-ment

Coursedesign Outcomes Staff Support

University X X X X X

Field of education X X X X X

Qualification X X X X

Fees X X X

Gender

Age X X X X

Attendance type X X

Attendance mode X X X X

Year data X X X

Australian residence X X X X

ATSI

Conditions that enhance engagement• A sound, responsive, flexible, relevant, clear and mutually

reinforcing course design—a design that uses an appropriate variety of interactive, practice-oriented and problem-based learning methods

• Capable, committed, accessible and responsive staff being in place to deliver and improve the design during implementation

• Efficient and responsive administrative, IT, library and student support systems actively working together to support its operation

• Relevant, consistent and integrated assessment of a university standard that the course’s design, learning methods and resources specifically enable students to complete

Key findings• It is the total experience that retains students• Graduates see certain experiences are more ‘important’

than others, and identify high quality areas as well as those in need of improvement

• Areas with highest hits identify key QA checkpoints for course design, accreditation and review

• Variation between field of education and institution suggests opportunities for benchmarking

• Should address the areas with low odds of a best aspect comment (assessment, course structure, support)

• Follow up specific issues, such as the low level mention of ICT as a best aspect

Focus of the presentation

• A strategic review of educational data• Nature and use of existing data• Student engagement: A lens for analysis• A tool for analysis: CEQuery• What can be done?

Using findings to enhance practice

• Student engagement data can be used to enhance the quality of education and the university experience

• Strategic use of ICT/LMS/portals in ways that supplment and enhance the student experience

• We need to develop measures of engagement, and make better use of the CEQuery tool

• We need to develop ways of factoring evidence-based perspectives on student engagement into practice

Use CEQuery• Available to all universities• Analyse feedback data from many survey

instruments• Can be tailored for local contexts• Need to enter open-text response• Need to position within strategic reporting and

planning cycles• Identify good practice, internal benchmarking,

priorities for improvement

Review student engagement• Ensure feedback instruments are harnessing

data on what really matters for high quality learning and support – we’re not measuring key aspects of higher education

• Ensure analysis and reporting approaches are appropriate

• Weave ‘student engagement’ ideas into operational and strategic plans, and practices

Help students learn to engage • Produce resources which help students

understand key aspects of being a university student

• Develop targeted programs and spaces to help students learn about university

• Develop practices which encourage engagement

• Provide students with feedback on their views

Manage student engagement• Shape support systems in response to known

individual and cohort differences, perceptions and reports

• Shape adaptive spaces, portals, services, cultures, activities, programs, approaches, people…

• Study student readiness, identify at risk students, studying student transitions through study

• Develop seamless links between academic and non-academic aspects of university

Benchmarking

• Make comparisons within and between institutions

• Develop meaningful points of reference and for improvement

• Develop ‘learning communities’ to generate and disseminate key ideas

At your institution…

• How could student engagement data be better used at your institution?

• What changes in data collection, analysis and reporting may be required?

• What programs or practices could be altered, and how?