Post on 30-Nov-2021
transcript
TRANSFORMING LIVES
INSPIRING CHANGE
ACC3015 Accounting & Finance for Managers
Gail Capper | Pearson
KEY
FINDINGS
Engagement, motivation and activity patterns are generally seen to be linked to the mode of course delivery. Campus and Distance Learners use MyAccountingLab (MAL) with differing levels of participation and achievement.
The findings in this report relate specifically to the campus and distance learning cohorts of Northampton Business School (NBS) ACC3015. This is an exploratory study within which cohorts vary from 8 to 144 students and whose findings assist us in understanding how MAL helps in supporting learners at NBS
Course: Accounting & Finance - 7 different cohorts both on campus and distance learners (see below for detail)
Resource: MyAccountingLab for Atrill/McLaney Accounting & Finance for Non-Specialists 8/e
Data Collection: MAL Gradebook on NILE (Northampton VLE)
Data Type & Period: Quantitative, 2013-2014 Summary of Results
Students engage differently across courses. Distance Learning and Campus learning seems to be intrinsically
different.
The exception to this was the general DL course which seemed to perform differently to all the other distance
learning courses. Motivation, activity and achievement all differed for this group. Their achievement was far
more diverse, and their engagement in the resource far higher on self-study than any other Distance Learning
cohort and their dropout rate was the highest at 18%.
Participation in the homework and homework scores are higher on campus than in any distance learning
course. Motivation appears to drop after the first two homework assignments have been completed for
distance learners.
More specific findings include:
Distance & Campus students show clear differences in the level and continuity of engagement with MyAccountingLab (MAL)
Deadlines and credit- bearing assessment influence activity online
Cohorts who completed more homework, gained higher average scores in their formative homework assignments
Cohorts showing most activity on the resource, were not those who achieved the highest grades in summative assessment
Cohorts who were actively engaged with the resource show sustained motivation to access the resource throughout the course
Pag
e2
Summary & Study Purpose
This study was completed in order to better understand how campus based and distance learners using
MyAccountingLab (MAL) were interacting with it. The University wanted to understand more about how
students were making use of the resource and whether it was effective in helping to engage them.
Northampton Business School (NBS) will move their campus location and begin a flipped classroom delivery
from September 2015 for their Year 1 intake. Understanding how students are using and engaging with
resources will help to inform the way in which online resources can be delivered most effectively.
Of particular interest to NBS was a report covering retention, achievement, activity patterns or peaks
(especially around deadlines) and overall student engagement on different courses. There was also a question
around how well the resource engaged campus versus distance learners.
NBS has invested significant amounts of time and money in MAL and so this study should inform a business
case considering whether the investment is well spent or how to improve the return on that investment by
improving the effectiveness of MAL in NBS.
General study question: ‘How is the resource helping and is the resource effective?’
Especially:
‘Are students engaging with MAL across all courses?
‘What differences are there in usage of and achievement in MAL between the courses? ‘
‘How can these findings inform NBS plans to deliver courses differently in e.g. flipped classrooms?’
NBS are unable to share final course marks for the purpose of this study and so there is no cross reference
between achievement on MAL tests and grades at the end of the course available.
MyAccountingLab – Overview of Resource
MyAccountingLab (MAL) is an online learning resource designed to work alongside specific textbooks as an
online homework, tutorial and assessment program. It provides a variety of different question types which can
be studied independently or assigned by the lecturer. Assigned work is created and set with a deadline either
as homework (for practice) or a test (for assessment). Settings can be adjusted to allow for single or multiple
attempts. In anything other than a test, students will have access to a range of features to provide support and
feedback such as ‘Help Me Solve This’ which offers a similar exercise, broken down into steps.
Independent study is completed in a study plan which contains practice and test questions. These tests are
called sample tests.
Achievement on the course can be measured in a number of ways and mastery is one way. Students gain
mastery points as they complete the study plan, usually to a maximum of 54 points. They must score 100% on
the relevant questions to gain 1 mastery point (the % required for a point can be adjusted manually to any
score the lecturer thinks appropriate). Mastery is represented as both points and translated as a percentage
for the overall study plan score. This allows us to compare it to assessed scores and homework scores.
Pag
e3
Integration of MAL in Northampton Business School
Overview:
NBS first started using MyLabs in 2010 and Pearson labs are integrated into the Northampton Integrated
Learning Environment (NILE – Blackboard Learn) for 6 disciplines in the school, one of which is Accounting. The
university integrated MyLabs into NILE following a couple of years of difficulty in self-registration using access
codes. Self-registration was time-consuming and ultimately unmanageable for the staff. VLE integration works
well, is cost efficient, gives reliable access for students and is time-saving for staff who are generally happy with
the setup.
The eLearning Development & IT Support Manager in NBS said that Northampton are committed to providing
students with a u-Learning (ubiquitous learning) experience in a timely, cost effective and cost transparent
way… a compelling eLearning experience.
On ACC3015 specifically:
This study reports on 7 courses run by NBS. They are all the same course delivered to different cohorts in
different locations: ACC3015 Accounting & Finance. All these courses used Atrill & McLaney’s 8/e textbook and
the accompanying MAL. Two of the courses are campus based, face-to-face courses while five courses are
Distance Learning (DL) courses delivered in Asia. The distance courses are classed as ‘supported distance
learning’ meaning they meet the lecturer and attend locally run workshops for the course with the exception of
the DL course who have no face-to-face contact with teaching staff. That said, students could be located in any
geography to attend the distance courses.
This VLE introduction to MAL on the Northampton VLE (NILE) sets out the intended use for all students.
Course & Assessment Structure
All 7 courses use MAL to set assessed tests for each of the cohorts. Test 1 represents 20% of the final mark to
assess ‘Financial Accounting’, Test 2 represents 20% of the final mark assessing ‘Management Accounting’ and
the last part of the course is assessed by an essay worth 60% which covers ‘Financial Management’.
In addition, students were assigned a mixture of practice tests, homework and given open access to additional
self-study contained in the study plan section of MAL. The study plan is open to all students and contains
exercises which are not used in homework or assessed tests.
One difference in implementation for DL and campus students is that DL students were set 9 homeworks for
the duration of their courses, October campus students were set 2 homeworks and January campus students
were set 2 homeworks with 4 additional homework assessments. None were credit bearing.
Campus courses: (short reference used throughout the study is in bold) • STD Oct - 144 students. Accounting & Finance for Managers • STD Jan - 24 students. Accounting & Finance for Managers
Pag
e4
Distance learning courses: • DL (General) - 40 students, Accounting & Finance for Managers
• DL 1 (H) - 35 students, Accounting & Finance for Managers
• DL 2 (S) – 16 students, Accounting & Finance for Managers
• DL 3 (XHC) – 8 students, Accounting & Finance for Managers
• DL 4 (HCX) – 10 students, Accounting & Finance for Managers
Methodology & Metrics
The metrics used in this report are drawn from each course Gradebook in MAL. As these cohorts have moved on there is no interview or survey data, but the students do complete Module Evaluation forms. Over time, the study can be added to as further years complete the course to allow for a more longitudinal study to be completed. All data provided is anonymised as stipulated by the university. The list below explains which data which has been accessed from the MAL Gradebook for each course.
• Mastery points: These points represent student achievement and ‘mastery’ of the content in MMLG. Mastery points are earned by answering questions correctly (100% correct) in the study plan (self-study). Assigned homework, tests and quizzes selected from chapter material may also contribute to mastery points.
• Average grades: available from sample tests (study plan tests), study plan, homework, tests and quizzes.
• Sample tests are used as a measure of self-study or independent study. They are accessed by individuals at any time in the course but are not assigned.
• Time spent on task: available in the study plan and assigned homework or tests. Total time represents logged in time to any part of the resource. It may not always indicate learner activity.
Pag
e5
Results
Student Access & Activity
Charts 1.1 and 1.2 below show that there is more activity on MAL when credit bearing assessments are due,
especially for distance learners.
Four out of the seven courses have been used as example activity patterns, the results show:
Distance Learners access materials in a need-to-do manner. There are peaks of activity around credit-
bearing assessment deadlines.
Campus-based learners also demonstrate peaks of activity around credit-bearing assessment deadlines.
However, there are more peaks and more sustained activity for campus based learners overall. Peaks
may be linked to lecture dates and/or being campus-based with more access to a learning
environment.
Campus-based learners have more peaks of activity at times when assessments are not due than
distance learners (over the duration of the course)
Announcements on the university VLE (NILE) show no impact on activity patterns on any courses.
How can we explain the other peaks of activity? For example, on the STD Oct course student activity shows peaks on the 3rd November when 23 students accessed the homeworks and on 24th November when 37 students accessed homework. Did the cohort attend lectures on that day, or the previous day where a lecturer reminded them to access the resource? Students were grouped in order to lecture on the course and had different days for their lectures. However, they appeared to access their work most often on a Sunday or Monday in preparation for the week – whatever day the lectures were. Did the availability of assigned homework affect activity? Homework can be completed at any time, and varies
by course. For example, STD Jan had to complete their Homework 1 & 2 by 7th March 2014. STD Sep had to
complete theirs by 15th November 2013. These dates did not seem to influence access to the homework.
The STD January cohort were given additional Homework Assessment 1-4 compared to STD September.
According to Chart 1.1, the STD January cohort had the highest percentage of students opening their
homework. The two campus cohorts also scored the highest average homework scores (see chart 1.4)
Chart 1.1 below tracks when homework was accessed, by date, for campus and distance students.
Pag
e6
Chart 1.1
Pag
e7
DL 2 (S) in particular has low levels of activity outside the deadlines for assessment. In addition, DL 2(S) also had
the lowest levels of access to testing within the study plan as shown in Chart 1.7. Combining this information
helps to draw a clearer picture of student engagement with MAL
Engagement with Homework & Assessed Tests (40% credit)
Accessing the assigned homework can be used as a measure of how engaged or motivated students are as it is
not credit-bearing, but is recommended for practice. The number of students failing to complete Test 1 AND
Test 2 shows a failure to retain the students on the course.
If we compare the average number of students who logged in and accessed assigned homework and also those
who failed to complete credit-bearing tests in ACC3015 it shows that for distance learning DL 1 (H) and DL
(General) have the highest non-completion rate and lowest number of homework assignments accessed.
STD Oct has comparatively high homework completion but is the highest non-completion number for the
campus courses.
Overall, retention is good on all courses except on the general DL (General) course.
Chart 1.2
N.B. Distance Learning courses, the students had more homework to complete.
What impact did this have on their sustained motivation to complete the homework and on what
scores they achieved?
Pag
e8
Table 1.1 shows the percentage of students who completed all homework set for their course against the total
number of students on the course. The yellow highlights show comparatively high or low numbers:
Course % Completing all
HW % Not Completing
Course
STD Oct 81% 4%
STD Jan 94% 0%
DL (General) 48% 18%
DL 1 (H) 40% 2%
DL 2 (S) 75% 0%
DL 3 (XHC) 75% 0%
DL 4 (HCX) 70% 0% Table 1.1
Table 1.2 shows the average number of students completing homework in more detail, showing how many
students completed each homework and giving an overall average for homework completion.
Cohort HW 1 HW 2 HW 3 HW 4 HW 5 HW 6 HW 7 HW 8 HW 9 Median Median % (rounded)
DL 1 (H) 21 14 12 11 13 10 15 15 17 14 40%
DL 2 (S) 11 11 11 15 12 11 14 14 12 12 75%
DL 3 (XHC) 8 7 6 6 6 6 8 7 5 6 75%
DL 4 (HCX) 7 7 7 6 6 5 7 7 7 7 70%
DL General 30 19 15 11 14 14 24 21 20 19 48%
HW 1 HW 2 Median Median % (rounded)
STD Oct 117 115 116 81%
STD Jan 23 22 22.5 94%
Table 1.2
Chart 1.3 shows that more students completed the first two homework assignments than the later homework
assignments. DL 2 (S) is the small, but notable exception. Is this linked to a decrease in motivation and
engagement in distance learning resources over the duration of a course?
Chart 1.3
Pag
e9
If we check the scores for those first two homework assignments against the scores for all homework on chart
1.4, it shows that students gain higher marks on their first homework assignments.
Chart 1.4
Engagement with the Study Plan.
Motivation to study without direction is a useful measure of how engaged students might be, in addition to
activity pattern for homework access which is completed at any time. It requires interpretation and those
students who choose to self—study might be struggling, but motivated or they may be highly motivated.
The number of sample tests opened in the study plan (which are not assigned by lecturers) can be used to
measure engagement and participation in the course over time. The charts below show how many sample
tests students from each course accessed.
Legend:
Pag
e10
Pag
e11
The DL (General) and STD Oct cohorts score well in self-study access as a positive sign of motivation:
DL (General) course cohort has the fewest number of students accessing zero sample tests.
DL (General) and STD Oct have the highest number of students opening more than 26 tests
On ALL other distance learning courses the majority of students do not open any sample tests for self-directed study.
Time Spent
As the study plan represents self-motivated, self-directed work it is interesting to look below the surface for
further trends in time spent and in how much of the resource was used.
Course Average Total Time on MAL Average Number of Questions (zero score students removed) Worked in Study Plan (rounded) STD Oct 779 minutes 61 STD Jan 1748 minutes 120 DL 1 (H) 818 minutes 1 DL 2 (S) 1761 minutes 6 DL 3 (XHC) 1712 minutes 5 DL 4 (HCX) 2244 minutes 2 DL (General) 1877 minutes 70 Table 1.3
The average time remains high for all courses. However, the number of questions students worked in their study plan is dramatically different for campus and DL courses.
DL 2 (S) is the least active / engaged and scored among the lowest in chart 1.4 for homework scores. However, in the achievement section below we can see that they performed well in their assessed work.
This raises the question ‘where is the time being spent and is it active engagement? ‘
In DL 2 (S) to continue the example, the time spent online is disproportionately affected by one or two students spending a large amount of time with their homework ‘open’ but not necessarily completing it - from 20 minutes per homework to 24 hours per homework Student A spent 86.5 hours in total time Student B spent 54 hours in total time
Chart 1.5
Pag
e12
The number of questions worked in the study plan is a better indicator of engagement or motivation than time spent on it. Chart 1.6 below demonstrates that STD Oct, DL (General) and STD Jan worked the greatest number of questions. However, the achievement in assessed work is higher in all the other supported distance learning courses.
Chart 1.6
Pag
e13
Motivation & Achievement Engagement and motivation are not shown to correlate with high scores in assessed tests in ACC3015. So how does participation in MAL compare to average course scores in the MAL resource?
If we use self-study as a measure of motivation on MAL (sample tests accessed, see page 4) and compare it to overall scores from all parts the MAL resource, Chart 1.7 shows no discernable link between the two. However, it does show how few distance learners were motivated to access the study plan sample tests.
Chart 1.7
If we look at the average number of ‘questions worked’ in the study plan which includes all exercises accessed
by students (see Table 1.3 & Table 1.4, Chart 1.6) it clearly shows supported distance learners* undertook
comparatively little self study. The reasons for this are not clear.
Course Number of students who worked Average number of questions
questions in study plan/total students worked in Study Plan (rounded)
STD Oct 69 / 144 (47.9%) 61 STD Jan 20 / 24 (83.3%) 120 DL (General) 21 / 40 (52.5%) 70 DL 1 (H)* 2/13 (15.4%) 1 DL 2 (S)* 1/16 (6.3%) 6 DL 3 (XHC)* 4/8 (50%) 5 DL 4 (HCX)* 2/10 (20%) 2 Table 1.4
This shows that half the DL (General) and half the DL 3 (XHC) accessed the study plan. Those who used it in DL (General) cohort simply did far more than DL 3 (XHC). Once again STD Jan have high numbers as 83.3% of the cohort worked questions and completed the highest average number of questions each. Why do distance learners access less of the digital learning resource and show less engagement in MAL despite the online delivery of the course?
Pag
e14
Achievement - Homework
Identifying where students perform well and where they perform less well informs any strategies for improving
engagement or achievement.
Chart 1.8 below shows the difference between homework and assessment scores for campus cohorts versus
Distance Learning cohorts. Campus students invariably gain higher average scores in homework than
assessments. However, Distance Learning courses gain higher average scores in assessed work rather than their
homework.
Chart 1.8
It is worth remembering that the Distance Learners have more homework than Campus students and that their
motivation to complete it appears to decrease across the duration of the course
The mastery scores are similar across courses – this is an average score for the study plan section of MAL (self-
study, non-assigned). If we were able to compare exam results we could compare achievement on the course.
Applying average examination results to this chart would be valuable in order to show how the cohort
performed on the course, not just on MAL. If they were to reflect the mastery score on the study plan,
for example, lecturers would be able to assign more activities on which students can gain mastery
points.
Pag
e15
0102030405060708090
100
0 10 20 30 40 50 60 70 80 90 100
Test
1 S
core
(%
)
Test 2 Score (%)
DL 2 (S) student achievement in assessed work
Pag
e16
Charts (all) 1.9
Achievement - Test 1 & 2 (20% credit each)
If we look at the spread of results over the assessed component of the course it shows the range of abilities in
the various cohorts over Test 1 and Test 2.
The trends which are most evident in Chart 1.9 below are:
Campus students have a wide range of average scores for both Test 1 and Test 2 in the October and January intake
Campus students tended to do better in Test 2 than in Test 1 Distance Learning students’ results were interesting.
DL 2 (S) and DL 3 (XHC) and DL 4 (HCX) achieved high scores in both Test 1 and Test 2
DL and DL 1 (H) had the greatest range of results in distance courses though they generally scored better in Test 2.
DL (General) had least students in the top, right quadrant (over 50% in both tests)
The questions this raises:
Why is it harder to engage the seemingly most able, best performing students in MAL?
Is it harder to engage these students in distance learning online generally?
Why is there a difference with sustaining participation for different groups of learners on MAL?
The high scores in the courses listed below
appear unusual considering the lower
homework scores in chart 1.8 and high total
time commitment in chart 1.10. It may
warrant some further investigation for NBS:
- DL 3 (XHC)
- DL 4 (HCX)
0102030405060708090
100
0 10 20 30 40 50 60 70 80 90 100
Test
1 S
core
(%
)
Test 2 Score (%)
DL (General) student achievement in assessed work
Pag
e17
Time Spent on MAL
Average total time spent on all aspects of MAL is shown below. It is important to remember that time spent
simply indicates that a particular activity was open on a student’s device. It is not an indication of activity and
so can be misleading e.g. 24 hours to do a homework is unlikely, even though one student spent this length of
time with the activity open.
STD Oct had the lowest average time in campus students
DL 1 (H) had the lowest average time in DL
DL 4 (HCX) had the highest average time online
Chart 1.10 How might time and achievement relate? The median averages have been used to represent time as it can be a misleading number. The lengths of time which students spend online varies to a large degree within each cohort e.g. from 131 hours at the highest down to just 11 minutes in STD Oct.
Course
Assessment Score %(based on students completing all assigned assessments)
Average Score % (all forms of testing, quizzes, assessed, homework, sample testing)
Mean Total Time (zero score students removed) (mins)
Median Total Time (zero score students removed) (mins)
STD Oct 70.9 72.6 799 474
STD Jan 73.5 56.2 1748 2007
DL (General) 67.7 44.6 1877 1075
DL 1 (H) 82.5 79.8 818 239
DL 2 (S) 86.9 62.7 1761 1775
DL 3 (XHC) 96.3 92.7 1712 1612
DL 4 (HCX) 86 54.1 2244 1054 Table 1.5
Pag
e18
Charts 1.11 and 1.12 below are show an overall trend upwards for total time spent on MAL in relation to the credit-bearing Test 1 & 2 scores i.e. the more time spent, the better the score (generally). However, the opposite picture is shown for total time and average scores on all forms of activity on MAL in chart 1.12. That is to say, more time in total on MAL, the worse the score.
Chart 1.11
Chart 1.12
Time as an average seems to be an unreliable marker of achievement or engagement at a course level – though perhaps if we used it for individual students it might be more revealing e.g. did the student who spent 131 hours online in STD Oct achieve high scores?
Pag
e19
Conclusion & Recommendations
1. Are students engaging with MAL across all courses? 2. What differences are there in usage of and achievement in MAL between the courses? Overall: How can these findings inform NBS plans to deliver courses differently in e.g. flipped classrooms?
1. Students engage differently across courses. Distance Learning and Campus learning seems different.
The exception here is the DL General course which seemed to perform differently to all the other distance
learning courses. Is this because the DL General cohort is a more diverse group of students or is not a
‘supported’ distance learning course? Motivation, activity and achievement all differed for this group. Their
achievement was far more diverse (Chart 1.9), their engagement in the resource far higher on self-study (Chart
1.6) than any other Distance Learning cohort and their dropout rate was the highest at 18%.
RECOMMENDATION. Consider how distance learning delivery needs to differ for different cohorts to
maximize participation.
RECOMMENDATION. Consider the general DL course for changes to ensure these students can achieve
their potential and be supported throughout the course.
RECOMMENDATION. Consider how best to motivate other distance learners to engage with MAL.
Consider setting fewer homework assignments, some ‘homework assessments’ and more testing
deadlines such as a practice test 1 and 2.
2. Participation in the homework and homework scores are higher on campus than in any distance learning
course. Why is this? Motivation drops after the first two homework assignments have been completed for
distance learners (Chart 1.4).
RECOMMENDATION. As NBS move to a flipped classroom delivery and have less students in proximity
to lecturers and learning resources consider how lecturers interact with distance students in order to
encourage participation in the way lectures might. Is a ‘live’ video of or group chat with a lecturer able
to recreate this motivation to participate in online activities for example?
RECOMMENDATION. Consider how to make announcements to students on distance learning courses
more personal and motivational.
RECOMMENDATION. Homework ‘assessment’ worked well in engaging the STD Jan cohort and may
have contributed to their high level of engagement with the work.
The questions this raises might be:
Why is it harder to engage the most able, best performing students in MAL?
Is it harder to engage these students in distance learning online generally?
Why is there a difference when sustaining participation for different cohorts on MAL?
Overall.
Recommendations in this report may offer insight into how flipped classrooms will work best with campus
students. There may also be insight into ways to ensure a consistent student experience with MAL on the same
course (by specification) delivered in different ways and at different times by different people. Alterations to
the course may be desirable for the different distance learning cohorts based on their activity, engagement
and achievement levels.
Pag
e20
Background Information
Northampton is proud of its achievements in university rankings as well as its social and entrepreneurial
achievements. Two of the highlights on the website are below along with recent highlights for context.
2014 Student satisfaction increases to 89% (National Student Survey 2014)
Maintains top 50 UK university placing in the Guardian University Guide
Midlands Business Awards Most Enterprising University of the Year (winner 2012, 2013 and 2014) 2013
Times Higher Education University of the Year (shortlisted) Business School of the Year (shortlisted)
Recognised as the highest climbing university in the Complete University Guide 2014 and the Times/Sunday Times University Guide 2014
Awarded Changemaker Campus status by AshokaU and recognised as number one for social enterprise.
2012
Times Higher Education Awards Entrepreneurial University of the Year (shortlisted) Recognised by HESA as number one for employability amongst full-status universities in England The Business School prides itself on the following statements which can be found on their website: ‘Northampton Business School (NBS) has over 3500 students across the globe, offering a wide choice of flexible courses at undergraduate, postgraduate and professional levels. The school is renowned for high quality teaching and research supported by strong links with industry and commerce’*
Flexible study – part-time & online learning
Professional development
Eligibility for professional body recognition on some courses
Teaching excellence
Improving employability through work, industry connections and social enterprise
*http://www.northampton.ac.uk/about-us/academic-schools/northampton-business-school 21st April 2015 16:17