Evaluating the Admission Process for a London based Business School

APRIL 2016

Using tests of cognitive ability to provide insights for a student recruitment programme

Evaluating the Admission Process for a London based Business School 2

About Pearson TalentLensPearson TalentLens is an international talent assessment business, using psychometrics to help organisations select and develop high performers with high potential.

Select Our ability tests are used for volume recruitment and targeted campaigns.

We work with our clients to help them ascertain candidate aptitude.

We offer numerical, abstract and verbal reasoning tests and of course, the industry standard measure of critical thinking, the Watson-Glaser and the renowned measure of fluid intelligence, Raven’s advanced Progressive Matrices.

DevelopOur personality trait and values tool helps to determine a candidate’s likely ‘fit’ to the role and as with our personality type tool, provides valuable developmental information and guidance.

We also offer the Honey & Mumford Learning Style Questionnaire

view the product matrix for our full range of assessments.

Graduate RecruitmentOur graduate solutions help organisations to manage their recruitment and onboarding campaigns.

Using ability tests to sift out early in the process, followed by a measure of traits and values prior to interview, we help our clients to identify, select and develop the best candidates.

TrainingWe offer official BPS Test User Ability and Personality training

Qualification courses in our trait and type personality tools

Combined, single and in-house options

Our lead trainer is Angus McDonald, Chair of the BPS Committee on Test Standards

see full details of training opportunities >

Registered office:Pearson education Ltd80 strandLondonWC2R 0RLRegistered in england 872828VAT Registration No. GB 278 5371 21Pearson assessment is a division of Pearson education Limited

Evaluating the Admission Process for a London based Business School 3

Contents

Executive Summary ............................3

Background ............................................... 4

Method ........................................................... 4

Results ............................................................ 6

Discussion .................................................. 8

Executive summaryChallenges for successful academic institutions include:

1) Whether there are additional measures to A-levels and other academic examgrades, that predict whether certain students are likely to do well on the course.

2) How to admit students in a fair, consistent and relevant way.

Ideally, admitted students should be motivated to complete their studies and be receptive of new knowledge. This study investigates efficient ways of measuring these traits in the admission process.

In this study, students at a London based business school were followed from their admission through to their final year of studies. Data was collected between 2012 and 2015.

The findings show that applicants with high critical thinking skills and high general cognitive ability, measured during the admission process, were more likely to complete their studies with high final awards 1-3 years later. A similar relationship was found for students’ performance on a written essay and an admission interview.

The study concludes that the best way of identifying high potential students is to ask applicants to complete two psychometric tests (Watson-Glaser™ Critical Thinking appraisal and Core abilities), a written essay and an interview before admission.

These measurements are complementary, and combining them provides a valid overall measure of applicants’ learning potential within the business school.

Evaluating the Admission Process for a London based Business School 4

BackgroundThe featured business school is a British private, not-for-profit degree provider based in London owned by a large multinational education company.

The business school is the only FTSE 100 company in the UK to design and deliver degrees, offering programmes with more industry involvement than any other Higher Education provider. The programmes are designed to be academically demanding and industry informed, to inspire a real interest in the contemporary professional world.

There are two possible routes to be admitted to a course – either the conventional route, where students will be admitted if they pass a UCAS Tariff criterion and an admission interview, or the alternative route, where students will be admitted based on their performance on two psychometric tests, a written essay and an admission interview.

after completing a course, the students’ achievements are evaluated, and a Final award is given to reflect their performance. This predictive validity study was conducted to investigate the relationships between the students’ results on the selection assessments and their academic achievements.

Research objectives:1. To estimate and compare the validity of the admission metrics (UCAS Tariffs,

Interview ratings, Essay scores and results on two psychometric tests) inpredicting academic performance one year later (Final Awards).

2. To compare Final Awards from the two admission routes (conventional tariffroute vs. alternative applicant route).

3. To find an optimal way of combining admission metrics to predict Final Awards.

MethodAssessmentsThe business school interviews all applicants who apply to its Accountancy, Business and Law courses, regardless of previous qualifications or background.

The admission interview is face to face with normally one academic member of staff. It lasts around 30 minutes. The interviewer is encouraged to make notes on the candidate response to each question.

At the end of the interview, the interviewer grades the candidate from 0 (low) to 100 (high) on overall performance. The interview is designed to measure the candidate’s motivation, oral communication skills and commercial awareness.

The admission essay forms part of the applicant Day (in addition to the Core abilities Test, Watson-Glaser Critical Thinking appraisal and the Interview). Candidates who do not meet the entry requirements and/or would like to be considered for scholarships are required to sit the essay.

The candidate is asked to answer a question, themed in relation to their course choice, in one hour. Unless reasonable adjustments apply, all answers are handwritten. Prior to the applicant Day the candidate is given limited guidance such as, ‘I would advise you to spend some time reading a reputable source of business news (eg. the BBC Business website).’

Evaluating the Admission Process for a London based Business School 5

Once completed, the essay is marked by an academic member of staff, who did not interview them. The marker scores the essay with an overall score from 0 (low) to 100 (high). The essay is not considered a test of subject knowledge, rather it is measuring the candidate’s ability with: the English language, reasoning, originality and commercial awareness.

Watson-Glaser Critical Thinking Appraisal (WGCTa) is a gold-standard measure of critical-thinking ability, built on more than 85 years of continuous development and application. It measures the ability to accurately recognize assumptions, evaluate arguments and draw conclusions, which is important for academic performance as well as future job performance. The test consists of 40 items in total with a 30 mins time limit.

Core Abilities is a quick online assessment of general abilities: verbal, numerical & abstract reasoning. The results are typically used as an efficient indicator of candidates’ general cognitive ability, making it useful to predict performance in many professional settings. It contains 20 items in total, with a time limit of 15 minutes.

Sample60 students were included in the study, all of whom completed their final year of studies with the business school. Prior to being admitted, all 60 students completed the admission Interview, 26 provided records of their UCAS Tariff, 47 submitted their Essays, 49 completed the Core Abilities assessments and 47 completed the WGCTa.

Academic performanceThe Final Awards given to the students were either 1st class, upper 2nd class, lower 2nd class, 3rd class or fail. These awards were coded into an ordinal scale to be used in estimating the predictive validity of the admission assessments, with 0 representing failing the course, 1 representing 3rd class, 2 representing lower 2nd class, 3 representing upper 2nd class and 4 representing 1st class.

AnalysesFirst, the average (median) assessment scores were calculated for students in each Final award group, to investigate research objective 1. Students with higher Final Awards were expected to perform better on the assessments compared to students with lower Final awards.

Secondly, Spearman’s rank correlation coefficients (ρ) between the admission assessments and the Final Awards were calculated to investigate research objective 1. Positive correlations were expected, where higher scores on the interview and the psychometric tests should be related to higher Final awards.

Thirdly, to investigate research objective 2, median Final awards from students admitted through the conventional route were compared to median Final awards from students admitted through the alternative route.

Finally, the scores from the assessments were summed, and this combined assessment score was correlated against the Final awards to estimate a global predictive validity.

Evaluating the Admission Process for a London based Business School 6

ResultsLooking at the median scores from each of the assessments reveals that students with higher awards achieved higher scores prior to being accepted, compared to students receiving lower awards. The effect is most notable for the Core Abilities and the WGCTA assessments. The median scores are shown in Figure 1 (excluding the UCAS Tariff, which is measured on a different scale). Note that the median score for the 3rd class wasn’t calculated since only one student received this grade.

Figure 1: Students’ median percentile scores on the assessments vs their Final Awards.

Significant correlations (predictive validity coefficients) were found between the students’ Final Awards and all assessment metrics, except for the UCAS Tariff Points. The highest predictive validity, with regards to students’ Final Awards, was their scores on Core Abilities (ρ=.46***), followed by Essay scores (ρ=.41***), WGCTA scores (ρ=.38**) and Interview ratings (ρ=.31**). The full correlation matrix is presented in Table 1.

For reference, the U.S. Department of Labor (1999) provides the following general guidelines for interpreting validity coefficients: values above .35 are considered “very beneficial”, .21-.35 are considered “likely to be useful”, .11-.20 “depends on the cicumstances”, and below .11 “unlikely to be useful”.

Med

ian

Perc

enti

le S

core

Final Award

TestCore

100

90

80

70

60

50

40

30

20

10

0

Fail Lower 2nd class Upper 2nd class 1st class

WGCTaInterviewessay

Evaluating the Admission Process for a London based Business School 7

Final Award Essay Core Abilities WGCTA Interview UCAS

Final Award .41*** .46*** .38** .31** (.17)Essay 47 .43*** .37** .44*** (.20)Core Abilities 49 46 .40** (.21) (.27)WGCTA 47 47 46 (.15) .53*Interview 60 47 49 47 (-.17)UCAS 26 17 18 17 26

Table 1: Spearman’s rank correlations (ρ) above diagonal and sample sizes (n) below diagonal.* p < .05; ** p < .01; *** p < .001; correlations in parentheses were not significant.

The median Final Award of the 52 students from the conventional admission route was upper 2nd class. In comparison, the median Final award of the 8 students from the alternative applicant route was 1st class. The number of students achieving each Final Award, split on the two admission routes, is presented in Table 2.

Conventional admission Alternative admission

1st class 13 5Upper 2nd class 19 2Lower 2nd class 11 03rd class 1 0Fail 8 1

Table 2: Students’ Final Awards, split on the two admission routes.

Finally, the total predictive validity for the sum of the assessment scores was ρ=.57, which was higher than the coefficients for each of the individual assessments. The linear relationship is shown in Figure 2. The assessments used to calculate the total score were the essay scores, Interview ratings, WGCTascores and Core abilities scores.

Fina

l Aw

ard

0

1.0

2.0

3.0

4.0

Core + WGCTA + Essay + Interview

80 100 120 140 160 180 200 220 240 260 280 300 320

Mode of entryTariff route

applicant route

Figure 2: Linear relationship between sum of admission scores and final awards (ρ=.57***).

Evaluating the Admission Process for a London based Business School 8

DiscussionThe trend in Figure 1, where higher Final Awards were associated with higher median assessment scores, shows that Core abilities, the Watson-Glaser Critical Thinking appraisal (WGCTa), the Interview ratings and the Essay scores are predictive of academic performance. The statistically significant correlations between these scores and the Final awards are also indicative of this.

The strongest correlations were found between the Core abilities scores and the Final awards. However, the differences between the observed correlations were small and not statistically meaningful (the confidence intervals for the ρ coefficients overlap greatly). The Core Abilities, the WGCTA, the Interview and the essay can all be considered predictive of academic performance.

The UCAS Tariff Points, however, did not perform as well as a predictor of academic performance. The correlations with the Final Awards were not significant.

Comparing students from the two admission routes, two observations can be made. First, there were significantly fewer students admitted from the alternative route compared to the conventional route. However, those who were admitted through this route did receive higher Final Awards on average.

a limitation of this study was the small sample size. When estimating correlations between the assessments and the Final awards, data was available for 47-60 students for most assessments. However, only 26 students had records of both UCAS Tariff points and Final Awards, limiting the power of this significance test.

Method discussionThe reason for using a non-parametric approach for the statistical analyses in this study was the nature of the data. While standard parametric methods, such as calculating means and standard correlations, are useful to describe variables that can be assumed to follow a continuous, normal distribution, they can give unreliable results for data that does not conform to this assumption.

Results on many of the assessments in this study were recorded on a percentile scale, which is assumed to follow a uniform distribution rather than a normal distribution. also, the Final award was recorded on an ordinal scale, which is neither continuous nor normally distributed. Because of this, the use of medians to describe average scores and calculating Spearman’s ρ to estimate correlation coefficients were preferred.

ReferencesU.S. Department of Labor. (1999). Testing and assessment: An employer’s guide to good practices. Washington, DC.

Contact Us

+44 (0) 345 099 1485

info@talentlens.co.uk

www.talentlens.co.uk

Connect twitter.com/talentlensuk

linkedin.com/company/pearson-talentlens-uk