ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE DECISION-MAKING

transcript

Nathan JonesAndrew CannHina PopalSaud Almashhadi

Agenda

1. Context

2. Problem & Need Statement

3. Design Alternatives

4. Simulation

5. Simulation Output

6. Utility Analysis

7. Conclusions

8. Management

Introduction to Soccer

Soccer is the world’s most popular sport.Generates the most revenue:

• In 2009-2010 season the English Premier League generated roughly 3.2 billion dollars

• European soccer generated 21.6 billion dollars

Highest average attendance for international club competitions:

• FIFA World Cup• UEFA Champions League

Information taken from: http://www.economist.com/blogs/gametheory/2011/09/ranking-sports%E2%80%99-popularity

European Soccer

NFL MLB NBA0

2521.6

Professional Sports vs. 2009-2010 Generated Revenue

Generated Rev-enue ($ Billions)

Introduction to Soccer

• The game is played by two teams (11 vs. 11).

• Field dimensions:

115 by 74 yards

• 2 – 45 minute periods

• 3 Referees – 1 main referee and 2 assistant referees• Responsible for upholding

the integrity of the game

Referee Responsibilities

Upholding the integrity of the game:

• Make accurate calls • Make calls that don’t interrupt

the flow of the game• Be in proper position, to

assess, process, and identify correct call

Referees are categorized as either junior referees (entry level) or senior referees (advanced level).

Current MLS referees make 86.1 % correct calls. (USSF)

Acknowledgement of Sponsor

Metro DC Virginia State Referee Program (MDCVSRP) oversees all soccer referees in the Commonwealth of Virginia (over 5400 referees)

Responsibilities:

1) Train and evaluate junior and senior referees

2) Assign Referees to officiate games

3) Promote high quality referees to senior ranks

Responsibilities 2 and 3 depend heavily on ability to assess referee call accuracy

Referee Call Making Process

Referee Assessment

On-Field Assessments

Written Exam on Knowledge of the

Fitness Test

Referee Assessment is Broken

Junior referees do not undergo fitness tests or on field assessments

(Preventing evaluation of Fitness or GFU attributes)

The evaluation process for referees is broken:

• 96% of total MDCVSRP Referees (junior level) do not receive assessment in two of three attributes.

Referee Attributes Assessment MethodFitness Fitness Test

(senior referees)

Call Decision Making (CDM) Written exam on rules (All referees)

Game Flow Understanding (GFU) Indirectly using on field assessment (senior referees)

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Problem Statement

96 % of MDCVSRP referees (Junior level) do not receive assessment for Game Flow Understanding and fitness attributes as predictors of call accuracy.

Need Statement

An assessment method is needed to evaluate referee accuracy in a cost effective manner utilizing fitness and/or Game Flow Understanding (GFU).

Scope: Our analysis will focus on determining the best system concept for assessing MDCVSRP junior referees.

Specifics of design and implementation are considered future work.

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Design Alternatives

# Alternative Description Tests Total Cost

(5,139 Referees)

1 Fitness Test

A baseline fitness test equivalent to those

administered at senior grades

Fitness $26,990

2 Game Flow Evaluation

Video performance assessments

conducted by official assessors

GFU $337,995

3 Combined EvaluationCombination of first

two evaluationsFitnessGFU $341,870

4 No AssessmentNot conducting any referee evaluations

(status quo)None $0.00

Costs defined as physical + implementation cost for one time evaluation of all junior referees.

Evaluation Of Alternatives

Utility of each alternative defined as:

Expected call accuracy of the top 100 referees identified using each alternative within junior referee pool (5000 referees).

To determine utilities, a two part analysis was conducted:

1) Function for call accuracy based on fitness and GFU levels developed using discrete soccer game simulator.

2) Using part 1 function, expected call accuracy of top 100 referees selected by each alternative computed through Monte Carlo analysis.

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Simulation: Input / Outputs

Expansion on Prior Work

Simulation was re-designed and re-coded from scratch.

Simulation Element Solomon, et al. (2011) This Project

Probability Maps 1 map for all teams, all time, and all score 19 maps dependent on team, time, an score

Ball Position Function 1 event 4 state cycle scaled to time

Referee Position Function 1-D, chase ball on left diagonal 2-D, based on GFU, scaled to time

Fitness 3 levels 5 levels

GFU None 5 levels based on probability maps

Call Grids None Survey 16 senior state referees

Call Event Trigger Simple probability Calls grids and position in cycle

Distance vs. Call Accuracy Function Estimated Figure of MeritSurveyed 16 senior state referees and

generated regression

Number of Teams in Game Home vs. Home Home vs. Away (4 Options)

Number of Teams Simulated 1 4

Team Strategy Changes Never Time / Score

Referee/Ball Movement Scaled to Time No Yes

Simulation – Ball and Referee Position

• In the discrete event simulation, a soccer field is divided into a fine grid of cells.

8510 cells

Each Cell 1x1 yd

Cell Groupings:

• 60 Movement Polygons

• 24 Call Grids

• 0.5 s refresh rate (game time)

Two Teams - Possession Shifts

The ball shifts possession between two different teams, each executing its own unique strategy . Changes in possession occur due to failed passes or shot events.

Simulation: Ball Movement

Cycle of Events

Ball & Referee Movement Algorithm

• At any time in simulation, ball moving to set destination in straight line.

• Destination changes during dribbling / passing.• Time taken for ball to move incrementally to destination (#

Refreshes) is reflective of ball speed and distance:

Shot Events

Whenever ball finishes dribbling, probability determines if ball is shot at goal. Shot either results in goal or turnover.

Pass Events

• If no shot, Ball passed between polygons controlled by movement probability maps indicating destination and chance of success.

• Polygon (n+1) = Polygon (n) * Prob. Map• When new polygon selected, destination is set to random cell

within polygon

Map sets areformulated for:

• Manchester United• Arsenal• Wigan Athletic• Stoke City

Probability Maps

• Ball movement and shot data were gathered from the Guardian Chalkboard Website. 80 total games (over 35,000 pass & shot events) recorded for Stoke City, Manchester United, Arsenal, Wigan.

Data was analyzed for strategy and used to produce shot and movement probability maps

Probability Maps – Team Strategy

Two way ANOVA Analysis: Time + Score + Time*Score

• Time has an effect on pass accuracy: Arsenal(p = 0.777); United(p=0.142); Stoke (p=0.001); Wigan (p=0.001)

• Score has an effect on pass accuracy:

Arsenal(p = 0.231);United(p=0.001);Stoke(p=0.000); Wigan (p=0.000)

• Score*Time has an effect on pass accuracy:

Arsenal(p = 0.338);United(p=0.000);Stoke(p=0.000);Wigan(P= 0.116)

Strategy Analysis conducted to determine when strategy maps should be changed (Metric = % completed passes)

Simulation: Referee Movement

Referee Profile DefinitionTo determine the effect of Fitness and Game Flow Understanding on call performance:

• 25 referee “profiles” defined as combinations of fitness and game flow understanding.

Referee Game Flow Understanding

Referee Fitness

0 25 50 75 100

///////// 0.25 0.41 0.58 0.74 0.9

02.023 yds / s Ref 1,1 Ref 1,2 Ref 1,3 Ref 1,4 Ref 1,5

Simulation – Ref movement

One main referee running within left hand diagonal route area.

Referee movement speed depends on fitness level of profile tested.

Simulation - Ref Movement

At each refresh rate (0.5 s), referee will compute desired location relative to ball using one of 2 movement scripts:

1) No Prediction – Referee will set destination to closest cell within 11 – 13 yds of ball’s current location.

2) Prediction – If dribbling: Referee will set destination to closest cell within 11 – 13 yds of next most probable pass destination.

Once destination is set, referee will begin moving to destination (rate = speed).

Process repeats at each refresh

Simulation – Ref Movement

• Proportion of time referee utilizes script 2 depends on GFU level. • Referee with (GFU = 0.75) with remain in script 2

75% of time.

GFU also includes an ability of referee to recognize a build up to a call:

Probability that predicting referee anticipates the call and switches to script 1 until the call occurs.

Simulation: Call Events

Call Events

• Call grid probabilities used to generate events based on ball location whenever new cycle begins. Further probabilities determine where in cycle event occurs.

Source: Senior MDCVSRP referee surveys (n = 16)

Roughly 90 events per game

Passing: 0.21Dribbling: 0.44En-route: 0.15Receiving: 0.21

Simulation - Call Accuracy

• Whenever a call event occurs, referee must make a decision regarding the nature of the event (infraction, no infraction).

• The probability that he makes the correct call depends on the distance from the ball.

Referee Call Accuracy Function

Source: Senior MDCVSRP referee surveys (n = 16)

Distance > 20 ydsDistance <= 20 yds

Accuracy Peaks at 11 – 13 yds

Simulation: Output

Simulation - Output

Simulation output :• Each profiles simulated through 2,000 games (200 per team comb.)• Referee call accuracy was calculated for each game.

Simulation Demo

Validation of Simulator

STATISTIC Simulation Professional Soccer

Average Goals per game 0.8266 ̴� 1.553(EPL 4 team Average) [1]

Average Team Passes per game

449 ̴� 424 (EPL 4 team Average) [2]

Average Referee Distance Run per game (yds)

11, 686 11, 289 (NZFC) [3]

[3] - D.R.D. Mascarenhas et al. (2009) "Physical Performance and Decision Making in Association Football Referees: A Naturalistic Study" [online]. Available: http://www.benthamscience.com/open/tossj/articles/V002/1TOSSJ.pdf

[1] - http://soccernet.espn.go.com/stats/_/league/eng.1/year/2010/barclays-premier-league?cc=5901

[2] - http://www.whoscored.com

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Simulation - Call Accuracy Results

1000.7

Call Accuracy(Fitness,GFU)

Fitness

Average2000 games per profile

Simulation Results - Regression

Accuracy (Fitness, GFU):

0.713491 + 0.000923486 *Fitness + 1.28791e-005*GFU 6.4846e-005*Fitness^2 + 1.12504e-006*GFU^2 + 1.26193e-006*Fitness^3- 6.75305e- 009*Fitness^4

R-Sq = 99.51%

Fitness, GFU nonlinearNo interaction (p = 0.813)

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Defining Referees for Utility Analysis

• Referees are defined as a combination of two independent traits (Fitness, GFU)

• Each trait is scaled from worst (0) to best (100) possible• The distribution of referees for each trait is Normal at mean

50 and st. dev 15

Call accuracy for each referee defined using Call Accuracy Regression

Utility Analysis Method – Monte Carlo

• 5000 Referees (Junior level) were generated .• For each alternative, a cutoff was defined on each attribute assessed where

if a referee preformed above the cutoff on all attributes, he would be selected by program.

• Cutoff developed using Normal CDF to ensure top 100 referees selected

Alternatives assumed to have perfect ability to identify if referees make the cutoff

Alternative Attributes Cutoff Avg. # Referees Chosen

Fitness Test Fitness Fitness > 81 97

Game Flow Evaluation GFU GFU > 81 97

Combined Evaluation Fitness, GFU Fitness >66 & GFU > 66

No Assessment N/A N/A 100

Analysis Method – Monte Carlo

For each alternative, referees are identified that meet the selection cutoff. The average call accuracy of referees selected (% correct calls) is used to

determine alternative utility.

Utility Analysis Results

Alternative Cutoff Avg. Call Accuracy 95 % Half-Width

Call Accuracy

Fitness Test Fitness > 81 0.74926 0.00012

Game Flow Evaluation GFU > 81 0.72693 0.00028

Combined EvaluationFitness >66 &

GFU > 66 0.74174 0.00021

No Assessment N/A 0.72099 0.00004

Based on n = 30 trials

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Alternative Cost vs. Benefit

“Fitness Test” dominates all other assessment based alternatives.

Recommendations for MDCVSRP

Recommendation: It is not cost effective to implement assessments on junior referees within MDCVSRP.

Fitness Test vs. No Assessment (status quo)

Marginal Cost Fitness Test: $26,990

Marginal Utility Fitness Test: Accuracy improvement of 2.8% for top 100 referees identified

Further Findings – Impact of Teams

Impact of different team strategies on game flow has noteworthy effect on referee performance

Impact of Teams – Call Distance

776655443322110

Call Distance

Call Distances (United vs. United)

847260483624120

Call Distance

Call Distances (Stoke vs. Stoke)

Same Referee Profile (GFU = 50, Fitness = 50)

500 Simulated games (30,000 calls) per team combination

Team combination has substantial effect on distribution of call distances.

Additional Findings – Recommendation for USSF

• When comparing the quality of multiple referees based on in-game performance, match difficulty in terms of game flow and team combination must be taken into consideration.

Agenda

1. Context

4. Simulation

6. Utility Analysis

7. Conclusions

8. Management

Work Breakdown Structure

Work Breakdown: Systems 490

Work Breakdown: Systems 495

BudgetTask Predicted Velocity Cost

Research 135 hours $4,050

Referee/Game Data 244 hours $7,320

Referee Evaluation Simulator 303 hours $9,090

Formulation of Conclusions 40 hours $1,200

Communication of Results 415 hours $12,450

Project Management 330 hours $9,900

Total 1467 hours $44,010

Earn Value Management

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 350

Earned Value Chart

Planned ValueActual CostEarned Value

Week (starting 8/29/2011)

Cost Performance Index = .9289Schedule Performance Index = .954

Sponsor Testimony

“ The analysis done by the students has been incredibly eye-opening. They have changed the way our management at MDCVSRP think about referee development and where to use our budget.”

-Pat Delaney

MDCVSRP Chairman

ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE DECISION-MAKING

Documents