Measuring education - 1 What to measure? International comparisons.

Measuring education - 1

What to measure?

International comparisons

1 – Inputs, outputs, outcomes

The National Accounts framework

National accounting framework

One aim of National Accounts = value production in the economy as a whole

Production = transformation of inputs (L, K) into outputs (goods & services: G & S)

Measuring production begins with identifying all the units which produce the G & S in question.

Next, consider which of the transactions of these producers need to be identified in measuring production in the National Accounts

Then, data on these transactions need to be obtained and aggregated

Education as a non market service

In the National Accounts, producers are allocated to industries according to a classification system of industries twinned with a classification system of products.

International Standard Industrial Classification (ISIC),

linked with the Central Product Classification (CPC)

Education is Division 85


Education (Division 85)– 851 Pre-primary and primary education– 852 Secondary education

• 8521 General secondary• 8522 Technical and vocational secondary

– 853 Higher secondary– 854 Other education– 855 Educational support activities

Source: International Standard Industrial Classification, revision 4


Education and health services: government provisions them free of charge or at

not economically significant prices= non-market output.

Price is “not economically significant” if it covers less than half of the costs of the service

Price paid cannot be the basis for valuing it Valuation for NA = sum of the costs incurred in its

production


Costs = sum of:– Intermediate consumption (the goods and services

used up in producing the service)– Compensation of employees (costs of teachers,

doctors, nurses, etc.)– Consumption of fixed capital (depreciation of

school and buildings, of medical equipment)– Other taxes, less subsidies, on production


How to measure the output of education & health?

(1) Use costs in place of prices

(2) Impute some value for the relative valuation by the consumer to generate aggregation weights.

Solution (1) Assume that, on average, the relative valuation by the consumer equals the relative cost

incurred by producers?

Government producers: Argument = in a democracy, via the electoral

process, consumers have influence over the production decision of governments so that the provision of non-market services is socially optimal.

relative costs = relative utility / willingness to pay

Solution (2) Impute some value for the valuation by the consumer ?

(2) Impute some value for the relative valuation by the consumer to generate aggregation weights.

implies a fully developed consumer or welfare perspective in the estimation of the volumes of health and education services.

= controversial – from a NA perspective if total value of health or

education services total expenditure– if the construction of weights entails delicate

questions such as the necessity to put a monetary value on human life

Back to the pb of aggregation

How to aggregate the production of 300 cars and 10,000 salads?

Answer for market products = market price Idea that it reflects the relative utility of G & S

Back to the pb of aggregation

But how do you aggregate 300 heart surgery operations and 10,000 eye treatments with no market price?

To describe the volume or price of a product category, you have to assume all goods in that category fulfill the same need

+ the price deflator implies an evaluation of the increases in quality

Output in value / in volume

GDP at current prices is not so interesting as growth of GDP and its components in volume

Growth in volume excludes changes in price levels between two periods

Only elements of the National Accounts that can be disaggregated in terms of prices and volumes are useful in analyzing economic growth, productivity and inflation

Previous methods and problems

Previous methodologies : measuring the volumes or the prices of inputs to obtain a measure of the volume or price of outputs

Pb: Such input-based methods cannot capture any productivity change

Productivity growth exists when more or better output can be produced with the same resources.

Quality of output does change in education services ignoring productivity changes = foregoing important information and measuring volume growth inaccurately

Reminder: volume and price indexes

Remember that in principle, deflating sales to consumers by a consumer price index = calculating directly a volume index by weighting together quantity indices which represent the volume of each homogeneous service

The amount spent on each service – a function of its price - provide the weights to aggregate these quantity indices

No difference in concept to construct a price index and then deflates sales or to construct a volume index directly

Reminder: volume and price indexes

With no quality change, the price index for education services would be a weighted average of the price indices of teaching service activities

This price index can then be used to obtain a volume measure of output by deflation

The issue of quality adjustments

Require the identification of characteristics such as speed, engine size for a car or the processor speed for a computer

Idea = isolate price changes due to changes in characteristics from those price changes that constitute ‘inflation’


One possibility to measure quality = to put together several characteristics into a single indicator that reflects the contribution of the product to outcome.

Ex: price indices for health care. Triplett (1998) suggests quality-adjusted life years (QALYs) as a single dimensional measure could be used for quality-adjustment of different treatments within a product group


Should not be affected by any other factors that influence consumer outcome (e.g., socioeconomic background of students or lifestyle of patients) and

Should reflect the normal / average /expected effect of the activity on “state of knowledge” of the consumer.

‘Normal’, ‘average’ or ‘expected’ because to the extent possible, measures of service production should not be influenced by the individual capacity of the consumer to make use of these services


Discussion on the necessity to invoke utility and outcome

Characteristics that matter for consumers have to be identified for quality adjustment

“glance over the national accounts production boundary” to deal with quality change in education services

The quality of services cannot be identified without considering indicators falling outside the production boundary

OECD conference in June 2007: current issue!

In-, output, outcome, utility

General links between output and outcome

Hill services and result of the production processPeter Hill (1975) says the production of services is inseparable from their

use or consumption: two economic units are simultaneously involved in service production whereas production of a good takes place within a single producer unit.

Output of services should be viewed as the attributable contribution of the producer to the outcome.

UnitProduct Producer Consumer

Good Output

Service Outcome

Definitions: education output Eurostat handbook on prices and volume in NA“the quantity of teaching received by the students, adjusted to allow for the

qualities of the services provided, for each type of education”

“In the area of education, the output can be defined as the quantity of teaching (that is, the transfer of knowledge, successfully or not) for a pupil, whereas the related outcomes are the skill and knowledge that a pupil achieves.”

ISCED definition“organised and sustained communication designed to bring about

learning”, where

“Communication” involves the transfer of information (messages, ideas, knowledge, strategies, etc.) between two or more persons; […]

“Learning” is taken as any change in behaviour, information, knowledgeknowledge, understanding, attitudes, skills, or capabilities which can be retained and cannot be ascribed to physical growth or to the development of inherited behaviour patterns

Definitions: education output (2)

OECD (2007) proposal “expected transfer of knowledge and skills provided by an

education unit, for each type of level of education / curriculum”

Transfer of knowledge and skills can be seen as the true quantity indicator, and change in academic scores due to school could provide a right measurement.

Expected = « average », means that each pupil receiving the same teaching is supposed to consume the same output, although some pupils will assimilate more than others (ES handbook: « successfully or not »). This difference in outcome is supposed due to the pupil/consumer.

Basic formula

Ivolume

= Iquantity (number of enrolled pupils)

x Iquality ed (average transfer of knowledge)

x Iquality non ed (total expenditures / core educational) Without information on the transfer of knowledge,

we find the number of pupils, 2nd best quantity indicator for Eurostat.

If we assume that the transfer is continuous and proportional to time of teaching, we get the “pupil x hours”, 1st best quantity indicator for Eurostat.

Overview of inputs, output, outcomes in volume

Inputs Activities / Processes

Output

Direct outcome

Indirect

outcomes

Number of teachers and

staff, by qualification

Pupil-hours, class size, pedagogy

Employment, future real

earnings from labour market

growth of GDP

Number of teachers

Pupil-hours, class size, pedagogy

Transfer of skills and

knowledge by levels of education

Knowledge and skills:

estimated by scores, degrees

Well-rounded

citizens ?

other envir.

factors

Pupils: inherited skills, family backgr.,

entry educ. status, gender?

Methods Input

Output w/o expl. outcome-

based quality adjustment

Output with expl. outcome-based

quality adjustment

Educ. outcome: assessment

and achievement

Human capital, others

Aggregates National accounts Welfare / policy

productivity

Cost-effectiveness

Education output = average transfer of knowledge and skills, some remarks

No accounting of the distribution of individual levels of knowledge and skills, although “reduction of inequalities” is usually one of the targets of the education policy: consistent with the idea of “individual services” in NA

No accounting of healthy, wealthy, well-rounded people / citizens, which are sometimes difficult to measure and which are more “indirect” outcomes.

Education status (outcome) is measured at the end of a curriculum, rarely at the beginning we need a model (role of the pupil / consumer) in order to estimate the part of the output in the outcome.

Other outcome-based quality indicators

OECD: academic scores as first best direct education outcomes

But some curricula could be better estimated by obtainment of a degree or professional insertion (indirect outcome).

Professional insertion: employment rate (if the right qualification if possible) and real earnings can be used (multiplied)

Coherent with human capital theory (discounted sum of future real earnings as return on “GFCF” in education),

Pb of evaluation: what is obtained after the curriculum minus what would have been obtained without, if we want to measure the “output”.

2 – International comparisons: historical landmarks

A little bit of history

Until after WW2, educational systems were evaluated without any indicators

Inspectors judged what went on in classes according to what they had learnt to be “good teaching”

1) The 1970s OECD, 1973: “a system of indicators aiming at

orienting policy”. Goal: increase social well-being. Social scientists

A set of 47 indicators…


… that were never calculated because there was no data to do so and governments were not willing to spend money to collect it

Failure

2) The 1980s

1983: A Nation at Risk (the US). Strong criticism

1986: Education Indicators. A guide for policymakers

= a period when the focus is on the quality of teaching (pedagogy). Heated debates on indicators


3) The 1990s

1991: OECD conference Making Education Counts: Developing and using international indicators.

At first, nothing on students’ achievement

Only the educational system: costs, enrolments

= a management approach: reducing costs, optimizing resources

First volumes of Education at a Glance: 1992


4) The 2000s

Strong focus on students attainment. “educ at a glance”:- A: output- B: resources- C: access- D: learning environment, organization of schools

Chapter on output was last until 2002: moved to 1st position. From 5 indicators of students attainments to 14

= a dramatic change in perspective

Psychometric approach, cognitive sciences have become dominant

1995: International Adult Literacy Survey / 2000: PISA


The Lisbon criteria (2000): “knowledge economy” Reduce to 10 % the proportion of early school leavers

(indicator 23). Reduce by at least 20% the proportion of students with bad

reading results (indicator 5). At least 85 % of youth should complete secondary education

(indicator 4). Increqe by at least 15 % the number of graduates from

scientific fields, while simultaneously reduging the gender gap (indicator 12).

Participation rate to continuing education: 12,5% of the adult population (indicator 19).

3 – ISCED

The International Standard Classification of EDucation

The ISCED classification

1) A little bit of history (again) UNESCO, 1975-1978 ISCED 1978 From the start, goal = int. comparisons

1990: need for change More adult education, more distance learning, more

various education providers On-the-job training (Germany) not taken into

account

ISCED 1997


2) What makes building an international classification so difficult?

Definitions must be precise but also work in very different contexts

– Ex: “completed” education when there is no exam/ diploma?

– Ex: what is “basic education”?

– Ex: “Vocational” training: theoretical or actual ?

Risk = countries including very different things in a given category meaningless comparisons



The right number of items

– 30 is not usable, 4 is too little…

– Usually: 2 to 6 different levels, hierarchical

Being understandable across the world: translation problems

– Ex ISCED: 6 official languages + 100 countries using it as a tool in a foreign language



Accepting differences within a category

– 2 exact same programs do not exist = need to define what is the more important criteria.

– ISCED: level rather than length or organisation of education

– Ex: primary education may last 4 to 8 years

… but clearly defining boundaries

– Implies spending much time discussing borderline, infrequent cases

ISCED-97 levels of education

Level 0 Pre-primary education

Level 1 Primary educationor first stage of basic education

Level 2 Lower secondaryor second stage of basic education

Level 3 (Upper) secondary education

Level 4 Post-secondary non-tertiary education

Level 5 First stage of tertiary education

Level 6 Second stage of tertiary education

ISIC rev. 4 and ISCED 97

ISIC rev 4 classes ISCED-97 levels of education

8510 Pre-primary and primary education

Levels 0 and 1

8521 General secondary education Levels 2 and 3 oriented general

8522 Technical and vocational secondary education

Levels 2 and 3 oriented vocational and technical

8530 Higher education Levels 4, 5 and 6

8541 Sports and recr. education

Not classified in ISCED-97 levels of education

8542 Cultural education

8549 Other education n.e.c.

8550 Educational support activities Not explicitly mentioned in ISCED-97 levels of education

Fo

rma

l edu

catio

n (o

ur fo

cus)

info

rma

l

Su

ppo

rt

act

iviti

es

But no R&D or adm., like in COFOG


3) The basics of the ISCED classification

0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated

Cannot simply be based on number of years? No: number of hours, curriculum, goals of one year very different from one country to another

– Ex: more time, same goals for students with disabilities : same level in more years

– Ex: selective tracks that go faster: less years, more skills


3) The basics of the ISCED classification

0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated

OK but hierarchy must be made explicit

– Ex: theoretical knowledge higher than practical skills?

Definition of “education”: the object of the classification

The ISCED classification : defining education

“ISCED does not intend to provide a comprehensive definition of education, still less to impose an internationally standardized concept of the philosophy, aims or content of education, or to reflect its cultural aspects. Indeed, for any given country the interplay of cultural traditions, local customs, socio-economic conditions, at the very least, will have resulted in a concept of education in many ways unique to that country, and any attempt to impose a common definition would not be productive.

However, for the purposes of ISCED, it is necessary to prescribe the scope and coverage of the educational activities to be covered by the classification.”

The ISCED classification: defining education

“Within the framework of ISCED, the term education is thus taken to comprise all deliberate and systematic activities designed to meet learning needs. This includes what in some countries is referred to as cultural activities or training. Whatever the name given to it, education is understood to involve organized and sustained communication designed to bring about learning. The key words in this formulation are to be understood as follows:”

- organized

- sustained

- communication

- learning


4) Ex of problems: level 4 “Post-secondary non-tertiary education”

Was created to put short programs that were classified as secondary or as tertiary depending on countries made comparisons invalid

A category that was created to get rid of those programs, making comparisons between the rest relevant

But not understood, not used as was expected

5) Ex of debate: cut-off btw 5 & 6?

Has to be well defined because countries will always want to put more in the highest level

6 does not include research Masters, only PhD

4 – What outcome at what level?

What are the expected outcomes of

Preschool ?

– Not easily measured. Ex: evaluation of Perry School Project, Abecedarian Project, Head Start…

– Range from grade repetition before 15, to wage at 25, to number of arrests before 40…


Primary education ?– “basic knowledge”: literacy and numeracy. But when do

you measure them (at what age?) and how ( next class)

Middle and high school ?– Goal of international comparisons not possible to rely

on “what students know” (ex: history, geography, literature… national, hence not comparable)

– Define a number of “skills” + measure them without cultural bias? ( next class)

These = goals of PISA


Universities ?

5 – International rankings of higher education institutions (“universities”)

ARWU

Since when does it exist?

- answer: 2003

Why such a success?

- Context: focus on education and research as investment

- Internet: more widespread information

Increased international competition

Who?

Which institutions should be included?

- ARWU: all institutions with Nobel Prizes, Fields Medals or frequently cited researchers

- Times: all institutions quoted by the 5010 experts

- Financial times: Masters programs

- Leiden: universities only

What should count?

How do you calculate a score?

- ARWU: awards, citations (researchers, articles, top 20 reviews)

- Times: experts quote 30 establishments they consider as excellent in their field + survey of recruiters + citations + staff/students ratio

- Financial times: Experts + indicators on international openness

- Leiden: bibliometrics

What do these rankings do?

Bibliometrics important size effect: More researchers, more papers

Countries where research done outside of teaching establishments (“pure” research) : not counted disadvantaged

The difficulty lies in the many aspects of higher education– Teaching (basics, research or business oriented?)

– Research (applied, fundamental?)

– Implementation (patents, consulting…)

Put all on the same scale implies weighting

What do these rankings do?

ARWU stresses research only, and excellence in research (Nobel Prizes). Leaves students, their careers, apart

Times: research = 20%

CC: Depends on who the ranking is aimed at:

- Students? Can they be Misleading?

- University managers?

- Policy makers?

Consequences

These rankings have massive institutional, practical consequences

– Ex of France: quest for “Critical size” (Paristech, PSE…)

– Ex on research practice: pressure to publish under this or that affiliation

Pb: indicators, rankings are quickly translated into straightforward evaluation: “x is better than y (in general)”. Nevermind sizes, means, goals…

Trade-off between simplicity and completeness

6 – A few important figures

A few important figures

OECD avg

Share of GDP spent on initial education: OECD


The share of educ/GDP is the result of 3 factors:

- Size of the school-age population

- Percentage going to school

- Spending per pupil


Yearly spending per pupil, primary to higher education, ppp$, 2005 (OECD)


Yearly spending per pupil, primary education, ppp$, 2005 (OECD)


Yearly spending per pupil, higher education, ppp$, 2005 (OECD)


Spending per pupil in sec and higher, relative to spending on primary education (=100)

2005 (OECD)

A few important figuresNumber of science graduates for 100 000 25-34 year-

olds in the labor force

« science » here includes biology, physics, maths, statistics, computer sciences, engineering and technical professions, making and transformation, building and architecture

Men Wmen Total


Spending on higher education institutions, GDP% (2005)

Grey = private / red = government


Proportion of 15-yr-olds having repeated a grade at least once (2003)

Lessons to be learnt

Output / outcome

Indicators on processes / on outcomes: what is “evaluation”?

Indicators sometimes shape the debate… but sometimes don’t (ex: debate on education in Fr since 1990)

Rankings of universities on a global scale: recent. Competition within/ between countries.

Importance of the design of classifications (ex: ESEC for occupations)

Date post:	25-Dec-2015
Category:	Documents
Upload:	leslie-stokes
View:	217 times
Download:	1 times

Measuring education - 1 What to measure? International comparisons.

Documents