Conducting a World Class Review of the Literature in Your Field

Dr. Philip HallingerJoseph Lau Chair Professor

Asia Pacific Centre for Leadership and Change

Hong Kong Institute of Education

hallinger@gmail.com

2

Teachers Create the Future

I believe that a I believe that a teacher lives on teacher lives on and on through and on through the lives of her the lives of her students. students.

Good teaching Good teaching lasts forever and lasts forever and the teacher is the teacher is immortal.immortal.

Jesse Stuart, 1937Jesse Stuart, 1937 To Teach, To LoveTo Teach, To Love

I hope this is not me!

Conducting a World Class Review of the Literature3

Research and Theory as a Source for Evidence Based Practice

Our beliefs and value are useful, but evidence is another source of guidance for practice.

Today we will focus on literature review as a way of examining evidence and describing the boundaries of knowledge in a field.

This is highly relevant for Malaysia where there are relatively few published reviews of the indigenous literature that has grown from the proliferation of Master Doctoral these over the past 15 years

5

Example used in this Presentation My colleague Ron Heck and I conducted a review of research on

principal leadership effects on learning in the mid-1990s Several reports from this review were published as journal articles

(2) and book chapters (3) The chapter was included as an exemplary literature review in the

volume, “Learning to Read Critically in Educational Leadership and Management”

It has become the most frequently cited research review in the educational leadership literature (1,500+ citations in journal articles and books)

I will also refer to a more recent review I conducted of literature on instructional leadership in 2011

7Conducting a World Class Review of the Literature

Preparing for a Review of the Literature

Why Conduct a ‘Lit Review’?

Map knowledge terrain Define your research topic Justifies research questions Justifies methodology Provides a point of

comparison for your findings

Identify your study’s contribution to knowledge

Conducting a World Class Review of the Literature10

Map the Terrain

Criteria for a high quality review

1. Focused on a clear question(s)

2. Organized to address the question

3. Clear, structured method of review

4. Critical in evaluating theory and empirical evidence

5. Accurately referenced

6. Reader friendly (clear organization and language)

7. Informative -- drawing clear conclusions

8. Balanced in expressing points of view(Wallace and Poulson, 2003)

13

Select a Focus for your ReviewUnclear focus gives readers a headache

Every review must define its scope or “intellectual space” i.e., questions, topic etc.

The review should also clarify what features of the topic it will focus on:• Theory development• Methodological issues • Findings• Types of studies• Some combination

14Conducting a World Class Review of the Literature

Stare at this Graphic for 12 Seconds

Look at other reviews done in your field in order to. . .

Find out what prior trends they identified

Identify exemplary or seminal studies

Locate persisting problems See potentially fruitful

directions to take on methods and questions

Look for accumulation of knowledge over time

time

productivity

16Conducting a World Class Review of the Literature

Select a Theoretical Perspective A theoretical perspective helps to organize the review – where and

what to look for in the literature Places diverse approaches in a common framework Helps make sense of the trend of studies &findings Examples of theoretical frameworks:

• Cross-culture• Cognitive or Information processing• Rational, natural, open systems models• Transformational and instructional leadership

18Conducting a World Class Review of the Literature

Example of Theoretical Models in Studies of Leadership Effects

Adapted from from Pitner, 1988, 105-108Adapted from from Pitner, 1988, 105-108

Model A: Direct-effects ModelModel A: Direct-effects Model

Principal Principal LeadershipLeadership

Student AchievementStudent Achievement

Model C: Reciprocal-effects ModelModel C: Reciprocal-effects Model

Principal Principal LeadershipLeadership

Student AchievementStudent AchievementMediating VariablesMediating Variables

Student AchievementStudent AchievementMediating VariablesMediating Variables

Model B: Mediated-effects Model B: Mediated-effects

Principal Principal LeadershipLeadership

Principal Principal LeadershipLeadership

Student Student AchievementAchievement

Context and/or Context and/or Personal Personal

CharacteristicsCharacteristicsMediating VariablesMediating Variables

Model D: Mediated-effects with Antecedent EffectsModel D: Mediated-effects with Antecedent Effects

Select a Review Methodology

Purpose

• Exploratory

• Explanatory Method of Review *

• Descriptive/analytical

• Critical Synthesis

• Meta-analysis

*Not mutually exclusive categories

20Conducting a World Class Review of the Literature

Selecting Sources for Your Review You will not be able to cover every study in the scope of your review Key decision is to determine what studies and papers you will review Selection of studies should be systematic, explainable, and

justifiable:• Timeframe for the research (For example, I am doing a review of

all articles in ed leadership journals from Asia from 2000-2011)• Criteria embedded in research questions and scope (e.g.,

variables)• Type of reports (e.g., theoretical, practice, empirical, qualitative,

quantitative, reviews. . . )• Sources (type of journal, or specific journals, conference papers,

dissertations, national and/or international)21

Conducting a World Class Review of the Literature

Doing the review. . .

Reading and Analyzing the Literature

23Conducting a World Class Review of the Literature

Distinguish Between Types of Studies and Reports

Theoretical papers Empirical research studies

• Descriptive

• Analytical Descriptions of practice Policy analyses/reports Critiques, advocacy or opinion

24Conducting a World Class Review of the Literature

Don’t Just Report Findings:Evaluate What You Read

Read critically Don’t treat all reports or

findings equally Evaluate methods and

assumptions Base evaluation on quality

of the study and support from other research

Look for counter-evidence

Researchers in the USA during the 1980s and 1990s identified small but significant positive effects of instructional leadership on student learning outcomes. However, several high quality studies conducted in the Netherlands failed to replicate this finding (van de Grift, 1989, 1990; Sleegers et al., 1996; Witziers et al., 2003). Possible explanations for this discrepant finding include. . .

25Conducting a World Class Review of the Literature

Features to Include in Your NotesSee the Research Review Template

Participants’ characteristics, criteria for inclusion Sample size Location of study Type of study – experiment, survey, longitudinal, case Type of method: observation, interview, survey Nature of task – what was done, details of tests Findings – numerical, descriptive, conclusions Notes/comments on strength of the study Implications for your research study

26Conducting a World Class Review of the Literature

Tracking What you Read

Hallinger & Heck (2009) found that distributed leadership had a positive impact on school improvement and student learning

How? Why? How strong? Relation to findings in other

studies?

Hallinger & Heck (2009) studied the impact of distributed leadership in 199 primary schools over a period of four years. Using SEM they assessed how changes in distributed leadership were associated with changes in the schools’ academic capacity and math and reading achievement. They found a significant indirect impact of DL on student learning through changes in academic capacity. These findings from longitudinal research support the trend of findings from prior cross sectional studies. Still to be addressed are . . .

27Conducting a World Class Review of the Literature

Tracking Data in Your Review

28

Contributing to the Accumulation of Knowledge

Outcomes of a Literature Review

42Conducting a World Class Review of the Literature

Build the Story as You Go Identify areas of consensus:

• Thus, there was agreement among researchers in the 90s regarding the indirect effects of principal leadership on student learning. . .

Identify areas of divergence: • However, this finding has yet to be replicated by longitudinal or experimental

research and scholars still speculate as to the means by which these effects are achieved.

Use summary paragraphs throughout the review, not just at the end.

43

In summary, the literature on school leadership effects suggests that leadership is an important influence on school improvement. However, at the same time, the literature confirms that leadership is influenced by the organizational and cultural context of the school. Most recently, a series of international case studies provide useful insight into how perceptions of effective leadership practices are shaped by the cultural context. These findings suggest that. . .

Draw Conclusions

Remember Bloom’s taxonomy: Know, Understand Analyze, Apply, Synthesize, Evaluate

Simple description of studies does not meet requirements of a review

Your review should analyze and evaluate the individual studies

In the end it must summarize and synthesize what you learned

44Conducting a World Class Review of the Literature

Tell the Reader What the Review Found

Findings Overall picture of

research in the field Use tables and figures as

needed to show trends Highlight key findings Theories and methods Cite weaknesses and

limitations of findings Gaps in knowledge

Be Careful Link conclusions to your

findings Avoid the tendency to

overstate and draw unsupported conclusions

The purpose of the review is not to prove a particular point

45Conducting a World Class Review of the Literature

Example:General Conclusions

Principal leadership makes a difference Leadership effects are achieved by impacting in-school

processes rather than directly on students Strongest effects from instructional leadership Quality of research has improved since reviews by

Bridges, Bossert and Leithwood in the early 1980s Stronger theory, designs and analyses are yielding

more consistent results. Notable accumulation of knowledge

46Conducting a World Class Review of the Literature

Interpretation and Implications:Tell readers what you think the findings mean

Place the findings in perspective Discuss limitations Relate back to the earlier

literature review Implications can be drawn for

relevant dimensions covered by the review• Theory• Methods• Practice• Policy

48

Example: Implications for Research (what/how we study)

Expand qualitative research – explore the key mediators (e.g., vision, teacher learning) and moderators (e.g., SES, school size, gender) identified in quantitative research

Reduce studies of effects that focus on bi-variate relationships and statistics

Use statistical techniques appropriate to more comprehensive theoretical models

Explore how these relationships change over time through longitudinal time series studies

49Conducting a World Class Review of the Literature

Example: Implications for Theory

Use intermediate outcomes for effects (e.g., climate and culture, teacher commitment) as well as measures of learning

Explore reciprocal effects models and related contingency theories of leadership

Unpack the practices within vision, mission, goals and how they impact practice

Expand theorizing on how leadership behavior in schools is moderated by the context

50Conducting a World Class Review of the Literature

Example: Implications for Policy

Since instructional leadership does appear to have an effect on learning outcomes, systems of selection of school leaders that focus on this competency is warranted

Ensure that instructional leadership is part of the preparation curriculum

Experiment with policies that support shared instructional leadership

51Conducting a World Class Review of the Literature

Example: Implications for Practice

1. Work with others to develop a common vision for learning in the school.

2. Keep staff focused on teaching and learning.

3. Adapt your leadership style to the needs of the context in which you are working.

4. Focus on school-wide processes that build capacity for improvement.

52Conducting a World Class Review of the Literature

In Conclusion. . . Reviews of research play an

important role in knowledge accumulation in any field

Your own review should be systematic and provide a map of the intellectual territory in which you are working

Over time your own knowledge of your field will deepen as you continue to read critically in your field

Indeed – if you are like me -- 10 years from now you will be shocked at how little of your field you knew today!

61Conducting a World Class Review of the Literature