How to Analyze Systematic Reviews: practical session

Akbar Soltani.MD.Tehran University of Medical Sciences (TUMS)

Shariati Hospitalwww.soltaniebm.com

Clinical question

• In older people (> 50 years), do corticosteroid injections into the knee joint, compared with no injections, reduce the symptoms of osteoarthritis?

Search

• A search in PubMed: ClinicalQueries using the terms:• corticosteroid AND knee* AND osteoarthrit*

• … brings up a few review papers, of which the most recent and largest is a Cochrane review published in 2005. While you might wish to go straight to the Cochrane review, for this exercise, we want you to imagine that the only published review for this issue is one published in the BMJ in 2004.

Search

• Arroll F, Goodyear-Smith F (2004). Corticosteroid injections for osteoarthritis of the knee: meta-analysis. British Medical Journal 328:869–873.

Authors’ conclusion

• The authors of this paper concluded that: Evidence supports short-term (up to two weeks) improvement of symptoms after intra-articular corticosteroid injection for osteoarthritis of the knee.…

Is the PICO of the corticosteroid study close enough to your PICO?

• Looking at the corticosteroids review, the I and O of the PICO can be identified from the Abstract and Introduction to the paper, while a quick look at Table 2 shows the population for the included studies. It is much the same as our PICO. Therefore, it certainly looks as though it is worth continuing.

Three Step Guide in Using an Article to Assess Therapy

1. Are the results of the study valid?

2. What are the results? What measures of precision of effects were reported (CIs, p-values)?

3. How can I apply these results to patient care?

1) Question — Did the corticosteroid reviewers ask a focused research question?

• The corticosteroid review includes the following information on the objectives of the review and the selection of papers:

• ‘Objectives: To determine the efficacy of intra-articular corticosteroid injections for osteoarthritis of the knee…’

See ‘Abstract’ (CS review p1).

1) Question — Did the corticosteroid reviewers ask a focused research question?

• ‘Efficacy’ is also defined as ‘improving the symptoms of osteoarthritis of the knee’.

See ‘Introduction’, last paragraph (CS review p1).

2) Find — Did the corticosteroid reviews find all the best evidence?

Search protocol• ‘We searched MEDLINE (1966 to 2003), and EMBASE

(1980 to 2003)’ ‘The reference lists [of included studies] were scrutinised for relevant papers.’ ‘We searched the Cochrane controlled trials register’

How did the corticosteroid reviewers overcome publication bias?

• ‘Authors of included studies were contacted for details of any further work.’

See ‘Methods’ (CS review p1).

2) Find — Did the corticosteroid reviews find al the best evidence?

See ‘Methods’ (CS review p1).

2) Find — Did the corticosteroid reviews find al the best evidence?

• Best evidence• ‘Our selection criterion was randomised placebo

controlled trials in which the efficacy of intra- articular corticosteroids for osteoarthritis of the knee, of any duration, could be assessed.’

• 10 RCTs were included on this basis from 36 papers initially identified.

See ‘Methods’ (CS review p1) and Fig 1 ‘Summary of search results’ (CS review p2).

3) Appraise — How did the corticosteroidreviewers appraise the studies?

• ‘The two authors independently assessed the methodological quality using the Jadad scoring system. Consensus was reached through discussion.’

See ‘Methods’ (CS review p1) and Table 1 ‘Jadad quality scores…’ (CS review p2).

Three Step Guide in Using an Article to Assess Therapy

1. Are the results of the study valid?

2. What are the results? What measures of precision of effects were reported (CIs, p-values)?

3. How can I apply these results to patient care?

4) Synthesise — Did the corticosteroid reviewers synthesise the results using appropriate summary tables and plots?

• ‘Table 2 Details of included studies with outcomes on improvement in osteoarthritis of the knee ‘

• This table shows a summary of the 10 included studies with information about the patients and type of osteoarthritis suffered, intervention and control groups, and outcomes.

See ‘Results’, Table 2 (CS review p3).

4) Synthesise — Did the corticosteroid reviewers synthesise the results using appropriate summary tables and plots?

See ‘Results’, Table 2 (CS review p3).

4)Synthesise — Did the corticosteroid reviewers synthesise the results using appropriate summary tables and plots?

• Figures 2, 3 and 4 show forest plots for 3 improvements in osteoarthritis symptoms and include statistical analysis of heterogeneity.

See ‘Results’, Figs 2–4 (CS review p4).

Fig 2: Improvements up to two weeks after steroid injection in knee

Figure 2: Improvements up to two weeks after steroid injection in knee

Heterogeneity score (Cochran Q): P = 0.12 (not significant); df = 5; Q/df = 0.024 (< 1)

This indicates that heterogeneity is unlikely.

Around 45% of the patients improved with placebo (the control event rate). The number needed to treat (NNT) to obtain one improvement based on the summary estimate was 3.5.

Figure 2: Improvements up to two weeks after steroid injection in knee

• Six studies included this outcome. Only three of the individual studies are statistically significant and one of these has a very large CI.

• However, the summary estimate shows RR = 1.66 (95% CI 1.37 to 2.01), which does not cross 1 (the ratio for ‘no effect’) and hence is a statistically significant improvement.

Fig 2: Improvements up to two weeks after steroid injection in knee

Fig 3 Improvements at 16-24 weeks after high dose steroid injection in knee for two high quality studies

Figure 3: Improvements at 16–24 weeks after high-dose steroid injection in knee for two high-quality studies

• Two high-quality studies included this outcome. Neither of the studies is statistically significant.

However, the summary estimate shows RR = 2.09 (95% CI 1.20 to 3.65), which is a statistically significant improvement

Figure 3: Improvements at 16–24 weeks after high-dose steroid injection in knee for two high-quality studies

Heterogeneity score (Cochran Q): P = 0.83 (not significant); df = 1; Q/df = 0.83 (< 1)

This indicates that heterogeneity is unlikely.

Around 21% of the patients improved with placebo (control event rate); NNT = 4.4.

Fig 3 Improvements at 16-24 weeks after high dose steroid injection in knee for two high quality studies

5) What do the results of the corticosteroid review mean?

• The results show statistically significant improvement in symptoms, including reduction in pain, for up to 2 weeks after corticosteroid injections at a range of doses. Two studies also showed statistically significant improvement in symptoms at 16–24 weeks after injection of a higher dose. Heterogeneity analysis in each case showed that heterogeneity was unlikely.

• There was no mention of side-effects in the paper, which means we may have to look elsewhere for this. For example, how common were local reactions or infection?

Overall conclusion

• The study is a good-quality systematic review that shows a statistically significant reduction in symptoms of osteoarthritis after corticosteroid injections at various doses. Further work is needed on the relationship between the duration of symptom relief and dose.

Thank you!