Occupational Therapy and Physical Therapy
Goal Attainment ScalingClinical Utility and Training Methods
Jen Angeli, Karen Harpster, Amber Sheehan
Occupational Therapy and Physical Therapy
FINANCIAL DISCLOSURE
AACPDM 72nd Annual Meeting
October 9-13, 2018
Speaker Names: Amber Sheehan, Karen Harpster, Jen Angeli
1. Disclosure of Relevant Financial Relationships
We have no financial relationships to disclose.
2. Disclosure of Off-Label and/or investigative uses:
We will not discuss off label use and/or investigational use in my presentation.
Occupational Therapy and Physical Therapy
Grab your phone.
In the internet browser, type:
kahoot.it
Occupational Therapy and Physical Therapy
https://kahoot.com/welcomeback/
https://create.kahoot.it/login
Occupational Therapy and Physical Therapy
Learning objectives
• Synthesize the literature on GAS training and clinical utility
• Describe pilot effort to train therapists on GAS utilization, per best available evidence
• Propose best practice for GAS standardization
• Discuss feasibility of implementation
Occupational Therapy and Physical Therapy
First, we get on the same page.
1.Goal Attainment Scaling = GAS
2.GAS is a criterion-referenced measure
that helps us to quantify a degree to
which personal goals are achieved.
3.It was developed in 1968 and was first
used in mental health.
4.It is becoming more popular in
rehabilitation health, as an index of
intervention efficacy.
Occupational Therapy and Physical Therapy
The GAS parts
• Document a set of patient-identified goals
• Identify the construct of change that matters most to the patient/family
• Observe/discuss/document current performance on each goal
• Agree upon a desired short term outcome
• Discuss what performance looks like if it is a little better than the desired short term outcome, or much better than the desired short term outcome.
Occupational Therapy and Physical Therapy
For example:
I want to ride my bike to the end of
Victory Avenue (0.5 miles).Performance Interpretation GAS
interval
Completed MS150 Much better than expected +2
Rode to park for new zipline (1
mile)
Better than expected +1
See above Short term goal 0
Rode to Joe’s house for dinner
(3 houses down)
Progress, but did not achieve
goal
-1
Cannot ride bike Baseline level of performance -2
Occupational Therapy and Physical Therapy
Extra parts• Weight each goal for importance (patient/caregiver)
and difficulty (therapist)
• Put it all in a big formula that will calculate a T score.
• Use the T score or change in T scores to interpret
or Minimal Detectable Change = 10 points
Occupational Therapy and Physical Therapy
What’s nice about GAS
• It’s patient-centric
• It provides a single, objective summary of
performance
• Progress reflects meaningful change in a
prioritized area
Occupational Therapy and Physical Therapy
What’s not nice about GAS
• It’s hard to learn.
• It feels like it’s not clinically feasible.
• It’s only valid if you follow the rules,
particularly those associated with scaling.
Occupational Therapy and Physical Therapy
What we learned about GAS
in a systematic review
1. Is it being used in pediatric rehab?
2. What’s the rigor of published pediatric
studies employing the GAS?
3. Is GAS a responsive outcome measure?
Occupational Therapy and Physical Therapy
• 51 studies in pediatric rehab were included
in our review
• They weren’t very strong with respect to
rigor and quality
• Despite this, GAS appeared to detect
meaningful change in ~ 60% of studies
Occupational Therapy and Physical Therapy
Occupational Therapy and Physical Therapy
• Higher Sackett levels = more rigorous study
design
Occupational Therapy and Physical Therapy
• 17 quality criteria that group broadly into categories describing:– content validity of the scale
– reliability of scale construction
– reliability of scale rating
– additional items related to training and examiner bias
• Higher quality scores = greater validity in GAS application
• See handout for criteria!
Occupational Therapy and Physical Therapy
Quality summary
• Of the 51 studies included, less than half (19 studies) were categorized as Sackett levels I, II, or III.
• Average quality score = 4.57 points (/17)
Individual studies lost the most points for failure of Goal Attainment Scaling scales to adhere to the criteria of:
-unidimensionality (1/51)
-time-specificity (3/51)
-reliability of scale construction-equidistance of levels (3/51)
-inter-rater reliability (4/51)
Occupational Therapy and Physical Therapy
Why does it matter?
• GAS T scores may inflate (and indicate
progress) simply because of bias
introduced from failure to create valid and
reliable scales.
Occupational Therapy and Physical Therapy
What did we do?
Occupational Therapy and Physical Therapy
Our Goal:
• To utilize the GAS as an objective
outcome measure with a high level of
reliability in clinical practice
– Start with episodic care summer programs
• Clinic-based GAS session
http://www.acmsoft.com/Services/think-big-start-small.html
Occupational Therapy and Physical Therapy
https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-
oZ74Bg&q=let%27s+do+it&oq=let%27s+do+it&gs_l=img.3..0l10.18304.28268..28880...0.0..0.105.1091.7j4......2....1..gws-wiz-
img.....0..35i39j0i67.3puL26l830Q#imgrc=FjAo6UDp_7yAjM:&spf=1539226018390
https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=hLm-W5SoFqGp_Qb-
oZ74Bg&q=Reading+research+articles&oq=Reading+research+articles&gs_l=img.3..35i39j0i24.3052.5474..5884...0.0..0.99.10
78.12......1....1..gws-wiz-img.......0j0i7i30j0i7i5i30j0i8i7i30j0i5i30.4Kmw2Xj2-aY
Occupational Therapy and Physical Therapy
https://www.google.com/search?biw=1518&bih=662&tbm=isch&sa=1&ei=tby-
W8HkDMqf_QbHgLToAg&q=variation&oq=variation&gs_l=img.3..35i39j0l9.2142.2834..3640...0.0..0.102.191.1j1......1..
..1..gws-wiz-img.0D_0uKckWAo#imgrc=nTk8YeFP9SO0NM:&spf=1539226998417
• Level of parent and child involvement
• Video tape GAS performance vs no video
• Documentation
• Setting the construct of change
Occupational Therapy and Physical Therapy
Develop a process
• Video tape all GAS goal performances
• Watch videos during team meetings
– Develop documentation (importance, difficulty,
level of change, etc)
• Review goals using GAS checklist
• Review GAS levels with caregivers
http://www.10minutebiztools.com/Startup-Trial-and-Error.html
Occupational Therapy and Physical Therapy
GAS in Research: Refine the
process even further• Independent teams
– COPM/treatment team
– GAS team
• Training to ensure specificity of COPM goal E.g. “John runs back and forth and puts holes in his socks” “John will sit for 5 minutes to complete a fine motor task”
• Develop a script to elicit meaningfulness of goals
Occupational Therapy and Physical Therapy
What happened next:
Therapist Training
• Developed a training manual, based on current evidence– includes procedures and criteria for scale
construction, scoring, and interpretation of outcome data
• Provide formal training – Introduced the training manual and demonstrate
application via case study
– Trainees independently review materials and construct a scales based on the same video
– Review scale with each trainee using quality criteria to promote adequate interrater reliability
Occupational Therapy and Physical Therapy
What happened next:
GAS Visit via telehealth
• GAS scales are developed by 2
independent raters based on video taped
performance of goals
– GAS scales compared for sameness
• Child scored on both
– GAS checklist implemented
– Feedback given regarding scale construction
within 1 week of GAS completion
Occupational Therapy and Physical Therapy
GAS through telehealth....
• Volume control
• Test video call with caregiver
– Ensure they understand how to use
technology, camera angle, etc.
• Make sure the caregiver knows the child
needs to be present for session
Occupational Therapy and Physical Therapy
GAS through telehealth....
• Make sure the supplies for the goals are readily available
• Making sure the video stays on the child during goal completion
• Considering an informal scale to capture how much assistance the caregiver needs to participate in co-creation of scale
– seems to be impacting validity of the scales we create
Occupational Therapy and Physical Therapy
of doing GAS through
telehealth
• Materials usually readily available
• Natural environment
• Generalization occurs naturally
• Empowering the caregiver
Occupational Therapy and Physical Therapy
Criteria for GAS appraisal
• Eliminated items
– Goals specific to aim of intervention
– Pretest score same across all goals/ patients
– Specification of follow up time
– Adequate therapist training
– Report of inter-rater reliability
– Report of examples of goals and scaling
• Added items suggested by other sources
Occupational Therapy and Physical Therapy
Results thus far
• 2/5 trained therapists had no prior GAS
background
• Telehealth goal scaling completed with 6
patients, each by one of 3 therapists
– Average 9.4/15 scaling quality criteria met
– After feedback provided, no additional sessions
yet to assess potential change in quality
– Two trained therapists have not yet had a goal
scaling session scheduled
Occupational Therapy and Physical Therapy
Results thus far, cont.
• Most easily achieved criteria
– Mutually exclusive goals
– All five levels defined
• similar to the literature
Occupational Therapy and Physical Therapy
Results thus far, cont.
• Other easily achieved criteria
• Scale uni-dimensionality
• Verification of pre-intervention performance
• Equidistance of scale levels
– frequent absence in the literature, achieved here due to
initial training and study design
Occupational Therapy and Physical Therapy
Future plans
• Study design
– Confirm post-test scale score with caregiver
Occupational Therapy and Physical Therapy
Future plans
• Study design
– Determine interrater reliability by correlating
the ratings of the original goal scaler with a
second rater on 20-30% of goals (Krasny-
Pacini et al., 2017; King et al., 1999)
Occupational Therapy and Physical Therapy
Future plans
• Reporting results
– Adequate example of goals and scaling reported
– Results in literature were most commonly stated
as:
• a raw post-intervention T-score (27 of 51 studies)
• change in T-score (3 of 51)
• number or % of goals that were achieved (i.e., at or
above the “0” level on the scale; 23 of 51 studies)
• effect size (4 of 51)
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
• Construct of Change: Level
of independence
Occupational Therapy and Physical Therapy
Let's Practice
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
2:
1:
0:
-1:
-2: Needs help with engaging zipper and stabilizing coat to pull up zipper
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
2:
1:
0:
-1: Stabilize coat and pull up zipper with coat on the table
-2: Needs help with engaging zipper andstabilizing coat to pull up
zipper
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
2:
1:
0: Stabilize coat and pull zipper up with coat on
-1: Stabilize coat and pull up zipper with coat on the table
-2: Needs help with engaging zipper andstabilizing coat to pull up zipper
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
2:
1: Engage and pull up zipper with coat on the table
0: Stabilize coat and pull zipper up with coat on
-1: Stabilize coat and pull up zipper with coat on the table
-2: Needs help with engaging zipper andstabilizing coat to pull up zipper
Occupational Therapy and Physical Therapy
Zip Coat
• Goal: Mom would like Peter to be able
to zip his coat independently
2: Engaging and pull up zipper while the coat is on
1: Engage and pull up zipper with coat on the table
0: Stabilize coat and pull zipper up with coat on
-1: Stabilize coat and pull up zipper with coat on the table
-2: Needs help with engaging zipper andstabilizing coat to pull up zipper
Occupational Therapy and Physical Therapy
Jennifer Angeli, DPT, PhD
Physical Therapist, Cincinnati Children’s Hospital Medical Center
Assistant Professor, Rehabilitation Sciences, University of Cincinnati
513.252.5722
Amber Sheehan, OTR/L
Occupational Therapist, Cincinnati Children’s Hospital Medical Center
513.636.8526
Karen Harpster, OTD, PhD
Occupational Therapist, Cincinnati Children’s Hospital Medical Center
Assistant Professor, Rehabilitation Sciences, University of Cincinnati
513.803.3604
Occupational Therapy and Physical Therapy
References
• Grant, M., & Ponsford, J. (2014). Goal attainment scaling in brain injury rehabilitation: strengths, limitations and recommendations for future applications. [KH, SS]. Neuropsychol Rehabil, 24(5), 661-677. doi: 10.1080/09602011.2014.901228
• Harpster, K., Sheehan, A., Foster, E. A., Leffler, E., Schwab, S. M., & Angeli, J. M. (2018). The methodological application of goal attainment scaling in pediatric rehabilitation research: a systematic review. Disability and rehabilitation, 1-10.
• Kiresuk T.J. & Sherman R.E. (1968). Goal Attainment Scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4(6), 443–453.
• Kiresuk, T.J., Smith, A., & Cardillo, J.E. 1994. Goal Attainment Scaling: Applications, Theory, and Measurement. Hillsdale, NJ: Lawrence Erlbaum Associates.
• Mailloux, Z., May-Benson, T. A., Summers, C. A., Miller, L. J., Brett-Green, B., Burke, J. P., et al. (2007). The Issue Is—Goal attainment scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61, 254–259.
• McDougall, J. & Wright, V. (2009). The ICF-CY and Goal Attainment Scaling: Benefits of their combined use for pediatric practice. Disability and Rehabilitation, 31 (16): 1362–1372.
• Ruble, L., McGrew, J.H., & Toland, M.D. (2012). Goal Attainment Scaling as an outcome measure in randomized controlled trials of psychosocial interventions in autism. J Autism Dev Disord, 42(9): 1974–1983
• Schaaf, R.C. et al. (2013). An intervention for sensory difficulties in children with autism: A randomized trial. J Autism Dev Disord, online.
Occupational Therapy and Physical Therapy