© 2016 | Trilogy Education Services - All Rights Reserved
Lessons from University Network Learner AnalyticsTrilogy Education Services
Becoming A “Data Driven” Education Provider
| October, 2018
WORKFORCE ACCELERATOR: Trilogy partners with leading
universities to help companies bridge the Digital Skills Gap
Learner Analytics Today
© 2016 | Trilogy Education Services - All Rights Reserved
Tracking Academic Experience
● In order to deliver a high-quality learning experience, we must help students with
technical skills and their motivation levels along the way. Thus, we invest significant effort in tracking both in every one of our classrooms.
How are students feeling?
How well do students see the program?
How supported do students feel?
How motivated do students feel?
Student Perception Student Performance
How well are students comprehending?
How quickly can students recall concepts?
How able are students to apply skills?
How well can they deal with complexity?
Coding Success = Motivation + Skills
Trilogy Learning Analytics
© 2016 | Trilogy Education Services - All Rights Reserved
Rationale – KPI Targets
KPI Target Rationale
Pace 3.0 - 4.25 / 5.0
Our program aims to maintain an accelerated pace without leavingstudents overwhelmed. “Pace tracking” offers us a way to identify areas of the curriculum needing to be modulated.
Self-Mastery >3.25 / 5.0
Our program continually introduces students to new subjects on a
weekly pace. Throughout this process, we expect a healthy balance between self-perceived mastery and early knowledge formation.
Support >4.0 / 5.0
Our program aims to deliver very highly visible degrees of support for students. An average of >4/5 means that as a whole students rate our program as providing high or extremely high degrees of academic support. This is a key indicator in our learning analytics system as it provides us early perspective on intervention needs.
% Students Noting “Low
Support”
<5% / 100%
Our program also aims to ensure that at weak students do not feel “left behind”. As such, we track the % of feedback forms in which students score the week as offering a <2/5 on academic support. If this KPI ever exceeds 5-10%, we intervene by offering these classes additional remedial support and instructor training.
Instructor Clarity
>4.0 / 5.0
Each of our instructors is expected to deliver very high levels of engagement, clarity, and knowledge of all subject matter. Student feedback on these metrics offer early perspective on when instructors require additional training.
Instructor
Engagement>4.0 / 5.0
Instructor Knowledge
>4.0 / 5.0
© 2016 | Trilogy Education Services - All Rights Reserved
Growing Student Feedback
Based on 15,372 student
feedback responses…
© 2016 | Trilogy Education Services - All Rights Reserved
Advantages of Data | – Clarity of Focus
“Omg… The curriculum is horrible in a million places!!!!”
© 2016 | Trilogy Education Services - All Rights Reserved
Advantages of Data | – Clarity of Focus
• When possible use data to separate meaningful patterns (signals) from sporadic occurrences (noise).
© 2016 | Trilogy Education Services - All Rights Reserved
Advantages of Data | – Clarity of Focus
“No. Just three. We got this!”
JavaScript / jQuery
Express / Sequelize
ReactJS
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #1: Maintaining Consistency Across Programs
• We regularly analyze data across our university programs to quality-control our university
programs with regards to the student experience.
• Our mark of success is to see consistency across university programs.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #1: Maintaining Consistency Across Programs
Strong Local Field TeamsEvery local program is actively managed by well-
trained field teams who ensure the quality of the
program. Field teams operate under clear SOPs that
articulate program management strategies.
Standardized CurriculumUnlike many other Bootcamps, Trilogy’s programs are backed by tightly defined curricula that provide
a detailed playbook of activities and lectures for
instructors to manage the classroom experience.
These playbooks allow the program to be replicated nationally with consistency.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #2: Maintaining Quality Over Time
• Not only do we benchmark across our programs, but we also benchmark against
our past. By looking back at historic student data, we can ensure we’re maintaining quality as we scale.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #2: Maintaining Quality Over Time
Highly Skilled (and Fun!) Instructors / TAsWe’ve gained a very deep expertise in recruiting Instructors and TAs with strong technical experience and the ability to connect with wide audiences. By year-end, we will have hired over 400 Instructors and TAs across the country.
Analytic-Driven InterventionsTrilogy relies heavily on data to drive interventions and provide guardrails against quality loss. This analytics comes in the form of student feedback tracking, student performance tracking, and instructor audits.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #3: Increasing Support Over Time
• Since we began tracking data in 2016 Q1, we have consistently increased the student perception of academic support inside and outside the classroom.
• Since we began tracking we’ve halved the percentage of student responses expressing low support to less than 3% of all responses.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #3: Increasing Support Over Time
Supplemental Video ContentWe created a video library composed of over 129 supplemental videos (over 25 hours of content) intended to help with self-guided learning. Videos have over 500 views on average.
Centralized SupportWe created a central tutoring service that provides 1-on-1 and large group review sessions. The Support desk has provided over 500 tutoring sessions with an average score of 4.62 / 5.0 on helpfulness.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #4: Decreasing Variability in the Portfolio
• Since we’ve begun we’ve tightened our instructor hiring and onboarding processes
leading to an overall reduction in quality variations across classrooms.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #4: Decreasing Variability in the Portfolio
Optimized Instructor RecruitmentWe built an expanded team around hiring and created multiple rounds of interviews for instructor approval. All instructors must pass three rounds of interviews testing their personality, technical ability, empathy, and teaching skills. (Our hiring rate is now less than 10% with an average time to hire of 17 days).
Optimized Instructor Onboarding ProcessWe created an extensive onboarding process for instructors on teaching strategies, classroom management techniques, screen management, and mechanics of running a classroom.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #5: Tweaking the Curriculum
Students’ Self-Perceived Mastery by Curriculum Unit
Q3:2016
Yeah… This was rocky.
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #5: Tweaking the Curriculum
Students’ Self-Perceived Mastery by Curriculum Unit
Q4:2017
Across the board, we improved the curriculum by:
•Creating better exercises
•Better instructor training on the material
•Supplementary walkthroughs
•In-depth pre-work curriculum
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #6: Helping Non-College Grads Succeed
Approximately 25% of our
students enter the program
without a Bachelor’s Degree
© 2016 | Trilogy Education Services - All Rights Reserved
Data Story #6: Helping Non-College Grads Succeed
+/- 25%
Data has led us to change tactics for
non-college grads:
•Emphasize the challenges upfront
during admissions. Encouraging the need for 3x effort of college grads.
•Encouraging a more personal and direct approach to job applications (direct
messages, real-world contacts).
•Working with hiring partners to identify
more creative opportunities for entry into the field. (Apprenticeships, etc.)
• College grads placing at 2-to-1 but
no difference during academic
interviews, attendance, grades, and homework submissions…. The market
perception of the ability of these students was different….
© 2016 | Trilogy Education Services - All Rights Reserved
“Count what is countable.”
“Measure what is measurable.”
“And what is not measurable, make measurable.”
- Galileo Galilei